swiftclient

” OpenStack Swift Python client binding.

swiftclient.client

OpenStack Swift client library used internally

class swiftclient.client.Connection(authurl=None, user=None, key=None, retries=5, preauthurl=None, preauthtoken=None, snet=False, starting_backoff=1, max_backoff=64, tenant_name=None, os_options=None, auth_version='1', cacert=None, insecure=False, ssl_compression=True, retry_on_ratelimit=False)

Bases: object

Convenience class to make requests that will also retry the request

Parameters:
  • authurl – authentication URL
  • user – user name to authenticate as
  • key – key/password to authenticate with
  • retries – Number of times to retry the request before failing
  • preauthurl – storage URL (if you have already authenticated)
  • preauthtoken – authentication token (if you have already authenticated) note authurl/user/key/tenant_name are not required when specifying preauthtoken
  • snet – use SERVICENET internal network default is False
  • starting_backoff – initial delay between retries (seconds)
  • max_backoff – maximum delay between retries (seconds)
  • auth_version – OpenStack auth version, default is 1.0
  • tenant_name – The tenant/account name, required when connecting to an auth 2.0 system.
  • os_options – The OpenStack options which can have tenant_id, auth_token, service_type, endpoint_type, tenant_name, object_storage_url, region_name
  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
  • ssl_compression – Whether to enable compression at the SSL layer. If set to ‘False’ and the pyOpenSSL library is present an attempt to disable SSL compression will be made. This may provide a performance increase for https upload/download operations.
  • retry_on_ratelimit – by default, a ratelimited connection will raise an exception to the caller. Setting this parameter to True will cause a retry after a backoff.
close()
delete_container(container, response_dict=None)

Wrapper for delete_container()

delete_object(container, obj, query_string=None, response_dict=None)

Wrapper for delete_object()

get_account(marker=None, limit=None, prefix=None, end_marker=None, full_listing=False)

Wrapper for get_account()

get_auth()
get_capabilities(url=None)
get_container(container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, full_listing=False)

Wrapper for get_container()

get_object(container, obj, resp_chunk_size=None, query_string=None, response_dict=None, headers=None)

Wrapper for get_object()

head_account()

Wrapper for head_account()

head_container(container)

Wrapper for head_container()

head_object(container, obj)

Wrapper for head_object()

http_connection()
post_account(headers, response_dict=None)

Wrapper for post_account()

post_container(container, headers, response_dict=None)

Wrapper for post_container()

post_object(container, obj, headers, response_dict=None)

Wrapper for post_object()

put_container(container, headers=None, response_dict=None)

Wrapper for put_container()

put_object(container, obj, contents, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, query_string=None, response_dict=None)

Wrapper for put_object()

class swiftclient.client.HTTPConnection(url, proxy=None, cacert=None, insecure=False, ssl_compression=False, default_user_agent=None)

Make an HTTPConnection or HTTPSConnection

Parameters:
  • url – url to connect to
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
  • cacert – A CA bundle file to use in verifying a TLS server certificate.
  • insecure – Allow to access servers without checking SSL certs. The server’s certificate will not be verified.
  • ssl_compression – SSL compression should be disabled by default and this setting is not usable as of now. The parameter is kept for backward compatibility.
  • default_user_agent – Set the User-Agent header on every request. If set to None (default), the user agent will be “python-swiftclient-<version>”. This may be overridden on a per-request basis by explicitly setting the user-agent header on a call to request().
Raises ClientException:
 

Unable to handle protocol scheme

getresponse()

Adapt requests response to httplib interface

putrequest(full_path, data=None, headers=None, files=None)

Use python-requests files upload

Parameters:
  • data – Use data generator for chunked-transfer
  • files – Use files for default transfer
request(method, full_path, data=None, headers=None, files=None)

Encode url and header, then call requests.request

swiftclient.client.delete_container(url, token, container, http_conn=None, response_dict=None)

Delete a container

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to delete
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP DELETE request failed

swiftclient.client.delete_object(url, token=None, container=None, name=None, http_conn=None, headers=None, proxy=None, query_string=None, response_dict=None)

Delete object

Parameters:
  • url – storage URL
  • token – auth token; if None, no token will be sent
  • container – container name that the object is in; if None, the container name is expected to be part of the url
  • name – object name to delete; if None, the object name is expected to be part of the url
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • headers – additional headers to include in the request
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP DELETE request failed

swiftclient.client.encode_utf8(value)
swiftclient.client.get_account(url, token, marker=None, limit=None, prefix=None, end_marker=None, http_conn=None, full_listing=False)

Get a listing of containers for the account.

Parameters:
  • url – storage URL
  • token – auth token
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • end_marker – end_marker query
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
Returns:

a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase.

Raises ClientException:
 

HTTP GET request failed

swiftclient.client.get_auth(auth_url, user, key, **kwargs)

Get authentication/authorization credentials.

The snet parameter is used for Rackspace’s ServiceNet internal network implementation. In this function, it simply adds snet- to the beginning of the host name for the returned storage URL. With Rackspace Cloud Files, use of this network path causes no bandwidth charges but requires the client to be running on Rackspace’s ServiceNet network.

swiftclient.client.get_auth_1_0(url, user, key, snet, **kwargs)
swiftclient.client.get_capabilities(http_conn)

Get cluster capability infos.

Parameters:http_conn – HTTP connection
Returns:a dict containing the cluster capabilities
Raises ClientException:
 HTTP Capabilities GET failed
swiftclient.client.get_container(url, token, container, marker=None, limit=None, prefix=None, delimiter=None, end_marker=None, path=None, http_conn=None, full_listing=False)

Get a listing of objects for the container.

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to get a listing for
  • marker – marker query
  • limit – limit query
  • prefix – prefix query
  • delimiter – string to delimit the queries on
  • end_marker – marker query
  • path – path query (equivalent: “delimiter=/” and “prefix=path/”)
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • full_listing – if True, return a full listing, else returns a max of 10000 listings
Returns:

a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase.

Raises ClientException:
 

HTTP GET request failed

swiftclient.client.get_keystoneclient_2_0(auth_url, user, key, os_options, **kwargs)

Authenticate against an auth 2.0 server.

We are using the keystoneclient library for our 2.0 authentication.

swiftclient.client.get_object(url, token, container, name, http_conn=None, resp_chunk_size=None, query_string=None, response_dict=None, headers=None)

Get an object

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • resp_chunk_size – if defined, chunk size of data to read. NOTE: If you specify a resp_chunk_size you must fully read the object’s contents before making another request.
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
  • headers – an optional dictionary with additional headers to include in the request
Returns:

a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase.

Raises ClientException:
 

HTTP GET request failed

swiftclient.client.head_account(url, token, http_conn=None)

Get account stats.

Parameters:
  • url – storage URL
  • token – auth token
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a dict containing the response’s headers (all header names will be lowercase)

Raises ClientException:
 

HTTP HEAD request failed

swiftclient.client.head_container(url, token, container, http_conn=None, headers=None)

Get container stats.

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to get stats for
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a dict containing the response’s headers (all header names will be lowercase)

Raises ClientException:
 

HTTP HEAD request failed

swiftclient.client.head_object(url, token, container, name, http_conn=None)

Get object info

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – object name to get info for
  • http_conn – HTTP connection object (If None, it will create the conn object)
Returns:

a dict containing the response’s headers (all header names will be lowercase)

Raises ClientException:
 

HTTP HEAD request failed

swiftclient.client.http_connection(*arg, **kwarg)
Returns:tuple of (parsed url, connection object)
swiftclient.client.http_log(args, kwargs, resp, body)
swiftclient.client.post_account(url, token, headers, http_conn=None, response_dict=None)

Update an account’s metadata.

Parameters:
  • url – storage URL
  • token – auth token
  • headers – additional headers to include in the request
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP POST request failed

swiftclient.client.post_container(url, token, container, headers, http_conn=None, response_dict=None)

Update a container’s metadata.

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to update
  • headers – additional headers to include in the request
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP POST request failed

swiftclient.client.post_object(url, token, container, name, headers, http_conn=None, response_dict=None)

Update object metadata

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name that the object is in
  • name – name of the object to update
  • headers – additional headers to include in the request
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP POST request failed

swiftclient.client.prepare_unicode_headers(self, headers)
swiftclient.client.put_container(url, token, container, headers=None, http_conn=None, response_dict=None)

Create a container

Parameters:
  • url – storage URL
  • token – auth token
  • container – container name to create
  • headers – additional headers to include in the request
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Raises ClientException:
 

HTTP PUT request failed

swiftclient.client.put_object(url, token=None, container=None, name=None, contents=None, content_length=None, etag=None, chunk_size=None, content_type=None, headers=None, http_conn=None, proxy=None, query_string=None, response_dict=None)

Put an object

Parameters:
  • url – storage URL
  • token – auth token; if None, no token will be sent
  • container – container name that the object is in; if None, the container name is expected to be part of the url
  • name – object name to put; if None, the object name is expected to be part of the url
  • contents – a string or a file like object to read object data from; if None, a zero-byte put will be done
  • content_length – value to send as content-length header; also limits the amount read from contents; if None, it will be computed via the contents or chunked transfer encoding will be used
  • etag – etag of contents; if None, no etag will be sent
  • chunk_size – chunk size of data to write; it defaults to 65536; used only if the contents object has a ‘read’ method, e.g. file-like objects, ignored otherwise
  • content_type – value to send as content-type header; if None, an empty string value will be sent
  • headers – additional headers to include in the request, if any
  • http_conn – HTTP connection object (If None, it will create the conn object)
  • proxy – proxy to connect through, if any; None by default; str of the format ‘http://127.0.0.1:8888‘ to set one
  • query_string – if set will be appended with ‘?’ to generated path
  • response_dict – an optional dictionary into which to place the response - status, reason and headers
Returns:

etag

Raises ClientException:
 

HTTP PUT request failed

swiftclient.client.quote(value, safe='/')

Patched version of urllib.quote that encodes utf8 strings before quoting. On Python 3, call directly urllib.parse.quote().

swiftclient.client.store_response(resp, response_dict)

store information about an operation into a dict

Parameters:
  • resp – an http response object containing the response headers
  • response_dict – a dict into which are placed the status, reason and a dict of lower-cased headers

swiftclient.exceptions

exception swiftclient.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=0, http_reason='', http_device='', http_response_content='')

Bases: exceptions.Exception

exception swiftclient.exceptions.SkipTest

Bases: exceptions.Exception

swiftclient.multithreading

class swiftclient.multithreading.MultiThreadingManager(print_stream=<open file '<stdout>', mode 'w' at 0x7f19171e9150>, error_stream=<open file '<stderr>', mode 'w' at 0x7f19171e91e0>)

Bases: object

One object to manage context for multi-threading. This should make bin/swift less error-prone and allow us to test this code.

This object is a context manager and returns itself into the context. When entering the context, two printing threads are created (see below) and they are waited on and cleaned up when exiting the context.

A convenience method, queue_manager(), is provided to create a QueueFunctionManager context manager (a thread-pool with an associated input queue for work items).

Also, thread-safe printing to two streams is provided. The print_msg() method will print to the supplied print_stream (defaults to sys.stdout) and the error() method will print to the supplied error_stream (defaults to sys.stderr). Both of these printing methods will format the given string with any supplied *args (a la printf). On Python 2, Unicode messages are encoded to utf8.

The attribute self.error_count is incremented once per error message printed, so an application can tell if any worker threads encountered exceptions or otherwise called error() on this instance. The swift command-line tool uses this to exit non-zero if any error strings were printed.

Parameters:
  • print_stream – The stream to which print_msg() sends formatted messages
  • error_stream – The stream to which error() sends formatted messages

On Python 2, Unicode messages are encoded to utf8.

error(msg, *fmt_args)
print_headers(headers, meta_prefix='', exclude_headers=None, offset=14)
print_items(items, offset=14, skip_missing=False)
print_msg(msg, *fmt_args)
queue_manager(func, thread_count, *args, **kwargs)
class swiftclient.multithreading.QueueFunctionManager(func, thread_count, thread_manager, thread_args=None, thread_kwargs=None, error_counter=None, connection_maker=None)

Bases: object

A context manager to handle the life-cycle of a single Queue and a list of associated QueueFunctionThread instances.

This class is not usually instantiated directly. Instead, call the MultiThreadingManager.queue_manager() object method, which will return an instance of this class.

When entering the context, thread_count QueueFunctionThread instances are created and started. The input queue is returned. Inside the context, any work item put into the queue will get worked on by one of the QueueFunctionThread instances.

When the context is exited, all threads are sent a StopWorkerThreadSignal instance and then all threads are waited upon. Finally, any exceptions from any of the threads are reported on via the supplied thread_manager‘s error() method. If an error_counter list was supplied on instantiation, its first element is incremented once for every exception which occurred.

Parameters:
  • func – The worker function which will be passed into each QueueFunctionThread‘s constructor.
  • thread_count – The number of worker threads to run.
  • thread_manager – An instance of MultiThreadingManager.
  • thread_args – Optional positional arguments to be passed into each invocation of func after the de-queued work item.
  • thread_kwargs – Optional keyword arguments to be passed into each invocation of func. If a list is supplied as the store_results keyword argument, it will be filled with every result of invoking func in all threads.
  • error_counter – Optional list containing one integer. If supplied, the list’s first element will be incremented once for each exception in any thread. This happens only when exiting the context.
  • connection_maker – Optional callable. If supplied, this callable will be invoked once per created thread, and the result will be passed into func after the de-queued work item but before thread_args and thread_kwargs. This is used to ensure each thread has its own connection to Swift.
class swiftclient.multithreading.QueueFunctionThread(queue, func, *args, **kwargs)

Bases: threading.Thread

Calls func` for each item in queue; func is called with a de-queued item as the first arg followed by *args and **kwargs.

Any exceptions raised by func are stored in self.exc_infos.

If the optional kwarg store_results is specified, it must be a list and each result of invoking func will be appended to that list.

Putting a StopWorkerThreadSignal instance into queue will cause this thread to exit.

Parameters:
  • queue – A Queue object from which work jobs will be pulled.
  • func – A callable which will be invoked with a dequeued item followed by *args and **kwargs.
  • *args – Optional positional arguments for func.
  • **kwargs – Optional kwargs for func. If the kwarg store_results is specified, its value must be a list, and every result from invoking func will be appended to the supplied list. The kwarg store_results will not be passed into func.
run()
class swiftclient.multithreading.StopWorkerThreadSignal

Bases: object

Table Of Contents

Previous topic

SwiftClient Web

This Page