Datastore API for Python Documentation

Datastores are an easy way to keep an app's per-user data — like settings, bookmarks, or game state — in sync across multiple devices and operating systems. Datastores are simple embedded databases, which are synced to Dropbox.

This reference details the full set of classes needed when working with datastores. You can also read the Datastore API tutorial for a detailed example of how to use them.

General information

The classes in the Common section live in the package dropbox.client. The classes in the Datastores section live in the package dropbox.datastore. Typical imports look like this:

from dropbox.client import DropboxClient, DropboxOAuth2Flow, DropboxOAuth2FlowNoRedirect
from dropbox.rest import ErrorResponse, RESTSocketError
from dropbox.datastore import DatastoreError, DatastoreManager, Date, Bytes

Unlike the client-side datastore APIs (iOS, OS X, Android and JavaScript), the Python datastore API does not implement automatic conflict resolution. Instead, if commit() fails, you must start over. You can use the transaction() method for this, which allows you to retry the transaction several times before giving up.

The Python API is not thread-safe. If you want to use the same Datastore object from multiple threads you should manage your own locking. The exception is the DatastoreManager class; all its methods are thread-safe. Also, static methods are thread-safe.

Shared datastores

Apps may want to share data between users. With the Datastore API, apps can share a datastore across multiple Dropbox accounts. The unit of sharing is a single datastore, and one or more datastores may be shared between accounts. Any datastore with a shareable ID (see "Private or shareable datastores" below) can be shared by assigning roles to principals, creating an access control list. Any Dropbox account with the correct permissions will then be able to open the shared datastore by ID.

There are two available principals to whom you may apply a role:

  • Datastore.PUBLIC – The role will apply to all Dropbox users.
  • Datastore.TEAM – The role will apply to everyone on the user's team (only applicable for Dropbox for Business accounts).

There are four available roles:

  • Datastore.NONE – The principal has no access to this datastore.
  • Datastore.VIEWER – The principal is able to view this datastore.
  • Datastore.EDITOR – The principal is able to edit this datastore.
  • Datastore.OWNER – The principal is the owner of this datastore. This role cannot be assigned directly. The user who created a datastore is always that datastore's owner.

An example of sharing a datastore is included in the Datastore API tutorial.

Private or shareable datastores

Datastores can be created in 2 different ways, and the choice affects how they can be used. Each type of datastore can be identified by its ID.

  • Datastores with private IDs are created using DatastoreManager open_or_create_datastore(id). Private IDs are meaningful to the developer of the app, such as "default" (for the default datastore) or "settings". The scope of private IDs is the current user-app pair. Two different devices can create datastores with the same private ID while offline, and their data will be merged when they come online.

    Private datastore IDs can be 1-64 characters containing only lowercase letters, digits, dot, hyphen or underscore, and they must not begin or end with dot.

  • Datastores with shareable IDs are created using DatastoreManager create_datastore() which allows them to be shared between users. Their IDs are auto-generated and are not only unique for the user-app pair, they're also unique across Dropbox. Shareable IDs are more appropriate when treating each datastore as an individual document where users may create an unknown number of them.

    Shareable datastore IDs (generated by the SDK) are a dot followed by 1-63 characters containing uppercase letters, lowercase letters, hyphen or underscore.

Storage size limits

Datastores have limits on their maximum size to ensure good performance across platforms. You should keep these in mind as guidelines when modeling your datastores.

Your app can store up to 5MB of data across all its datastores without counting against the user's storage quota. Any data beyond the first 5MB is factored into the user's Dropbox storage quota, and writing can be limited in these cases when a user is over quota. Sizes are calculated as:

  • The size of a datastore is calculated by summing the size of all records, plus 1000 bytes for the datastore itself.
  • The size of a record is calculated by summing the size of all values in all fields, plus 100 bytes for the record itself.
  • The size of a field is a fixed 100 bytes for the field itself plus:
    • for string or bytes values, the length in bytes of the value.
    • for List values, 20 bytes for each list element plus the size of each element.
    • for all other types, no additional contribution.
  • The size of changes made in a transaction is calculated by summing the size of each change, plus 100 bytes for the delta itself. The size of each change is calculated by summing the size of the values contained in the change (for field updates and list put and insert operations) plus 100 bytes for the change itself.

Maximum record size100 KiB
Maximum number of records per datastore100,000
Maximum datastore size10 MiB
Maximum size of a single transaction2 MiB

DropboxClient

This class lets you make Dropbox API calls. You'll need to obtain an OAuth 2 access token first. You can get an access token using either DropboxOAuth2Flow or DropboxOAuth2FlowNoRedirect.

All of the API call methods can raise a dropbox.rest.ErrorResponse exception if the server returns a non-200 or invalid HTTP response. Note that a 401 return status at any point indicates that the access token you're using is no longer valid and the user must be put through the OAuth 2 authorization flow again.

Constructors

  • DropboxClient(oauth2_access_token, locale=None, rest_client=None)

    Construct a DropboxClient instance.

    Parameters
    oauth2_access_token
    An OAuth 2 access token (string). For backwards compatibility this may also be a DropboxSession object (see create_oauth2_access_token()).
    locale
    The locale of the user of your application. For example "en" or "en_US". Some API calls return localized data and error messages; this setting tells the server which locale to use. By default, the server uses "en_US".
    rest_client
    Optional dropbox.rest.RESTClient-like object to use for making requests.

DropboxOAuth2Flow

OAuth 2 authorization helper. Use this for web apps.

OAuth 2 has a two-step authorization process. The first step is having the user authorize your app. The second involves getting an OAuth 2 access token from Dropbox.

Example:

from dropbox.client import DropboxOAuth2Flow, DropboxClient

def get_dropbox_auth_flow(web_app_session):
    redirect_uri = "https://my-web-server.org/dropbox-auth-finish"
    return DropboxOAuth2Flow(APP_KEY, APP_SECRET, redirect_uri,
                             web_app_session, "dropbox-auth-csrf-token")

# URL handler for /dropbox-auth-start
def dropbox_auth_start(web_app_session, request):
    authorize_url = get_dropbox_auth_flow(web_app_session).start()
    redirect_to(authorize_url)

# URL handler for /dropbox-auth-finish
def dropbox_auth_finish(web_app_session, request):
    try:
        access_token, user_id, url_state = \
                get_dropbox_auth_flow(web_app_session).finish(request.query_params)
    except DropboxOAuth2Flow.BadRequestException, e:
        http_status(400)
    except DropboxOAuth2Flow.BadStateException, e:
        # Start the auth flow again.
        redirect_to("/dropbox-auth-start")
    except DropboxOAuth2Flow.CsrfException, e:
        http_status(403)
    except DropboxOAuth2Flow.NotApprovedException, e:
        flash('Not approved?  Why not?')
        return redirect_to("/home")
    except DropboxOAuth2Flow.ProviderException, e:
        logger.log("Auth error: %s" % (e,))
        http_status(403)
Constructors

  • DropboxOAuth2Flow(consumer_key, consumer_secret, redirect_uri, session, csrf_token_session_key, locale=None, rest_client=None)

    Construct an instance.

    Parameters
    consumer_key
    Your API app's "app key".
    consumer_secret
    Your API app's "app secret".
    redirect_uri
    The URI that the Dropbox server will redirect the user to after the user finishes authorizing your app. This URI must be HTTPS-based and pre-registered with the Dropbox servers, though localhost URIs are allowed without pre-registration and can be either HTTP or HTTPS.
    session
    A dict-like object that represents the current user's web session (will be used to save the CSRF token).
    csrf_token_session_key
    The key to use when storing the CSRF token in the session (for example: "dropbox-auth-csrf-token").
    locale
    The locale of the user of your application. For example "en" or "en_US". Some API calls return localized data and error messages; this setting tells the server which locale to use. By default, the server uses "en_US".
    rest_client
    Optional dropbox.rest.RESTClient-like object to use for making requests.

DropboxOAuth2FlowNoRedirect

OAuth 2 authorization helper for apps that can't provide a redirect URI (such as the command-line example apps).

Example:

from dropbox.client import DropboxOAuth2FlowNoRedirect, DropboxClient
from dropbox import rest as dbrest

auth_flow = DropboxOAuth2FlowNoRedirect(APP_KEY, APP_SECRET)

authorize_url = auth_flow.start()
print "1. Go to: " + authorize_url
print "2. Click \"Allow\" (you might have to log in first)."
print "3. Copy the authorization code."
auth_code = raw_input("Enter the authorization code here: ").strip()

try:
    access_token, user_id = auth_flow.finish(auth_code)
except dbrest.ErrorResponse, e:
    print('Error: %s' % (e,))
    return

c = DropboxClient(access_token)
Constructors

  • DropboxOAuth2FlowNoRedirect(consumer_key, consumer_secret, locale=None, rest_client=None)

    Construct an instance.

    Parameters
    consumer_key
    Your API app's "app key"
    consumer_secret
    Your API app's "app secret"
    locale
    The locale of the user of your application. For example "en" or "en_US". Some API calls return localized data and error messages; this setting tells the server which locale to use. By default, the server uses "en_US".
    rest_client
    Optional dropbox.rest.RESTClient-like object to use for making requests.

ErrorResponse

Exception raised when DropboxClient exeriences a problem.

For example, this is raised when the server returns an unexpected non-200 HTTP response.

Properties

  • status

    HTTP response status (an int).

  • reason

    HTTP response reason (a string).

  • headers

    HTTP response headers (a list of (header, value) tuples).

  • body

    HTTP response body (string or JSON dict).

  • error_msg

    Error message for developer (optional).

  • user_error_msg

    Error message for end user (optional).

RESTSocketError

A light wrapper for socket.error that adds some more information.

DatastoreError

Exception raised for datastore-specific error conditions.

This is the base class for more specific exception classes.

Properties

  • resp

    The JSON dict that was returned by the server.

DatastoreNotFoundError

Exception raised when attempting to open a non-existent datastore.

Derives from DatastoreError.

DatastoreConflictError

Exception raised when the server reports a conflict.

Derives from DatastoreError.

DatastoreManager

A manager for datastores.

In order to work with datastores you must first create an instance of this class, passing its constructor a dropbox.client.DropboxClient instance.

The methods here let you open or create datastores and retrieve the list of datastores.

This class has no state except for a reference to the dropbox.client.DropboxClient, which itself is thread-safe; hence, all methods of this class are thread-safe.

Static Methods

  • make_cursor_map(datastores, deltamap)

    Utility to construct a datastores argument for await().

    Parameters
    datastores
    A list of Datastore objects.
    deltamap
    An data structure as returned by await() in its deltamap return value. This may be None or it may be a dict mapping Datastore objects to values that are either lists of deltas or None.
    Returns
    A dict mapping Datastore objects to revisions, suitable to pass as the datastores parameter to await(). This will normally just map the datastores from the datastores parameter to their current revision; however, datastores that are deleted or invalid according to deltamap are excluded from the dict, and for datastores that have one or more deltas in deltamap, the revision is set to one higher than the revision of the last delta.

    Using this function will reduce redundant server roundtrips in multi-threaded apps that call await() in a background thread and then pass the received deltas to the main thread through some kind of queue.

Constructors
Instance Methods

  • get_client()

    Return the dropbox.client.DropboxClient object used.

  • open_default_datastore()

    Open the default datastore for this account, creating it if needed.

    This is a shorthand for open_or_create_datastore(DEFAULT_DATASTORE_ID).

    Returns
    A Datastore instance.

  • open_datastore(id)

    Open an existing datastore given its ID (a string).

    Returns
    A Datastore instance.

  • open_or_create_datastore(id)

    Open a private datastore, creating it if it does not yet exist.

    The ID must not start with a dot.

    Returns
    A Datastore instance.

  • create_datastore()

    Create a new datastore with a randomly assigned ID.

    The assigned ID will start with a dot.

    Returns
    A Datastore instance.

  • open_raw_datastore(id, handle, role=None)

    Create a new Datastore object without going to the server.

    You can use this to save a server roundtrip when opening a datastore given a DatastoreInfo object returned by list_datastores():

    def open_from_info(mgr, info):
    ds = mgr.open_raw_datastore(info.id, info.handle, info.role)
    ds.load_snapshot()
    return ds

  • delete_datastore(id, should_purge=False)

    Delete a datastore given its ID.

  • list_datastores()

    List the existing datastores for this account.

    Returns
    A list of DatastoreInfo objects.

  • await(token=None, datastores=None)

    Wait until certain changes occur.

    This methods implements a flexible and efficient long-polling mechanism that can be used to be notified of changes to specific datastores and/or to the list of datastores itself (for the current account).

    Parameters
    token
    An optional token that represents a hash of the list of datastores, computed by the server. If this parameter is present and non-empty, await() will return when the list of datastores has changed in a way that would cause a different token to be computed, such as when a datastore is created or deleted. The token should be obtained from the previous await() call; as a special case, the value '.' forces the call to return immediately with a fresh token (as does any outdated token).
    datastores
    An optional list of Datastore instances or dict mapping such instances to revision numbers. The instances represents currently open datastores for which you are interested in receiving updates. If this parameter is a list of instances, the revision to compare is retrieved from each instance using Datastore.get_rev(). If this parameter is present and non-empty, await() will return whenever a new revision is available for any of those datastores.

    The call also returns after a certain amount of time passes without activity. The timeout is controlled by the server; it is currently approximately one minute.

    Returns

    A (token, dsinfos, deltamap) tuple. The items are as follows:

    token
    A new token, or the value of the token parameter if there are no changes to the list of datastores. You should pass this to the next await() call.
    dsinfo
    The full list of DatastoreInfo objects (as returned by list_datastores()) if there is a new token, otherwise None.
    deltamap
    Either a mapping indicating which of the given datastores were changed, or None if there are no changes to datastores to report. If it is a mapping, each key is a Datastore, and the corresponding value is either a non-empty list of deltas, or None if that datastore is deleted or is otherwise invalid. Datastores that were not changed (and are still valid) will not be present.

    Unlike Datastore.load_deltas() and Datastore.await_deltas(), await() does not apply the deltas returned in deltamap to the respective datastores; that is the caller's responsibility. For example:

    for ds, deltas in deltamap.items():
    if deltas is not None:
        ds.apply_deltas(deltas)
    else:
        # ds has been deleted
Constants

DatastoreInfo

A read-only record of information about a Datastore.

Instances of this class are returned by DatastoreManager.list_datastores().

Properties

  • id

    The datastore ID (a string).

  • handle

    The datastore handle (a string).

  • rev

    The datastore revision (an integer >= 0).

  • title

    The datastore title (string or None).

  • mtime

    The time of last modification (Date or None).

  • effective_role

    The current user's effective role (Datastore.OWNER, Datastore.EDITOR or Datastore.VIEWER).

Datastore

An object representing a datastore.

A datastore holds a set of tables identified by table IDs, each of which holds a set of records identified by record IDs. A record holds a set of field values identified by field names. The Datastore object keeps a snapshot of the current content (all tables, records and fields) in memory and supports simple queries.

Changes to a datastore are made through methods on the Table and Record classes, as well as the List class (which represents a composite field value).

Changes are not immediately sent to the server. Instead, the datastore keeps a list of changes in memory; these are sent to the server by the commit() method. The load_deltas() method retrieves new changes from the server and incorporates them into the current snapshot. Those changes that have not yet been sent to the server can be undone using the rollback() method. Finally, the transaction() method combines the functionality of these into a more powerful operation that can retry sets of changes specified by a callback function.

Do not instantiate this class directly. Use the methods on DatastoreManager instead.

Static Methods

  • is_valid_id(id)

    A helper method to check for a valid datastore ID.

    There are actually two types of datastore IDs, which are called private IDs and shareable IDs.

    Private datastores are created with DatastoreManager.open_default_datastore() or DatastoreManager.open_or_create_datastore(), and the app has control over the name. Valid private datastore IDs are 1-64 characters long and may contain the following characters: a-z 0-9 . - _ . However the first and last character cannot be dots. Note that upper case is not allowed.

    Shareable datastores are created with DatastoreManager.create_datastore(); the name is a dot followed by a random-looking sequence of characters assigned by the SDK. Valid shareable datastore IDs are a dot followed by 1-63 dbase64 characters (which are a-z A-Z 0-9 - _). Note that upper case is allowed.

    The DatastoreManager.open_datastore() and DatastoreManager.open_raw_datastore() methods can open either type of datastores.

  • is_valid_shareable_id(id)

    A helper method to check for a valid shareable datastore ID.

    This is a valid datastore ID starting with a '.'.

Instance Methods

  • get_id()

    Return the ID of this datastore (a string).

  • is_shareable()

    Return whether this is a shareable datastore.

  • is_writable()

    Return whether this datastore is writable.

    Always true for private datastores. False iff role==:const:VIEWER for shareable datastores.

  • get_handle()

    Return the handle of this datastore (a string).

  • get_rev()

    Return the current revision of this datastore (an integer >= 0).

  • get_manager()

    Return the DatastoreManager to which this datastore belongs.

  • get_mtime()

    Return time this datastore was last modified, if known.

    This value is automatically set to the current time by commit().

    Returns
    A Date or None.

  • get_title()

    Return the title of this datastore (a string or None).

    The title is primarily useful for apps that use shareable datastores to represent documents created by the user. Using set_title() the title can be set to a string chosen by the user, and DatastoreManager.list_datastores() will return the titles (see DatastoreInfo). The app can then show the user a list of documents containing the title and time of last modification for each document without needing to open all datastores.

  • set_title(title)

    Set the title of this datastore (a string or None).

    Since this operation is implemented by updating a reserved table, you must call commit() to send this change to the server.

  • get_record_count()

    Return the number of records in this datastore.

  • get_size()

    Return the size in bytes of this datastore.

    The overall size of a datastore is calculated by summing the size of all records, plus the base size of an empty datastore itself.

  • get_pending_changes_size()

    Return the size in bytes of changes made since the last commit().

    If there are any pending changes, the total size is given by summing the size of those changes and BASE_DELTA_SIZE. If there are no pending changes, the total size is zero.

  • get_effective_role()

    Return the effective role for the current user.

    This can return OWNER, EDITOR or VIEWER.

    For a private datastore this always returns OWNER.

  • list_roles()

    Return the full ACL, as a dict mapping principals to roles.

    This is only supported for shareable datastores.

  • get_role(principal)

    Return the role for a principal.

    This can return OWNER, EDITOR, VIEWER, or None.

    The principal must be TEAM or PUBLIC.

    This is only supported for shareable datastores.

    This method only returns the role explicitly set for the given principal in the ACL; it is equivalent to ds.list_roles().get(principal). The effective role for a principal may be different; it is affected by the full ACL as well as by team membership and ownership.

    To get the effective role for the current user, use get_effective_role().

  • set_role(principal, role)

    Set a principal's role.

    The principal must be TEAM or PUBLIC. The role must be EDITOR or VIEWER.

    If the principal already has a role it is updated.

    This is only supported for writable, shareable datastores.

  • delete_role(principal)

    Delete a principal's role.

    The principal must be TEAM or PUBLIC.

    The principal may but need not have a role.

    This is only supported for writable, shareable datastores.

  • load_snapshot()

    Load the datastore with a snapshot retrieved from the server.

    All previously loaded content of the datastore is discarded, including pending changes.

    This is automatically called by most of the open_*() methods, so there is normally no reason to call this.

  • apply_snapshot(rev, snapshot)

    Restore the datastore from a revision and a snapshot.

    All previously loaded content of the Datastore object is discarded, including pending changes.

    Normally this method is called internally by load_snapshot(). It may also be called with a revision and snapshot obtained previously from get_rev() and get_snapshot().

  • get_snapshot()

    Return a snapshot of the datastore.

    A snapshot is a list of dicts with keys 'tid', 'rowid', and 'data', where 'tid' maps to the table ID, 'rowid' maps to a record ID, and 'data' maps to a JSON-encoded record, i.e. a dict mapping field names to JSON-encoded values.

    Together with the revision (which you can obtain from get_rev()) this comprises the mutable state of a datastore. You may restore a Datastore object to a given state using apply_snapshot().

  • await_deltas()

    Wait for and incorporate changes to this datastore.

    It is an error to call this method if the datastore has pending changes.

    Returns
    A dict mapping table IDs to sets of records, see apply_deltas().

  • load_deltas()

    Load new changes retrieved from the server into the datastore.

    All previously loaded content is preserved, unless explicitly deleted or modified by one of the loaded changes.

    It is an error to call this method if the datastore has pending changes.

    Calling ds.load_deltas() is equivalent to:

    deltas = ds.fetch_deltas()
ds.apply_deltas(deltas)
    Returns
    A dict mapping table IDs to sets of records, see apply_deltas().

  • fetch_deltas()

    Retrieve new changes from the server without applying them.

    This is one of the building blocks of load_deltas(); you probably want to use that instead.

    Returns
    A list of deltas suitable to be passed directly to apply_deltas().

  • apply_deltas(deltas)

    Apply deltas retrieved by some other means.

    It is an error to call this method if the datastore has pending changes.

    Normally this method is called internally by await_deltas() or load_deltas().

    The deltas should be received from the server. Under certain conditions (e.g. when DatastoreManager.await() is called in a background thread) it is possible that the server sends a delta that has already been applied locally. Such deltas are silently ignored.

    Returns
    A dict mapping table IDs to sets of records, indicating the records that were inserted, updated or deleted by the loaded deltas.

  • get_table(tid)

    Get a Table object with the given table ID.

  • list_table_ids()

    List the non-empty tables for this datastore.

    Returns
    A set of strings table IDs (strings).

  • rollback()

    Discard all pending changes since the last successful commit().

  • commit()

    Attempt to commit all pending changes.

    Pending changes are all mutations to a datastore made through Table.insert(), Record.set() and similar methods (inluding mutating List methods).

    To upload pending changes to the server you must use commit(), or transaction(), which calls it.

    This method raises DatastoreConflictError when the server detects a conflict and refuses to accept the changes. The proper response to this exception is to call rollback(), then load_deltas(), and then retry the transaction from the top, or give up and report an error to the user. (The transaction() method implements this higher-level control flow.)

    If there are any changes, this method adds a change that updates the datastore's mtime. If there are no changes, this method is a no-op (and no empty delta will be sent to the server).

  • transaction(callback, *args, max_tries=1)

    Call a callback function and commit changes, with retries.

    When multiple clients try to update a datastore concurrently, it is possible for commit() to raise DatastoreConflictError, indicating a conflict. This function handles the details of handling such failures and retrying the updates. You pass it a callback function which will be called repeatedly until commit() succeeds, or the maximum number of tries is reached.

    The keyword-only parameter max_tries specifies how many times the callback is called before giving up. The default is 1, i.e. call it only once; the recommended value is 4.

    Generally, if you plan to modify a datastore, you should do all your reads and writes in a transaction. On entry, there should be no pending changes.

    Example:

    def do_stuff(record_id):
    record = tasks_table.get(record_id)
    user_count = record.get('user_count')
    record.update(user_count=user_count+1)

datastore.transaction(do_stuff, some_record_id, max_tries=4)

    Extra positional arguments are passed on to the callback function. On success, the return value of the callback is returned.

    When a commit attempt fails, uncommitted changes are rolled back using rollback(), and new changes are retrieved from the server and loaded into the datastore using load_deltas(). This is done before checking whether we are out of tries.

    When giving up, DatastoreError is raised.

    When any other exception occurs (either in the callback or in the commit), uncommitted changes are rolled back and the last exception is re-raised.

  • close()

    Close the datastore.

    The datastore should not be used after this call.

    All pending changes are lost.

Constants

  • DATASTORE_SIZE_LIMIT

    The maximum size in bytes of a datastore.

  • PENDING_CHANGES_SIZE_LIMIT

    The maximum size in bytes of changes that can be queued up between calls to commit().

  • RECORD_COUNT_LIMIT

    The maximum number of records in a datastore.

  • BASE_DATASTORE_SIZE

    The size in bytes of a datastore before accounting for the size of its records.

    The overall size of a datastore is this value plus the size of all records.

  • BASE_DELTA_SIZE

    The size in bytes of a delta before accounting for the size of each change.

    The overall size of a delta is this value plus the size of each change.

  • BASE_CHANGE_SIZE

    The size in bytes of a change before including the size of its values.

    The overall size of a change is this value plus the size of the values in the change.

  • TEAM

    The principal used to get or modify the team role for a datastore.

  • PUBLIC

    The principal used to get or modify the public role for a datastore.

  • OWNER

    The role indicating ownership of a datastore. Owners have full access and their role cannot be changed or removed.

  • EDITOR

    The role indicating edit (i.e., read-write) access. Editors can also modify the role for other principals (except owners).

  • VIEWER

    The role indicating view (i.e. read-only) access. Viewers cannot change any aspect of a datastore.

  • NONE

    The role indicating no access at all.

Table

An object representing a table in a datastore.

You need a Table in order to query or modify the content of the datastore.

Do not instantiate this class directly. Use Datastore.get_table() instead. Calls with the same ID will return the same object.

Static Methods

  • is_valid_id(id)

    A helper method to check for a valid table ID.

    Valid table IDs are 1-64 characters long and may contain the following characters: a-z A-Z 0-9 _ - / . + = . Reserved IDs start with a colon followed by 1-63 characters from that set.

Instance Methods

  • get_id()

    Return the ID of this table (a string).

  • get_datastore()

    Return the Datastore to which this table belongs.

  • get(recordid)

    Return the record with the given record ID.

    If no such record exists, return None.

  • get_or_insert(recordid, **fields)

    Return the record with the given record ID, or create it.

    If a record with the given record ID already exists, it is returned, and the keyword arguments are ignored. If no such record exists, this inserts a record with the given record ID, setting its fields from the keyword arguments.

  • insert(**fields)

    Insert a new record into the table and return it.

    The new record's fields are set from the keyword arguments. A unique record ID is assigned automatically.

  • query(**kwds)

    Query the records in the table.

    If called without arguments, this returns a set of all records in the table.

    If called with keyword arguments, each keyword argument specifies a required value for the corresponding field; only records that have the required field values for all keyword arguments are returned.

    The following example retrieves all records in the 'tasks' table that have a 'done' field whose type is bool and whose value is False:

    to_do = tasks.query(done=False)

    For the purpose of queries, integers and floats are compared using the standard Python equality comparisons.

    Tip: specifying multiple keyword arguments implements a logical 'AND' operation; to implement a logical 'OR' operation, use the union of multiple queries. For example:

    # Assume priority can be 1 (low), 2 (normal), 3 (high)
urgent = tasks.query(done=False, priority=3)
normal = tasks.query(done=False, priority=2)
to_do = urgent | normal

  • _update_record_fields(recordid, fields, change_in_size)

    Update the fields of the record, or delete the record if fields is None.

    This method updates the fields for the recordid and also updates its cached size in bytes and the cached size of the datastore.

Record

An object representing a record in a table in a datastore.

A record has a record ID and zero or more fields. A record belongs to a specific table. Two records are considered equal when they belong to the same table and have the same record ID; equal records by definition have the same fields. Records are hashable.

A field value can be an atomic type or a list of atomic types.

Atomic types are bool, integer (int or long), float, string (unicode or 8-bit str; the latter must be a valid UTF-8 string), or an instance of the special classes Date or Bytes. Note that None is not a valid field value.

Do not instantiate this class directly. Use Table.get(), Table.insert(), Table.get_or_insert() or Table.query() instead.

Static Methods

  • is_valid_id(id)

    A helper method to check for a valid record ID.

    Valid record IDs are 1-64 characters long and may contain the following characters: a-z A-Z 0-9 _ - / . + = . Reserved IDs start with a colon followed by 1-63 characters from that set.

  • is_valid_field(field)

    A helper method to check for a valid field name.

    Valid field names are 1-64 characters long and may contain the following characters: a-z A-Z 0-9 _ - / . + = . Reserved field names start with a colon followed by 1-63 characters from that set.

Instance Methods

  • get_id()

    Return the ID of this record (a string).

  • get_table()

    Return the Table to which this record belongs.

  • get_size()

    Return the size in bytes of this record.

    The overall size of a record is calculated by summing the size of all values in all fields, plus the base size of an empty record itself. A deleted record has a size of zero.

  • get(field)

    Return the value of a field in the record.

    If the record does not have a field by that name, return None.

    If the field value is a list, this returns a List object; mutating that object will modify the field's value in the record.

  • set(field, value)

    Set the value of a field in the record.

    Setting the value to None deletes the field.

  • delete(field)

    Delete the value of a field in the record.

    If the field does not exist this is a no-op.

  • get_fields()

    Return a dict mapping all the fields in the record to their values.

    Modifying the dict will not affect the record in the datastore.

    To enforce this, list values are returned as tuples.

  • update(**kwds)

    Set the value of multiple fields in the record.

    For each keyword argument, the field by that name is set to the corresponding value, except that if the value is None, the field is deleted.

  • delete_record()

    Delete the record from the table.

    If the record is already marked as deleted, this is a no-op.

    A record marked as deleted cannot be re-inserted, cannot be modified, and no longer has any fields. To check for a deleted record, use is_deleted().

  • get_or_create_list(field)

    Get a list field, possibly setting it to an empty list.

    If the field exists, it must be a list. If it does not exist, it is set to an empty list. In either case, a List object representing the field is returned.

  • has(field)

    Inquire whether the record has a given field.

    Return True if the field exists, False if not.

  • is_deleted()

    Inquire whether the record is marked as deleted.

    Return True if the record has been deleted, False if not.

Constants

  • RECORD_SIZE_LIMIT

    The maximum size in bytes of a record.

  • BASE_RECORD_SIZE

    The size in bytes of a record before accounting for the sizes of its fields.

    The overall size of a record is this value plus the sum of the sizes of its fields.

  • BASE_FIELD_SIZE

    The size in bytes of a field before accounting for the sizes of its values.

    The overall size of a field is this value plus:

    • For string and Bytes: the length in bytes of the value.
    • For List: the sum of the size of each list item, where each item's size is computed as the size of the item value plus List.BASE_ITEM_SIZE.
    • For other atomic types: no additional contribution to the size of the field.

Date

A simple immutable object representing a timestamp.

Datastores store timestamps as milliseconds since the Epoch (1/1/1970) in UTC.

To store a timestamp, you must set a field to a Date object; if a field value is a timestamp, getting the value will return a Date.

To construct a Date, pass the constructor a POSIX timestamp as returned by time.time() (and many other standard Python APIs).

You can convert a Date back to a POSIX timestamp by calling float() or int() on it. These conversions take care of the conversion between seconds and milliseconds; milliseconds map to fractions when converting to/from float, and are truncated when converting to int.

You can also convert between Date and naive (tzinfo-less) datetime objects using a choice of UTC or local time, using to_datetime_utc(), from_datetime_utc(), to_datetime_local(), and from_datetime_local(). Note that datetime objects using an explicit tzinfo field are not supported; if you need to work with those you must convert to/from naive datetime objects yourself.

Class Methods

  • from_datetime_utc(dt)

    Convert a datetime.datetime object in UTC to a Date.

    The tzinfo field must be None.

  • from_datetime_local(dt)

    Convert a datetime.datetime object in UTC to a Date.

    The tzinfo field must be None.

Constructors

  • Date(timestamp=None)

    Construct a Date from a timestamp.

    The timestamp is an integer or float specifying seconds since the epoch. It defaults to the current time.

Instance Methods

  • to_datetime_utc()

    Convert a Date to a datetime.datetime object in UTC.

    This sets the tzinfo field to None.

  • to_datetime_local()

    Convert a Date to a datetime.datetime object in local time.

    This set the tzinfo field to None.

Bytes

A simple immutable object representing a binary string.

Datastores transmit binary strings using a base64 encoding.

Because Python 2 uses ambiguous representations of binary strings, you must wrap binary strings in this class in order to store them in a datastore. 8-bit strings not wrapped this way are assumed to represent text and must use the UTF-8 encoding.

To construct a Bytes, pass the constructor a str instance, a buffer instance, or an array.array instance whose typecode indicate a one-byte-wide data type (i.e. 'c', 'b' or 'B').

To convert a Bytes to a raw byte string, call bytes() on it.

Constructors

  • Bytes(blob)

    Construct a Bytes from an 8-bit string.

List

A wrapper for a list value.

When a field contains a list value, retrieving the field using Record.get() returns a List object. This object behaves like a mutable sequence, but mutating it (e.g., replacing an item with a new value) will mutate the list value in the record.

A List object knows the record and field to which it refers. Multiple List objects may refer to the same record and field.

List objects are compared by value (i.e., the sequence of items they contain, not the record and field to which they refer). They can also be compared to regular tuples and lists.

Several methods available for regular lists are available for List objects, when in doubt, consult the documentation below. Some methods unique to List objects also exist.

Negative indices are supported in the usual fashion.

Do not instantiate this class directly. Use Record.get() or Record.get_or_create_list() instead.

Instance Methods

  • get_record()

    Return the Record to which this List refers.

  • get_field()

    Return the field name (a string) to which this List refers.

  • insert(index, value)

    Insert a value into the list at a given index.

  • append(value)

    Append a value to the end of the list.

  • move(index, newindex)

    Move the list item at index to position newindex.

    This is most easily explained as follows: first delete the item at position index; then re-insert it at position newindex.

Constants

  • BASE_ITEM_SIZE

    The size in bytes of a list item.

    The overall size of a list item is this value plus the size of the item value.