Using Webhooks

Webhooks are a way for web apps to get real-time notifications when users' files and datastores change in Dropbox.

Once you register a URI to receive webhooks, Dropbox will send an HTTP request to that URI every time there's a change for any of your app's registered users.

In this tutorial, we'll walk through a very simple example app that uses webhooks to convert Markdown files to HTML. Thanks to webhooks, the app will convert files as soon as they're added to Dropbox. The sample app is running at mdwebhook.herokuapp.com, and the full source code is available on GitHub.

The verification request

To set up a new webhook, find your app in the App Console, and add the full URI for your webhook (e.g. https://www.example.com/dropbox-webhook) in the "Webhooks" section. Note that the URI needs to be one publicly accessible over the internet. For example, 127.0.0.1 and localhost URIs will not work, since Dropbox's servers will not be able to contact your local computer.

Once you enter your webhook URI, an initial "verification request" will be made to that URI. This verification is an HTTP GET request with a query parameter called challenge. Your app needs to respond by echoing back that challenge parameter. The purpose of this verification request is to demonstrate that your app really does want to receive notifications at that URI. If you accidentally entered the wrong URI (or if someone maliciously entered your server as their webhook), your app would fail to respond correctly to the challenge request, and Dropbox would not send any notifications to that URI.

The following is Python code (using the Flask framework) that responds correctly to a verification request:

@app.route('/webhook', methods=['GET'])
def verify():
    '''Respond to the webhook verification (GET request) by echoing back the challenge parameter.'''

    return request.args.get('challenge')

If your app responds correctly to the challenge request, Dropbox will start sending notifications to your webhook URI every time one of your users adds, removes, or changes a file. If your app fails to respond correctly, you'll see an error message in the App Console telling you about the problem.

Receiving notifications

Once your webhook URI is added, your app will start receiving "notification requests" every time a user's files change. A notification request is an HTTP POST request with a JSON body. The JSON has the following format:

{
    "delta": {
        "users": [
            12345678,
            23456789,
            ...
        ]
    }
}

Note that the payload of the notification request does not include the actual file changes. It only informs your app of which users have changes. You will typically want to call /delta to get the latest changes for each user in the notification, keeping track of the latest cursor for each user as you go.

Every notification request will include a header called X-Dropbox-Signature that includes an HMAC-SHA256 signature of the request body, using your app secret as the signing key. This lets your app verify that the notification really came from Dropbox. It's a good idea (but not required) that you check the validity of the signature before processing the notification.

Below is Python code (again using Flask) that validates a notification request and then calls a function process_user once for each user ID in the payload:

from hashlib import sha256
import hmac
import threading

@app.route('/webhook', methods=['POST'])
def webhook():
    '''Receive a list of changed user IDs from Dropbox and process each.'''

    # Make sure this is a valid request from Dropbox
    signature = request.headers.get('X-Dropbox-Signature')
    if signature != hmac.new(APP_SECRET, request.data, sha256).hexdigest():
        abort(403)

    for uid in json.loads(request.data)['delta']['users']:
        # We need to respond quickly to the webhook request, so we do the
        # actual work in a separate thread. For more robustness, it's a
        # good idea to add the work to a reliable queue and process the queue
        # in a worker process.
        threading.Thread(target=process_user, args=(uid,)).start()
    return ''

Typically, the code you run in response to a notification will make a call to /delta to get the latest changes for a user. In our sample Markdown-to-HTML converter, we're keeping track of each user's OAuth access token and their latest delta cursor in Redis (a key-value store). This is the implementation of process_user for our Markdown-to-HTML converter sample app:

def process_user(uid):
    '''Call /delta for the given user ID and process any changes.'''

    # OAuth token for the user
    token = redis_client.hget('tokens', uid)

    # /delta cursor for the user (None the first time)
    cursor = redis_client.hget('cursors', uid)

    client = DropboxClient(token)
    has_more = True

    while has_more:
        result = client.delta(cursor)

        for path, metadata in result['entries']:

            # Ignore deleted files, folders, and non-markdown files
            if (metadata is None or
                    metadata['is_dir'] or
                    not path.endswith('.md')):
                continue

            # Convert to Markdown and store as <basename>.html
            html = markdown(client.get_file(path).read())
            client.put_file(path[:-3] + '.html', html, overwrite=True)

        # Update cursor
        cursor = result['cursor']
        redis_client.hset('cursors', uid, cursor)

        # Repeat only if there's more to do
        has_more = result['has_more']

Best practices

Always respond to webhooks quickly

Your app only has ten seconds to respond to webhook requests. For the verification request, this is never really an issue, since your app doesn't need to do any real work to respond. For notification requests, however, your app will usually do something that takes time in response to the request. For example, an app processing file changes will call /delta and then process the changed files. (In our Markdown example, we needed to download each Markdown file, convert it to HTML, and then upload the result.) It's important to keep in mind that delta payloads can sometimes be very large and require multiple round-trips to the Dropbox API, particularly when a new user first links your app and has a lot of files.

To make sure you can always respond within ten seconds, you should always do your work on a separate thread (as in the simple example above) or asynchronously using a queue.

Manage concurrency

When a user makes a number of changes in rapid succession, your app is likely to receive multiple notifications for the same user at roughly the same time. If you're not careful about how you manage concurrency, your app can end up processing the same changes for the same user more than once.

For some applications, this is not a serious issue. In our Markdown-to-HTML conversion app, if the same changes are processed more than once, a file just ends up getting overwritten with the same content, so no harm is done. Work that can be repeated without changing the outcome is called idempotent. If your app's actions are always idempotent, you don't need to worry much about concurrency.

Unfortunately, not every app can be made idempotent easily. For example, suppose you have an app that sends email every time a certain file or datastore is changed. To avoid sending duplicate emails, you need to make sure that your app never processes the same user on multiple threads/processes at the same time. The simplest solution is to use leases. When a thread or process starts processing a certain user, it will first acquire a lease on that user, giving it exclusive access.

Contents

Choose a language