English
Skip to page content Skip to main navigation

e.g. "templates" or "datastore"

# Accessing the datastore remotely with remote_api

Nick Johnson
February 27, 2009

## Introduction

Often when developing or supporting an App Engine application, it is useful to be able to manipulate the datastore in ways that are not well suited to the request/response model that works so well for serving Web Applications. Previously, doing these sort of operations has entailed workarounds such as app3 or App Rocket. Starting with release 1.1.9 of the App Engine SDK, however, there's a new way to interact with the datastore, in the form of the `remote_api` module. This module allows remote access to the App Engine datastore, using the same APIs you know and love from writing App Engine Apps.

In this article, we'll introduce you to the remote_api module, describe its basic functionality, and show you how to get an interactive console with access to your app's datastore. Then, we'll give an overview of the limitations of the remote_api module. Finally, we'll walk through a more sophisticated example: an implementation of the 'Map' part of a map/reduce operation, allowing you to execute a function on every entity of a kind.

## About the remote_api module

The remote_api module consists of two parts: A 'handler', which you install on the server to handle remote datastore requests, and a 'stub', which you set up on the client to translate datastore requests into calls to the remote handler. remote_api works at the lowest level of the datastore, so once you've set up the stub, you don't have to worry about the fact that you're operating on a remote datastore: With a few caveats, it works exactly the same as if you were accessing the datastore directly.

Installing the handler is easy. Simply add the following lines to your `app.yaml` under the 'handlers' key:

    handlers:
        - url: /remote_api
          script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
          login: admin

This installs the remote_api handler under the url '`/remote_api` '. You can choose a different endpoint, if you wish, but you'll need to change the examples below. Note that the handler specifies "login: admin". This is extremely important, since we don't want to give just anyone unfettered access to our datastore!

Once you've updated the `app.yaml` file, you'll need to execute `appcfg.py update` for your app to upload the new mapping.

## Creating an interactive console

One of the simplest and most useful things you can do with remote_api is to set up an interactive Python console that has access to your production datastore. Create a new Python file called "appengine_console.py" and put the following text in it:

    #!/usr/bin/python
    import code
    import getpass
    import sys

    sys.path.append("~/google_appengine")
    sys.path.append("~/google_appengine/lib/yaml/lib")

    from google.appengine.ext.remote_api import remote_api_stub
    from google.appengine.ext import db

    def auth_func():
      return raw_input('Username:'), getpass.getpass('Password:')

    if len(sys.argv) < 2:
      print "Usage: %s app_id [host]" % (sys.argv[0],)
    app_id = sys.argv[1]
    if len(sys.argv) > 2:
      host = sys.argv[2]
    else:
      host = '%s.appspot.com' % app_id

    remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api', auth_func, host)

    code.interact('App Engine interactive console for %s' % (app_id,), None, locals())

The code here should be fairly self explanatory. On line 6, we add the SDK directory to the Python path. The example given here will work fine if you're using Linux or OS X and installed the SDK in your home directory. In windows, change this to the path where you installed the SDK such as "`c:\google_appengine`". If you're using the launcher in OS X, you need to set the path to "`/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine`".

On line 11, we define a function for getting credentials from the user when they're required. In our example, we simply prompt for them over the console, but this could use any method you like. Starting on line 14, we extract the app ID and optional hostname from the command line, and on line 22, we use a helper function provided by remote_api to set up the environment to send all datastore requests to the remote datastore. Finally, we call code.Interact, which starts an interactive Python interpreter.

## Running the interactive console

Since you will probably want to access modules defined by your app, such as the model definitions, we need to make sure that your app is in the Python path. The easiest way to do this is to change directory to your app's root directory (the one with app.yaml in it) before running our new command. Then, just execute:

    python appengine_console.py myapp

Replace 'myapp' with the app ID of your app, and you should get a Python interactive console prompt.

For demonstration purposes, we'll use the Guestbook app from the Getting Started documentation. Assuming we're in the root directory for the guestbook app, issue:

    >>> import helloworld
    >>> from google.appengine.ext import db

Now we have access to the contents of the guestbook app, we can issue commands just like we would if we were writing code to run on the server:

    >>> # Fetch the most recent 10 guestbook entries
    >>> entries = helloworld.Greeting.all().order("-date").fetch(10)
    >>>
    >>> # Create our own guestbook entry
    >>> helloworld.Greeting(content="A greeting").put()

In general, the console will act exactly as if you were accessing the datastore directly, but because the script is running on your own machine, you don't have to worry about how long it takes to run, and you can access all the files and resources on your local machine as you normally would!

## Limitations of remote_api

The remote_api module goes to great lengths to make sure that as far as possible, it behaves exactly like the native App Engine datastore. In some cases, this means doing things that are less efficient than they might otherwise be. When using remote_api, here's a few things to keep in mind:

### Every datastore request requires a round-trip

Since you're accessing the datastore over HTTP, there's a bit more overhead and latency than when you access it locally. In order to speed things up and decrease load, try to limit the number of round-trips you do by batching gets and puts, and fetching batches of entities from queries. This is good advice not just for remote_api, but for using the datastore in general, since a batch operation is only considered to be a single Datastore operation. For example, instead of this:

    for key in keys:
      rec = MyModel.get(key)
      rec.foo = bar
      rec.put()

you can do this:

    records = MyModel.get(keys)
    for rec in records:
      rec.foo = bar
    db.Put(records)

Both examples have the same effect, but the latter requires only two roundtrips in total, while the former requires two roundtrips for each entity.

### Requests to remote_api use quota

Since remote_api operates over HTTP, every datastore call you make incurs quota usage for HTTP requests, bytes in and out, as well as the usual datastore quota you would expect. Bear this in mind if you're using remote_api to do bulk updates.

### 1 MB API limits apply

As when running natively, the 1MB limit on API requests and responses still applies. If your entities are particularly large, you may need to limit the number you fetch or put at a time to keep below this limit. This conflicts with minimising round-trips, unfortunately, so the best advice is to use the largest batches you can without going over the request or response size limitations. For most entities, this is unlikely to be an issue, however.

### Avoid iterating over queries

One common pattern with datastore access is the following:

    q = MyModel.all()
    for entity in q:
      # Do something with entity

When you do this, the SDK fetches entities from the datastore in batches of 20, fetching a new batch whenever it uses up the existing ones. Because each batch has to be fetched in a separate request by remote_api, it's unable to do this as efficiently. Instead, remote_api executes an entirely new query for each batch, using the offset functionality to get further into the results.

If you know how many entities you need, you can do the whole fetch in one request by asking for the number you need:

    entities = MyModel.all().fetch(100)
    for entity in entities:
      # Do something with entity

If you don't know how many entities you will want, you can use the __key__ pseudo-property to efficiently iterate over large result sets. This also allows you to avoid the 1000 entity limit imposed on normal datastore queries:

    entities = MyModel.all().fetch(100)
    while entities:
      for entity in entities:
        # Do something with entity
      entities = MyModel.all().filter('__key__ >', entities[-1].key()).fetch(100)

### Transactions are less efficient

In order to implement transactions via remote_api, it accumulates information on entities fetched inside the transaction, along with copies of entities that were put or deleted inside the transaction. When the transaction is committed, it sends all of this information off to the App Engine server, where it has to fetch all the entities that were used in the transaction again, verify that they have not been modified, then put and delete all the changes the transaction made and commit it. If there's a conflict, the server rolls back the transaction and notifies the client end, which then has to repeat the process all over again.

This approach works, and exactly duplicates the functionality provided by transactions on the local datastore, but is rather inefficient. By all means use transactions where they are necessary, but try to limit the number and complexity of the transactions you execute in the interest of efficiency.

## Putting remote_api to work

Now that we've demonstrated the power of remote_api and outlined its limitations, it's time to put what we've learned to work with a practical tool. Frequently it would be useful to be able to iterate over every entity of a given kind, be it to extract their data, or to modify them and store the updated entities back to the datastore.

In order to achieve this, we're going to implement a simple 'map' framework. We'll define a class, `Mapper`, that exposes a `map()` method for subclasses to extend, and a couple of fields - `KIND` and `FILTERS` - for them to define what kind to map over, and any filters to apply.

    class Mapper(object):
      # Subclasses should replace this with a model class (eg, model.Person).
      KIND = None

      # Subclasses can replace this with a list of (property, value) tuples to filter by.
      FILTERS = []

      def map(self, entity):
        """Updates a single entity.

        Implementers should return a tuple containing two iterables (to_update, to_delete).
        """
        return ([], [])

      def get_query(self):
        """Returns a query over the specified kind, with any appropriate filters applied."""
        q = self.KIND.all()
        for prop, value in self.FILTERS:
          q.filter("%s =" % prop, value)
        q.order("__key__")
        return q

      def run(self, batch_size=100):
        """Executes the map procedure over all matching entities."""
        q = self.get_query()
        entities = q.fetch(batch_size)
        while entities:
          to_put = []
          to_delete = []
          for entity in entities:
            map_updates, map_deletes = self.map(entity)
            to_put.extend(map_updates)
            to_delete.extend(map_deletes)
          if to_put:
            db.put(to_put)
          if to_delete:
            db.delete(to_delete)
          q = self.get_query()
          q.filter("__key__ >", entities[-1].key())
          entities = q.fetch(batch_size)

As you can see, there's not much to it. First, we define a convenience method, `get_query()`, that returns a query that matches the kind and filters specified in the class definition, sorted by the `__key__` pseudo-property. This method could optionally be overridden by a subclass, for example to support varying the filters at runtime, as long as it uses only equality filters, and orders the query by `__key__`. Then, we define an instance method, `run()` , which iterates over every matching entity in batches, calling the `map()` function on each one, and updating or deleting the entity as appropriate.

One caveat to our Mapper: The map process does not work from a snapshot of the datastore. So if you return new entities from `map()` that themselves meet the criteria for mapping, you may get them passed in to the `map()` function later in the process. Whether or not they do depends on where their key sorts compared to the current record's key. As a general rule, if you're going to create new entities of the same type in a `map()` function, you need some way to distinguish them from the original entities so you don't process them a second time.

In order to use this class, we define a subclass that implements the `map()` function. In this example, we're going to add the phrase 'Bar!' to any guestbook entry that contains the phrase 'foo':

    class GuestbookUpdater(Mapper):
      KIND = Greeting

      def map(self, entity):
        if entity.content.lower().find('foo') != -1:
          entity.content += ' Bar!'
          return ([entity], [])
        return ([], [])

Then, we instantiate our class and call run():

     mapper = MyMapper()
    mapper.run()

You can try this out for yourself easily: Just enter the code in the interactive console we set up earlier.

Finally, here's a practical - though trivial - example of where our new framework can be useful: Deleting all the entities of a given kind.

    class MyModelDeleter(Mapper):
      KIND = MyModel

      def map(self, entity):
        return ([], [entity])

Simple! Because Mapper takes care to always access `KIND` and `FILTERS` as instance variables, we can even generalize this to allow you to select the kind and filters at runtime:

    class BulkDeleter(Mapper):
      def __init__(self, kind, filters=None):
        self.KIND = kind
        if filters:
          self.FILTERS = filters

      def map(self, entity):
        return ([], [entity])

Of course, this is only the start of what you can do with remote_api and the Mapper framework. If you have a novel use you've come up with, please post it to the group - we'd love to hear about it.