English
Skip to page content Skip to main navigation

e.g. "templates" or "datastore"

# Using Django 1.0 on App Engine with Zipimport

Dan Sanderson
September 2008

## Introduction

Using a Python web application framework with your App Engine application is usually as simple as including the files for the framework with your application's code. However, there is a limit to the number of files that can be uploaded for an application, and the standard distributions for some frameworks exceed this limit or leave little room for application code. You can work around the file limit using Python's "zipimport" feature, which is supported by App Engine as of the 1.1.3 release (September 2008).

This article describes how to use Django 1.0 with Google App Engine using the "zipimport" feature. You can use similar techniques with other frameworks, libraries or large applications.

## Introducing zipimport

When your application imports a module, Python looks for the module's code in one of several directories. You can access and change the list of directories Python checks from Python code using `sys.path`. In App Engine, your handler is called with a path that includes the App Engine API and your application root directory.

If any of the items in `sys.path` refers to a ZIP-format archive, Python will treat the archive as a directory. The archive contains the `.py` source files for one or more modules. This feature is supported by a module in the standard library called `zipimport`, though this module is part of the default import process and you do not need to import this module directly to use it. For more information about zipimport, see the zipimport documentation.

To use module archives with your App Engine application:

1. Create a ZIP-format archive of the modules you want to bundle.
2. Put the archive in your application directory.
3. If necessary, in your handler scripts, add the archive file to `sys.path`.

For example, if you have a ZIP archive named `django.zip` with the following files in it:

    django/forms/__init__.py
    django/forms/fields.py
    django/forms/forms.py
    django/forms/formsets.py
    django/forms/models.py
    ...

A handler script can import a module from the archive as follows:

    import sys
    sys.path.insert(0, 'django.zip')

    import django.forms.fields

This example illustrates zipimport, but is not sufficient for loading Django 1.0 in App Engine. A more complete example follows.

## zipimport and App Engine

App Engine uses a custom version of the zipimport feature instead of the standard implementation. It generally works the usual way: add the Zip archive to `sys.path`, then import as usual.

Because it is a custom implementation, several features do not work with App Engine. For instance, App Engine can load `.py` files from the archive, but it can't load `.pyc` files like the standard version can. The SDK uses the standard version, so if you'd like to use features of zipimport beyond those discussed here, be sure to test them on App Engine.

Update, January 2009: A previous draft of this article mentioned an issue with the Python import path being reset for each request. This issue has been resolved as of release 1.1.8 of the Python SDK.

## Archiving Django 1.0

When App Engine launched in Summer 2008, it included the Django application framework as part of the environment to make it easy to get started. At the time, the latest release of Django was 0.96, so this is the version that is part of version "1" of the Python runtime environment. Since then, the Django project released version 1.0. For compatibility reasons, App Engine can't update its version of Django without also releasing a new version of the Python runtime environment. To use 1.0 with App Engine with version "1" of the runtime environment, an application must include the 1.0 distribution in its application directory.

The Django 1.0 distribution contains 1,582 files. An App Engine application is limited to 1,000 files, so the Django distribution can't be included directly. Of course, not every file in the distribution needs to be included with the application. You can prune the distribution to remove documentation files, unused locales, database interfaces and other components that don't work with App Engine (such as the Admin application) to get the file count below the limit.

Using zipimport, you can include Django 1.0 with your application using just 1 file, leaving plenty of room for your own application files in the 1,000 file limit. A single ZIP archive of Django 1.0 is about 3 MB. This fits within the 10 MB file size limit. You may wish to prune unused libraries from the Django distribution anyway to further reduce the size of the archive.

Update: Prior to the 1.1.9 release of the Python SDK in February 2009, the file size limit was 1 MB. With 1.1.9, the limit has been increased to 10 MB. These instructions produce a Django archive smaller than 1 MB.
To make an archive containing all of Django, replace steps 2, 3 and 4 below with the following command: `zip -r django.zip django`

To download and re-package Django 1.0 as a ZIP archives:

1. Download the Django 1.0 distribution from the Django website. Unpack this archive using an appropriate tool for your operating system (a tool that can unpack a `.tar.gz` file). For example, on the Linux or Mac OS X command line:

       tar -xzvf Django-1.0.tar.gz

2. Create a ZIP archive that contains everything in the `django/` directory except for the `.../conf/` and `.../contrib/` sub-directories. (You can also omit `bin/` and `test/`.) The path inside the ZIP must start with `django/`.

       cd Django-1.0
       zip -r django.zip django/__init__.py django/bin django/core \
                         django/db django/dispatch django/forms \
                         django/http django/middleware django/shortcuts \
                         django/template django/templatetags \
                         django/test django/utils django/views

3. The `conf` package contains a large number of localization files. Adding all of these files to the archive would increase the size of the archive beyond the 1 MB limit. However, there's room for a few files, and many Django packages need some parts of `conf`. Add everything in `conf` except the `locale` directory to the archive. If necessary, you can also add the specific locales you need, but be sure to check that the file size of the archive is below 1 MB.

   The following command adds everything in `conf` except `conf/locale` to the archive:

       zip -r django.zip django/conf -x 'django/conf/locale/*'

4. Similarly, if you need anything in `.../contrib/`, add it to the archive. The largest component in `contrib` is the Django Admin application, which doesn't work with App Engine, so you can safely omit the `admin` and `admindocs` directories. For example, to add `formtools`:

       zip -r django.zip django/contrib/__init__.py \
                         django/contrib/formtools

5. Put the archive file in your application directory.

       mv django.zip your-app-dir/

## Using the Module Archive

Tip: The latest version of the Django App Engine Helper (starting with version "r64") supports Django 1.0 with zipimport out of the box. Make sure your archive is named `django.zip` and is in your application root directory. All new projects created using the Google App Engine Helper for Django will automatically use django.zip if present. If you are upgrading an existing project you will need to copy the appengine_django, manage.py and main.py files from Google App Engine Helper for Django into your existing project. See Using the Google App Engine Helper for Django.
The following instructions only apply if you are using Django without the Helper, or if you are preparing another module archive.

To use a module archive, the `.zip` file must be on the Python module load path. The easiest way to do this is to modify the load path at the top of each handler script, and in each handler's `main()` routine. All other files that use modules in the archives will work without changes.

Because App Engine pre-loads Django 0.96 for all Python applications, using Django 1.0 requires one more step to make sure the `django` package refers to 1.0 and not the preloaded version. As described in the article Running Django on App Engine, the handler script must remove Django 0.96 from `sys.modules` before importing Django 1.0.

The following code uses the techniques described here to run Django 1.0 from an archive named `django.zip`:

    import sys
    from google.appengine.ext.webapp import util

    # Uninstall Django 0.96.
    for k in [k for k in sys.modules if k.startswith('django')]:
      del sys.modules[k]

    # Add Django 1.0 archive to the path.
    django_path = 'django.zip'
    sys.path.insert(0, django_path)

    # Django imports and other code go here...
    import os
    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
    import django.core.handlers.wsgi

    def main():
      # Run Django via WSGI.
      application = django.core.handlers.wsgi.WSGIHandler()
      util.run_wsgi_app(application)

    if __name__ == '__main__':
      main()

With appropriate `app.yaml`, `settings.py` and `urls.py` files, this handler displays the Django "It worked!" page. See Running Django on App Engine for more information on using Django.

## Using Multiple Archive Files for a Single Package

Since all of Django 1.0 is too large to fit into a single archive, can we split it into multiple archives, each on `sys.path`? Actually yes, with some bootstrapping code to help Python navigate the different locations.

When Python imports a module, it checks each location mentioned in `sys.path` for the package that contains the module. If a location does not contain the first package in the module's path, Python checks the next `sys.path` entry, and so on until it finds the first package or runs out of locations to check.

When Python finds the first package in the module's path, it assumes that wherever it found it is the definitive location for that package, and it won't bother looking for it elsewhere. If Python cannot find the rest of the module path in the package, it raises an import error and stops. Python does not check subsequent `sys.path` entries after the first package in the path has been found.

You can work around this by importing the package that is split across multiple archives from the first archive, then telling Python that the contents of the package can actually be found in multiple places. The `__path__` member of a package (module) object is a list of locations for the package's contents. For example, if the `django` package is split between two archives called `django1.zip` and `django2.zip`, the following code tells Python to look in both archives for the contents of the package:

    sys.path.insert(0, 'django1.zip')
    import django
    django.__path__.append('django2.zip/django')

This imports the `django` package from `django1.zip`, so make sure that archive contains `django/__init__.py`.

With the second archive on the package's `__path__`, subsequent imports of modules inside `django` will search both archives.

## Additional Notes

Some additional things to note about using zipimport with App Engine:

- Module archives use additional CPU time the first time a module is imported. Imports are cached in memory for future requests to the same application instance, and modules from archives are cached uncompressed and compiled, so subsequent imports on the same instance will not incur CPU overhead for decompression or compilation.
- The App Engine implementation of zipimport only supports `.py` files, not precompiled `.pyc` files.
- Because handler scripts are responsible for adding module archives to the path, handler scripts themselves cannot be stored in module archives. Any other Python code can be stored in module archives.