python-peps/pep-0273.txt

PEP: 273
Title: Import Modules from Zip Archives
Version: $Revision$
Last-Modified: $Date$
Author: jim@interet.com (James C. Ahlstrom)
Status: Final
Type: Standards Track
Created: 11-Oct-2001
Python-Version: 2.3
Post-History: 26-Oct-2001


Abstract

    This PEP adds the ability to import Python modules
    *.py, *.py[co] and packages from zip archives.  The
    same code is used to speed up normal directory imports
    provided os.listdir is available.


Note

    Zip imports were added to Python 2.3, but the final implementation
    uses an approach different from the one described in this PEP.
    The 2.3 implementation is SourceForge patch #652586, which adds
    new import hooks described in PEP 302.  

    The rest of this PEP is therefore only of historical interest.

   
Specification

    Currently, sys.path is a list of directory names as strings.  If
    this PEP is implemented, an item of sys.path can be a string
    naming a zip file archive.  The zip archive can contain a
    subdirectory structure to support package imports.  The zip
    archive satisfies imports exactly as a subdirectory would.

    The implementation is in C code in the Python core and works on
    all supported Python platforms.

    Any files may be present in the zip archive, but only files
    *.py and *.py[co] are available for import.  Zip import of
    dynamic modules (*.pyd, *.so) is disallowed.

    Just as sys.path currently has default directory names, a default
    zip archive name is added too.  Otherwise there is no way to
    import all Python library files from an archive.


Subdirectory Equivalence

    The zip archive must be treated exactly as a subdirectory tree so
    we can support package imports based on current and future rules.
    All zip data is taken from the Central Directory, the data must be
    correct, and brain dead zip files are not accommodated.

    Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip",
    and we are trying to import modfoo from the Q package.  Then
    import.c will generate a list of paths and extensions and will
    look for the file.  The list of generated paths does not change
    for zip imports.  Suppose import.c generates the path
    "/A/B/SubDir/Q/R/modfoo.pyc".  Then it will also generate the path
    "/C/D/E/Archive.zip/Q/R/modfoo.pyc".  Finding the SubDir path is
    exactly equivalent to finding "Q/R/modfoo.pyc" in the archive.

    Suppose you zip up /A/B/SubDir/* and all its subdirectories.  Then
    your zip file will satisfy imports just as your subdirectory did.

    Well, not quite.  You can't satisfy dynamic modules from a zip
    file.  Dynamic modules have extensions like .dll, .pyd, and .so.
    They are operating system dependent, and probably can't be loaded
    except from a file.  It might be possible to extract the dynamic
    module from the zip file, write it to a plain file and load it.
    But that would mean creating temporary files, and dealing with all
    the dynload_*.c, and that's probably not a good idea.

    When trying to import *.pyc, if it is not available then
    *.pyo will be used instead.  And vice versa when looking for *.pyo.
    If neither *.pyc nor *.pyo is available, or if the magic numbers
    are invalid, then *.py will be compiled and used to satisfy the
    import, but the compiled file will not be saved.  Python would
    normally write it to the same directory as *.py, but surely we
    don't want to write to the zip file.  We could write to the
    directory of the zip archive, but that would clutter it up, not
    good if it is /usr/bin for example.

    Failing to write the compiled files will make zip imports very slow,
    and the user will probably not figure out what is wrong.  So it
    is best to put *.pyc and *.pyo in the archive with the *.py.


Efficiency

    The only way to find files in a zip archive is linear search.  So
    for each zip file in sys.path, we search for its names once, and
    put the names plus other relevant data into a static Python
    dictionary.  The key is the archive name from sys.path joined with
    the file name (including any subdirectories) within the archive.
    This is exactly the name generated by import.c, and makes lookup
    easy.

    This same mechanism is used to speed up directory (non-zip) imports.
    See below.


zlib

    Compressed zip archives require zlib for decompression.  Prior to
    any other imports, we attempt an import of zlib.  Import of
    compressed files will fail with a message "missing zlib" unless
    zlib is available.


Booting

    Python imports site.py itself, and this imports os, nt, ntpath,
    stat, and UserDict.  It also imports sitecustomize.py which may
    import more modules.  Zip imports must be available before site.py
    is imported.

    Just as there are default directories in sys.path, there must be
    one or more default zip archives too.

    The problem is what the name should be.  The name should be linked
    with the Python version, so the Python executable can correctly
    find its corresponding libraries even when there are multiple
    Python versions on the same machine.

    We add one name to sys.path.  On Unix, the directory is
    sys.prefix + "/lib", and the file name is
    "python%s%s.zip" % (sys.version[0], sys.version[2]).
    So for Python 2.2 and prefix /usr/local, the path
    /usr/local/lib/python2.2/ is already on sys.path, and
    /usr/local/lib/python22.zip would be added.
    On Windows, the file is the full path to python22.dll, with
    "dll" replaced by "zip".  The zip archive name is always inserted
    as the second item in sys.path.  The first is the directory of the
    main.py (thanks Tim).


Directory Imports

    The static Python dictionary used to speed up zip imports can be
    used to speed up normal directory imports too.  For each item in
    sys.path that is not a zip archive, we call os.listdir, and add
    the directory contents to the dictionary.  Then instead of calling
    fopen() in a double loop, we just check the dictionary.  This
    greatly speeds up imports.  If os.listdir doesn't exist, the
    dictionary is not used.


Benchmarks

    Case  Original 2.2a3    Using os.listdir   Zip Uncomp  Zip Compr
    ---- -----------------  -----------------  ----------  ----------
      1  3.2 2.5 3.2->1.02  2.3 2.5 2.3->0.87  1.66->0.93  1.5->1.07
      2  2.8 3.9 3.0->1.32  Same as Case 1.
      3  5.7 5.7 5.7->5.7   2.1 2.1 2.1->1.8   1.25->0.99  1.19->1.13
      4  9.4 9.4 9.3->9.35  Same as Case 3.

    Case 1: Local drive C:, sys.path has its default value.
    Case 2: Local drive C:, directory with files is at the end of sys.path.
    Case 3: Network  drive, sys.path has its default value.
    Case 4: Network  drive, directory with files is at the end of sys.path.

    Benchmarks were performed on a Pentium 4 clone, 1.4 GHz, 256 Meg.
    The machine was running Windows 2000 with a Linux/Samba network server.
    Times are in seconds, and are the time to import about 100 Lib modules.
    Case 2 and 4 have the "correct" directory moved to the end of sys.path.
    "Uncomp" means uncompressed zip archive, "Compr" means compressed.

    Initial times are after a re-boot of the system; the time after
    "->" is the time after repeated runs.  Times to import from C:
    after a re-boot are rather highly variable for the "Original" case,
    but are more realistic.


Custom Imports

    The logic demonstrates the ability to import using default searching
    until a needed Python module (in this case, os) becomes available.
    This can be used to bootstrap custom importers.  For example, if
    "importer()" in __init__.py exists, then it could be used for imports.
    The "importer()" can freely import os and other modules, and these
    will be satisfied from the default mechanism.  This PEP does not
    define any custom importers, and this note is for information only.


Implementation

    A C implementation is available as SourceForge patch 492105.
    Superseded by patch 652586 and current CVS.
    http://python.org/sf/492105

    A newer version (updated for recent CVS by Paul Moore) is 645650.
    Superseded by patch 652586 and current CVS.
    http://python.org/sf/645650

    A competing implementation by Just van Rossum is 652586, which is
    the basis for the final implementation of PEP 302.  PEP 273 has
    been implemented using PEP 302's import hooks.
    http://python.org/sf/652586


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
fill-column: 70
End: