python-peps/pep-0382.txt

PEP: 382
Title: Namespace Packages
Version: $Revision$
Last-Modified: $Date$
Author: Martin v. Löwis <martin@v.loewis.de>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 02-Apr-2009
Python-Version: 3.2
Post-History:

Abstract
========

Namespace packages are a mechanism for splitting a single Python
package across multiple directories on disk. In current Python
versions, an algorithm to compute the packages __path__ must be
formulated. With the enhancement proposed here, the import machinery
itself will construct the list of directories that make up the
package.

Terminology
===========

Within this PEP, the term package refers to Python packages as defined
by Python's import statement. The term distribution refers to
separately installable sets of Python modules as stored in the Python
package index, and installed by distutils or setuptools. The term
vendor package refers to groups of files installed by an operating
system's packaging mechanism (e.g. Debian or Redhat packages install
on Linux systems).

The term portion refers to a set of files in a single directory (possibly
stored in a zip file) that contribute to a namespace package.

Namespace packages today
========================

Python currently provides the pkgutil.extend_path to denote a package as
a namespace package. The recommended way of using it is to put::

        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)

in the package's ``__init__.py``. Every distribution needs to provide
the same contents in its ``__init__.py``, so that extend_path is
invoked independent of which portion of the package gets imported
first. As a consequence, the package's ``__init__.py`` cannot
practically define any names as it depends on the order of the package
fragments on sys.path which portion is imported first. As a special
feature, extend_path reads files named ``<packagename>.pkg`` which
allow to declare additional portions.

setuptools provides a similar function pkg_resources.declare_namespace
that is used in the form::

    import pkg_resources
    pkg_resources.declare_namespace(__name__)

In the portion's __init__.py, no assignment to __path__ is necessary,
as declare_namespace modifies the package __path__ through sys.modules.
As a special feature, declare_namespace also supports zip files, and
registers the package name internally so that future additions to sys.path
by setuptools can properly add additional portions to each package.

setuptools allows declaring namespace packages in a distribution's
setup.py, so that distribution developers don't need to put the
magic __path__ modification into __init__.py themselves.

Rationale
=========

The current imperative approach to namespace packages has lead to
multiple slightly-incompatible mechanisms for providing namespace
packages. For example, pkgutil supports ``*.pkg`` files; setuptools
doesn't. Likewise, setuptools supports inspecting zip files, and
supports adding portions to its _namespace_packages variable, whereas
pkgutil doesn't.

In addition, the current approach causes problems for system vendors.
Vendor packages typically must not provide overlapping files, and an
attempt to install a vendor package that has a file already on disk
will fail or cause unpredictable behavior. As vendors might chose to
package distributions such that they will end up all in a single
directory for the namespace package, all portions would contribute
conflicting __init__.py files.

Specification
=============

Rather than using an imperative mechanism for importing packages, a
declarative approach is proposed here: A directory whose name ends 
with ``.pyp`` (for Python package) contains a portion of a package.

The import statement is extended so that computes the package's
``__path__`` attribute for a package named ``P`` as consisting of all
directories named ``P.pyp``, plus optionally a single directory named
``P`` containing a file ``__init__.py``. If either of these are
found on the path of the parent package (or sys.path for a top-level
package), search for additional portions of the package continues.

No other change to the importing mechanism is made; searching modules
(including __init__.py) will continue to stop at the first module
encountered. In summary, the process import a package foo works like
this:

 1. sys.path is searched for directories foo or foo.pyp, or a file foo.<ext>.
    If a file is found and no directory, it is treated as a module, and imported.
 2. If a directory foo is found, a check is made whether it contains __init__.py.
    If so, the location of the __init__.py is remembered. Otherwise, the directory
    is skipped. Once an __init__.py is found, further directories called foo are
    skipped.
 3. For both directories foo and foo.pyp, the directories are added to the package's
    __path__.
 4. If an __init__ module was found, it is imported, with __path__
    being initialized to the path computed all ``.pyp`` directories.

Impact on Import Hooks
----------------------

Both loaders and finders as defined in PEP 302 will need to be changed
to support namespace packages. Failure to comform to the protocol
below might cause a package not being recognized as a namespace
package; loaders and finders not supporting this protocol must raise
AttributeError when the functions below get accessed.

Finders need to support looking for \*.pth files in step 1 of above
algorithm. To do so, a finder used as a path hook must support a
method:

   finder.is_package_portion(fullname)

This method will be called in the same manner as find_module, and it
must return True if there is a package portion with that name; False
if fullname indicates a subpackage/submodule that is not a package
portion; otherwise, it shall raise an ImportError.

If any \*.pyp directories are found, but no loader was returned from
find_module, a package is created and initialized with the path.

If a loader was return, but no \*.pyp directories, load_module is
called as defined in PEP 302.

If both \*.pyp directories where found, and a loader was returned, a
new method is called on the loader:

   loader.load_module_with_path(load_module, path)

where the path parameter is the list that will become the __path__
attribute of the new package.


Discussion
==========

Original versions of this specification proposed the addition of
``*.pth`` files, similar to the way those files are used on sys.path.
With a wildcard marker (``*``), a package could indicate that the
entire path is derived by looking at the parent path, searching for
properly-named subdirectories.

People then observed that the support for the full .pth syntax is
inappropriate, and the .pth files were changed to be mere marker
files, indicating that a directories is a package. Peter Tröger
suggested that .pth is an unsuitable file extension, as all file
extensions related to Python should start with ``.py``. Therefore, the
marker file was renamed to be ``.pyp``.

Dinu Gherman then observed that using a marker file is not necessary,
and that a directoy extension could well serve as a such as a
marker. This is what this PEP currently proposes.

Phillip Eby designed PEP 402 as an alternative approach to this PEP,
after comparing Python's package syntax with that found in other
languages. PEP 402 proposes not to use a marker file at all. At the
discussion at PyCon DE 2011, people remarked that having an explicit
declaration of a directory as contributing to a package is a desirable
property, rather than an obstactle. In particular, Jython developers
noticed that Jython could easily mistake a directory that is a Java
package as being a Python package, if there is no need to declare
Python packages.

packages can stop filling out the namespace package's __init__.py.  As
a consequence, extend_path and declare_namespace become obsolete.

Namespace packages can start providing non-trivial __init__.py
implementations; to do so, it is recommended that a single distribution
provides a portion with just the namespace package's __init__.py
(and potentially other modules that belong to the namespace package
proper).

The mechanism is mostly compatible with the existing namespace
mechanisms. extend_path will be adjusted to this specification;
any other mechanism might cause portions to get added twice to
__path__.

It has been proposed to also add this feature to Python 2.7. Given
that 2.x reaches its end-of-life, it is questionable whether the
addition of the feature would really do more good than harm (in having
users and tools starting to special-case 2.7). Prospective users of
this feature are encouraged to comment on this particular question. In
addition, Python 2.7 is in bug-fix mode now, so adding new features to
it would be a violation of the established policies.


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: