228 lines
9.0 KiB
Plaintext
228 lines
9.0 KiB
Plaintext
PEP: 382
|
||
Title: Namespace Packages
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Martin v. Löwis <martin@v.loewis.de>
|
||
Status: Rejected
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 02-Apr-2009
|
||
Python-Version: 3.2
|
||
Post-History:
|
||
|
||
Rejection Notice
|
||
================
|
||
|
||
On the first day of sprints at US PyCon 2012 we had a long and
|
||
fruitful discussion about :pep:`382` and :pep:`402`. We ended up rejecting
|
||
both but a new PEP will be written to carry on in the spirit of PEP
|
||
402. Martin von Löwis wrote up a summary: [2]_.
|
||
|
||
Abstract
|
||
========
|
||
|
||
Namespace packages are a mechanism for splitting a single Python
|
||
package across multiple directories on disk. In current Python
|
||
versions, an algorithm to compute the packages __path__ must be
|
||
formulated. With the enhancement proposed here, the import machinery
|
||
itself will construct the list of directories that make up the
|
||
package. An implementation of this PEP is available at [1]_.
|
||
|
||
Terminology
|
||
===========
|
||
|
||
Within this PEP, the term package refers to Python packages as defined
|
||
by Python's import statement. The term distribution refers to
|
||
separately installable sets of Python modules as stored in the Python
|
||
package index, and installed by distutils or setuptools. The term
|
||
vendor package refers to groups of files installed by an operating
|
||
system's packaging mechanism (e.g. Debian or Redhat packages install
|
||
on Linux systems).
|
||
|
||
The term portion refers to a set of files in a single directory (possibly
|
||
stored in a zip file) that contribute to a namespace package.
|
||
|
||
Namespace packages today
|
||
========================
|
||
|
||
Python currently provides the pkgutil.extend_path to denote a package as
|
||
a namespace package. The recommended way of using it is to put::
|
||
|
||
from pkgutil import extend_path
|
||
__path__ = extend_path(__path__, __name__)
|
||
|
||
in the package's ``__init__.py``. Every distribution needs to provide
|
||
the same contents in its ``__init__.py``, so that extend_path is
|
||
invoked independent of which portion of the package gets imported
|
||
first. As a consequence, the package's ``__init__.py`` cannot
|
||
practically define any names as it depends on the order of the package
|
||
fragments on sys.path which portion is imported first. As a special
|
||
feature, extend_path reads files named ``<packagename>.pkg`` which
|
||
allow to declare additional portions.
|
||
|
||
setuptools provides a similar function pkg_resources.declare_namespace
|
||
that is used in the form::
|
||
|
||
import pkg_resources
|
||
pkg_resources.declare_namespace(__name__)
|
||
|
||
In the portion's __init__.py, no assignment to __path__ is necessary,
|
||
as declare_namespace modifies the package __path__ through sys.modules.
|
||
As a special feature, declare_namespace also supports zip files, and
|
||
registers the package name internally so that future additions to sys.path
|
||
by setuptools can properly add additional portions to each package.
|
||
|
||
setuptools allows declaring namespace packages in a distribution's
|
||
setup.py, so that distribution developers don't need to put the
|
||
magic __path__ modification into __init__.py themselves.
|
||
|
||
Rationale
|
||
=========
|
||
|
||
The current imperative approach to namespace packages has lead to
|
||
multiple slightly-incompatible mechanisms for providing namespace
|
||
packages. For example, pkgutil supports ``*.pkg`` files; setuptools
|
||
doesn't. Likewise, setuptools supports inspecting zip files, and
|
||
supports adding portions to its _namespace_packages variable, whereas
|
||
pkgutil doesn't.
|
||
|
||
In addition, the current approach causes problems for system vendors.
|
||
Vendor packages typically must not provide overlapping files, and an
|
||
attempt to install a vendor package that has a file already on disk
|
||
will fail or cause unpredictable behavior. As vendors might chose to
|
||
package distributions such that they will end up all in a single
|
||
directory for the namespace package, all portions would contribute
|
||
conflicting __init__.py files.
|
||
|
||
Specification
|
||
=============
|
||
|
||
Rather than using an imperative mechanism for importing packages, a
|
||
declarative approach is proposed here: A directory whose name ends
|
||
with ``.pyp`` (for Python package) contains a portion of a package.
|
||
|
||
The import statement is extended so that computes the package's
|
||
``__path__`` attribute for a package named ``P`` as consisting of
|
||
optionally a single directory name ``P`` containing a file
|
||
``__init__.py``, plus all directories named ``P.pyp``, in the order in
|
||
which they are found in the parent's package ``__path__`` (or
|
||
``sys.path``). If either of these are found, search for additional
|
||
portions of the package continues.
|
||
|
||
A directory may contain both a package in the ``P/__init__.py`` and
|
||
the ``P.pyp`` form.
|
||
|
||
No other change to the importing mechanism is made; searching modules
|
||
(including __init__.py) will continue to stop at the first module
|
||
encountered. In summary, the process import a package foo works like
|
||
this:
|
||
|
||
1. sys.path is searched for directories foo or foo.pyp, or a file foo.<ext>.
|
||
If a file is found and no directory, it is treated as a module, and imported.
|
||
2. If a directory foo is found, a check is made whether it contains __init__.py.
|
||
If so, the location of the __init__.py is remembered. Otherwise, the directory
|
||
is skipped. Once an __init__.py is found, further directories called foo are
|
||
skipped.
|
||
3. For both directories foo and foo.pyp, the directories are added to the package's
|
||
__path__.
|
||
4. If an __init__ module was found, it is imported, with __path__
|
||
being initialized to the path computed all ``.pyp`` directories.
|
||
|
||
Impact on Import Hooks
|
||
----------------------
|
||
|
||
Both loaders and finders as defined in :pep:`302` will need to be changed
|
||
to support namespace packages. Failure to conform to the protocol
|
||
below might cause a package not being recognized as a namespace
|
||
package; loaders and finders not supporting this protocol must raise
|
||
AttributeError when the functions below get accessed.
|
||
|
||
Finders need to support looking for \*.pth files in step 1 of above
|
||
algorithm. To do so, a finder used as a path hook must support a
|
||
method:
|
||
|
||
finder.find_package_portion(fullname)
|
||
|
||
This method will be called in the same manner as find_module, and it
|
||
must return a string to be added to the package's ``__path__``.
|
||
If the finder doesn't find a portion of the package, it shall return
|
||
``None``. Raising ``AttributeError`` from above call will be treated
|
||
as non-conformance with this PEP, and the exception will be ignored.
|
||
All other exceptions are reported.
|
||
|
||
A finder may report both success from ``find_module`` and from
|
||
``find_package_portion``, allowing for both a package containing
|
||
an ``__init__.py`` and a portion of the same package.
|
||
|
||
All strings returned from ``find_package_portion``, along with all
|
||
path names of ``.pyp`` directories are added to the new package's
|
||
``__path__``.
|
||
|
||
Discussion
|
||
==========
|
||
|
||
Original versions of this specification proposed the addition of
|
||
``*.pth`` files, similar to the way those files are used on sys.path.
|
||
With a wildcard marker (``*``), a package could indicate that the
|
||
entire path is derived by looking at the parent path, searching for
|
||
properly-named subdirectories.
|
||
|
||
People then observed that the support for the full .pth syntax is
|
||
inappropriate, and the .pth files were changed to be mere marker
|
||
files, indicating that a directories is a package. Peter Tröger
|
||
suggested that .pth is an unsuitable file extension, as all file
|
||
extensions related to Python should start with ``.py``. Therefore, the
|
||
marker file was renamed to be ``.pyp``.
|
||
|
||
Dinu Gherman then observed that using a marker file is not necessary,
|
||
and that a directory extension could well serve as a such as a
|
||
marker. This is what this PEP currently proposes.
|
||
|
||
Phillip Eby designed :pep:`402` as an alternative approach to this PEP,
|
||
after comparing Python's package syntax with that found in other
|
||
languages. :pep:`402` proposes not to use a marker file at all. At the
|
||
discussion at PyCon DE 2011, people remarked that having an explicit
|
||
declaration of a directory as contributing to a package is a desirable
|
||
property, rather than an obstacle. In particular, Jython developers
|
||
noticed that Jython could easily mistake a directory that is a Java
|
||
package as being a Python package, if there is no need to declare
|
||
Python packages.
|
||
|
||
Packages can stop filling out the namespace package's __init__.py. As
|
||
a consequence, extend_path and declare_namespace become obsolete.
|
||
|
||
Namespace packages can start providing non-trivial __init__.py
|
||
implementations; to do so, it is recommended that a single distribution
|
||
provides a portion with just the namespace package's __init__.py
|
||
(and potentially other modules that belong to the namespace package
|
||
proper).
|
||
|
||
The mechanism is mostly compatible with the existing namespace
|
||
mechanisms. extend_path will be adjusted to this specification;
|
||
any other mechanism might cause portions to get added twice to
|
||
__path__.
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] PEP 382 branch
|
||
(http://hg.python.org/features/pep-382-2#pep-382)
|
||
|
||
.. [2] Namespace Packages resolution
|
||
(https://mail.python.org/pipermail/import-sig/2012-March/000421.html)
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|