199 lines
7.0 KiB
Plaintext
199 lines
7.0 KiB
Plaintext
PEP: 420
|
||
Title: Implicit Namespace Packages
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Eric V. Smith <eric@trueblade.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 19-Apr-2012
|
||
Python-Version: 3.3
|
||
Post-History:
|
||
|
||
Abstract
|
||
========
|
||
|
||
Namespace packages are a mechanism for splitting a single Python
|
||
package across multiple directories on disk. In current Python
|
||
versions, an algorithm to compute the packages __path__ must be
|
||
formulated. With the enhancement proposed here, the import machinery
|
||
itself will construct the list of directories that make up the
|
||
package. This PEP builds upon the work started in rejected PEPs 382
|
||
and 402.
|
||
|
||
Terminology
|
||
===========
|
||
|
||
Within this PEP, the term "package" refers to Python packages as defined
|
||
by Python's import statement. The term "distribution" refers to
|
||
separately installable sets of Python modules as stored in the Python
|
||
package index, and installed by distutils or setuptools. The term
|
||
"vendor package" refers to groups of files installed by an operating
|
||
system's packaging mechanism (e.g. Debian or Redhat packages install
|
||
on Linux systems).
|
||
|
||
The term "portion" refers to a set of files in a single directory (possibly
|
||
stored in a zip file) that contribute to a namespace package.
|
||
|
||
The term "regular package" refers to packages as they are implemented
|
||
in Python 3.2.
|
||
|
||
This PEP describes a new type of package, the "namespace package".
|
||
|
||
Namespace packages today
|
||
========================
|
||
|
||
Python currently provides the pkgutil.extend_path to denote a package as
|
||
a namespace package. The recommended way of using it is to put::
|
||
|
||
from pkgutil import extend_path
|
||
__path__ = extend_path(__path__, __name__)
|
||
|
||
in the package's ``__init__.py``. Every distribution needs to provide
|
||
the same contents in its ``__init__.py``, so that extend_path is
|
||
invoked independent of which portion of the package gets imported
|
||
first. As a consequence, the package's ``__init__.py`` cannot
|
||
practically define any names as it depends on the order of the package
|
||
fragments on sys.path which portion is imported first. As a special
|
||
feature, extend_path reads files named ``<packagename>.pkg`` which
|
||
allow to declare additional portions.
|
||
|
||
setuptools provides a similar function named
|
||
pkg_resources.declare_namespace that is used in the form::
|
||
|
||
import pkg_resources
|
||
pkg_resources.declare_namespace(__name__)
|
||
|
||
In the portion's __init__.py, no assignment to __path__ is necessary,
|
||
as declare_namespace modifies the package __path__ through sys.modules.
|
||
As a special feature, declare_namespace also supports zip files, and
|
||
registers the package name internally so that future additions to sys.path
|
||
by setuptools can properly add additional portions to each package.
|
||
|
||
setuptools allows declaring namespace packages in a distribution's
|
||
setup.py, so that distribution developers don't need to put the
|
||
magic __path__ modification into __init__.py themselves.
|
||
|
||
Rationale
|
||
=========
|
||
|
||
The current imperative approach to namespace packages has lead to
|
||
multiple slightly-incompatible mechanisms for providing namespace
|
||
packages. For example, pkgutil supports ``*.pkg`` files; setuptools
|
||
doesn't. Likewise, setuptools supports inspecting zip files, and
|
||
supports adding portions to its _namespace_packages variable, whereas
|
||
pkgutil doesn't.
|
||
|
||
Namespace packages need to be installed in one of two ways: either all
|
||
portions of a namespace will be combined into a single directory (and
|
||
therefore a single entry in sys.path), or each portion will be
|
||
installed in its own directory (and each portion will have a distinct
|
||
sys.path entry).
|
||
|
||
The current imperative approach causes problems for system vendors.
|
||
Vendor packages typically must not provide overlapping files, and an
|
||
attempt to install a vendor package that has a file already on disk
|
||
will fail or cause unpredictable behavior. As vendors might choose to
|
||
package distributions such that they will end up all in a single
|
||
directory for the namespace package, all portions would contribute
|
||
conflicting __init__.py files.
|
||
|
||
Specification
|
||
=============
|
||
|
||
Regular packages will continue to have an __init__.py and will reside
|
||
in a single directory.
|
||
|
||
Namespace packages cannot contain an __init__.py. As a consequence,
|
||
extend_path and declare_namespace become obsolete. There will be no
|
||
marker file or directory for specifing a namespace package.
|
||
|
||
During import processing, the import machinery will continue to
|
||
iterate over the parent path as it does in Python 3.2. While looking
|
||
for a module or package named "foo":
|
||
|
||
* If foo/__init__.py is found, a regular package is imported.
|
||
* If not, but foo.{py,pyc,so,pyd} is found, a module is imported.
|
||
* If not, but foo is found and is a directory, it is recorded.
|
||
|
||
If the scan along the parent path completes without finding a module
|
||
or package, then a namespace package is created. The new namespace
|
||
package:
|
||
|
||
* Has a __file__ set to the first directory that was found during the
|
||
scan, including the trailing path separator.
|
||
* Has a __path__ set to the list of directories there were found and
|
||
recorded during the scan.
|
||
|
||
There is no mechanism to recompute the __path__ once a namespace
|
||
package has been created.
|
||
|
||
Note that if "import foo" is executed and "foo" is found as a
|
||
namespace package (using the above rules), then "foo" is immediately
|
||
created as a package. The creation of the namespace package is not
|
||
deferred until a sub-level import occurs.
|
||
|
||
Impact on Import Hooks
|
||
----------------------
|
||
|
||
To be determined in the sample implementation.
|
||
|
||
Discussion
|
||
==========
|
||
|
||
There is no intention to remove support of regular packages. If there
|
||
is no intention that a package is a namespace package, then there is a
|
||
performance advantage to it being a regular package. Creation and
|
||
loading of the package can take place once it is located along the
|
||
path. With namespace packages, all entries in the path must be
|
||
scanned.
|
||
|
||
Note that an ImportWarning will no longer be raised for a directory
|
||
lacking an ``__init__.py`` file. Such a directory will now be imported
|
||
as a namespace package, whereas in prior Python versions an
|
||
ImportError would be raised.
|
||
|
||
At PyCon 2012, we had a discussion about namespace packages at which
|
||
PEP 382 and PEP 402 were rejected, to be replaced by this PEP [1]_.
|
||
|
||
Nick Coglan presented a list of his objections to this proposal
|
||
[2]_. They are:
|
||
|
||
* Implicit package directories go against the Zen of Python
|
||
|
||
* Implicit package directories pose awkward backwards compatibility
|
||
challenges
|
||
|
||
* Implicit package directories introduce ambiguity into filesystem
|
||
layouts
|
||
|
||
* Implicit package directories will permanently entrench current
|
||
newbie-hostile behaviour in __main__
|
||
|
||
(These need to be addressed here.)
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] Namespace Packages resolution
|
||
(http://mail.python.org/pipermail/import-sig/2012-March/000421.html)
|
||
|
||
.. [2] Nick Coglan's objection to the lack of marker files or directories
|
||
(http://mail.python.org/pipermail/import-sig/2012-March/000423.html)
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|