248 lines
8.8 KiB
Plaintext
248 lines
8.8 KiB
Plaintext
PEP: 406
|
|
Title: Improved Encapsulation of Import State
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Nick Coghlan <ncoghlan@gmail.com>, Greg Slodkowicz <jergosh@gmail.com>
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 4-Jul-2011
|
|
Post-History: 31-Jul-2011, 13-Nov-2011
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes the introduction of a new 'ImportEngine' class as part of
|
|
``importlib`` which would encapsulate all state related to importing modules
|
|
into a single object. Creating new instances of this object would then provide
|
|
an alternative to completely replacing the built-in implementation of the
|
|
import statement, by overriding the ``__import__()`` function. To work with
|
|
the builtin import functionality and importing via import engine objects,
|
|
module importers and loaders must accept an optional ``engine`` parameter. In
|
|
that sense, this PEP constitutes a revision of finder and loader interfaces
|
|
described in PEP 302 [1]_. However, the standard import process will not
|
|
supply the additional argument, so this proposal remains fully backwards
|
|
compatible.
|
|
|
|
The PEP also proposes inclusion of a ``GlobalImportEngine`` subclass and a
|
|
globally accessible instance of that class, which "writes through" to the
|
|
process global state and invokes importers and loaders without the additional
|
|
``engine`` argument. This provides a backwards compatible bridge between the
|
|
proposed encapsulated API and the legacy process global state.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Currently, most state related to the import system is stored as module level
|
|
attributes in the ``sys`` module. The one exception is the import lock, which
|
|
is not accessible directly, but only via the related functions in the ``imp``
|
|
module. The current process global import state comprises:
|
|
|
|
* sys.modules
|
|
* sys.path
|
|
* sys.path_hooks
|
|
* sys.meta_path
|
|
* sys.path_importer_cache
|
|
* the import lock (imp.lock_held()/acquire_lock()/release_lock())
|
|
|
|
Isolating this state would allow multiple import states to be
|
|
conveniently stored within a process. Placing the import functionality
|
|
in a self-contained object would also allow subclassing to add additional
|
|
features (e.g. module import notifications or fine-grained control
|
|
over which modules can be imported). The engine would also be
|
|
subclassed to make it possible to use the import engine API to
|
|
interact with the existing process-global state.
|
|
|
|
The namespace PEPs (especially PEP 402) raise a potential need for
|
|
*additional* process global state, in order to correctly update package paths
|
|
as ``sys.path`` is modified.
|
|
|
|
|
|
Proposal
|
|
========
|
|
|
|
We propose introducing an ImportEngine class to encapsulate import
|
|
functionality. This includes an ``__import__()`` method which can
|
|
be used as an alternative to the built-in ``__import__()`` when
|
|
desired and also an ``import_module()`` method, equivalent to
|
|
``importlib.import_module()`` [3]_.
|
|
|
|
Since the new style finders and loaders should also have the option to
|
|
modify the global import state, we introduce a ``GlobalImportState``
|
|
class with an interface identical to ``ImportEngine`` but taking
|
|
advantage of the global state. This can be easily implemented using
|
|
class properties.
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
ImportEngine API
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The proposed extension consists of the following objects:
|
|
|
|
``importlib.engine.ImportEngine``
|
|
|
|
``from_engine(self, other)``
|
|
|
|
Create a new import object from another ImportEngine instance. The
|
|
new object is initialised with a copy of the state in ``other``. When
|
|
called on ``importlib engine.sysengine``, ``from_engine()`` can be
|
|
used to create an ``ImportEngine`` object with a **copy** of the
|
|
global import state.
|
|
|
|
``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)``
|
|
|
|
Reimplementation of the builtin ``__import__()`` function. The
|
|
import of a module will proceed using the state stored in the
|
|
ImportEngine instance rather than the global import state. For full
|
|
documentation of ``__import__`` funtionality, see [2]_ .
|
|
``__import__()`` from ``ImportEngine`` and its subclasses can be used
|
|
to customise the behaviour of the ``import`` statement by replacing
|
|
``__builtin__.__import__`` with ``ImportEngine().__import__``.
|
|
|
|
``import_module(name, package=None)``
|
|
|
|
A reimplementation of ``importlib.import_module()`` which uses the
|
|
import state stored in the ImportEngine instance. See [3]_ for a full
|
|
reference.
|
|
|
|
``modules, path, path_hooks, meta_path, path_importer_cache``
|
|
|
|
Instance-specific versions of their process global ``sys`` equivalents
|
|
|
|
|
|
``importlib.engine.GlobalImportEngine(ImportEngine)``
|
|
|
|
Convenience class to provide engine-like access to the global state.
|
|
Provides ``__import__()``, ``import_module()`` and ``from_engine()``
|
|
methods like ``ImportEngine`` but writes through to the global state
|
|
in ``sys``.
|
|
|
|
|
|
Global variables
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
``importlib.engine.sysengine``
|
|
|
|
A precreated instance of ``GlobalImportEngine``. Intended for use by
|
|
importers and loaders that have been updated to accept optional ``engine``
|
|
parameters and with ``ImportEngine.from_engine(sysengine)`` to start with
|
|
a copy of the process global import state.
|
|
|
|
|
|
Necessary changes to finder/loader interfaces:
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
``find_module (cls, fullname, path=None, engine=None)``
|
|
|
|
``load_module (cls, fullname, path=None, engine=None)``
|
|
|
|
The only difference between engine compatible and PEP 302 compatible
|
|
finders/loaders is the presence of an additional ``engine`` parameter.
|
|
This is intended to specify an ImportEngine instance or subclass thereof.
|
|
This parameter is optional so that engine compatible finders and
|
|
loaders can be made backwards compatible with PEP 302 calling conventions by
|
|
falling back on ``engine.sysengine`` with the following simple pattern::
|
|
|
|
def find_module(cls, fullname, path=None, engine=None):
|
|
if not engine:
|
|
engine = importlib.engine.sysengine
|
|
...
|
|
|
|
|
|
Open Issues
|
|
===========
|
|
|
|
|
|
API design for falling back to global import state
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The current proposal relies on the ``from_engine()`` API to fall back to the
|
|
global import state. It may be desirable to offer a variant that instead falls
|
|
back to the global import state dynamically.
|
|
|
|
However, one big advantage of starting with an "as isolated as possible"
|
|
design is that it becomes possible to experiment with subclasses that blur
|
|
the boundaries between the engine instance state and the process global state
|
|
in various ways.
|
|
|
|
|
|
Builtin and extension modules must be process global
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Due to platform limitations, only one copy of each builtin and extension
|
|
module can readily exist in each process. Accordingly, it is impossible for
|
|
each ``ImportEngine`` instance to load such modules independently.
|
|
|
|
The simplest solution is for ``ImportEngine`` to refuse to load such modules,
|
|
raising ``ImportError``. ``GlobalImportEngine`` would be able to load them
|
|
normally.
|
|
|
|
``ImportEngine`` will still return such modules from a prepopulated module
|
|
cache - it's only loading them directly which causes problems.
|
|
|
|
|
|
Nested imports
|
|
~~~~~~~~~~~~~~
|
|
|
|
The reference implementation currently applies only to the outermost import.
|
|
Any imports by the module being imported will be handled using the standard
|
|
import machinery.
|
|
|
|
One way to handle this is to place the burden on the implementation of module
|
|
loaders to set ``module.__dict__["__import__"] = engine.__import__`` before
|
|
running the module's code. The ``importlib`` design facilities this by
|
|
allowing the change to be made in one place (``_LoaderBasics._load_module``).
|
|
|
|
|
|
Scope of API updates
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The reference implementation focuses on finding and loading modules. There
|
|
may be other PEP 302 APIs that should also be updated to accept an optional
|
|
``engine`` parameter.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
A reference implementation [4]_ based on Brett Cannon's importlib has been
|
|
developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. Note
|
|
that the current implementation avoids modifying existing code, and hence
|
|
duplicates a lot of things unnecessarily. An actual implementation would just
|
|
modify any such affected code in place.
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [1] PEP 302, New Import Hooks, J van Rossum, Moore
|
|
(http://www.python.org/dev/peps/pep-0302)
|
|
|
|
.. [2] __import__() builtin function, The Python Standard Library documentation
|
|
(http://docs.python.org/library/functions.html#__import__)
|
|
|
|
.. [3] Importlib documentation, Cannon
|
|
(http://docs.python.org/dev/library/importlib)
|
|
|
|
.. [4] Reference implentation
|
|
(https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py)
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|