python-peps/peps/pep-0369.rst

PEP: 369
Title: Post import hooks
Version: $Revision$
Last-Modified: $Date$
Author: Christian Heimes <christian@python.org>
Status: Withdrawn
Type: Standards Track
Content-Type: text/x-rst
Created: 02-Jan-2008
Python-Version: 2.6, 3.0
Post-History: 02-Dec-2012


Withdrawal Notice
=================

This PEP has been withdrawn by its author, as much of the detailed design
is no longer valid following the migration to importlib in Python 3.3.


Abstract
========

This PEP proposes enhancements for the import machinery to add
post import hooks. It is intended primarily to support the wider
use of abstract base classes that is expected in Python 3.0.

The PEP originally started as a combined PEP for lazy imports and
post import hooks. After some discussion on the python-dev mailing
list the PEP was parted in two separate PEPs. [1]_


Rationale
=========

Python has no API to hook into the import machinery and execute code
*after* a module is successfully loaded. The import hooks of :pep:`302` are
about finding modules and loading modules but they were not designed to
as post import hooks.


Use cases
=========

A use case for a post import hook is mentioned in Alyssa (Nick) Coghlan's initial
posting [2]_. about callbacks on module import. It was found during the
development of Python 3.0 and its ABCs. We wanted to register classes
like decimal.Decimal with an ABC but the module should not be imported
on every interpreter startup. Alyssa came up with this example::

   @imp.when_imported('decimal')
   def register(decimal):
       Inexact.register(decimal.Decimal)

The function ``register`` is registered as callback for the module named
'decimal'. When decimal is imported the function is called with the
module object as argument.

While this particular example isn't necessary in practice, (as
decimal.Decimal will inherit from the appropriate abstract Number base
class in 2.6 and 3.0), it still illustrates the principle.


Existing implementations
========================

PJE's peak.util.imports [3]_ implements post load hooks. My
implementation shares a lot with his and it's partly based on his ideas.


Post import hook implementation
===============================

Post import hooks are called after a module has been loaded. The hooks
are callable which take one argument, the module instance. They are
registered by the dotted name of the module, e.g. 'os' or 'os.path'.

The callable are stored in the dict ``sys.post_import_hooks`` which
is a mapping from names (as string) to a list of callables or None.


States
------

No hook was registered
''''''''''''''''''''''

sys.post_import_hooks contains no entry for the module


A hook is registered and the module is not loaded yet
'''''''''''''''''''''''''''''''''''''''''''''''''''''

The import hook registry contains an entry
sys.post_import_hooks["name"] = [hook1]


A module is successfully loaded
'''''''''''''''''''''''''''''''

The import machinery checks if sys.post_import_hooks contains post import
hooks for the newly loaded module. If hooks are found then the hooks are
called in the order they were registered with the module instance as first
argument. The processing of the hooks is stopped when a method raises an
exception. At the end the entry for the module name set to None, even
when an error has occurred.

Additionally the new ``__notified__`` slot of the module object is set
to ``True`` in order to prevent infinity recursions when the notification
method is called inside a hook. For object which don't subclass from
``PyModule`` a new attribute is added instead.


A module can't be loaded
''''''''''''''''''''''''

The import hooks are neither called nor removed from the registry. It
may be possible to load the module later.


A hook is registered but the module is already loaded
'''''''''''''''''''''''''''''''''''''''''''''''''''''

The hook is fired immediately.


Invariants
----------

The import hook system guarantees certain invariants. XXX


Sample Python implementation
----------------------------

A Python implementation may look like::

      def notify(name):
          try:
              module = sys.modules[name]
          except KeyError:
              raise ImportError("Module %s has not been imported" % (name,))
          if module.__notified__:
              return
          try:
              module.__notified__ = True
              if '.' in name:
                  notify(name[:name.rfind('.')])
              for callback in post_import_hooks[name]:
                 callback(module)
          finally:
              post_import_hooks[name] = None

    XXX


C API
-----

New C API functions
'''''''''''''''''''

``PyObject* PyImport_GetPostImportHooks(void)``
    Returns the dict sys.post_import_hooks or NULL

``PyObject* PyImport_NotifyLoadedByModule(PyObject *module)``
   Notify the post import system that a module was requested. Returns the
   a borrowed reference to the same module object or NULL if an error has
   occurred. The function calls only the hooks for the module itself and not
   its parents. The function must be called with the import lock acquired.

``PyObject* PyImport_NotifyLoadedByName(const char *name)``
   ``PyImport_NotifyLoadedByName("a.b.c")`` calls
   ``PyImport_NotifyLoadedByModule()`` for ``a``, ``a.b`` and ``a.b.c``
   in that particular order. The modules are retrieved from
   ``sys.modules``. If a module can't be retrieved, an exception is raised
   otherwise the a borrowed reference to ``modname`` is returned.
   The hook calls always start with the prime parent module.
   The caller of PyImport_NotifyLoadedByName() must hold the import lock!

``PyObject* PyImport_RegisterPostImportHook(PyObject *callable, PyObject *mod_name)``
   Register a new hook ``callable`` for the module ``mod_name``

``int PyModule_GetNotified(PyObject *module)``
   Returns the status of the ``__notified__`` slot / attribute.

``int PyModule_SetNotified(PyObject *module, int status)``
   Set the status of the ``__notified__`` slot / attribute.


The ``PyImport_NotifyLoadedByModule()`` method is called inside
``import_submodule()``. The import system makes sure that the import lock
is acquired and the hooks for the parent modules are already called.


Python API
----------

The import hook registry and two new API methods are exposed through the
``sys`` and ``imp`` module.

``sys.post_import_hooks``
    The dict contains the post import hooks::

       {"name" : [hook1, hook2], ...}

``imp.register_post_import_hook(hook: "callable", name: str)``
   Register a new hook *hook* for the module *name*

``imp.notify_module_loaded(module: "module instance") -> module``
   Notify the system that a module has been loaded. The method is provided
   for compatibility with existing lazy / deferred import extensions.

``module.__notified__``
   A slot of a module instance. XXX

The when_imported function decorator is also in the imp module,
which is equivalent to::

   def when_imported(name):
       def register(hook):
           register_post_import_hook(hook, name)
       return register

imp.when_imported(name) -> decorator function
   for @when_imported(name) def hook(module): pass


Open issues
===========

The when_imported decorator hasn't been written.

The code contains several XXX comments. They are mostly about error
handling in edge cases.


Backwards Compatibility
=======================

The new features and API don't conflict with old import system of Python
and don't cause any backward compatibility issues for most software.
However systems like PEAK and Zope which implement their own lazy import
magic need to follow some rules.

The post import hooks carefully designed to cooperate with existing
deferred and lazy import systems. It's the suggestion of the PEP author
to replace own on-load-hooks with the new hook API. The alternative
lazy or deferred imports will still work but the implementations must
call the ``imp.notify_module_loaded`` function.


Reference Implementation
========================

A reference implementation is already written and is available in the
*py3k-importhook* branch. [4]_ It still requires some cleanups,
documentation updates and additional unit tests.


Acknowledgments
===============

Alyssa Coghlan, for proof reading and the initial discussion
Phillip J. Eby, for his implementation in PEAK and help with my own implementation


Copyright
=========

This document has been placed in the public domain.


References
==========

.. [1] PEP: Lazy module imports and post import hook
   http://permalink.gmane.org/gmane.comp.python.devel/90949

.. [2] Interest in PEP for callbacks on module import
   http://permalink.gmane.org/gmane.comp.python.python-3000.devel/11126

.. [3] peak.utils.imports
   http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup

.. [4] py3k-importhook branch
   http://svn.python.org/view/python/branches/py3k-importhook/


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: