python-peps/peps/pep-3121.rst

PEP: 3121
Title: Extension Module Initialization and Finalization
Version: $Revision$
Last-Modified: $Date$
Author: Martin von Löwis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Apr-2007
Python-Version: 3.0
Post-History:

.. canonical-doc:: :external+python:c:func:`PyInit_modulename` and
                   :external+python:c:type:`PyModuleDef`

Abstract
========

Extension module initialization currently has a few deficiencies.
There is no cleanup for modules, the entry point name might give
naming conflicts, the entry functions don't follow the usual calling
convention, and multiple interpreters are not supported well. This PEP
addresses these issues.

Problems
========

Module Finalization
-------------------

Currently, extension modules are initialized usually once and then
"live" forever. The only exception is when Py_Finalize() is called:
then the initialization routine is invoked a second time. This is bad
from a resource management point of view: memory and other resources
might get allocated each time initialization is called, but there is
no way to reclaim them. As a result, there is currently no way to
completely release all resources Python has allocated.

Entry point name conflicts
--------------------------

The entry point is currently called init<module>. This might conflict
with other symbols also called init<something>. In particular,
initsocket is known to have conflicted in the past (this specific
problem got resolved as a side effect of renaming the module to
_socket).

Entry point signature
---------------------

The entry point is currently a procedure (returning void).  This
deviates from the usual calling conventions; callers can find out
whether there was an error during initialization only by checking
PyErr_Occurred. The entry point should return a PyObject*, which will
be the module created, or NULL in case of an exception.

Multiple Interpreters
---------------------

Currently, extension modules share their state across all
interpreters. This allows for undesirable information leakage across
interpreters: one script could permanently corrupt objects in an
extension module, possibly breaking all scripts in other interpreters.

Specification
=============

The module initialization routines change their signature
to::

  PyObject *PyInit_<modulename>()

The initialization routine will be invoked once per
interpreter, when the module is imported. It should
return a new module object each time.

In order to store per-module state in C variables,
each module object will contain a block of memory
that is interpreted only by the module. The amount
of memory used for the module is specified at
the point of creation of the module.

In addition to the initialization function, a module
may implement a number of additional callback
functions, which are invoked when the module's
tp_traverse, tp_clear, and tp_free functions are
invoked, and when the module is reloaded.

The entire module definition is combined in a struct
PyModuleDef::

  struct PyModuleDef{
    PyModuleDef_Base m_base;  /* To be filled out by the interpreter */
    Py_ssize_t m_size; /* Size of per-module data */
    PyMethodDef *m_methods;
    inquiry m_reload;
    traverseproc m_traverse;
    inquiry m_clear;
    freefunc m_free;
  };

Creation of a module is changed to expect an optional
PyModuleDef*. The module state will be
null-initialized.

Each module method will be passed the module object
as the first parameter. To access the module data,
a function::

  void* PyModule_GetState(PyObject*);

will be provided. In addition, to lookup a module
more efficiently than going through sys.modules,
a function::

  PyObject* PyState_FindModule(struct PyModuleDef*);

will be provided. This lookup function will use an
index located in the m_base field, to find the
module by index, not by name.

As all Python objects should be controlled through
the Python memory management, usage of "static"
type objects is discouraged, unless the type object
itself has no memory-managed state. To simplify
definition of heap types, a new method::

  PyTypeObject* PyType_Copy(PyTypeObject*);

is added.

Example
=======

xxmodule.c would be changed to remove the initxx
function, and add the following code instead::

  struct xxstate{
    PyObject *ErrorObject;
    PyObject *Xxo_Type;
  };

  #define xxstate(o) ((struct xxstate*)PyModule_GetState(o))

  static int xx_traverse(PyObject *m, visitproc v,
                         void *arg)
  {
    Py_VISIT(xxstate(m)->ErrorObject);
    Py_VISIT(xxstate(m)->Xxo_Type);
    return 0;
  }

  static int xx_clear(PyObject *m)
  {
    Py_CLEAR(xxstate(m)->ErrorObject);
    Py_CLEAR(xxstate(m)->Xxo_Type);
    return 0;
  }

  static struct PyModuleDef xxmodule = {
    {}, /* m_base */
    sizeof(struct xxstate),
    &xx_methods,
    0,  /* m_reload */
    xx_traverse,
    xx_clear,
    0,  /* m_free - not needed, since all is done in m_clear */
  }

  PyObject*
  PyInit_xx()
  {
    PyObject *res = PyModule_New("xx", &xxmodule);
    if (!res) return NULL;
    xxstate(res)->ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
    if (!xxstate(res)->ErrorObject) {
      Py_DECREF(res);
      return NULL;
    }
    xxstate(res)->XxoType = PyType_Copy(&Xxo_Type);
    if (!xxstate(res)->Xxo_Type) {
      Py_DECREF(res);
      return NULL;
    }
    return res;
  }


Discussion
==========

Tim Peters reports in [1]_ that PythonLabs considered such a feature
at one point, and lists the following additional hooks which aren't
currently supported in this PEP:

* when the module object is deleted from sys.modules

* when Py_Finalize is called

* when Python exits

* when the Python DLL is unloaded (Windows only)


References
==========

.. [1] Tim Peters, reporting earlier conversation about such a feature
   https://mail.python.org/pipermail/python-3000/2006-April/000726.html


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: