python-peps/peps/pep-0547.rst

PEP: 547
Title: Running extension modules using the -m option
Version: $Revision$
Last-Modified: $Date$
Author: Marcel Plch <gmarcel.plch@gmail.com>,
        Petr Viktorin <encukou@gmail.com>
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 25-May-2017
Python-Version: 3.7
Post-History:


Deferral Notice
===============

Cython -- the most important use case for this PEP and the only explicit
one -- is not ready for multi-phase initialization yet.
It keeps global state in C-level static variables.
See discussion at `Cython issue 1923`_.

The PEP is deferred until the situation changes.


Abstract
========

This PEP proposes implementation that allows built-in and extension
modules to be executed in the ``__main__`` namespace using
the :pep:`489` multi-phase initialization.

With this, a multi-phase initialization enabled module can be run
using following command::

    $ python3 -m _testmultiphase
    This is a test module named __main__.


Motivation
==========

Currently, extension modules do not support all functionality of
Python source modules.
Specifically, it is not possible to run extension modules as scripts using
Python's ``-m`` option.

The technical groundwork to make this possible has been done for :pep:`489`,
and enabling the ``-m`` option is listed in that PEP's
“Possible Future Extensions” section.
Technically, the additional changes proposed here are relatively small.


Rationale
=========

Extension modules' lack of support for the ``-m`` option has traditionally
been worked around by providing a Python wrapper.
For example, the ``_pickle`` module's command line interface is in the
pure-Python ``pickle`` module (along with a pure-Python reimplementation).

This works well for standard library modules, as building command line
interfaces using the C API is cumbersome.
However, other users may want to create executable extension modules directly.

An important use case is Cython, a Python-like language that compiles to
C extension modules.
Cython is a (near) superset of Python, meaning that compiling a Python module
with Cython will typically not change the module's functionality, allowing
Cython-specific features to be added gradually.
This PEP will allow Cython extension modules to behave the same as their Python
counterparts when run using the ``-m`` option.
Cython developers consider the feature worth implementing (see
`Cython issue 1715`_).


Background
==========

Python's ``-m`` option is handled by the function
``runpy._run_module_as_main``.

The module specified by ``-m`` is not imported normally.
Instead, it is executed in the namespace of the ``__main__`` module,
which is created quite early in interpreter initialization.

For Python source modules, running in another module's namespace is not
a problem: the code is executed with ``locals`` and ``globals`` set to the
existing module's ``__dict__``.
This is not the case for extension modules, whose ``PyInit_*`` entry point
traditionally both created a new module object (using ``PyModule_Create``),
and initialized it.

Since Python 3.5, extension modules can use :pep:`489` multi-phase initialization.
In this scenario, the ``PyInit_*`` entry point returns a ``PyModuleDef``
structure: a description of how the module should be created and initialized.
The extension can choose to customize creation of the module object using
the ``Py_mod_create`` callback, or opt to use a normal module object by not
specifying ``Py_mod_create``.
Another callback, ``Py_mod_exec``, is then called to initialize the module
object, e.g. by populating it with methods and classes.


Proposal
========

Multi-phase initialization makes it possible to execute an extension module in
another module's namespace: if a ``Py_mod_create`` callback is not specified,
the ``__main__`` module can be passed to the ``Py_mod_exec`` callback to be
initialized, as if ``__main__`` was a freshly constructed module object.

One complication in this scheme is C-level module state.
Each module has a ``md_state`` pointer that points to a region of memory
allocated when an extension module is created.
The ``PyModuleDef`` specifies how much memory is to be allocated.

The implementation must take care that ``md_state`` memory is allocated at most
once.
Also, the ``Py_mod_exec`` callback should only be called once per module.
The implications of multiply-initialized modules are too subtle to require
expecting extension authors to reason about them.
The ``md_state`` pointer itself will serve as a guard: allocating the memory
and calling ``Py_mod_exec`` will always be done together, and initializing an
extension module will fail if ``md_state`` is already non-NULL.

Since the ``__main__`` module is not created as an extension module,
its ``md_state`` is normally ``NULL``.
Before initializing an extension module in ``__main__``'s context, its module
state will be allocated according to the ``PyModuleDef`` of that module.

While :pep:`489` was designed to make these changes generally possible,
it's necessary to decouple module discovery, creation, and initialization
steps for extension modules, so that another module can be used instead of
a newly initialized one, and the functionality needs to be added to
``runpy`` and ``importlib``.


Specification
=============

A new optional method for importlib loaders will be added.
This method will be called ``exec_in_module`` and will take two
positional arguments: module spec and an already existing module.
Any import-related attributes, such as ``__spec__`` or ``__name__``,
already set on the module will be ignored.

The ``runpy._run_module_as_main`` function will look for this new
loader method.
If it is present, ``runpy`` will execute it instead of trying to load and
run the module's Python code.
Otherwise, ``runpy`` will act as before.


ExtensionFileLoader Changes
---------------------------

importlib's ``ExtensionFileLoader`` will get an implementation of
``exec_in_module`` that will call a new function, ``_imp.exec_in_module``.

``_imp.exec_in_module`` will use existing machinery to find and call an
extension module's ``PyInit_*`` function.

The ``PyInit_*`` function can return either a fully initialized module
(single-phase initialization) or a ``PyModuleDef`` (for :pep:`489` multi-phase
initialization).

In the single-phase initialization case, ``_imp.exec_in_module`` will raise
``ImportError``.

In the multi-phase initialization case, the ``PyModuleDef`` and the module to
be initialized will be passed to a new function, ``PyModule_ExecInModule``.

This function raises ``ImportError`` if the ``PyModuleDef`` specifies
a ``Py_mod_create`` slot, or if the module has already been initialized
(i.e. its ``md_state`` pointer is not ``NULL``).
Otherwise, the function will initialize the module according to the
``PyModuleDef``.


Backwards Compatibility
=======================

This PEP maintains backwards compatibility.
It only adds new functions, and a new loader method that is added for
a loader that previously did not support running modules as ``__main__``.


Reference Implementation
========================

The reference implementation of this PEP is available at GitHub_.


References
==========

.. _GitHub: https://github.com/python/cpython/pull/1761
.. _Cython issue 1715: https://github.com/cython/cython/issues/1715
.. _Cython issue 1923: https://github.com/cython/cython/pull/1923


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: