diff --git a/pep-3149.txt b/pep-3149.txt new file mode 100644 index 000000000..eaf76b810 --- /dev/null +++ b/pep-3149.txt @@ -0,0 +1,267 @@ +PEP: 3149 +Title: ABI version tagged .so files +Version: $Revision: 81577 $ +Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $ +Author: Barry Warsaw +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 2010-07-09 +Python-Version: 3.2 +Post-History: 2010-07-14 +Resolution: TBD + + +Abstract +======== + +PEP 3147 [1]_ described an extension to Python's import machinery that +improved the sharing of Python source code, by allowing more than one +byte compilation file (.pyc) to be co-located with each source file. + +This PEP defines an adjunct feature which allows the co-location of +extension module files (.so) in a similar manner. This optional, +build-time feature will enable downstream distributions of Python to +more easily provide more than one Python major version at a time. + + +Background +========== + +PEP 3147 defined the file system layout for a pure-Python package, +where multiple versions of Python are available on the system. For +example, where the `alpha` package containing source modules `one.py` +and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte +compilation file system layout would be:: + + alpha/ + __pycache__/ + __init__.cpython-32.pyc + __init__.cpython-33.pyc + one.cpython-32.pyc + one.cpython-33.pyc + two.cpython-32.pyc + two.cpython-33.pyc + __init__.py + one.py + two.py + +For packages with extension modules, a similar differentiation is +needed for the module's .so files. Extension modules compiled for +different Python major versions are incompatible with each other due +to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it +will minimize, but not eliminate extension module incompatibilities +between Python major versions. Thus a mechanism for discriminating +extension module file names is proposed. + + +Rationale +========= + +Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more +than one Python version at the same time to their users. For example, +Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, +with Python 2.6 being the default. + +In order to share as much as possible between the available Python +versions, these distributions install third party (i.e. non-standard +library) packages into `/usr/share/pyshared` and symlink to them from +`/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a +pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from +byte compilation by the various installed Pythons will name collide +with each other. For Python versions >= 3.2, all pure-Python packages +can be shared, because the `.pyc` files will no longer cause file +system naming conflicts. Eliminating these symlinks makes for a +simpler, more robust Python distribution. + +A similar situation arises with shared library extensions. Because +extension modules are typically named `foo.so` for a `foo` extension +module, these would also name collide if `foo` was provided for more +than one Python version. There are several approaches that could be +taken to avoid this, which will be explored below, but this PEP +proposes a fairly simple compile-time option to allow extension +modules to live in the same file system directory and avoid any name +collisions. + + +Proposal +======== + +A new configure option is added for building Python, called +`--with-so-abi-tag`. This takes as an argument a unique, but +arbitrary string, e.g.:: + + ./configure --with-so-abi-tag=cpython-32 + +This string is passed into the `Makefile` and affects two aspects of +the Python build. First, it is compiled into `Python/dynload_shlib.c` +where it defines some additional `.so` file names to search for when +importing extension modules. Second, it modifies the `Makefile`'s +`$SO` variable, which in turn controls the `distutils` module's default +filename when compiling extension modules. + +When `--with-so-abi-tag` is not given to `configure` nothing changes +in the way the Python executable is built, or acts. Thus, this +configure switch is completely optional and has no effect if not used. + +What this allows is for distributions that want to distinguish among +extension modules built for different versions of Python, but shared +in the same file system path, to arrange for `.so` names that are +unique and non-colliding. + +For example, let's say Python 3.2 was built with:: + + ./configure --with-so-abi-tag=cpython-32 + +and Python 3.3 was built with:: + + ./configure --with-so-abi-tag=cpython-33 + +For an arbitrary package `foo`, you would see these files when the +distribution package was installed:: + + /usr/share/pyshared/foo.cpython-32.so + /usr/share/pyshared/foo.cpython-33.so + + +Proven approach +=============== + +The approach described here is already proven, in a sense, on Debian +and Ubuntu system where different extensions are used for debug builds +of Python and extension modules. Debug builds on Windows also already +use a different file extension for dynamic libraries. + + +PEP 384 +======= + +PEP 384 defines a stable ABI for extension modules. Universal +adoption of PEP 384 would eliminate the need for this PEP because all +extension modules would be compatible with any Python version. In +practice of course, it will be impossible to achieve universal +adoption. Older extensions may not be ported to PEP 384, or an +extension may require Python APIs outside of PEP 384 definition. +Therefore there will always be a (hopefully diminishing, but never +zero) need for ABI version tagged shared libraries. + +Further, it is anticipated that the stable ABI will evolve over time, +meaning that existing PEP 384 compatible extension modules may be +incompatible with future versions of Python. While a complete +specification is reserved for PEP 384, here is a discussion of the +relevant issues. + +PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is +passed as the API version if the extension was complied with +``Py_LIMITED_API``. This should be formalized into an official macro +called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If +and when the ABI changes in an incompatible way, this version number +would be bumped. To facilitate sharing, Python would be extended to +search for extension modules with the ``PYTHON_ABI_VERSION`` number in +its name. The prefix ``abi`` is reserved for Python's use. + +Thus for example, an initial implementation of PEP 384, compiled with +`--with-so-abi-tag=cpython-xy` would search for the following file +names when extension module `foo` is imported (in this order):: + + foo.abi3.so + foo.cpython-xy.so + foo.so + +The distutils [7]_ ``build_ext`` command would also have to be +extended to compile to shared library files with the ``abi3`` tag, +when the module author indicates that their extension supports that +version of the ABI. This could be done in a backward compatible way +by adding a keyword argument to the ``Extension`` class, such as:: + + Extension('foo', ['foo.c'], abi=3) + + +Alternatives +============ + +In the initial python-dev thread [8]_ where this idea was first +introduced, several alternatives were suggested. For completeness +they are listed here, along with the reasons for not adopting them. + + +Independent directories or symlinks +----------------------------------- + +Debian and Ubuntu could simply add a version-specific directory to +``sys.path`` that would contain just the extension modules for that +version of Python. Or the symlink trick eliminated in PEP 3147 could +be retained for just shared libraries. This approach is rejected +because it propagates the essential complexity that PEP 3147 tries to +avoid, and adds yet another directory to search for all modules, even +when the number of extension modules is much fewer than the total +number of Python packages. It also makes for more robust management +when all of a package's module files live in the same directory, +because it allows systems such as `dpkg` to detect file conflicts +between distribution packages. + + +Don't share packages with extension modules +------------------------------------------- + +It has been suggested that Python packages with extension modules not +be shared among all supported Python versions on a distribution. Even +with adoption of PEP 3149, extension modules will have to be compiled +for every supported Python version, so perhaps sharing of such +packages isn't useful anyway. Not sharing packages with extensions +though is infeasible for several reasons. + +If a pure-Python package is shared in one version, should it suddenly +be not-shared if the next release adds an extension module for speed? +Also, even though all extension shared libraries will be compiled and +distributed once for every supported Python, there's a big difference +between duplicating the `.so` files and duplicating all `.py` files. +The extra space increases the download time for such packages, and +more immediately, increases the space pressures on already constrained +distribution CD-ROMs. + + +Reference implementation +======================== + +Work on this code is tracked in a Bazaar branch on Launchpad [5]_ +until it's ready for merge into Python 3.2. The work-in-progress diff +can also be viewed [6]_ and is updated automatically as new changes +are uploaded. + + +References +========== + +.. [1] PEP 3147 + +.. [2] PEP 384 + +.. [3] Ubuntu: + +.. [4] Debian: + +.. [5] https://code.edge.launchpad.net/~barry/python/sovers + +.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411 + +.. [7] http://docs.python.org/py3k/distutils/index.html + +.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: