268 lines
9.7 KiB
Plaintext
268 lines
9.7 KiB
Plaintext
PEP: 3149
|
||
Title: ABI version tagged .so files
|
||
Version: $Revision: 81577 $
|
||
Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $
|
||
Author: Barry Warsaw <barry@python.org>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 2010-07-09
|
||
Python-Version: 3.2
|
||
Post-History: 2010-07-14
|
||
Resolution: TBD
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
PEP 3147 [1]_ described an extension to Python's import machinery that
|
||
improved the sharing of Python source code, by allowing more than one
|
||
byte compilation file (.pyc) to be co-located with each source file.
|
||
|
||
This PEP defines an adjunct feature which allows the co-location of
|
||
extension module files (.so) in a similar manner. This optional,
|
||
build-time feature will enable downstream distributions of Python to
|
||
more easily provide more than one Python major version at a time.
|
||
|
||
|
||
Background
|
||
==========
|
||
|
||
PEP 3147 defined the file system layout for a pure-Python package,
|
||
where multiple versions of Python are available on the system. For
|
||
example, where the `alpha` package containing source modules `one.py`
|
||
and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte
|
||
compilation file system layout would be::
|
||
|
||
alpha/
|
||
__pycache__/
|
||
__init__.cpython-32.pyc
|
||
__init__.cpython-33.pyc
|
||
one.cpython-32.pyc
|
||
one.cpython-33.pyc
|
||
two.cpython-32.pyc
|
||
two.cpython-33.pyc
|
||
__init__.py
|
||
one.py
|
||
two.py
|
||
|
||
For packages with extension modules, a similar differentiation is
|
||
needed for the module's .so files. Extension modules compiled for
|
||
different Python major versions are incompatible with each other due
|
||
to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it
|
||
will minimize, but not eliminate extension module incompatibilities
|
||
between Python major versions. Thus a mechanism for discriminating
|
||
extension module file names is proposed.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more
|
||
than one Python version at the same time to their users. For example,
|
||
Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
|
||
with Python 2.6 being the default.
|
||
|
||
In order to share as much as possible between the available Python
|
||
versions, these distributions install third party (i.e. non-standard
|
||
library) packages into `/usr/share/pyshared` and symlink to them from
|
||
`/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a
|
||
pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from
|
||
byte compilation by the various installed Pythons will name collide
|
||
with each other. For Python versions >= 3.2, all pure-Python packages
|
||
can be shared, because the `.pyc` files will no longer cause file
|
||
system naming conflicts. Eliminating these symlinks makes for a
|
||
simpler, more robust Python distribution.
|
||
|
||
A similar situation arises with shared library extensions. Because
|
||
extension modules are typically named `foo.so` for a `foo` extension
|
||
module, these would also name collide if `foo` was provided for more
|
||
than one Python version. There are several approaches that could be
|
||
taken to avoid this, which will be explored below, but this PEP
|
||
proposes a fairly simple compile-time option to allow extension
|
||
modules to live in the same file system directory and avoid any name
|
||
collisions.
|
||
|
||
|
||
Proposal
|
||
========
|
||
|
||
A new configure option is added for building Python, called
|
||
`--with-so-abi-tag`. This takes as an argument a unique, but
|
||
arbitrary string, e.g.::
|
||
|
||
./configure --with-so-abi-tag=cpython-32
|
||
|
||
This string is passed into the `Makefile` and affects two aspects of
|
||
the Python build. First, it is compiled into `Python/dynload_shlib.c`
|
||
where it defines some additional `.so` file names to search for when
|
||
importing extension modules. Second, it modifies the `Makefile`'s
|
||
`$SO` variable, which in turn controls the `distutils` module's default
|
||
filename when compiling extension modules.
|
||
|
||
When `--with-so-abi-tag` is not given to `configure` nothing changes
|
||
in the way the Python executable is built, or acts. Thus, this
|
||
configure switch is completely optional and has no effect if not used.
|
||
|
||
What this allows is for distributions that want to distinguish among
|
||
extension modules built for different versions of Python, but shared
|
||
in the same file system path, to arrange for `.so` names that are
|
||
unique and non-colliding.
|
||
|
||
For example, let's say Python 3.2 was built with::
|
||
|
||
./configure --with-so-abi-tag=cpython-32
|
||
|
||
and Python 3.3 was built with::
|
||
|
||
./configure --with-so-abi-tag=cpython-33
|
||
|
||
For an arbitrary package `foo`, you would see these files when the
|
||
distribution package was installed::
|
||
|
||
/usr/share/pyshared/foo.cpython-32.so
|
||
/usr/share/pyshared/foo.cpython-33.so
|
||
|
||
|
||
Proven approach
|
||
===============
|
||
|
||
The approach described here is already proven, in a sense, on Debian
|
||
and Ubuntu system where different extensions are used for debug builds
|
||
of Python and extension modules. Debug builds on Windows also already
|
||
use a different file extension for dynamic libraries.
|
||
|
||
|
||
PEP 384
|
||
=======
|
||
|
||
PEP 384 defines a stable ABI for extension modules. Universal
|
||
adoption of PEP 384 would eliminate the need for this PEP because all
|
||
extension modules would be compatible with any Python version. In
|
||
practice of course, it will be impossible to achieve universal
|
||
adoption. Older extensions may not be ported to PEP 384, or an
|
||
extension may require Python APIs outside of PEP 384 definition.
|
||
Therefore there will always be a (hopefully diminishing, but never
|
||
zero) need for ABI version tagged shared libraries.
|
||
|
||
Further, it is anticipated that the stable ABI will evolve over time,
|
||
meaning that existing PEP 384 compatible extension modules may be
|
||
incompatible with future versions of Python. While a complete
|
||
specification is reserved for PEP 384, here is a discussion of the
|
||
relevant issues.
|
||
|
||
PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
|
||
passed as the API version if the extension was complied with
|
||
``Py_LIMITED_API``. This should be formalized into an official macro
|
||
called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If
|
||
and when the ABI changes in an incompatible way, this version number
|
||
would be bumped. To facilitate sharing, Python would be extended to
|
||
search for extension modules with the ``PYTHON_ABI_VERSION`` number in
|
||
its name. The prefix ``abi`` is reserved for Python's use.
|
||
|
||
Thus for example, an initial implementation of PEP 384, compiled with
|
||
`--with-so-abi-tag=cpython-xy` would search for the following file
|
||
names when extension module `foo` is imported (in this order)::
|
||
|
||
foo.abi3.so
|
||
foo.cpython-xy.so
|
||
foo.so
|
||
|
||
The distutils [7]_ ``build_ext`` command would also have to be
|
||
extended to compile to shared library files with the ``abi3`` tag,
|
||
when the module author indicates that their extension supports that
|
||
version of the ABI. This could be done in a backward compatible way
|
||
by adding a keyword argument to the ``Extension`` class, such as::
|
||
|
||
Extension('foo', ['foo.c'], abi=3)
|
||
|
||
|
||
Alternatives
|
||
============
|
||
|
||
In the initial python-dev thread [8]_ where this idea was first
|
||
introduced, several alternatives were suggested. For completeness
|
||
they are listed here, along with the reasons for not adopting them.
|
||
|
||
|
||
Independent directories or symlinks
|
||
-----------------------------------
|
||
|
||
Debian and Ubuntu could simply add a version-specific directory to
|
||
``sys.path`` that would contain just the extension modules for that
|
||
version of Python. Or the symlink trick eliminated in PEP 3147 could
|
||
be retained for just shared libraries. This approach is rejected
|
||
because it propagates the essential complexity that PEP 3147 tries to
|
||
avoid, and adds yet another directory to search for all modules, even
|
||
when the number of extension modules is much fewer than the total
|
||
number of Python packages. It also makes for more robust management
|
||
when all of a package's module files live in the same directory,
|
||
because it allows systems such as `dpkg` to detect file conflicts
|
||
between distribution packages.
|
||
|
||
|
||
Don't share packages with extension modules
|
||
-------------------------------------------
|
||
|
||
It has been suggested that Python packages with extension modules not
|
||
be shared among all supported Python versions on a distribution. Even
|
||
with adoption of PEP 3149, extension modules will have to be compiled
|
||
for every supported Python version, so perhaps sharing of such
|
||
packages isn't useful anyway. Not sharing packages with extensions
|
||
though is infeasible for several reasons.
|
||
|
||
If a pure-Python package is shared in one version, should it suddenly
|
||
be not-shared if the next release adds an extension module for speed?
|
||
Also, even though all extension shared libraries will be compiled and
|
||
distributed once for every supported Python, there's a big difference
|
||
between duplicating the `.so` files and duplicating all `.py` files.
|
||
The extra space increases the download time for such packages, and
|
||
more immediately, increases the space pressures on already constrained
|
||
distribution CD-ROMs.
|
||
|
||
|
||
Reference implementation
|
||
========================
|
||
|
||
Work on this code is tracked in a Bazaar branch on Launchpad [5]_
|
||
until it's ready for merge into Python 3.2. The work-in-progress diff
|
||
can also be viewed [6]_ and is updated automatically as new changes
|
||
are uploaded.
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] PEP 3147
|
||
|
||
.. [2] PEP 384
|
||
|
||
.. [3] Ubuntu: <http://www.ubuntu.com>
|
||
|
||
.. [4] Debian: <http://www.debian.org>
|
||
|
||
.. [5] https://code.edge.launchpad.net/~barry/python/sovers
|
||
|
||
.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
|
||
|
||
.. [7] http://docs.python.org/py3k/distutils/index.html
|
||
|
||
.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|