268 lines
9.7 KiB
Plaintext
268 lines
9.7 KiB
Plaintext
|
PEP: 3149
|
|||
|
|
Title: ABI version tagged .so files
|
|||
|
|
Version: $Revision: 81577 $
|
|||
|
|
Last-Modified: $Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) $
|
|||
|
|
Author: Barry Warsaw <barry@python.org>
|
|||
|
|
Status: Draft
|
|||
|
|
Type: Standards Track
|
|||
|
|
Content-Type: text/x-rst
|
|||
|
|
Created: 2010-07-09
|
|||
|
|
Python-Version: 3.2
|
|||
|
|
Post-History: 2010-07-14
|
|||
|
|
Resolution: TBD
|
|||
|
|
|
|||
|
|
|
|||
|
|
Abstract
|
|||
|
|
========
|
|||
|
|
|
|||
|
|
PEP 3147 [1]_ described an extension to Python's import machinery that
|
|||
|
|
improved the sharing of Python source code, by allowing more than one
|
|||
|
|
byte compilation file (.pyc) to be co-located with each source file.
|
|||
|
|
|
|||
|
|
This PEP defines an adjunct feature which allows the co-location of
|
|||
|
|
extension module files (.so) in a similar manner. This optional,
|
|||
|
|
build-time feature will enable downstream distributions of Python to
|
|||
|
|
more easily provide more than one Python major version at a time.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Background
|
|||
|
|
==========
|
|||
|
|
|
|||
|
|
PEP 3147 defined the file system layout for a pure-Python package,
|
|||
|
|
where multiple versions of Python are available on the system. For
|
|||
|
|
example, where the `alpha` package containing source modules `one.py`
|
|||
|
|
and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte
|
|||
|
|
compilation file system layout would be::
|
|||
|
|
|
|||
|
|
alpha/
|
|||
|
|
__pycache__/
|
|||
|
|
__init__.cpython-32.pyc
|
|||
|
|
__init__.cpython-33.pyc
|
|||
|
|
one.cpython-32.pyc
|
|||
|
|
one.cpython-33.pyc
|
|||
|
|
two.cpython-32.pyc
|
|||
|
|
two.cpython-33.pyc
|
|||
|
|
__init__.py
|
|||
|
|
one.py
|
|||
|
|
two.py
|
|||
|
|
|
|||
|
|
For packages with extension modules, a similar differentiation is
|
|||
|
|
needed for the module's .so files. Extension modules compiled for
|
|||
|
|
different Python major versions are incompatible with each other due
|
|||
|
|
to changes in the ABI. While PEP 384 [2]_ defines a stable ABI, it
|
|||
|
|
will minimize, but not eliminate extension module incompatibilities
|
|||
|
|
between Python major versions. Thus a mechanism for discriminating
|
|||
|
|
extension module file names is proposed.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Rationale
|
|||
|
|
=========
|
|||
|
|
|
|||
|
|
Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more
|
|||
|
|
than one Python version at the same time to their users. For example,
|
|||
|
|
Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
|
|||
|
|
with Python 2.6 being the default.
|
|||
|
|
|
|||
|
|
In order to share as much as possible between the available Python
|
|||
|
|
versions, these distributions install third party (i.e. non-standard
|
|||
|
|
library) packages into `/usr/share/pyshared` and symlink to them from
|
|||
|
|
`/usr/lib/pythonX.Y/dist-packages`. The symlinks exist because in a
|
|||
|
|
pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files resulting from
|
|||
|
|
byte compilation by the various installed Pythons will name collide
|
|||
|
|
with each other. For Python versions >= 3.2, all pure-Python packages
|
|||
|
|
can be shared, because the `.pyc` files will no longer cause file
|
|||
|
|
system naming conflicts. Eliminating these symlinks makes for a
|
|||
|
|
simpler, more robust Python distribution.
|
|||
|
|
|
|||
|
|
A similar situation arises with shared library extensions. Because
|
|||
|
|
extension modules are typically named `foo.so` for a `foo` extension
|
|||
|
|
module, these would also name collide if `foo` was provided for more
|
|||
|
|
than one Python version. There are several approaches that could be
|
|||
|
|
taken to avoid this, which will be explored below, but this PEP
|
|||
|
|
proposes a fairly simple compile-time option to allow extension
|
|||
|
|
modules to live in the same file system directory and avoid any name
|
|||
|
|
collisions.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Proposal
|
|||
|
|
========
|
|||
|
|
|
|||
|
|
A new configure option is added for building Python, called
|
|||
|
|
`--with-so-abi-tag`. This takes as an argument a unique, but
|
|||
|
|
arbitrary string, e.g.::
|
|||
|
|
|
|||
|
|
./configure --with-so-abi-tag=cpython-32
|
|||
|
|
|
|||
|
|
This string is passed into the `Makefile` and affects two aspects of
|
|||
|
|
the Python build. First, it is compiled into `Python/dynload_shlib.c`
|
|||
|
|
where it defines some additional `.so` file names to search for when
|
|||
|
|
importing extension modules. Second, it modifies the `Makefile`'s
|
|||
|
|
`$SO` variable, which in turn controls the `distutils` module's default
|
|||
|
|
filename when compiling extension modules.
|
|||
|
|
|
|||
|
|
When `--with-so-abi-tag` is not given to `configure` nothing changes
|
|||
|
|
in the way the Python executable is built, or acts. Thus, this
|
|||
|
|
configure switch is completely optional and has no effect if not used.
|
|||
|
|
|
|||
|
|
What this allows is for distributions that want to distinguish among
|
|||
|
|
extension modules built for different versions of Python, but shared
|
|||
|
|
in the same file system path, to arrange for `.so` names that are
|
|||
|
|
unique and non-colliding.
|
|||
|
|
|
|||
|
|
For example, let's say Python 3.2 was built with::
|
|||
|
|
|
|||
|
|
./configure --with-so-abi-tag=cpython-32
|
|||
|
|
|
|||
|
|
and Python 3.3 was built with::
|
|||
|
|
|
|||
|
|
./configure --with-so-abi-tag=cpython-33
|
|||
|
|
|
|||
|
|
For an arbitrary package `foo`, you would see these files when the
|
|||
|
|
distribution package was installed::
|
|||
|
|
|
|||
|
|
/usr/share/pyshared/foo.cpython-32.so
|
|||
|
|
/usr/share/pyshared/foo.cpython-33.so
|
|||
|
|
|
|||
|
|
|
|||
|
|
Proven approach
|
|||
|
|
===============
|
|||
|
|
|
|||
|
|
The approach described here is already proven, in a sense, on Debian
|
|||
|
|
and Ubuntu system where different extensions are used for debug builds
|
|||
|
|
of Python and extension modules. Debug builds on Windows also already
|
|||
|
|
use a different file extension for dynamic libraries.
|
|||
|
|
|
|||
|
|
|
|||
|
|
PEP 384
|
|||
|
|
=======
|
|||
|
|
|
|||
|
|
PEP 384 defines a stable ABI for extension modules. Universal
|
|||
|
|
adoption of PEP 384 would eliminate the need for this PEP because all
|
|||
|
|
extension modules would be compatible with any Python version. In
|
|||
|
|
practice of course, it will be impossible to achieve universal
|
|||
|
|
adoption. Older extensions may not be ported to PEP 384, or an
|
|||
|
|
extension may require Python APIs outside of PEP 384 definition.
|
|||
|
|
Therefore there will always be a (hopefully diminishing, but never
|
|||
|
|
zero) need for ABI version tagged shared libraries.
|
|||
|
|
|
|||
|
|
Further, it is anticipated that the stable ABI will evolve over time,
|
|||
|
|
meaning that existing PEP 384 compatible extension modules may be
|
|||
|
|
incompatible with future versions of Python. While a complete
|
|||
|
|
specification is reserved for PEP 384, here is a discussion of the
|
|||
|
|
relevant issues.
|
|||
|
|
|
|||
|
|
PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
|
|||
|
|
passed as the API version if the extension was complied with
|
|||
|
|
``Py_LIMITED_API``. This should be formalized into an official macro
|
|||
|
|
called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If
|
|||
|
|
and when the ABI changes in an incompatible way, this version number
|
|||
|
|
would be bumped. To facilitate sharing, Python would be extended to
|
|||
|
|
search for extension modules with the ``PYTHON_ABI_VERSION`` number in
|
|||
|
|
its name. The prefix ``abi`` is reserved for Python's use.
|
|||
|
|
|
|||
|
|
Thus for example, an initial implementation of PEP 384, compiled with
|
|||
|
|
`--with-so-abi-tag=cpython-xy` would search for the following file
|
|||
|
|
names when extension module `foo` is imported (in this order)::
|
|||
|
|
|
|||
|
|
foo.abi3.so
|
|||
|
|
foo.cpython-xy.so
|
|||
|
|
foo.so
|
|||
|
|
|
|||
|
|
The distutils [7]_ ``build_ext`` command would also have to be
|
|||
|
|
extended to compile to shared library files with the ``abi3`` tag,
|
|||
|
|
when the module author indicates that their extension supports that
|
|||
|
|
version of the ABI. This could be done in a backward compatible way
|
|||
|
|
by adding a keyword argument to the ``Extension`` class, such as::
|
|||
|
|
|
|||
|
|
Extension('foo', ['foo.c'], abi=3)
|
|||
|
|
|
|||
|
|
|
|||
|
|
Alternatives
|
|||
|
|
============
|
|||
|
|
|
|||
|
|
In the initial python-dev thread [8]_ where this idea was first
|
|||
|
|
introduced, several alternatives were suggested. For completeness
|
|||
|
|
they are listed here, along with the reasons for not adopting them.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Independent directories or symlinks
|
|||
|
|
-----------------------------------
|
|||
|
|
|
|||
|
|
Debian and Ubuntu could simply add a version-specific directory to
|
|||
|
|
``sys.path`` that would contain just the extension modules for that
|
|||
|
|
version of Python. Or the symlink trick eliminated in PEP 3147 could
|
|||
|
|
be retained for just shared libraries. This approach is rejected
|
|||
|
|
because it propagates the essential complexity that PEP 3147 tries to
|
|||
|
|
avoid, and adds yet another directory to search for all modules, even
|
|||
|
|
when the number of extension modules is much fewer than the total
|
|||
|
|
number of Python packages. It also makes for more robust management
|
|||
|
|
when all of a package's module files live in the same directory,
|
|||
|
|
because it allows systems such as `dpkg` to detect file conflicts
|
|||
|
|
between distribution packages.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Don't share packages with extension modules
|
|||
|
|
-------------------------------------------
|
|||
|
|
|
|||
|
|
It has been suggested that Python packages with extension modules not
|
|||
|
|
be shared among all supported Python versions on a distribution. Even
|
|||
|
|
with adoption of PEP 3149, extension modules will have to be compiled
|
|||
|
|
for every supported Python version, so perhaps sharing of such
|
|||
|
|
packages isn't useful anyway. Not sharing packages with extensions
|
|||
|
|
though is infeasible for several reasons.
|
|||
|
|
|
|||
|
|
If a pure-Python package is shared in one version, should it suddenly
|
|||
|
|
be not-shared if the next release adds an extension module for speed?
|
|||
|
|
Also, even though all extension shared libraries will be compiled and
|
|||
|
|
distributed once for every supported Python, there's a big difference
|
|||
|
|
between duplicating the `.so` files and duplicating all `.py` files.
|
|||
|
|
The extra space increases the download time for such packages, and
|
|||
|
|
more immediately, increases the space pressures on already constrained
|
|||
|
|
distribution CD-ROMs.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Reference implementation
|
|||
|
|
========================
|
|||
|
|
|
|||
|
|
Work on this code is tracked in a Bazaar branch on Launchpad [5]_
|
|||
|
|
until it's ready for merge into Python 3.2. The work-in-progress diff
|
|||
|
|
can also be viewed [6]_ and is updated automatically as new changes
|
|||
|
|
are uploaded.
|
|||
|
|
|
|||
|
|
|
|||
|
|
References
|
|||
|
|
==========
|
|||
|
|
|
|||
|
|
.. [1] PEP 3147
|
|||
|
|
|
|||
|
|
.. [2] PEP 384
|
|||
|
|
|
|||
|
|
.. [3] Ubuntu: <http://www.ubuntu.com>
|
|||
|
|
|
|||
|
|
.. [4] Debian: <http://www.debian.org>
|
|||
|
|
|
|||
|
|
.. [5] https://code.edge.launchpad.net/~barry/python/sovers
|
|||
|
|
|
|||
|
|
.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
|
|||
|
|
|
|||
|
|
.. [7] http://docs.python.org/py3k/distutils/index.html
|
|||
|
|
|
|||
|
|
.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
|
|||
|
|
|
|||
|
|
|
|||
|
|
Copyright
|
|||
|
|
=========
|
|||
|
|
|
|||
|
|
This document has been placed in the public domain.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
..
|
|||
|
|
Local Variables:
|
|||
|
|
mode: indented-text
|
|||
|
|
indent-tabs-mode: nil
|
|||
|
|
sentence-end-double-space: t
|
|||
|
|
fill-column: 70
|
|||
|
|
coding: utf-8
|
|||
|
|
End:
|