Updates based on python-dev discussion.

pep-3149.txt

@@ -8,7 +8,7 @@ Type: Standards Track
 Content-Type: text/x-rst
 Created: 2010-07-09
 Python-Version: 3.2
-Post-History: 2010-07-14
+Post-History: 2010-07-14, 2010-07-22
 Resolution: TBD
 
 
@@ -49,10 +49,14 @@ compilation file system layout would be::
 For packages with extension modules, a similar differentiation is
 needed for the module's .so files.  Extension modules compiled for
 different Python major versions are incompatible with each other due
-to changes in the ABI.  While PEP 384 [2]_ defines a stable ABI, it
+to changes in the ABI.  Different configuration/compilation options
-will minimize, but not eliminate extension module incompatibilities
+for the same Python version can result in different ABIs
-between Python major versions.  Thus a mechanism for discriminating
+(e.g. --with-wide-unicode).
-extension module file names is proposed.
+
+While PEP 384 [2]_ defines a stable ABI, it will minimize, but not
+eliminate extension module incompatibilities between Python builds or
+major versions.  Thus a mechanism for discriminating extension module
+file names is proposed.
 
 
 Rationale
@@ -77,51 +81,85 @@ simpler, more robust Python distribution.
 A similar situation arises with shared library extensions.  Because
 extension modules are typically named `foo.so` for a `foo` extension
 module, these would also name collide if `foo` was provided for more
-than one Python version.  There are several approaches that could be
+than one Python version.
-taken to avoid this, which will be explored below, but this PEP
+
-proposes a fairly simple compile-time option to allow extension
+In addition, because different configuration/compilation options for
-modules to live in the same file system directory and avoid any name
+the same Python version can cause different ABIs to be presented to
-collisions.
+extension modules.  On POSIX systems for example, the configure
+options ``--with-pydebug``, ``--with-pymalloc``, and
+``--with-wide-unicode`` all change the ABI.  This PEP proposes to
+encode build-time options in the file name of the ``.so`` extension
+module files.
+
+PyPy [5]_ can also benefit from this PEP, allowing it to avoid name
+collisions in extension modules built for its API, but with a
+different `.so` tag.
 
 
 Proposal
 ========
 
-A new configure option is added for building Python, called
+The configure/compilation options chosen at Python interpreter
-`--with-so-abi-tag`.  This takes as an argument a unique, but
+build-time will be encoded in the shared library file name for
-arbitrary string, e.g.::
+extension modules.  This "tag" will appear between the module base
+name and the operation file system extension for shared libraries.
 
-    ./configure --with-so-abi-tag=cpython-32
+The following information *MUST* be included in the shared library
+file name:
 
-This string is passed into the `Makefile` and affects two aspects of
+ * The Python implementation (e.g. cpython, pypy, jython, etc.)
-the Python build.  First, it is compiled into `Python/dynload_shlib.c`
+ * The interpreter's major and minor version numbers
-where it defines some additional `.so` file names to search for when
-importing extension modules.  Second, it modifies the `Makefile`'s
-`$SO` variable, which in turn controls the `distutils` module's default
-filename when compiling extension modules.
 
-When `--with-so-abi-tag` is not given to `configure` nothing changes
+These two fields are separated by a hyphen and no dots are to appear
-in the way the Python executable is built, or acts.  Thus, this
+between the major and minor version numbers.  E.g. ``cpython-32``.
-configure switch is completely optional and has no effect if not used.
 
-What this allows is for distributions that want to distinguish among
+Python implementations *MAY* include additional flags in the file name
-extension modules built for different versions of Python, but shared
+tag as appropriate.  For example, on POSIX systems these flags will
-in the same file system path, to arrange for `.so` names that are
+also contribute to the file name:
-unique and non-colliding.
 
-For example, let's say Python 3.2 was built with::
+ * ``--with-pydebug`` (flag: ``d``)
+ * ``--with-pymalloc`` (flag: ``m``)
+ * ``--with-wide-unicode`` (flag: ``u``)
 
-    ./configure --with-so-abi-tag=cpython-32
+By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so
+shared library file names would appear as ``foo.cpython-32m.so``.
+When the other two flags are also enabled, the file names would be
+``foo.cpython-32dmu.so``.
 
-and Python 3.3 was built with::
+(This PEP only addresses build issues on POSIX systems that use the
+``configure`` script.  While Windows or other platform support is not
+explicitly disallowed under this PEP, platform expertise is needed in
+order to evaluate, describe, and implement support on such platforms.)
 
-    ./configure --with-so-abi-tag=cpython-33
+The shared library file name tag is used unconditionally; it cannot be
+changed.  The tag and extension module suffix are available through
+the ``sysconfig`` modules via the following variables::
+
+    >>> sysconfig.get_config_var('SO')
+    '.cpython-32mu.so'
+    >>> sysconfig.get_config_var('SOABI')
+    'cpython-32mu'
+
+Note that ``$SOABI`` contains just the tag, while ``$SO`` includes the
+platform extension for shared library files, and is the exact suffix
+added to the extension module name.
 
 For an arbitrary package `foo`, you would see these files when the
 distribution package was installed::
 
-    /usr/share/pyshared/foo.cpython-32.so
+    /usr/share/pyshared/foo.cpython-32m.so
-    /usr/share/pyshared/foo.cpython-33.so
+    /usr/share/pyshared/foo.cpython-33m.so
+
+Python's dynamic module loader will recognize and import shared
+library extension modules with a tag that matches its build-time
+options.  For backward compatibility, Python will also continue to
+import untagged extension modules, e.g. ``foo.so``.
+
+This shared library tag would be used globally for all distutils-based
+extension modules, regardless of where on the file system they are
+built.  Extension modules built by means other than distutils would
+either have to calculate the tag manually, or fallback to the
+non-tagged `.so` file name.
 
 
 Proven approach
@@ -130,26 +168,22 @@ Proven approach
 The approach described here is already proven, in a sense, on Debian
 and Ubuntu system where different extensions are used for debug builds
 of Python and extension modules.  Debug builds on Windows also already
-use a different file extension for dynamic libraries.
+use a different file extension for dynamic libraries, and in fact
+encoded (in a different way than proposed in this PEP) the Python
+major and minor version in the `.dll` file name.
 
 
 PEP 384
 =======
 
-PEP 384 defines a stable ABI for extension modules.  Universal
+PEP 384 defines a stable ABI for extension modules.  In theory,
-adoption of PEP 384 would eliminate the need for this PEP because all
+universal adoption of PEP 384 would eliminate the need for this PEP
-extension modules would be compatible with any Python version.  In
+because all extension modules could be compatible with any Python
-practice of course, it will be impossible to achieve universal
+version.  In practice of course, it will be impossible to achieve
-adoption.  Older extensions may not be ported to PEP 384, or an
+universal adoption, and as described above, different built-time flags
-extension may require Python APIs outside of PEP 384 definition.
+still affect the ABI.  Thus even with a stable ABI, this PEP may still
-Therefore there will always be a (hopefully diminishing, but never
+be necessary.  While a complete specification is reserved for PEP 384,
-zero) need for ABI version tagged shared libraries.
+here is a discussion of the relevant issues.
-
-Further, it is anticipated that the stable ABI will evolve over time,
-meaning that existing PEP 384 compatible extension modules may be
-incompatible with future versions of Python.  While a complete
-specification is reserved for PEP 384, here is a discussion of the
-relevant issues.
 
 PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
 passed as the API version if the extension was complied with
@@ -160,15 +194,15 @@ would be bumped.  To facilitate sharing, Python would be extended to
 search for extension modules with the ``PYTHON_ABI_VERSION`` number in
 its name.  The prefix ``abi`` is reserved for Python's use.
 
-Thus for example, an initial implementation of PEP 384, compiled with
+Thus, an initial implementation of PEP 384, when Python is configured
-`--with-so-abi-tag=cpython-xy` would search for the following file
+with the default set of flags, would search for the following file
 names when extension module `foo` is imported (in this order)::
 
+    foo.cpython-XYm.so
     foo.abi3.so
-    foo.cpython-xy.so
     foo.so
 
-The distutils [7]_ ``build_ext`` command would also have to be
+The distutils [6]_ ``build_ext`` command would also have to be
 extended to compile to shared library files with the ``abi3`` tag,
 when the module author indicates that their extension supports that
 version of the ABI.  This could be done in a backward compatible way
@@ -180,7 +214,7 @@ by adding a keyword argument to the ``Extension`` class, such as::
 Alternatives
 ============
 
-In the initial python-dev thread [8]_ where this idea was first
+In the initial python-dev thread [7]_ where this idea was first
 introduced, several alternatives were suggested.  For completeness
 they are listed here, along with the reasons for not adopting them.
 
@@ -193,12 +227,12 @@ Debian and Ubuntu could simply add a version-specific directory to
 version of Python.  Or the symlink trick eliminated in PEP 3147 could
 be retained for just shared libraries.  This approach is rejected
 because it propagates the essential complexity that PEP 3147 tries to
-avoid, and adds yet another directory to search for all modules, even
+avoid, and adds potentially several additional directories to search
-when the number of extension modules is much fewer than the total
+for all modules, even when the number of extension modules is much
-number of Python packages.  It also makes for more robust management
+fewer than the total number of Python packages.  For example, builds
-when all of a package's module files live in the same directory,
+were made available both with and without wide unicode, with and
-because it allows systems such as `dpkg` to detect file conflicts
+without pydebug, and with and without pymalloc, the total number of
-between distribution packages.
+directories search increases substantially.
 
 
 Don't share packages with extension modules
@@ -216,17 +250,17 @@ be not-shared if the next release adds an extension module for speed?
 Also, even though all extension shared libraries will be compiled and
 distributed once for every supported Python, there's a big difference
 between duplicating the `.so` files and duplicating all `.py` files.
-The extra space increases the download time for such packages, and
+The extra size increases the download time for such packages, and more
-more immediately, increases the space pressures on already constrained
+immediately, increases the space pressures on already constrained
 distribution CD-ROMs.
 
 
 Reference implementation
 ========================
 
-Work on this code is tracked in a Bazaar branch on Launchpad [5]_
+Work on this code is tracked in a Bazaar branch on Launchpad [8]_
 until it's ready for merge into Python 3.2.  The work-in-progress diff
-can also be viewed [6]_ and is updated automatically as new changes
+can also be viewed [9]_ and is updated automatically as new changes
 are uploaded.
 
 
@@ -241,13 +275,15 @@ References
 
 .. [4] Debian: <http://www.debian.org>
 
-.. [5] https://code.edge.launchpad.net/~barry/python/sovers
+.. [5] http://codespeak.net/pypy/dist/pypy/doc/
 
-.. [6] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
+.. [6] http://docs.python.org/py3k/distutils/index.html
 
-.. [7] http://docs.python.org/py3k/distutils/index.html
+.. [7] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
 
-.. [8] http://mail.python.org/pipermail/python-dev/2010-June/100998.html
+.. [8] https://code.edge.launchpad.net/~barry/python/sovers
+
+.. [9] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411
 
 
 Copyright