diff --git a/pep-0513.txt b/pep-0513.txt index 21bce1484..934b242bb 100644 --- a/pep-0513.txt +++ b/pep-0513.txt @@ -126,8 +126,7 @@ For these reasons, to achieve broad portability, Python wheels To be eligible for the ``manylinux1`` platform tag, a Python wheel must therefore both (a) contain binary executables and compiled code that links -*only* to libraries (other than the appropriate ``libpython`` library, which is -always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs +*only* to libraries with SONAMEs included in the following list: :: libpanelw.so.5 @@ -153,11 +152,7 @@ included in the following list: :: libglib-2.0.so.0 and, (b) work on a stock CentOS 5.11 [6]_ system that contains the system -package manager's provided versions of these libraries. In addition, -for wheels targeting CPython 3.2 and earlier (including all 2.x -versions), there is an extra requirement that (c) the wheel be -built against a version of CPython compiled with 4-byte unicode -support (i.e. one where ``sys.maxunicode > 0xFFFF``). +package manager's provided versions of these libraries. Because CentOS 5 is only available for x86_64 and i686 architectures, these are the only architectures currently supported by the ``manylinux1`` @@ -216,6 +211,81 @@ the list above to remove libraries that have turned out to be problematic or add libraries that have turned out to be safe. +libpythonX.Y.so.1 +----------------- + +Note that ``libpythonX.Y.so.1`` is *not* on the list of libraries that +a ``manylinux1`` extension is allowed to link to. Explicitly linking +to ``libpythonX.Y.so.1`` is unnecessary in almost all cases: the way +ELF linking works, extension modules that are loaded into the +interpreter automatically get access to all of the interpreter's +symbols, regardless of whether or not the extension itself is +explicitly linked against libpython. Furthermore, explicit linking to +libpython creates problems in the common configuration where Python is +not built with ``--enable-shared``. In particular, on Debian and +Ubuntu systems, ``apt install pythonX.Y`` does not even install +``libpythonX.Y.so.1``, meaning that any wheel that *did* depend on +``libpythonX.Y.so.1`` could fail to import. + +There is one situation where extensions that are linked in this way +can fail to work: if a host program (e.g., ``apache2``) uses +``dlopen()`` to load a module (e.g., ``mod_wsgi``) that embeds the +CPython interpreter, and the host program does *not* pass the +``RTLD_GLOBAL`` flag to ``dlopen()``, then the embedded CPython will +be unable to load any extension modules that do not themselves link +explicitly to ``libpythonX.Y.so.1``. Fortunately, ``apache2`` *does* +set the ``RTLD_GLOBAL`` flag, as do all the other programs that +embed-CPython-via-a-dlopened-plugin that we could locate, so this does +not seem to be a serious problem in practice. The incompatibility with +Debian/Ubuntu is more of an issue than the theoretical incompatibility +with a rather obscure corner case. + +This is a rather complex and subtle issue that extends beyond +the scope of ``manylinux1``; for more discussion see: [9]_, [10]_, +[11]_. + +UCS-2 vs UCS-4 builds +--------------------- + +All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can be +built in two ABI-incompatible modes: builds using the +``--enable-unicode=ucs2`` configure flag store Unicode data in UCS-2 +(or really UTF-16) format, while builds using the +``--enable-unicode=ucs4`` configure flag store Unicode data in +UCS-4. (CPython 3.3 and greater use a different storage method that +always supports UCS-4.) If we want to make sure ``ucs2`` wheels don't +get installed into ``ucs4`` CPythons and vice-versa, then something +must be done. + +An earlier version of this PEP included a requirement that +``manylinux1`` wheels targeting these older CPython versions should +always use the ``ucs4`` ABI. But then, in between the PEP's initial +acceptance and its implementation, ``pip`` and ``wheel`` gained +first-class support for tracking and checking this aspect of ABI +compatibility for the relevant CPython versions, which is a better +solution. So we now allow the ``manylinux1`` platform tags to be used +in combination with any ABI tag. However, to maintain compatibility it +is crucial to ensure that all ``manylinux1`` wheels include a +non-trivial abi tag. For example, a wheel built against a ``ucs4`` +CPython might have a name like:: + + PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl + ^^^^^^ Good! + +While a wheel built against the ``ucs2`` ABI might have a name like:: + + PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl + ^^^^^ Okay! + +But you should never have a wheel with a name like:: + + PKG-VERSION-cp27-none-manylinux1_x86_64.whl + ^^^^ BAD! Don't do this! + +We note for information that the ``ucs4`` ABI appears to be much more +widespread among Linux CPython distributors. + + Compilation of Compliant Wheels =============================== @@ -237,7 +307,7 @@ Docker Image The first tool is a Docker image based on CentOS 5.11, which is recommended as an easy to use self-contained build box for compiling ``manylinux1`` wheels -[9]_. Compiling on a more recently-released linux distribution will generally +[12]_. Compiling on a more recently-released linux distribution will generally introduce dependencies on too-new versioned symbols. The image comes with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. @@ -245,7 +315,7 @@ well as the latest releases of Python and ``pip``. Auditwheel ---------- -The second tool is a command line executable called ``auditwheel`` [10]_ that +The second tool is a command line executable called ``auditwheel`` [13]_ that may aid in package maintainers in dealing with third-party external dependencies. @@ -285,7 +355,7 @@ While we acknowledge many approaches for dealing with third-party library dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1`` policy encourages bundling external dependencies, a practice which runs counter to the package management policies of many linux -distributions' system package managers [11]_, [12]_. The primary purpose of +distributions' system package managers [14]_, [15]_. The primary purpose of this is cross-distro compatibility. Furthermore, ``manylinux1`` wheels on PyPI occupy a different niche than the Python packages available through the system package manager. @@ -352,8 +422,6 @@ think otherwise. We know of four main sources of potential incompatibility that are likely to arise in practice: -* "Narrow" unicode builds of Python 3.2 and earlier (including all - versions of Python 2) * Eventually, in the future, there may exist distributions that break compatibility with this profile (e.g., if one of the libraries in the profile changes its ABI in a backwards-incompatible way) @@ -361,8 +429,7 @@ likely to arise in practice: * A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is based on musl ``libc``, or Android) -Checking for unicode configuration compatibility is straightforward, -but the other cases are more subtle. We propose a two-pronged +To address these we propose a two-pronged approach. To handle potential future incompatibilities, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with ``manylinux1``: @@ -389,11 +456,6 @@ Specifically, the algorithm we propose is:: if get_platform() not in ["linux-x86_64", "linux-i686"]: return False - # "wide" Unicode mode is mandatory (always true on CPython 3.3+) - import sys - if sys.maxunicode <= 0xFFFF: - return False - # Check for presence of _manylinux module try: import _manylinux @@ -522,13 +584,21 @@ References (https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U) .. [8] distutils-sig discussion (https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html) -.. [9] manylinux1 docker image - (https://quay.io/repository/manylinux/manylinux) -.. [10] auditwheel tool +.. [9] distutils-sig discussion + (https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html) +.. [10] github issue discussion + (https://github.com/pypa/manylinux/issues/30) +.. [11] python bug tracker discussion + (https://bugs.python.org/issue21536) +.. [12] manylinux1 docker images + (Source: https://github.com/pypa/manylinux; + x86-64: https://quay.io/repository/pypa/manylinux1_x86_64; + x86-32: https://quay.io/repository/pypa/manylinux1_i686) +.. [13] auditwheel tool (https://pypi.python.org/pypi/auditwheel) -.. [11] Fedora Bundled Software Policy +.. [14] Fedora Bundled Software Policy (https://fedoraproject.org/wiki/Bundled_Software_policy) -.. [12] Debian Policy Manual -- 4.13: Convenience copies of code +.. [15] Debian Policy Manual -- 4.13: Convenience copies of code (https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)