diff --git a/pep-0566.rst b/pep-0566.rst index 74924e042..a3aa46e39 100644 --- a/pep-0566.rst +++ b/pep-0566.rst @@ -2,7 +2,7 @@ PEP: 566 Title: Metadata for Python Software Packages 2.1 Version: $Revision$ Last-Modified: $Date$ -Author: Dustin Ingram +Author: Dustin Ingram BDFL-Delegate: Daniel Holth Discussions-To: distutils-sig Status: Final diff --git a/pep-0599.rst b/pep-0599.rst new file mode 100644 index 000000000..59d83cc6d --- /dev/null +++ b/pep-0599.rst @@ -0,0 +1,316 @@ +PEP: 599 +Title: The manylinux2014 Platform Tag +Version: $Revision$ +Last-Modified: $Date$ +Author: Dustin Ingram +Sponsor: Paul Moore +BDFL-Delegate: Paul Moore +Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 29-April-2019 +Post-History: 29-April-2019 + + +Abstract +======== + +This PEP proposes the creation of a ``manylinux2014`` platform tag to +succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also +proposes that PyPI and ``pip`` both be updated to support uploading, +downloading, and installing ``manylinux2014`` distributions on +compatible platforms. + +Rationale +========= + +CentOS 6 is now the oldest supported CentOS release, and will receive +maintenance updates through November 30th, 2020, [1]_ at which point +it will reach end-of-life, and no further updates such as security +patches will be made available. All wheels built under the +``manylinux2010`` images will remain at obsolete versions after that +point. + +Therefore, we propose the continuation of the existing manylinux +standard, and that a new PEP 425-style [2]_ platform tag called +``manylinux2014`` be derived from CentOS 7 and that the ``manylinux`` +toolchain, PyPI, and ``pip`` be updated to support it. + +Similar to how :pep:`571` and :pep:`513` drew allowed shared +libraries and their symbol versions from CentOS 5.11 and CentOS 6, +respectively, a ``manylinux2014`` platform tag will draw its libraries +and symbol versions from CentOS 7, which will reach end-of-life on +June 30th, 2024. [1]_ + +The ``manylinuxYYYY`` pattern has a number of advantages that motivate +continuing with the current status quo: + +- Well-defined Docker images with clearly specified compatible + libraries; +- No need to survey for compatibility issues across multiple releases; +- A single build image and ``auditwheel`` profile per architecture. + +There are also some disadvantages: + +- Requires drafting a new PEP for every new standard; +- Requires adding the new platform tag to installers (e.g., ``pip``); +- Installers are unable to install a platform tag which predates a + given release. + +There are also challenges which would exist for any proposal, +including the time and effort it takes to define, prepare and release +the Docker images and corresponding ``auditwheel`` profiles. These +challenges were experienced in the long rollout period for +``manylinux2010``, which took approximately 1 year from PEP acceptance +to compatible build environment published. [3]_ + +However, if this PEP can be an indicator, the process is now +well-defined and easily repeatable, which should increase the timeline +for rollout of a newer, updated platform tag. + +The ``manylinux2014`` policy +============================ + +The following criteria determine a ``linux`` wheel's eligibility for +the ``manylinux2014`` tag: + +1. The wheel may only contain binary executables and shared objects + compiled for one of the following architectures supported by CentOS + 7, or a CentOS 7 compatible base image (such as ubi7): [4]_ :: + + x86_64 + i686 + aarch64 + armhfp + ppc64 + ppc64le + s390x + + This list adds support for ARM (aarch64, armhfp) and PowerPC + (ppc64, ppc64le) architectures supported by the CentOS Alternative + Architecture Special Interest Group, as well as the IBM Z (s390x) + architecture. [5]_ + +2. The wheel's binary executables or shared objects may not link + against externally-provided libraries except those in the following + list: :: + + libgcc_s.so.1 + libstdc++.so.6 + libm.so.6 + libdl.so.2 + librt.so.1 + libc.so.6 + libnsl.so.1 + libutil.so.1 + libpthread.so.0 + libresolv.so.2 + libX11.so.6 + libXext.so.6 + libXrender.so.1 + libICE.so.6 + libSM.so.6 + libGL.so.1 + libgobject-2.0.so.0 + libgthread-2.0.so.0 + libglib-2.0.so.0 + + This list is identical to the externally-provided libraries + originally allowed for ``manylinux2010``, with one exception: + ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30. + ``libpythonX.Y`` remains ineligible for inclusion for the same + reasons outlined in :pep:`513`. + + On Debian-based systems, these libraries are provided by the + packages: + + ============ ======================================================= + Package Libraries + ============ ======================================================= + libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, + libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1 + libgcc1 libgcc_s.so.1 + libgl1 libGL.so.1 + libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 + libice6 libICE.so.6 + libsm6 libSM.so.6 + libstdc++6 libstdc++.so.6 + libx11-6 libX11.so.6 + libxext6 libXext.so.6 + libxrender1 libXrender.so.1 + ============ ======================================================= + + On RPM-based systems, they are provided by these packages: + + ============ ======================================================= + Package Libraries + ============ ======================================================= + glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 + glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, + libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 + libICE libICE.so.6 + libX11 libX11.so.6 + libXext: libXext.so.6 + libXrender libXrender.so.1 + libgcc: libgcc_s.so.1 + libstdc++ libstdc++.so.6 + mesa libGL.so.1 + ============ ======================================================= + +3. If the wheel contains binary executables or shared objects linked + against any allowed libraries that also export versioned symbols, + they may only depend on the following maximum versions:: + + GLIBC_2.17 + CXXABI_1.3.7 + GLIBCXX_3.4.19 + GCC_4.8.5 + + As an example, ``manylinux2014`` wheels may include binary + artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``, + because this an earlier version than the maximum of ``GLIBC_2.17``. +4. If a wheel is built for any version of CPython 2 or CPython + versions 3.0 up to and including 3.2, it *must* include a CPython + ABI tag indicating its Unicode ABI. A ``manylinux2014`` wheel + built against Python 2, then, must include either the ``cpy27mu`` + tag indicating it was built against an interpreter with the UCS-4 + ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2 + ABI. [6]_ [7]_ +5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is + achieved by building it against a Python compiled *without* the + ``--with-fpectl`` ``configure`` flag. + +Compilation of Compliant Wheels +=============================== + +Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014`` +platform tags to ``linux`` wheels built by ``pip wheel`` or +``bdist_wheel`` in a ``manylinux2014`` Docker container. + +Docker Images +------------- + +A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be +provided for building binary ``linux`` wheels that can reliably be +converted to ``manylinux2014`` wheels. This image will come with a +full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` +4.8.5) as well as the latest releases of Python and ``pip``. + +Compatibility with kernels that lack ``vsyscall`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Similar to CentOS 6, CentOS 7 includes a version of ``glibc`` that +depends on the ``vsyscall`` page. As described in :pep:`571`, this +breaks the assumption that a Docker containers userland is compatible +wtih its host's kernel. Similar to ``manylinux2010``, Docker images +for ``manylinux2014`` require patching ``glibc`` to remove all +dependencies on ``vsyscall`` in the version of ``glibc`` included with +our Docker image. + +Auditwheel +---------- + +The ``auditwheel`` tool will also be updated to produce +``manylinux2014`` wheels. [8]_ Its behavior and purpose will be +otherwise unchanged from :pep:`513`. + +Platform Detection for Installers +================================= + +Platforms may define a ``manylinux2014_compatible`` boolean attribute +on the ``_manylinux`` module described in :pep:`513`. A platform is +considered incompatible with ``manylinux2014`` if the attribute is +``False``. + +If the ``_manylinux`` module is not found, or it does not have the +attribute ``manylinux2014_compatible``, tools may fall back to +checking for glibc. If the platform has glibc 2.17 or newer, it is +assumed to be compatible unless the ``_manylinux`` module says +otherwise. + +Specifically, the algorithm we propose is:: + + def is_manylinux2014_compatible(): + # Only Linux, and only supported architectures + from distutils.util import get_platform + + if get_platform() not in [ + "linux-x86_64", + "linux-i686", + "linux-aarch64", + "linux-armhfp", + "linux-ppc64", + "linux-ppc64le", + "linux-s390x", + ]: + return False + + # Check for presence of _manylinux module + try: + import _manylinux + + return bool(_manylinux.manylinux2014_compatible) + except (ImportError, AttributeError): + # Fall through to heuristic check below + pass + + # Check glibc version. CentOS 7 uses glibc 2.17. + # PEP 513 contains an implementation of this function. + return have_compatible_glibc(2, 17) + +Backwards compatibility with ``manylinux2010`` wheels +===================================================== + +As explained in :pep:`513`, the specified symbol versions for +``manylinux1`` allowed libraries constitute an *upper bound*. The +same is true for the symbol versions defined for ``manylinux2014`` in +this PEP. As a result, ``manylinux1`` and ``manylinux2010`` wheels +are considered ``manylinux2014`` wheels. A ``pip`` that recognizes +the ``manylinux2014`` platform tag will thus install ``manylinux2010`` +wheels for ``manylinux2014`` platforms -- even when explicitly set -- +when no ``manylinux2014`` wheels are available. + +PyPI Support +============ + +PyPI should permit wheels containing the ``manylinux2014`` platform +tag to be uploaded in the same way that it permits ``manylinux2010``. +It should not attempt to verify the compatibility of ``manylinux2014`` +wheels. + +References +========== + +.. [1] CentOS Product Specifications + (https://wiki.centos.org/About/Product) +.. [2] PEP 425 -- Compatibility Tags for Built Distributions + (https://www.python.org/dev/peps/pep-0425/) +.. [3] Tracking issue for manylinux2010 rollout + (https://github.com/pypa/manylinux/issues/179) +.. [4] Red Hat Universal Base Image 7 + (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7) + (https://wiki.centos.org/SpecialInterestGroup/AltArch) +.. [5] The CentOS Alternative Architecture Special Interest Group + (https://wiki.centos.org/SpecialInterestGroup/AltArch) +.. [6] PEP 3149 + https://www.python.org/dev/peps/pep-3149/ +.. [7] SOABI support for Python 2.X and PyPy + https://github.com/pypa/pip/pull/3075 +.. [8] auditwheel + (https://github.com/pypa/auditwheel/) + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: