PEP: 599 Title: The manylinux2014 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Dustin Ingram Sponsor: Paul Moore BDFL-Delegate: Paul Moore Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ Status: Accepted Type: Informational Content-Type: text/x-rst Created: 29-April-2019 Post-History: 29-April-2019 Resolution: https://discuss.python.org/t/the-next-manylinux-specification/1043/199 Abstract ======== This PEP proposes the creation of a ``manylinux2014`` platform tag to succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2014`` distributions on compatible platforms. Rationale ========= CentOS 6 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020, [1]_ at which point it will reach end-of-life, and no further updates such as security patches will be made available. All wheels built under the ``manylinux2010`` images will remain at obsolete versions after that point. Therefore, we propose the continuation of the existing manylinux standard, and that a new PEP 425-style [2]_ platform tag called ``manylinux2014`` be derived from CentOS 7 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. Similar to how :pep:`571` and :pep:`513` drew allowed shared libraries and their symbol versions from CentOS 5.11 and CentOS 6, respectively, a ``manylinux2014`` platform tag will draw its libraries and symbol versions from CentOS 7, which will reach end-of-life on June 30th, 2024. [1]_ The ``manylinuxYYYY`` pattern has a number of advantages that motivate continuing with the current status quo: - Well-defined Docker images with clearly specified compatible libraries; - No need to survey for compatibility issues across multiple releases; - A single build image and ``auditwheel`` profile per architecture. There are also some disadvantages: - Requires drafting a new PEP for every new standard; - Requires adding the new platform tag to installers (e.g., ``pip``); - Installers are unable to install a platform tag which predates a given release. There are also challenges which would exist for any proposal, including the time and effort it takes to define, prepare and release the Docker images and corresponding ``auditwheel`` profiles. These challenges were experienced in the long rollout period for ``manylinux2010``, which took approximately 1 year from PEP acceptance to compatible build environment published. [3]_ However, if this PEP can be an indicator, the process is now well-defined and easily repeatable, which should increase the timeline for rollout of a newer, updated platform tag. The ``manylinux2014`` policy ============================ The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2014`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the following architectures supported by CentOS 7, or a CentOS 7 compatible base image (such as ubi7): [4]_ :: x86_64 i686 aarch64 armhfp ppc64 ppc64le s390x This list adds support for ARM (aarch64, armhfp) and PowerPC (ppc64, ppc64le) architectures supported by the CentOS Alternative Architecture Special Interest Group, as well as the IBM Z (s390x) architecture. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following list: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries originally allowed for ``manylinux2010``, with one exception: ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30. ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in :pep:`513`. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any allowed libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.17 CXXABI_1.3.7 GLIBCXX_3.4.19 GCC_4.8.5 As an example, ``manylinux2014`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``, because this an earlier version than the maximum of ``GLIBC_2.17``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2014`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2 ABI. [6]_ [7]_ 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` in a ``manylinux2014`` Docker container. Docker Images ------------- A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2014`` wheels. This image will come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.5) as well as the latest releases of Python and ``pip``. Auditwheel ---------- The ``auditwheel`` tool will also be updated to produce ``manylinux2014`` wheels. [8]_ Its behavior and purpose will be otherwise unchanged from :pep:`513`. Platform Detection for Installers ================================= Platforms may define a ``manylinux2014_compatible`` boolean attribute on the ``_manylinux`` module described in :pep:`513`. A platform is considered incompatible with ``manylinux2014`` if the attribute is ``False``. If the ``_manylinux`` module is not found, or it does not have the attribute ``manylinux2014_compatible``, tools may fall back to checking for glibc. If the platform has glibc 2.17 or newer, it is assumed to be compatible unless the ``_manylinux`` module says otherwise. Specifically, the algorithm we propose is:: def is_manylinux2014_compatible(): # Only Linux, and only supported architectures from distutils.util import get_platform if get_platform() not in [ "linux-x86_64", "linux-i686", "linux-aarch64", "linux-armhfp", "linux-ppc64", "linux-ppc64le", "linux-s390x", ]: return False # Check for presence of _manylinux module try: import _manylinux return bool(_manylinux.manylinux2014_compatible) except (ImportError, AttributeError): # Fall through to heuristic check below pass # Check glibc version. CentOS 7 uses glibc 2.17. # PEP 513 contains an implementation of this function. return have_compatible_glibc(2, 17) Backwards compatibility with ``manylinux2010`` wheels ===================================================== As explained in :pep:`513`, the specified symbol versions for ``manylinux1`` allowed libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2014`` in this PEP. As a result, ``manylinux1`` and ``manylinux2010`` wheels are considered ``manylinux2014`` wheels. A ``pip`` that recognizes the ``manylinux2014`` platform tag will thus install ``manylinux2010`` wheels for ``manylinux2014`` platforms -- even when explicitly set -- when no ``manylinux2014`` wheels are available. PyPI Support ============ PyPI should permit wheels containing the ``manylinux2014`` platform tag to be uploaded in the same way that it permits ``manylinux2010``. If technically feasible, PyPI should attempt to verify the compatibility of ``manylinux2014`` wheels, but that capability is not a requirement for adoption of this PEP. Package authors should not upload non-compliant ``manylinux2014`` wheels to PyPI, and should be aware that PyPI may begin blocking non-compliant wheels from being uploaded. References ========== .. [1] CentOS Product Specifications (https://wiki.centos.org/About/Product) .. [2] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [3] Tracking issue for manylinux2010 rollout (https://github.com/pypa/manylinux/issues/179) .. [4] Red Hat Universal Base Image 7 (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7) (https://wiki.centos.org/SpecialInterestGroup/AltArch) .. [5] The CentOS Alternative Architecture Special Interest Group (https://wiki.centos.org/SpecialInterestGroup/AltArch) .. [6] PEP 3149 https://www.python.org/dev/peps/pep-3149/ .. [7] SOABI support for Python 2.X and PyPy https://github.com/pypa/pip/pull/3075 .. [8] auditwheel (https://github.com/pypa/auditwheel/) Acceptance ========== PEP 599 was `accepted by Paul Moore on July 31, 2019 `_. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: