python-peps/pep-0599.rst

PEP: 599
Title: The manylinux2014 Platform Tag
Version: $Revision$
Last-Modified: $Date$
Author: Dustin Ingram <di@python.org>
Sponsor: Paul Moore <p.f.moore@gmail.com>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/
Status: Superseded
Type: Informational
Content-Type: text/x-rst
Created: 29-Apr-2019
Post-History: 29-April-2019
Superseded-By: 600
Resolution: https://discuss.python.org/t/the-next-manylinux-specification/1043/199


Abstract
========

This PEP proposes the creation of a ``manylinux2014`` platform tag to
succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also
proposes that PyPI and ``pip`` both be updated to support uploading,
downloading, and installing ``manylinux2014`` distributions on
compatible platforms.

Rationale
=========

CentOS 6 is now the oldest supported CentOS release, and will receive
maintenance updates through November 30th, 2020, [1]_ at which point
it will reach end-of-life, and no further updates such as security
patches will be made available. All wheels built under the
``manylinux2010`` images will remain at obsolete versions after that
point.

Therefore, we propose the continuation of the existing manylinux
standard, and that a new PEP 425-style [2]_ platform tag called
``manylinux2014`` be derived from CentOS 7 and that the ``manylinux``
toolchain, PyPI, and ``pip`` be updated to support it.

Similar to how :pep:`571` and :pep:`513` drew allowed shared
libraries and their symbol versions from CentOS 5.11 and CentOS 6,
respectively, a ``manylinux2014`` platform tag will draw its libraries
and symbol versions from CentOS 7, which will reach end-of-life on
June 30th, 2024. [1]_

The ``manylinuxYYYY`` pattern has a number of advantages that motivate
continuing with the current status quo:

- Well-defined Docker images with clearly specified compatible
  libraries;
- No need to survey for compatibility issues across multiple releases;
- A single build image and ``auditwheel`` profile per architecture.

There are also some disadvantages:

- Requires drafting a new PEP for every new standard;
- Requires adding the new platform tag to installers (e.g., ``pip``);
- Installers are unable to install a platform tag which predates a
  given release.

There are also challenges which would exist for any proposal,
including the time and effort it takes to define, prepare and release
the Docker images and corresponding ``auditwheel`` profiles. These
challenges were experienced in the long rollout period for
``manylinux2010``, which took approximately 1 year from PEP acceptance
to compatible build environment published. [3]_

However, if this PEP can be an indicator, the process is now
well-defined and easily repeatable, which should increase the timeline
for rollout of a newer, updated platform tag.

The ``manylinux2014`` policy
============================

The following criteria determine a ``linux`` wheel's eligibility for
the ``manylinux2014`` tag:

1. The wheel may only contain binary executables and shared objects
   compiled for one of the following architectures supported by CentOS
   7, or a CentOS 7 compatible base image (such as ubi7): [4]_ ::

       x86_64
       i686
       aarch64
       armv7l
       ppc64
       ppc64le
       s390x

   This list adds support for ARMv7 (armv7l), ARMv8 (aarch64) and PowerPC
   (ppc64, ppc64le) architectures supported by the CentOS Alternative
   Architecture Special Interest Group, as well as the IBM Z (s390x)
   architecture. [5]_

2. The wheel's binary executables or shared objects may not link
   against externally-provided libraries except those in the following
   list: ::

       libgcc_s.so.1
       libstdc++.so.6
       libm.so.6
       libdl.so.2
       librt.so.1
       libc.so.6
       libnsl.so.1
       libutil.so.1
       libpthread.so.0
       libresolv.so.2
       libX11.so.6
       libXext.so.6
       libXrender.so.1
       libICE.so.6
       libSM.so.6
       libGL.so.1
       libgobject-2.0.so.0
       libgthread-2.0.so.0
       libglib-2.0.so.0

   This list is identical to the externally-provided libraries
   originally allowed for ``manylinux2010``, with one exception:
   ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30.
   ``libpythonX.Y`` remains ineligible for inclusion for the same
   reasons outlined in :pep:`513`.

   On Debian-based systems, these libraries are provided by the
   packages:

   ============  =======================================================
   Package       Libraries
   ============  =======================================================
   libc6         libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
                 libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
   libgcc1       libgcc_s.so.1
   libgl1        libGL.so.1
   libglib2.0-0  libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
   libice6       libICE.so.6
   libsm6        libSM.so.6
   libstdc++6    libstdc++.so.6
   libx11-6      libX11.so.6
   libxext6      libXext.so.6
   libxrender1   libXrender.so.1
   ============  =======================================================

   On RPM-based systems, they are provided by these packages:

   ============  =======================================================
   Package       Libraries
   ============  =======================================================
   glib2         libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
   glibc         libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,
                 libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
   libICE        libICE.so.6
   libX11        libX11.so.6
   libXext:      libXext.so.6
   libXrender    libXrender.so.1
   libgcc:       libgcc_s.so.1
   libstdc++     libstdc++.so.6
   mesa          libGL.so.1
   ============  =======================================================

3. If the wheel contains binary executables or shared objects linked
   against any allowed libraries that also export versioned symbols,
   they may only depend on the following maximum versions::

       GLIBC_2.17
       CXXABI_1.3.7, CXXABI_TM_1 is also allowed
       GLIBCXX_3.4.19
       GCC_4.8.0

   As an example, ``manylinux2014`` wheels may include binary
   artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``,
   because this an earlier version than the maximum of ``GLIBC_2.17``.
4. If a wheel is built for any version of CPython 2 or CPython
   versions 3.0 up to and including 3.2, it *must* include a CPython
   ABI tag indicating its Unicode ABI.  A ``manylinux2014`` wheel
   built against Python 2, then, must include either the ``cpy27mu``
   tag indicating it was built against an interpreter with the UCS-4
   ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2
   ABI. [6]_ [7]_
5. A wheel *must not* require the ``PyFPE_jbuf`` symbol.  This is
   achieved by building it against a Python compiled *without* the
   ``--with-fpectl`` ``configure`` flag.

Compilation of Compliant Wheels
===============================

Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014``
platform tags to ``linux`` wheels built by ``pip wheel`` or
``bdist_wheel`` in a ``manylinux2014`` Docker container.

Docker Images
-------------

A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be
provided for building binary ``linux`` wheels that can reliably be
converted to ``manylinux2014`` wheels. This image will come with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran``
4.8.5) as well as the latest releases of Python and ``pip``.

Auditwheel
----------

The ``auditwheel`` tool will also be updated to produce
``manylinux2014`` wheels. [8]_ Its behavior and purpose will be
otherwise unchanged from :pep:`513`.

Platform Detection for Installers
=================================

Platforms may define a ``manylinux2014_compatible`` boolean attribute
on the ``_manylinux`` module described in :pep:`513`. A platform is
considered incompatible with ``manylinux2014`` if the attribute is
``False``.

If the ``_manylinux`` module is not found, or it does not have the
attribute ``manylinux2014_compatible``, tools may fall back to
checking for glibc. If the platform has glibc 2.17 or newer, it is
assumed to be compatible unless the ``_manylinux`` module says
otherwise.

Specifically, the algorithm we propose is::

    def is_manylinux2014_compatible():
        # Only Linux, and only supported architectures
        from distutils.util import get_platform

        if get_platform() not in [
            "linux-x86_64",
            "linux-i686",
            "linux-aarch64",
            "linux-armv7l",
            "linux-ppc64",
            "linux-ppc64le",
            "linux-s390x",
        ]:
            return False

        # Check for presence of _manylinux module
        try:
            import _manylinux

            return bool(_manylinux.manylinux2014_compatible)
        except (ImportError, AttributeError):
            # Fall through to heuristic check below
            pass

        # Check glibc version. CentOS 7 uses glibc 2.17.
        # PEP 513 contains an implementation of this function.
        return have_compatible_glibc(2, 17)

Backwards compatibility with ``manylinux2010`` wheels
=====================================================

As explained in :pep:`513`, the specified symbol versions for
``manylinux1`` allowed libraries constitute an *upper bound*.  The
same is true for the symbol versions defined for ``manylinux2014`` in
this PEP.  As a result, ``manylinux1`` and ``manylinux2010`` wheels
are considered ``manylinux2014`` wheels.  A ``pip`` that recognizes
the ``manylinux2014`` platform tag will thus install ``manylinux2010``
wheels for ``manylinux2014`` platforms -- even when explicitly set --
when no ``manylinux2014`` wheels are available.

PyPI Support
============

PyPI should permit wheels containing the ``manylinux2014`` platform
tag to be uploaded in the same way that it permits ``manylinux2010``.

If technically feasible, PyPI should attempt to verify the
compatibility of ``manylinux2014`` wheels, but that capability is not
a requirement for adoption of this PEP.

Package authors should not upload non-compliant ``manylinux2014``
wheels to PyPI, and should be aware that PyPI may begin blocking
non-compliant wheels from being uploaded.

References
==========

.. [1] CentOS Product Specifications
   (https://wiki.centos.org/About/Product)
.. [2] PEP 425: Compatibility Tags for Built Distributions
   (https://www.python.org/dev/peps/pep-0425/)
.. [3] Tracking issue for manylinux2010 rollout
   (https://github.com/pypa/manylinux/issues/179)
.. [4] Red Hat Universal Base Image 7
   (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)
.. [5] The CentOS Alternative Architecture Special Interest Group
   (https://wiki.centos.org/SpecialInterestGroup/AltArch)
.. [6] PEP 3149: ABI version tagged .so files
   (https://www.python.org/dev/peps/pep-3149/)
.. [7] SOABI support for Python 2.X and PyPy
   (https://github.com/pypa/pip/pull/3075)
.. [8] auditwheel
   (https://github.com/pypa/auditwheel/)

Acceptance
==========

PEP 599 was `accepted by Paul Moore on July 31, 2019
<https://discuss.python.org/t/the-next-manylinux-specification/1043/199>`_.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: