PEP 599: manylinux2014 PEP (#1121)
Updated manylinux baseline, to be considered in combination with the (competing/complementary?) perennial manylinux proposal in PEP 600.
This commit is contained in:
Dustin Ingram
Nick Coghlan
parent
455f8e4e11
commit
a5edd3def8
|
|
@ -2,7 +2,7 @@ PEP: 566
|
|||
Title: Metadata for Python Software Packages 2.1
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Dustin Ingram <di@di.codes>
|
||||
Author: Dustin Ingram <di@python.org>
|
||||
BDFL-Delegate: Daniel Holth
|
||||
Discussions-To: distutils-sig <distutils-sig at python.org>
|
||||
Status: Final
|
||||
|
|
|
|||
|
|
@ -0,0 +1,316 @@
|
|||
PEP: 599
|
||||
Title: The manylinux2014 Platform Tag
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Dustin Ingram <di@python.org>
|
||||
Sponsor: Paul Moore <p.f.moore@gmail.com>
|
||||
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
|
||||
Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/
|
||||
Status: Draft
|
||||
Type: Informational
|
||||
Content-Type: text/x-rst
|
||||
Created: 29-April-2019
|
||||
Post-History: 29-April-2019
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes the creation of a ``manylinux2014`` platform tag to
|
||||
succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also
|
||||
proposes that PyPI and ``pip`` both be updated to support uploading,
|
||||
downloading, and installing ``manylinux2014`` distributions on
|
||||
compatible platforms.
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
CentOS 6 is now the oldest supported CentOS release, and will receive
|
||||
maintenance updates through November 30th, 2020, [1]_ at which point
|
||||
it will reach end-of-life, and no further updates such as security
|
||||
patches will be made available. All wheels built under the
|
||||
``manylinux2010`` images will remain at obsolete versions after that
|
||||
point.
|
||||
|
||||
Therefore, we propose the continuation of the existing manylinux
|
||||
standard, and that a new PEP 425-style [2]_ platform tag called
|
||||
``manylinux2014`` be derived from CentOS 7 and that the ``manylinux``
|
||||
toolchain, PyPI, and ``pip`` be updated to support it.
|
||||
|
||||
Similar to how :pep:`571` and :pep:`513` drew allowed shared
|
||||
libraries and their symbol versions from CentOS 5.11 and CentOS 6,
|
||||
respectively, a ``manylinux2014`` platform tag will draw its libraries
|
||||
and symbol versions from CentOS 7, which will reach end-of-life on
|
||||
June 30th, 2024. [1]_
|
||||
|
||||
The ``manylinuxYYYY`` pattern has a number of advantages that motivate
|
||||
continuing with the current status quo:
|
||||
|
||||
- Well-defined Docker images with clearly specified compatible
|
||||
libraries;
|
||||
- No need to survey for compatibility issues across multiple releases;
|
||||
- A single build image and ``auditwheel`` profile per architecture.
|
||||
|
||||
There are also some disadvantages:
|
||||
|
||||
- Requires drafting a new PEP for every new standard;
|
||||
- Requires adding the new platform tag to installers (e.g., ``pip``);
|
||||
- Installers are unable to install a platform tag which predates a
|
||||
given release.
|
||||
|
||||
There are also challenges which would exist for any proposal,
|
||||
including the time and effort it takes to define, prepare and release
|
||||
the Docker images and corresponding ``auditwheel`` profiles. These
|
||||
challenges were experienced in the long rollout period for
|
||||
``manylinux2010``, which took approximately 1 year from PEP acceptance
|
||||
to compatible build environment published. [3]_
|
||||
|
||||
However, if this PEP can be an indicator, the process is now
|
||||
well-defined and easily repeatable, which should increase the timeline
|
||||
for rollout of a newer, updated platform tag.
|
||||
|
||||
The ``manylinux2014`` policy
|
||||
============================
|
||||
|
||||
The following criteria determine a ``linux`` wheel's eligibility for
|
||||
the ``manylinux2014`` tag:
|
||||
|
||||
1. The wheel may only contain binary executables and shared objects
|
||||
compiled for one of the following architectures supported by CentOS
|
||||
7, or a CentOS 7 compatible base image (such as ubi7): [4]_ ::
|
||||
|
||||
x86_64
|
||||
i686
|
||||
aarch64
|
||||
armhfp
|
||||
ppc64
|
||||
ppc64le
|
||||
s390x
|
||||
|
||||
This list adds support for ARM (aarch64, armhfp) and PowerPC
|
||||
(ppc64, ppc64le) architectures supported by the CentOS Alternative
|
||||
Architecture Special Interest Group, as well as the IBM Z (s390x)
|
||||
architecture. [5]_
|
||||
|
||||
2. The wheel's binary executables or shared objects may not link
|
||||
against externally-provided libraries except those in the following
|
||||
list: ::
|
||||
|
||||
libgcc_s.so.1
|
||||
libstdc++.so.6
|
||||
libm.so.6
|
||||
libdl.so.2
|
||||
librt.so.1
|
||||
libc.so.6
|
||||
libnsl.so.1
|
||||
libutil.so.1
|
||||
libpthread.so.0
|
||||
libresolv.so.2
|
||||
libX11.so.6
|
||||
libXext.so.6
|
||||
libXrender.so.1
|
||||
libICE.so.6
|
||||
libSM.so.6
|
||||
libGL.so.1
|
||||
libgobject-2.0.so.0
|
||||
libgthread-2.0.so.0
|
||||
libglib-2.0.so.0
|
||||
|
||||
This list is identical to the externally-provided libraries
|
||||
originally allowed for ``manylinux2010``, with one exception:
|
||||
``libcrypt.so.1`` was removed due to being deprecated in Fedora 30.
|
||||
``libpythonX.Y`` remains ineligible for inclusion for the same
|
||||
reasons outlined in :pep:`513`.
|
||||
|
||||
On Debian-based systems, these libraries are provided by the
|
||||
packages:
|
||||
|
||||
============ =======================================================
|
||||
Package Libraries
|
||||
============ =======================================================
|
||||
libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
|
||||
libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
|
||||
libgcc1 libgcc_s.so.1
|
||||
libgl1 libGL.so.1
|
||||
libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
|
||||
libice6 libICE.so.6
|
||||
libsm6 libSM.so.6
|
||||
libstdc++6 libstdc++.so.6
|
||||
libx11-6 libX11.so.6
|
||||
libxext6 libXext.so.6
|
||||
libxrender1 libXrender.so.1
|
||||
============ =======================================================
|
||||
|
||||
On RPM-based systems, they are provided by these packages:
|
||||
|
||||
============ =======================================================
|
||||
Package Libraries
|
||||
============ =======================================================
|
||||
glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
|
||||
glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,
|
||||
libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
|
||||
libICE libICE.so.6
|
||||
libX11 libX11.so.6
|
||||
libXext: libXext.so.6
|
||||
libXrender libXrender.so.1
|
||||
libgcc: libgcc_s.so.1
|
||||
libstdc++ libstdc++.so.6
|
||||
mesa libGL.so.1
|
||||
============ =======================================================
|
||||
|
||||
3. If the wheel contains binary executables or shared objects linked
|
||||
against any allowed libraries that also export versioned symbols,
|
||||
they may only depend on the following maximum versions::
|
||||
|
||||
GLIBC_2.17
|
||||
CXXABI_1.3.7
|
||||
GLIBCXX_3.4.19
|
||||
GCC_4.8.5
|
||||
|
||||
As an example, ``manylinux2014`` wheels may include binary
|
||||
artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``,
|
||||
because this an earlier version than the maximum of ``GLIBC_2.17``.
|
||||
4. If a wheel is built for any version of CPython 2 or CPython
|
||||
versions 3.0 up to and including 3.2, it *must* include a CPython
|
||||
ABI tag indicating its Unicode ABI. A ``manylinux2014`` wheel
|
||||
built against Python 2, then, must include either the ``cpy27mu``
|
||||
tag indicating it was built against an interpreter with the UCS-4
|
||||
ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2
|
||||
ABI. [6]_ [7]_
|
||||
5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is
|
||||
achieved by building it against a Python compiled *without* the
|
||||
``--with-fpectl`` ``configure`` flag.
|
||||
|
||||
Compilation of Compliant Wheels
|
||||
===============================
|
||||
|
||||
Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014``
|
||||
platform tags to ``linux`` wheels built by ``pip wheel`` or
|
||||
``bdist_wheel`` in a ``manylinux2014`` Docker container.
|
||||
|
||||
Docker Images
|
||||
-------------
|
||||
|
||||
A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be
|
||||
provided for building binary ``linux`` wheels that can reliably be
|
||||
converted to ``manylinux2014`` wheels. This image will come with a
|
||||
full compiler suite installed (``gcc``, ``g++``, and ``gfortran``
|
||||
4.8.5) as well as the latest releases of Python and ``pip``.
|
||||
|
||||
Compatibility with kernels that lack ``vsyscall``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Similar to CentOS 6, CentOS 7 includes a version of ``glibc`` that
|
||||
depends on the ``vsyscall`` page. As described in :pep:`571`, this
|
||||
breaks the assumption that a Docker containers userland is compatible
|
||||
wtih its host's kernel. Similar to ``manylinux2010``, Docker images
|
||||
for ``manylinux2014`` require patching ``glibc`` to remove all
|
||||
dependencies on ``vsyscall`` in the version of ``glibc`` included with
|
||||
our Docker image.
|
||||
|
||||
Auditwheel
|
||||
----------
|
||||
|
||||
The ``auditwheel`` tool will also be updated to produce
|
||||
``manylinux2014`` wheels. [8]_ Its behavior and purpose will be
|
||||
otherwise unchanged from :pep:`513`.
|
||||
|
||||
Platform Detection for Installers
|
||||
=================================
|
||||
|
||||
Platforms may define a ``manylinux2014_compatible`` boolean attribute
|
||||
on the ``_manylinux`` module described in :pep:`513`. A platform is
|
||||
considered incompatible with ``manylinux2014`` if the attribute is
|
||||
``False``.
|
||||
|
||||
If the ``_manylinux`` module is not found, or it does not have the
|
||||
attribute ``manylinux2014_compatible``, tools may fall back to
|
||||
checking for glibc. If the platform has glibc 2.17 or newer, it is
|
||||
assumed to be compatible unless the ``_manylinux`` module says
|
||||
otherwise.
|
||||
|
||||
Specifically, the algorithm we propose is::
|
||||
|
||||
def is_manylinux2014_compatible():
|
||||
# Only Linux, and only supported architectures
|
||||
from distutils.util import get_platform
|
||||
|
||||
if get_platform() not in [
|
||||
"linux-x86_64",
|
||||
"linux-i686",
|
||||
"linux-aarch64",
|
||||
"linux-armhfp",
|
||||
"linux-ppc64",
|
||||
"linux-ppc64le",
|
||||
"linux-s390x",
|
||||
]:
|
||||
return False
|
||||
|
||||
# Check for presence of _manylinux module
|
||||
try:
|
||||
import _manylinux
|
||||
|
||||
return bool(_manylinux.manylinux2014_compatible)
|
||||
except (ImportError, AttributeError):
|
||||
# Fall through to heuristic check below
|
||||
pass
|
||||
|
||||
# Check glibc version. CentOS 7 uses glibc 2.17.
|
||||
# PEP 513 contains an implementation of this function.
|
||||
return have_compatible_glibc(2, 17)
|
||||
|
||||
Backwards compatibility with ``manylinux2010`` wheels
|
||||
=====================================================
|
||||
|
||||
As explained in :pep:`513`, the specified symbol versions for
|
||||
``manylinux1`` allowed libraries constitute an *upper bound*. The
|
||||
same is true for the symbol versions defined for ``manylinux2014`` in
|
||||
this PEP. As a result, ``manylinux1`` and ``manylinux2010`` wheels
|
||||
are considered ``manylinux2014`` wheels. A ``pip`` that recognizes
|
||||
the ``manylinux2014`` platform tag will thus install ``manylinux2010``
|
||||
wheels for ``manylinux2014`` platforms -- even when explicitly set --
|
||||
when no ``manylinux2014`` wheels are available.
|
||||
|
||||
PyPI Support
|
||||
============
|
||||
|
||||
PyPI should permit wheels containing the ``manylinux2014`` platform
|
||||
tag to be uploaded in the same way that it permits ``manylinux2010``.
|
||||
It should not attempt to verify the compatibility of ``manylinux2014``
|
||||
wheels.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] CentOS Product Specifications
|
||||
(https://wiki.centos.org/About/Product)
|
||||
.. [2] PEP 425 -- Compatibility Tags for Built Distributions
|
||||
(https://www.python.org/dev/peps/pep-0425/)
|
||||
.. [3] Tracking issue for manylinux2010 rollout
|
||||
(https://github.com/pypa/manylinux/issues/179)
|
||||
.. [4] Red Hat Universal Base Image 7
|
||||
(https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)
|
||||
(https://wiki.centos.org/SpecialInterestGroup/AltArch)
|
||||
.. [5] The CentOS Alternative Architecture Special Interest Group
|
||||
(https://wiki.centos.org/SpecialInterestGroup/AltArch)
|
||||
.. [6] PEP 3149
|
||||
https://www.python.org/dev/peps/pep-3149/
|
||||
.. [7] SOABI support for Python 2.X and PyPy
|
||||
https://github.com/pypa/pip/pull/3075
|
||||
.. [8] auditwheel
|
||||
(https://github.com/pypa/auditwheel/)
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document is placed in the public domain or under the
|
||||
CC0-1.0-Universal license, whichever is more permissive.
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue