python-peps/pep-0600.rst

PEP: 600
Title: Future 'manylinux' Platform Tags for Portable Linux Built Distributions
Version: $Revision$
Last-Modified: $Date$
Author: Nathaniel J. Smith <njs@pobox.com>
        Thomas Kluyver <thomas@kluyver.me.uk>
Sponsor: Paul Moore <p.f.moore@gmail.com>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: Discourse https://discuss.python.org/t/the-next-manylinux-specification/1043
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 3-May-2019
Post-History: 3-May-2019

Abstract
========

This PEP proposes a scheme for new 'manylinux' distribution tags to be defined
without requiring a PEP for every specific tag. The naming scheme is based on
glibc versions, with profiles in the auditwheel tool defining what other
external libraries and symbols a compatible wheel may link against.

While there is interest in defining tags for non-glibc Linux platforms,
this PEP does not attempt to address that.

Rationale
=========

Distributing compiled code for Linux is more complicated than for other popular
operating systems, because the Linux kernel and the libraries which typically
accompany it are built in different configurations and combinations for different
distributions. However, there are certain core libraries which can be expected in
many common distributions and are reliably backwards compatible, so binaries
built with older versions of these libraries will work with newer versions.
:pep:`513` describes these ideas in much more detail.

The ``manylinux1`` (:pep:`513`) and ``manylinux2010`` (:pep:`571`) tags make
use of these features. They define a set of core libraries and symbol versions
which wheels may expect on the system, based on CentOS 5 and 6 respectively.
Typically, packages are built in Docker containers based on these old CentOS
versions, and then the ``auditwheel`` tool is used to check them and bundle any
other linked libraries into the wheel.

If we were to define a ``manylinux2014`` tag based on CentOS 7, there would be
five steps involved to make it practically useful:

1. Write a PEP
2. Prepare docker images based on CentOS 7.
3. Add the definition to auditwheel
4. Allow uploads with the new tag on PyPI
5. Add code to pip to recognise the new tag and check if the platform is
   compatible

Although preparing the docker images and updating auditwheel take more work,
these parts can be used as soon as that work is complete. The changes to pip
are more straightforward, but not all users will promptly install a new version
of pip, so distributors are concerned about moving to a new tag too quickly.

This PEP aims to remove the need for steps 1 and 5 above, so new manylinux tags
can be adopted more easily.

Naming
======

Tags using the new scheme will look like::

    manylinux_glibc_2_17_x86_64

Where ``2_17`` is the major and minor version of glibc. I.e. for this example,
the platform must have glibc 2.17 or newer. Installer tools should be prepared
to handle any numeric values here, but building and publishing wheels to PyPI
will probably be constrained to specific profiles defined by auditwheel.

The existing manylinux tags can also be represented in the new scheme,
for instance:

- ``manylinux1_x86_64`` becomes ``manylinux_glibc_2_5_x86_64``
- ``manylinux2010_x86_64`` becomes ``manylinux_glibc_2_12_x86_64``

``x86_64`` refers to the CPU architecture, as in previous tags.

While this PEP does not attempt to define tags for non-glibc Linux, the name
glibc is included to leave room for future efforts in that direction.

Wheel compatibility
===================

There are two components to a tag definition: a specification of what makes a
compatible wheel, and of what makes a compatible platform.

A wheel may never use symbols from a newer version of glibc than that indicated
by its tag. Likewise, a wheel with a glibc tag under this scheme may not be
linked against another libc implementation.

As with the previous manylinux tags, wheels will be allowed to link against
a limited set of external libraries and symbols. These will be defined by
profiles documented on https://packaging.python.org/ and implemented in
auditwheel. At least initially, they will likely be similar to
the list for manylinux2010 (:pep:`571`), and based on library versions in
newer versions of CentOS.

The overall goal is to ensure that if a wheel is tagged as
``manylinux_glibc_2_Y``, then users can be reasonably confident that this wheel
will work in any real-world linux-based python environment that uses
``glibc 2.Y`` or later and matches the other wheel compatibility tags.
For example, this includes making sure that the wheel only uses symbols that
are available in the oldest supported glibc, and doesn't rely on the system to
provide any libraries that aren't universally available.

One of the central points of this PEP is to move away from defining each
compatibility profile in its own PEP.
In part, this is to acknowledge that the details of compatibility profiles
evolve over time as the Linux distribution landscape changes and as we learn
more about real-world compatibility pitfalls.
For instance, Fedora 30 `removed <https://github.com/pypa/manylinux/issues/305>`__
``libcrypt.so.1``, which both ``manylinux1`` and ``manylinux2010`` previously
allowed wheels to externally link.
Auditwheel and the manylinux build images will be updated to avoid new wheels
relying on this as an external library.

As with the previous manylinux tags, required libraries which are not on
the whitelist will need to be bundled into the wheel.

Building compatible wheels
--------------------------

For each profile defined on https://packaging.python.org/, we plan to provide
a canonical build environment, such as a Docker image, available for people to
build wheels for that profile.
People can build in other environments, so long as the
resulting wheels can be verified by auditwheel, but the canonical environments
hopefully provide an easy answer for most packages.

The definition of a new profile may well precede the construction of its
build environment; it's not expected that the definition in auditwheel
is held up until a corresponding environment is ready to use.

Verification on upload
----------------------

In the future, PyPI may begin using auditwheel to automatically validate
uploaded manylinux wheels, and reject wheels that it can't determine are
compliant. If PyPI does this, then it will mean that only wheels that have a
corresponding auditwheel profile can be distributed publicly.

If you need manylinux support for a platform that currently has no profile
in auditwheel, then you're encouraged to contribute a profile to auditwheel.
If that's not possible for some reason, then other tools can be used,
as long as you try to meet the same goal as auditwheel (i.e., the wheel should
work in all environments with the given glibc version and architecture) –
though you may not be able to upload these wheels to PyPI.

Platform compatibility
======================

The checks for a compatible platform on installation consist of a heuristic
and an optional override. The heuristic is that the platform is compatible if
and only if it has a version of glibc equal to or greater than that indicated
in the tag name.

The override is defined in an importable ``_manylinux`` module,
the same as already used for manylinux1 and manylinux2010 overrides.
For the new scheme, this module must define a function rather than an
attribute. ``manylinux_glibc_compatible(major, minor)`` takes two integers
for the glibc version number in the tag, and returns True, False or None.
If it is not defined or it returns None, the default heuristic is used.

The compatibility check could be implemented like this::

    def is_manylinux_glibc_compatible(major, minor):
        # Check for presence of _manylinux module
        try:
            import _manylinux
            f = _manylinux.manylinux_glibc_compatible
        except (ImportError, AttributeError):
            # Fall through to heuristic check below
            pass
        else:
            compat = f(major, minor)
            if compat is not None:
                return bool(compat)

        # Check glibc version.
        # PEP 513 contains an implementation of this function.
        return have_compatible_glibc(major, minor)

The installer should also check that the platform is Linux and that the
architecture in the tag matches that of the running interpreter.
These checks are not illustrated here.

Next steps
==========

There is pressure to create a new manylinux tag to supersede manylinux2010.
If this PEP is accepted, a priority would be to define and support a
``manylinux_glibc_2_17`` profile, based on CentOS 7, equivalent to manylinux2014
in the previous numbering system.
This would be essentially the same process as defining manylinux2014, and
could build upon the research already started for this.

Beyond that, pip would need changes to allow it to recognise any
``manylinux_glibc_2_*`` tag and check platform compatibility, rather than
handling only a whitelist of specific tags.

Rejected alternatives
=====================

Early versions of this proposed specifying only that wheels using a given tag
must work on all mainstream platforms with glibc at or above the version
which the tag referred to. This would have left the library and symbol
whitelists as implementation details of auditwheel. This was felt to be
insufficient, so the proposal now includes an intent to document specific
compatibility profiles as well as defining them in auditwheel.
In keeping with the goal of reducing the work involved in defining a new tag,
it does not attempt to prescribe exactly how to create and update profiles:
the ways to do this should evolve with experience.