347 lines
14 KiB
Plaintext
347 lines
14 KiB
Plaintext
|
PEP: 513
|
||
|
Title: A Platform Tag for Portable Linux Built Distributions
|
||
|
Version: $Revision$
|
||
|
Last-Modified: $Date$
|
||
|
Author: Robert T. McGibbon <rmcgibbo@gmail.com>, Nathaniel J. Smith <njs@pobox.com>
|
||
|
BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com>
|
||
|
Status: Draft
|
||
|
Type: Informational
|
||
|
Content-Type: text/x-rst
|
||
|
Created: 19-Jan-2016
|
||
|
Post-History: 19-Jan-2016
|
||
|
|
||
|
|
||
|
Abstract
|
||
|
========
|
||
|
|
||
|
This PEP proposes the creation of a new platform tag for Python package built
|
||
|
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
|
||
|
external dependencies limited restricted to a standardized subset of
|
||
|
the Linux kernel and core userspace ABI. It proposes that PyPI support
|
||
|
uploading and distributing Wheels with this platform tag, and that ``pip``
|
||
|
support downloading and installing these packages on compatible platforms.
|
||
|
|
||
|
|
||
|
Rationale
|
||
|
=========
|
||
|
|
||
|
Currently, distribution of binary Python extensions for Windows and OS X is
|
||
|
straightforward. Developers and packagers build wheels, which are assigned
|
||
|
platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
|
||
|
wheels to PyPI. Users can download and install these wheels using tools such
|
||
|
as ``pip``.
|
||
|
|
||
|
For Linux, the situation is much more delicate. In general, compiled Python
|
||
|
extension modules built on one Linux distribution will not work on other Linux
|
||
|
distributions, or even on the same Linux distribution with different system
|
||
|
libraries installed.
|
||
|
|
||
|
Build tools using PEP 425 platform tags [1]_ do not track information about the
|
||
|
particular Linux distribution or installed system libraries, and instead assign
|
||
|
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
|
||
|
this ambiguity, there is no expectation that ``linux``-tagged built
|
||
|
distributions compiled on one machine will work properly on another, and for
|
||
|
this reason, PyPI has not permitted the uploading of wheels for Linux.
|
||
|
|
||
|
It would be ideal if wheel packages could be compiled that would work on *any*
|
||
|
linux system. But, because of the incredible diversity of Linux systems -- from
|
||
|
PCs to Android to embedded systems with custom libcs -- this cannot
|
||
|
be guaranteed in general.
|
||
|
|
||
|
Instead, we define a standard subset of the kernel+core userspace ABI that,
|
||
|
in practice, is compatible enough that packages conforming to this standard
|
||
|
will work on *many* linux systems, including essentially all of the desktop
|
||
|
and server distributions in common use. We know this because there are
|
||
|
companies who have been distributing such widely-portable pre-compiled Python
|
||
|
extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
|
||
|
Analytics with Anaconda [3]_.
|
||
|
|
||
|
Building on the compability lessons learned from these companies, we thus
|
||
|
define a baseline ``manylinux1`` platform tag for use by binary Python
|
||
|
wheels, and introduce the implementation of preliminary tools to aid in the
|
||
|
construction of these ``manylinux1`` wheels.
|
||
|
|
||
|
|
||
|
Key Causes of Inter-Linux Binary Incompatibility
|
||
|
================================================
|
||
|
|
||
|
To properly define a standard that will guarantee that wheel packages meeting
|
||
|
this specification will operate on *many* linux platforms, it is necessary to
|
||
|
understand the root causes which often prevent portability of pre-compiled
|
||
|
binaries on Linux. The two key causes are dependencies on shared libraries
|
||
|
which are not present on users' systems, and dependencies on particular
|
||
|
versions of certain core libraries like ``glibc``.
|
||
|
|
||
|
|
||
|
External Shared Libraries
|
||
|
-------------------------
|
||
|
|
||
|
Most desktop and server linux distributions come with a system package manager
|
||
|
(examples include ``APT`` on Debian-based systems, ``yum`` on
|
||
|
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
|
||
|
responsibilities, the installation of shared libraries installed to system
|
||
|
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
|
||
|
on one or more of these shared libraries, and thus function properly only on
|
||
|
systems where the user has the proper libraries (and the proper
|
||
|
versions thereof), either installed using their package manager, or installed
|
||
|
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
|
||
|
to notify the runtime linker of the location of the depended-upon shared
|
||
|
libraries.
|
||
|
|
||
|
|
||
|
Versioning of Core Shared Libraries
|
||
|
-----------------------------------
|
||
|
|
||
|
Even if author or maintainers of a Python extension module with to use no
|
||
|
external shared libraries, the modules will generally have a dynamic runtime
|
||
|
dependency on the GNU C library, ``glibc``. While it is possible, statically
|
||
|
linking ``glibc`` is usually a bad idea because of bloat, and because certain
|
||
|
important C functions like ``dlopen()`` cannot be called from code that
|
||
|
statically links ``glibc``. A runtime shared library dependency on a
|
||
|
system-provided ``glibc`` is unavoidable in practice.
|
||
|
|
||
|
The maintainers of the GNU C library follow a strict symbol versioning scheme
|
||
|
for backward compatibility. This ensures that binaries compiled against an older
|
||
|
version of ``glibc`` can run on systems that have a newer ``glibc``. The
|
||
|
opposite is generally not true -- binaries compiled on newer Linux
|
||
|
distributions tend to rely upon versioned functions in glibc that are not
|
||
|
available on older systems.
|
||
|
|
||
|
This generally prevents built distributions compiled on the latest Linux
|
||
|
distributions from being portable.
|
||
|
|
||
|
|
||
|
The ``manylinux1`` policy
|
||
|
=========================
|
||
|
|
||
|
For these reasons, to achieve broad portability, Python wheels
|
||
|
|
||
|
* should depend only on an extremely limited set of external shared
|
||
|
libraries; and
|
||
|
* should depend only on ``old`` symbol versions in those external shared
|
||
|
libraries.
|
||
|
|
||
|
The ``manylinux1`` policy thus encompasses a standard for what the
|
||
|
permitted external shared libraries a wheel may depend on, and the maximum
|
||
|
depended-upon symbol versions therein.
|
||
|
|
||
|
The permitted external shared libraries are: ::
|
||
|
|
||
|
libpanelw.so.5
|
||
|
libncursesw.so.5
|
||
|
libgcc_s.so.1
|
||
|
libstdc++.so.6
|
||
|
libm.so.6
|
||
|
libdl.so.2
|
||
|
librt.so.1
|
||
|
libcrypt.so.1
|
||
|
libc.so.6
|
||
|
libnsl.so.1
|
||
|
libutil.so.1
|
||
|
libpthread.so.0
|
||
|
libX11.so.6
|
||
|
libXext.so.6
|
||
|
libXrender.so.1
|
||
|
libICE.so.6
|
||
|
libSM.so.6
|
||
|
libGL.so.1
|
||
|
libgobject-2.0.so.0
|
||
|
libgthread-2.0.so.0
|
||
|
libglib-2.0.so.0
|
||
|
|
||
|
On Debian-based systems, these libraries are provided by the packages ::
|
||
|
|
||
|
libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
|
||
|
libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0
|
||
|
|
||
|
On RPM-based systems, these libraries are provided by the packages ::
|
||
|
|
||
|
ncurses libgcc libstdc++ glibc libXext libXrender
|
||
|
libICE libSM mesa-libGL glib2
|
||
|
|
||
|
This list was compiled by checking the external shared library dependencies of
|
||
|
the Canopy [1]_ and Anaconda [2]_ distributions, which both include a wide array
|
||
|
of the most popular Python modules and have been confirmed in practice to work
|
||
|
across a wide swath of Linux systems in the wild.
|
||
|
|
||
|
For dependencies on externally-provided versioned symbols in the above shared
|
||
|
libraries, the following symbol versions are permitted: ::
|
||
|
|
||
|
GLIBC <= 2.5
|
||
|
CXXABI <= 3.4.8
|
||
|
GLIBCXX <= 3.4.9
|
||
|
GCC <= 4.2.0
|
||
|
|
||
|
These symbol versions were determined by inspecting the latest symbol version
|
||
|
provided in the libraries distributed with CentOS 5, a Linux distribution
|
||
|
released in April 2007. In practice, this means that Python wheels which conform
|
||
|
to this policy should function on almost any linux distribution released after
|
||
|
this date.
|
||
|
|
||
|
|
||
|
Compilation and Tooling
|
||
|
=======================
|
||
|
|
||
|
To support the compilation of wheels meeting the ``manylinux1`` standard, we
|
||
|
provide initial drafts of two tools.
|
||
|
|
||
|
The first is a Docker image based on CentOS 5.11, which is recommended as an
|
||
|
easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
|
||
|
Compiling on a more recently-released linux distribution will generally
|
||
|
introduce dependencies on too-new versioned symbols. The image comes with a
|
||
|
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
|
||
|
well as the latest releases of Python and pip.
|
||
|
|
||
|
The second tool is a command line executable called ``auditwheel`` [5]_. First,
|
||
|
it inspects all of the ELF files inside a wheel to check for dependencies on
|
||
|
versioned symbols or external shared libraries, and verifies conformance with
|
||
|
the ``manylinux1`` policy. This includes the ability to add the new platform
|
||
|
tag to conforming wheels.
|
||
|
|
||
|
In addition, ``auditwheel`` has the ability to automatically modify wheels that
|
||
|
depend on external shared libraries by copying those shared libraries from
|
||
|
the system into the wheel itself, and modifying the appropriate RPATH entries
|
||
|
such that these libraries will be picked up at runtime. This accomplishes a
|
||
|
similar result as if the libraries had been statically linked without requiring
|
||
|
changes to the build system.
|
||
|
|
||
|
Neither of these tools are necessary to build wheels which conform with the
|
||
|
``manylinux1`` policy. Similar results can usually be achieved by statically
|
||
|
linking external dependencies and/or using certain inline assembly constructs
|
||
|
to instruct the linker to prefer older symbol versions, however these tricks
|
||
|
can be quite esoteric.
|
||
|
|
||
|
|
||
|
Platform Detection for Installers
|
||
|
=================================
|
||
|
|
||
|
Because the ``manylinux1`` profile is already known to work for the many
|
||
|
thousands of users of popular commercial Python distributions, we suggest that
|
||
|
installation tools like ``pip`` should error on the side of assuming that a
|
||
|
system *is* compatible, unless there is specific reason to think otherwise.
|
||
|
|
||
|
We know of three main sources of potential incompatibility that are likely to
|
||
|
arise in practice:
|
||
|
|
||
|
* A linux distribution that is too old (e.g. RHEL 4)
|
||
|
* A linux distribution that does not use glibc (e.g. Alpine Linux, which is
|
||
|
based on musl libc, or Android)
|
||
|
* Eventually, in the future, there may exist distributions that break
|
||
|
compatibility with this profile
|
||
|
|
||
|
To handle the first two cases, we propose the following simple and reliable
|
||
|
check: ::
|
||
|
|
||
|
def have_glibc_version(major, minimum_minor):
|
||
|
import ctypes
|
||
|
|
||
|
process_namespace = ctypes.CDLL(None)
|
||
|
try:
|
||
|
gnu_get_libc_version = process_namespace.gnu_get_libc_version
|
||
|
except AttributeError:
|
||
|
# We are not linked to glibc.
|
||
|
return False
|
||
|
|
||
|
gnu_get_libc_version.restype = ctypes.c_char_p
|
||
|
version_str = gnu_get_libc_version()
|
||
|
# py2 / py3 compatibility:
|
||
|
if not isinstance(version_str, str):
|
||
|
version_str = version_str.decode("ascii")
|
||
|
|
||
|
version = [int(piece) for piece in version_str.split(".")]
|
||
|
assert len(version) == 2
|
||
|
if major != version[0]:
|
||
|
return False
|
||
|
if minimum_minor > version[1]:
|
||
|
return False
|
||
|
return True
|
||
|
|
||
|
# CentOS 5 uses glibc 2.5.
|
||
|
is_manylinux1_compatible = have_glibc_version(2, 5)
|
||
|
|
||
|
To handle the third case, we propose the creation of a file
|
||
|
``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
|
||
|
contents: ::
|
||
|
|
||
|
[manylinux1]
|
||
|
compatible = true
|
||
|
|
||
|
where the supported values for the ``manylinux1.compatible`` entry are the
|
||
|
same as those supported by the ConfigParser ``getboolean`` method.
|
||
|
|
||
|
The proposed logic for ``pip`` or related tools, then, is:
|
||
|
|
||
|
0) If ``distutils.util.get_platform()`` does not start with the string
|
||
|
``"linux"``, then assume the current system is not ``manylinux1``
|
||
|
compatible.
|
||
|
1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
|
||
|
key, then trust that.
|
||
|
2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
|
||
|
current system can handle ``manylinux1`` wheels.
|
||
|
3) Otherwise, assume that the current system cannot handle ``manylinux1``
|
||
|
wheels.
|
||
|
|
||
|
|
||
|
Security Implications
|
||
|
=====================
|
||
|
|
||
|
One of the advantages of dependencies on centralized libraries in Linux is
|
||
|
that bugfixes and security updates can be deployed system-wide, and
|
||
|
applications which depend on on these libraries will automatically feel the
|
||
|
effects of these patches when the underlying libraries are updated. This can
|
||
|
be particularly important for security updates in packages communication
|
||
|
across the network or cryptography.
|
||
|
|
||
|
``manylinux1`` wheels distributed through PyPI that bundle security-critical
|
||
|
libraries like OpenSSL will thus assume responsibility for prompt updates in
|
||
|
response disclosed vulnerabilities and patches. This closely parallels the
|
||
|
security implications of the distribution of binary wheels on Windows that,
|
||
|
because the platform lacks a system package manager, generally bundle their
|
||
|
dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
|
||
|
included in the ``manylinux1`` profile.
|
||
|
|
||
|
|
||
|
Rejected Alternatives
|
||
|
=====================
|
||
|
|
||
|
One alternative would be to provide separate platform tags for each Linux
|
||
|
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
|
||
|
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
|
||
|
adding such platform tags in the future, or of further extensions to wheel
|
||
|
metadata that would allow wheels to declare dependencies on external
|
||
|
system-installed packages. However, such extensions would require substantially
|
||
|
more work than this proposal, and still might not be appreciated by package
|
||
|
developers who would prefer not to have to maintain multiple build environments
|
||
|
and build multiple wheels in order to cover all the common Linux distributions.
|
||
|
Therefore we consider such proposals to be out-of-scope for this PEP.
|
||
|
|
||
|
|
||
|
References
|
||
|
==========
|
||
|
|
||
|
.. [1] PEP 425 -- Compatibility Tags for Built Distributions
|
||
|
(https://www.python.org/dev/peps/pep-0425/)
|
||
|
.. [2] Enthought Canopy Python Distribution
|
||
|
(https://store.enthought.com/downloads/)
|
||
|
.. [3] Continuum Analytics Anaconda Python Distribution
|
||
|
(https://www.continuum.io/downloads)
|
||
|
.. [4] manylinux1 docker image
|
||
|
(https://quay.io/repository/manylinux/manylinux)
|
||
|
.. [5] auditwheel
|
||
|
(https://pypi.python.org/pypi/auditwheel)
|
||
|
|
||
|
Copyright
|
||
|
=========
|
||
|
|
||
|
This document has been placed into the public domain.
|
||
|
|
||
|
..
|
||
|
|
||
|
Local Variables:
|
||
|
mode: indented-text
|
||
|
indent-tabs-mode: nil
|
||
|
sentence-end-double-space: t
|
||
|
fill-column: 70
|
||
|
coding: utf-8
|
||
|
End:
|