PEP 513: Portable Built Linux Distributions
This commit is contained in:
Nick Coghlan
parent
571813608f
commit
d161460a4b
|
|
@ -0,0 +1,346 @@
|
|||
PEP: 513
|
||||
Title: A Platform Tag for Portable Linux Built Distributions
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Robert T. McGibbon <rmcgibbo@gmail.com>, Nathaniel J. Smith <njs@pobox.com>
|
||||
BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com>
|
||||
Status: Draft
|
||||
Type: Informational
|
||||
Content-Type: text/x-rst
|
||||
Created: 19-Jan-2016
|
||||
Post-History: 19-Jan-2016
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes the creation of a new platform tag for Python package built
|
||||
distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with
|
||||
external dependencies limited restricted to a standardized subset of
|
||||
the Linux kernel and core userspace ABI. It proposes that PyPI support
|
||||
uploading and distributing Wheels with this platform tag, and that ``pip``
|
||||
support downloading and installing these packages on compatible platforms.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
Currently, distribution of binary Python extensions for Windows and OS X is
|
||||
straightforward. Developers and packagers build wheels, which are assigned
|
||||
platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these
|
||||
wheels to PyPI. Users can download and install these wheels using tools such
|
||||
as ``pip``.
|
||||
|
||||
For Linux, the situation is much more delicate. In general, compiled Python
|
||||
extension modules built on one Linux distribution will not work on other Linux
|
||||
distributions, or even on the same Linux distribution with different system
|
||||
libraries installed.
|
||||
|
||||
Build tools using PEP 425 platform tags [1]_ do not track information about the
|
||||
particular Linux distribution or installed system libraries, and instead assign
|
||||
all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of
|
||||
this ambiguity, there is no expectation that ``linux``-tagged built
|
||||
distributions compiled on one machine will work properly on another, and for
|
||||
this reason, PyPI has not permitted the uploading of wheels for Linux.
|
||||
|
||||
It would be ideal if wheel packages could be compiled that would work on *any*
|
||||
linux system. But, because of the incredible diversity of Linux systems -- from
|
||||
PCs to Android to embedded systems with custom libcs -- this cannot
|
||||
be guaranteed in general.
|
||||
|
||||
Instead, we define a standard subset of the kernel+core userspace ABI that,
|
||||
in practice, is compatible enough that packages conforming to this standard
|
||||
will work on *many* linux systems, including essentially all of the desktop
|
||||
and server distributions in common use. We know this because there are
|
||||
companies who have been distributing such widely-portable pre-compiled Python
|
||||
extension modules for Linux -- e.g. Enthought with Canopy [2]_ and Continuum
|
||||
Analytics with Anaconda [3]_.
|
||||
|
||||
Building on the compability lessons learned from these companies, we thus
|
||||
define a baseline ``manylinux1`` platform tag for use by binary Python
|
||||
wheels, and introduce the implementation of preliminary tools to aid in the
|
||||
construction of these ``manylinux1`` wheels.
|
||||
|
||||
|
||||
Key Causes of Inter-Linux Binary Incompatibility
|
||||
================================================
|
||||
|
||||
To properly define a standard that will guarantee that wheel packages meeting
|
||||
this specification will operate on *many* linux platforms, it is necessary to
|
||||
understand the root causes which often prevent portability of pre-compiled
|
||||
binaries on Linux. The two key causes are dependencies on shared libraries
|
||||
which are not present on users' systems, and dependencies on particular
|
||||
versions of certain core libraries like ``glibc``.
|
||||
|
||||
|
||||
External Shared Libraries
|
||||
-------------------------
|
||||
|
||||
Most desktop and server linux distributions come with a system package manager
|
||||
(examples include ``APT`` on Debian-based systems, ``yum`` on
|
||||
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
|
||||
responsibilities, the installation of shared libraries installed to system
|
||||
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
|
||||
on one or more of these shared libraries, and thus function properly only on
|
||||
systems where the user has the proper libraries (and the proper
|
||||
versions thereof), either installed using their package manager, or installed
|
||||
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
|
||||
to notify the runtime linker of the location of the depended-upon shared
|
||||
libraries.
|
||||
|
||||
|
||||
Versioning of Core Shared Libraries
|
||||
-----------------------------------
|
||||
|
||||
Even if author or maintainers of a Python extension module with to use no
|
||||
external shared libraries, the modules will generally have a dynamic runtime
|
||||
dependency on the GNU C library, ``glibc``. While it is possible, statically
|
||||
linking ``glibc`` is usually a bad idea because of bloat, and because certain
|
||||
important C functions like ``dlopen()`` cannot be called from code that
|
||||
statically links ``glibc``. A runtime shared library dependency on a
|
||||
system-provided ``glibc`` is unavoidable in practice.
|
||||
|
||||
The maintainers of the GNU C library follow a strict symbol versioning scheme
|
||||
for backward compatibility. This ensures that binaries compiled against an older
|
||||
version of ``glibc`` can run on systems that have a newer ``glibc``. The
|
||||
opposite is generally not true -- binaries compiled on newer Linux
|
||||
distributions tend to rely upon versioned functions in glibc that are not
|
||||
available on older systems.
|
||||
|
||||
This generally prevents built distributions compiled on the latest Linux
|
||||
distributions from being portable.
|
||||
|
||||
|
||||
The ``manylinux1`` policy
|
||||
=========================
|
||||
|
||||
For these reasons, to achieve broad portability, Python wheels
|
||||
|
||||
* should depend only on an extremely limited set of external shared
|
||||
libraries; and
|
||||
* should depend only on ``old`` symbol versions in those external shared
|
||||
libraries.
|
||||
|
||||
The ``manylinux1`` policy thus encompasses a standard for what the
|
||||
permitted external shared libraries a wheel may depend on, and the maximum
|
||||
depended-upon symbol versions therein.
|
||||
|
||||
The permitted external shared libraries are: ::
|
||||
|
||||
libpanelw.so.5
|
||||
libncursesw.so.5
|
||||
libgcc_s.so.1
|
||||
libstdc++.so.6
|
||||
libm.so.6
|
||||
libdl.so.2
|
||||
librt.so.1
|
||||
libcrypt.so.1
|
||||
libc.so.6
|
||||
libnsl.so.1
|
||||
libutil.so.1
|
||||
libpthread.so.0
|
||||
libX11.so.6
|
||||
libXext.so.6
|
||||
libXrender.so.1
|
||||
libICE.so.6
|
||||
libSM.so.6
|
||||
libGL.so.1
|
||||
libgobject-2.0.so.0
|
||||
libgthread-2.0.so.0
|
||||
libglib-2.0.so.0
|
||||
|
||||
On Debian-based systems, these libraries are provided by the packages ::
|
||||
|
||||
libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
|
||||
libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0
|
||||
|
||||
On RPM-based systems, these libraries are provided by the packages ::
|
||||
|
||||
ncurses libgcc libstdc++ glibc libXext libXrender
|
||||
libICE libSM mesa-libGL glib2
|
||||
|
||||
This list was compiled by checking the external shared library dependencies of
|
||||
the Canopy [1]_ and Anaconda [2]_ distributions, which both include a wide array
|
||||
of the most popular Python modules and have been confirmed in practice to work
|
||||
across a wide swath of Linux systems in the wild.
|
||||
|
||||
For dependencies on externally-provided versioned symbols in the above shared
|
||||
libraries, the following symbol versions are permitted: ::
|
||||
|
||||
GLIBC <= 2.5
|
||||
CXXABI <= 3.4.8
|
||||
GLIBCXX <= 3.4.9
|
||||
GCC <= 4.2.0
|
||||
|
||||
These symbol versions were determined by inspecting the latest symbol version
|
||||
provided in the libraries distributed with CentOS 5, a Linux distribution
|
||||
released in April 2007. In practice, this means that Python wheels which conform
|
||||
to this policy should function on almost any linux distribution released after
|
||||
this date.
|
||||
|
||||
|
||||
Compilation and Tooling
|
||||
=======================
|
||||
|
||||
To support the compilation of wheels meeting the ``manylinux1`` standard, we
|
||||
provide initial drafts of two tools.
|
||||
|
||||
The first is a Docker image based on CentOS 5.11, which is recommended as an
|
||||
easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
|
||||
Compiling on a more recently-released linux distribution will generally
|
||||
introduce dependencies on too-new versioned symbols. The image comes with a
|
||||
full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
|
||||
well as the latest releases of Python and pip.
|
||||
|
||||
The second tool is a command line executable called ``auditwheel`` [5]_. First,
|
||||
it inspects all of the ELF files inside a wheel to check for dependencies on
|
||||
versioned symbols or external shared libraries, and verifies conformance with
|
||||
the ``manylinux1`` policy. This includes the ability to add the new platform
|
||||
tag to conforming wheels.
|
||||
|
||||
In addition, ``auditwheel`` has the ability to automatically modify wheels that
|
||||
depend on external shared libraries by copying those shared libraries from
|
||||
the system into the wheel itself, and modifying the appropriate RPATH entries
|
||||
such that these libraries will be picked up at runtime. This accomplishes a
|
||||
similar result as if the libraries had been statically linked without requiring
|
||||
changes to the build system.
|
||||
|
||||
Neither of these tools are necessary to build wheels which conform with the
|
||||
``manylinux1`` policy. Similar results can usually be achieved by statically
|
||||
linking external dependencies and/or using certain inline assembly constructs
|
||||
to instruct the linker to prefer older symbol versions, however these tricks
|
||||
can be quite esoteric.
|
||||
|
||||
|
||||
Platform Detection for Installers
|
||||
=================================
|
||||
|
||||
Because the ``manylinux1`` profile is already known to work for the many
|
||||
thousands of users of popular commercial Python distributions, we suggest that
|
||||
installation tools like ``pip`` should error on the side of assuming that a
|
||||
system *is* compatible, unless there is specific reason to think otherwise.
|
||||
|
||||
We know of three main sources of potential incompatibility that are likely to
|
||||
arise in practice:
|
||||
|
||||
* A linux distribution that is too old (e.g. RHEL 4)
|
||||
* A linux distribution that does not use glibc (e.g. Alpine Linux, which is
|
||||
based on musl libc, or Android)
|
||||
* Eventually, in the future, there may exist distributions that break
|
||||
compatibility with this profile
|
||||
|
||||
To handle the first two cases, we propose the following simple and reliable
|
||||
check: ::
|
||||
|
||||
def have_glibc_version(major, minimum_minor):
|
||||
import ctypes
|
||||
|
||||
process_namespace = ctypes.CDLL(None)
|
||||
try:
|
||||
gnu_get_libc_version = process_namespace.gnu_get_libc_version
|
||||
except AttributeError:
|
||||
# We are not linked to glibc.
|
||||
return False
|
||||
|
||||
gnu_get_libc_version.restype = ctypes.c_char_p
|
||||
version_str = gnu_get_libc_version()
|
||||
# py2 / py3 compatibility:
|
||||
if not isinstance(version_str, str):
|
||||
version_str = version_str.decode("ascii")
|
||||
|
||||
version = [int(piece) for piece in version_str.split(".")]
|
||||
assert len(version) == 2
|
||||
if major != version[0]:
|
||||
return False
|
||||
if minimum_minor > version[1]:
|
||||
return False
|
||||
return True
|
||||
|
||||
# CentOS 5 uses glibc 2.5.
|
||||
is_manylinux1_compatible = have_glibc_version(2, 5)
|
||||
|
||||
To handle the third case, we propose the creation of a file
|
||||
``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
|
||||
contents: ::
|
||||
|
||||
[manylinux1]
|
||||
compatible = true
|
||||
|
||||
where the supported values for the ``manylinux1.compatible`` entry are the
|
||||
same as those supported by the ConfigParser ``getboolean`` method.
|
||||
|
||||
The proposed logic for ``pip`` or related tools, then, is:
|
||||
|
||||
0) If ``distutils.util.get_platform()`` does not start with the string
|
||||
``"linux"``, then assume the current system is not ``manylinux1``
|
||||
compatible.
|
||||
1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
|
||||
key, then trust that.
|
||||
2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
|
||||
current system can handle ``manylinux1`` wheels.
|
||||
3) Otherwise, assume that the current system cannot handle ``manylinux1``
|
||||
wheels.
|
||||
|
||||
|
||||
Security Implications
|
||||
=====================
|
||||
|
||||
One of the advantages of dependencies on centralized libraries in Linux is
|
||||
that bugfixes and security updates can be deployed system-wide, and
|
||||
applications which depend on on these libraries will automatically feel the
|
||||
effects of these patches when the underlying libraries are updated. This can
|
||||
be particularly important for security updates in packages communication
|
||||
across the network or cryptography.
|
||||
|
||||
``manylinux1`` wheels distributed through PyPI that bundle security-critical
|
||||
libraries like OpenSSL will thus assume responsibility for prompt updates in
|
||||
response disclosed vulnerabilities and patches. This closely parallels the
|
||||
security implications of the distribution of binary wheels on Windows that,
|
||||
because the platform lacks a system package manager, generally bundle their
|
||||
dependencies. In particular, because its lacks a stable ABI, OpenSSL cannot be
|
||||
included in the ``manylinux1`` profile.
|
||||
|
||||
|
||||
Rejected Alternatives
|
||||
=====================
|
||||
|
||||
One alternative would be to provide separate platform tags for each Linux
|
||||
distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
|
||||
``debian_jessie``, etc. Nothing in this proposal rules out the possibility of
|
||||
adding such platform tags in the future, or of further extensions to wheel
|
||||
metadata that would allow wheels to declare dependencies on external
|
||||
system-installed packages. However, such extensions would require substantially
|
||||
more work than this proposal, and still might not be appreciated by package
|
||||
developers who would prefer not to have to maintain multiple build environments
|
||||
and build multiple wheels in order to cover all the common Linux distributions.
|
||||
Therefore we consider such proposals to be out-of-scope for this PEP.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] PEP 425 -- Compatibility Tags for Built Distributions
|
||||
(https://www.python.org/dev/peps/pep-0425/)
|
||||
.. [2] Enthought Canopy Python Distribution
|
||||
(https://store.enthought.com/downloads/)
|
||||
.. [3] Continuum Analytics Anaconda Python Distribution
|
||||
(https://www.continuum.io/downloads)
|
||||
.. [4] manylinux1 docker image
|
||||
(https://quay.io/repository/manylinux/manylinux)
|
||||
.. [5] auditwheel
|
||||
(https://pypi.python.org/pypi/auditwheel)
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed into the public domain.
|
||||
|
||||
..
|
||||
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue