iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/pep-0600/index.html

598 lines
47 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 600 – Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0600/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 600 – Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions | peps.python.org'>
<meta property="og:description" content="This PEP proposes a scheme for new ‘manylinux’ wheel tags to be defined without requiring a PEP for every specific tag, similar to how Windows and macOS tags already work. This will allow package maintainers to take advantage of new tags more quickly, w...">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0600/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP proposes a scheme for new ‘manylinux’ wheel tags to be defined without requiring a PEP for every specific tag, similar to how Windows and macOS tags already work. This will allow package maintainers to take advantage of new tags more quickly, w...">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 600</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 600 – Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Nathaniel J. Smith &lt;njs&#32;&#97;t&#32;pobox.com&gt;,
Thomas Kluyver &lt;thomas&#32;&#97;t&#32;kluyver.me.uk&gt;</dd>
<dt class="field-even">Sponsor<span class="colon">:</span></dt>
<dd class="field-even">Paul Moore &lt;p.f.moore&#32;&#97;t&#32;gmail.com&gt;</dd>
<dt class="field-odd">BDFL-Delegate<span class="colon">:</span></dt>
<dd class="field-odd">Paul Moore &lt;p.f.moore&#32;&#97;t&#32;gmail.com&gt;</dd>
<dt class="field-even">Discussions-To<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/the-next-manylinux-specification/1043">Discourse thread</a></dd>
<dt class="field-odd">Status<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd>
<dt class="field-even">Type<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-odd">Topic<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="../topic/packaging/">Packaging</a></dd>
<dt class="field-even">Created<span class="colon">:</span></dt>
<dd class="field-even">03-May-2019</dd>
<dt class="field-odd">Post-History<span class="colon">:</span></dt>
<dd class="field-odd">03-May-2019</dd>
<dt class="field-even">Replaces<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="../pep-0513/">513</a>, <a class="reference external" href="../pep-0571/">571</a>, <a class="reference external" href="../pep-0599/">599</a></dd>
<dt class="field-odd">Resolution<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-600-future-manylinux-platform-tags-for-portable-linux-built-distributions/2414/27">Discourse message</a></dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#specification">Specification</a><ul>
<li><a class="reference internal" href="#core-definition">Core definition</a></li>
<li><a class="reference internal" href="#legacy-manylinux-tags">Legacy manylinux tags</a></li>
<li><a class="reference internal" href="#package-installers">Package installers</a></li>
<li><a class="reference internal" href="#package-indexes">Package indexes</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rejected-alternatives">Rejected alternatives</a></li>
</ul>
</details></section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP proposes a scheme for new ‘manylinux’ wheel tags to be
defined without requiring a PEP for every specific tag, similar to how
Windows and macOS tags already work. This will allow package
maintainers to take advantage of new tags more quickly, while making
better use of limited volunteer time.</p>
<p>Non-goals include: handling non-glibc-based platforms; integrating
with external package managers or handling external dependencies such
as CUDA; making manylinux tags more sophisticated than their
Windows/macOS equivalents; doing anything besides taking our existing
tried-and-tested approach and streamlining it. These are important
issues and other PEPs may address them in the future, but for this PEP
they’re out of scope.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>Python users appreciate it when PyPI has pre-compiled packages for
their platform, because it makes installation fast and simple. But
distributing pre-compiled binaries on Linux is challenging because of
the diversity of Linux-based platforms. For example, Debian, Android,
and Alpine all use the Linux kernel, but with radically different
userspace libraries, which makes it difficult or impossible to create
a single wheel that works on all three. This complexity has caused
many previous discussions of Linux wheels to stall out.</p>
<p>The “manylinux” project succeeded by adopting a strategy of ruthless
pragmatism. We chose a large but tractable set of Linux platforms –
specifically, mainstream glibc-based distributions like Debian,
OpenSuSE, Ubuntu, RHEL, etc. – and then we did whatever it takes to
make wheels that work across all these platforms.</p>
<p>This approach requires many compromises. Manylinux wheels can only
rely on external libraries that maintain a consistent ABI and are
universally available across all these distributions, which in
practice restricts them to a small set of core libraries like glibc
and a few others. Wheels have to be built on carefully-chosen
platforms of the oldest possible vintage, using a Python that is
itself built in a carefully-chosen configuration. Other shared library
dependencies have to be bundled into the wheel, which requires a
complex process to avoid collisions between unrelated wheels. And
finally, the details of these requirements change over time, as new
distro versions are released, and old ones fall out of use.</p>
<p>It turns out that these requirements are not too onerous: they’re
essentially equivalent to what you have to do to ship Windows or macOS
wheels, and the manylinux approach has achieved substantial uptake
among both package maintainers and end-users. But any manylinux PEP
needs some way to address these complexities.</p>
<p>In previous manylinux PEPs (<a class="pep reference internal" href="../pep-0513/" title="PEP 513 – A Platform Tag for Portable Linux Built Distributions">PEP 513</a>, <a class="pep reference internal" href="../pep-0571/" title="PEP 571 – The manylinux2010 Platform Tag">PEP 571</a>, <a class="pep reference internal" href="../pep-0599/" title="PEP 599 – The manylinux2014 Platform Tag">PEP 599</a>), we’ve
done this by attempting to write down in the PEP the exact set of
libraries, symbol versions, Python configuration, etc. that we
believed would lead to wheels that work on all mainstream glibc-based
Linux systems. But this created several problems:</p>
<p>First, PEPs are generally supposed to be normative references: if
software doesn’t match the PEP, then we fix the software. But in this
case, the PEPs are attempting to describe Linux distributions, which
are a moving target, and do not consider our PEPs to constrain their
behavior. This means that we’ve been taking on an unbounded commitment
to keep updating every manylinux PEP whenever the Linux distro
landscape changes. This is a substantial commitment for unfunded
volunteers to take on, and it’s not clear that this work produces
value for our users.</p>
<p>And second, every time we move manylinux forward to a newer range of
supported platforms, or add support for a new architecture, we have to
go through a fairly elaborate process: writing a new PEP, updating the
PyPI and pip codebases to recognize the new tag, waiting for the new
pip to percolate to users, etc. None of this happens on Windows/macOS;
it’s only a tax on Linux maintainers. This slows deployment of new
manylinux versions, and consumes part of our community’s limited PEP
review bandwidth, thus slowing progress of the Python packaging
ecosystem as a whole. This is especially problematic for less-popular
architectures, who have less volunteer resources to overcome these
barriers.</p>
<p>How can we fix it?</p>
<p>A manylinux PEP has to address three main audiences:</p>
<ul class="simple">
<li><strong>Package installers</strong>, like pip, need to be able to determine which
wheel tags are compatible with the system they find themselves
running on. This requires some automated process to introspect the
system and match it up with wheel tags.</li>
<li><strong>Package indexes</strong>, like PyPI, need to be able to validate which
wheel tags are valid. Generally, this just requires something like a
list of valid tags, or regex they match, with no need to know
anything about the actual semantics for individual tags. (But see
the discussion of upload verification below.)</li>
<li><strong>Package maintainers</strong> need to be able to build wheels that meet
the requirements for a given wheel tag.</li>
</ul>
<p>Here’s the key insight behind this new PEP: it’s crucial that
different <strong>package installers</strong> and <strong>package indexes</strong> all agree on
which manylinux tags are valid and which systems they install on, so
we need a PEP to specify these – but, these are straightforward, and
don’t really change between manylinux versions. The complicated part
that keeps changing is the process of actually <strong>building the wheels</strong>
– but, if there are multiple competing build environments, it <em>doesn’t
matter</em> whether they use exactly the same rules as each other, as long
as they all produce wheels that work on end-user systems. Therefore,
we don’t need an interoperability standard for building wheels, so we
don’t need to write the details into a PEP.</p>
<p>To further convince ourselves that this approach will work, let’s look
again at how we handle wheels on Windows and macOS: the PEPs describe
which tags are valid, and which systems they’re supposed to work on,
but not how to actually build wheels for those platforms. And in
practice, if you want to distribute Windows or macOS wheels, you might
have to jump through some complicated and poorly documented hoops in
order to bundle dependencies, target the right range of OS versions,
etc. But the system works, and the way to improve it is to write
better docs and build better tooling; no-one thinks that the way to
make Windows wheels work better is to publish a PEP describing
which symbols we think Microsoft should be including in their
libraries and how their linker ought to work. This PEP extends that
philosophy to manylinux as well.</p>
</section>
<section id="specification">
<h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2>
<section id="core-definition">
<h3><a class="toc-backref" href="#core-definition" role="doc-backlink">Core definition</a></h3>
<p>Tags using the new scheme will look like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">manylinux_2_17_x86_64</span>
</pre></div>
</div>
<p>Or more generally:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH}
</pre></div>
</div>
<p>This tag is a promise: the wheel’s creator promises that the wheel
will work on any mainstream Linux distro that uses glibc version
<code class="docutils literal notranslate"><span class="pre">${GLIBCMAJOR}.${GLIBCMINOR}</span></code> or later, and where the <code class="docutils literal notranslate"><span class="pre">${ARCH}</span></code>
matches the return value from <code class="docutils literal notranslate"><span class="pre">distutils.util.get_platform()</span></code>. (For
more detail about architecture tags, see <a class="pep reference internal" href="../pep-0425/" title="PEP 425 – Compatibility Tags for Built Distributions">PEP 425</a>.)</p>
<p>If a user installs this wheel into an environment that matches these
requirements and it doesn’t work, then that wheel does not comply with
this specification. This should be considered a bug in the wheel, and
it’s the wheel creator’s responsibility to look for a fix (possibly
with the help of the broader community).</p>
<p>The word “mainstream” is intentionally somewhat vague, and should be
interpreted expansively. The goal is to rule out weird homebrew Linux
systems; generally any distro you’ve actually heard of should be
considered “mainstream”. We also provide a way for maintainers of
“weird” distros to manually override this check, though based on
experience with previous manylinux PEPs, we don’t expect this feature
to see much use.</p>
<p>And finally, compliant wheels are required to “play well with others”,
i.e., installing a manylinux wheel must not cause other unrelated
packages to break.</p>
<p>Any method of producing wheels which meets these criteria is
acceptable. However, in practice we expect that the auditwheel project
will maintain an up-to-date set of tools and build images for
producing manylinux wheels, as well as documentation about how they
work and how to use them, and that most maintainers will want to use
those. For the latest information on building manylinux wheels,
including recommendations about which build images to use, see
<a class="reference external" href="https://packaging.python.org">https://packaging.python.org</a>.</p>
<p>Since these requirements are fairly high-level, here are some examples
of how they play out in specific situations:</p>
<p>Example: if a wheel is tagged as <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_x86_64</span></code>, but it
uses symbols that were only added in glibc 2.18, then that wheel won’t
work on systems with glibc 2.17. Therefore, we can conclude that this
wheel is in violation of this specification.</p>
<p>Example: Until ~2017, all major Linux distros included
<code class="docutils literal notranslate"><span class="pre">libncursesw.so.5</span></code> as part of their default install. Until that
date, a wheel that linked to <code class="docutils literal notranslate"><span class="pre">libncursesw.so.5</span></code> was compliant with
this specification. Then, distros started switching to ncurses 6,
which has a different name and incompatible ABI, and stopped
installing <code class="docutils literal notranslate"><span class="pre">libncursesw.so.5</span></code> by default. So after that date, a
wheel that links to <code class="docutils literal notranslate"><span class="pre">libncursesw.so.5</span></code> was no longer compliant with
this specification.</p>
<p>Example: The Linux ELF linker places all shared library SONAMEs into a
single process-global namespace. If independent wheels used the same
SONAME for their bundled libraries, they might end up colliding and
using the wrong library version, which would violate the “play well
with others” rule. Therefore, this specification requires that wheels
use globally-unique names for all bundled libraries. (Auditwheel
currently accomplishes this by renaming all bundled libraries to
include a globally-unique hash.)</p>
<p>Example: we’ve observed certain wheels using C++ in ways that
<a class="reference external" href="https://github.com/apache/arrow/pull/2210">interfere with other packages</a> via an unclear
mechanism. This is also a violation of the “play well with others”
rule, so those wheels aren’t compliant with this specification.</p>
<p>Example: The imaginary architecture LEG v7 has both big-endian and
little-endian variants. Big-endian binaries require a big-endian
system, and little-endian binaries require a little-endian system. But
unfortunately, it’s discovered that due to a bug in <a class="pep reference internal" href="../pep-0425/" title="PEP 425 – Compatibility Tags for Built Distributions">PEP 425</a>, both
variants use the same architecture tag, <code class="docutils literal notranslate"><span class="pre">legv7</span></code>. This makes it
impossible to create a compliant <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_legv7</span></code> wheel: no
matter what we do, it will crash on some user’s systems. So, we write
a new PEP defining architecture tags <code class="docutils literal notranslate"><span class="pre">legv7le</span></code> and <code class="docutils literal notranslate"><span class="pre">legv7be</span></code>; now
we can ship manylinux LEG v7 wheels.</p>
<p>Example: There’s also a LEG v8. It also has big-endian and
little-endian variants. But fortunately, it turns out that <a class="pep reference internal" href="../pep-0425/" title="PEP 425 – Compatibility Tags for Built Distributions">PEP 425</a>
already does the right thing LEG v8, so LEG v8 enthusiasts can start
shipping <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_legv8le</span></code> and <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_legv8be</span></code>
wheels immediately once this PEP is implemented, even though the
authors of this PEP don’t know anything at all about LEG v8.</p>
</section>
<section id="legacy-manylinux-tags">
<h3><a class="toc-backref" href="#legacy-manylinux-tags" role="doc-backlink">Legacy manylinux tags</a></h3>
<p>The existing manylinux tags are redefined as aliases for new-style
tags:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">manylinux1_x86_64</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_5_x86_64</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux1_i686</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_5_i686</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2010_x86_64</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_12_x86_64</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2010_i686</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_12_i686</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_x86_64</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_x86_64</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_i686</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_i686</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_aarch64</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_aarch64</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_armv7l</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_armv7l</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_ppc64</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_ppc64</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_ppc64le</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_ppc64le</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_s390x</span></code> is now an alias for <code class="docutils literal notranslate"><span class="pre">manylinux_2_17_s390x</span></code></li>
</ul>
<p>This redefinition is largely a no-op, but does affect a few things:</p>
<ul>
<li>Previously, we had an open-ended and growing commitment to keep
updating every manylinux PEP whenever a new Linux distro was
released, for the rest of time. By making this PEP normative for the
older tags, that obligation goes away. When this PEP is accepted,
the previous manylinux PEPs will receive a final update noting that
they are no longer maintained and referring to this PEP.</li>
<li>The “play well with others” rule was always intended, but previous
PEPs didn’t state it explicitly; now it’s explicit.</li>
<li>Previous PEPs assumed that glibc 3.x might be incompatible with
glibc 2.x, so we checked for compatibility between a system and a
tag using logic like:<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sys_major</span> <span class="o">==</span> <span class="n">tag_major</span> <span class="ow">and</span> <span class="n">sys_minor</span> <span class="o">&gt;=</span> <span class="n">tag_minor</span>
</pre></div>
</div>
<p>Recently the glibc maintainers <a class="reference external" href="https://sourceware.org/bugzilla/show_bug.cgi?id=24636">advised us</a> that we
should assume that glibc will maintain backwards-compatibility
indefinitely, even if they bump the major version number. So the new
check for compatibility is:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">(</span><span class="n">sys_major</span><span class="p">,</span> <span class="n">sys_minor</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="p">(</span><span class="n">tag_major</span><span class="p">,</span> <span class="n">tag_minor</span><span class="p">)</span>
</pre></div>
</div>
</li>
</ul>
</section>
<section id="package-installers">
<h3><a class="toc-backref" href="#package-installers" role="doc-backlink">Package installers</a></h3>
<p>Generally, package installers should install manylinux wheels on
systems that have an appropriate glibc and architecture, and not
otherwise. If there are multiple compatible manylinux wheels
available, then the wheel with the highest glibc version should be
preferred, in order to take advantage of newer compilers and glibc
features.</p>
<p>In addition, we follow previous specifications, and allow for Python
distributors to manually override this check by adding a
<code class="docutils literal notranslate"><span class="pre">_manylinux</span></code> module to their standard library. If this package is
importable, and if it defines a function called
<code class="docutils literal notranslate"><span class="pre">manylinux_compatible</span></code>, then package installers should call this
function, passing in the major version, minor version, and
architecture from the manylinux tag, and it will either return a
boolean saying whether wheels with the given tag should be considered
compatible with the current system, or else <code class="docutils literal notranslate"><span class="pre">None</span></code> to indicate that
the default logic should be used.</p>
<p>For compatibility with previous specifications, if the tag is
<code class="docutils literal notranslate"><span class="pre">manylinux1</span></code> or <code class="docutils literal notranslate"><span class="pre">manylinux_2_5</span></code> exactly, then we also check the
module for a boolean attribute <code class="docutils literal notranslate"><span class="pre">manylinux1_compatible</span></code>, if the
tag version is <code class="docutils literal notranslate"><span class="pre">manylinux2010</span></code> or <code class="docutils literal notranslate"><span class="pre">manylinux_2_12</span></code> exactly, then
we also check the module for a boolean attribute
<code class="docutils literal notranslate"><span class="pre">manylinux2010_compatible</span></code>, and if the tag version is
<code class="docutils literal notranslate"><span class="pre">manylinux2014</span></code> or <code class="docutils literal notranslate"><span class="pre">manylinux_2_17</span></code> exactly, then we also check
the module for a boolean attribute <code class="docutils literal notranslate"><span class="pre">manylinux2014_compatible</span></code>. If
both the new and old attributes are defined, then
<code class="docutils literal notranslate"><span class="pre">manylinux_compatible</span></code> takes precedence.</p>
<p>Here’s some example code. You don’t have to actually use this code,
but you can use it for reference if you have questions about the exact
semantics:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">LEGACY_ALIASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;manylinux1_x86_64&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_5_x86_64&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux1_i686&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_5_i686&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2010_x86_64&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_12_x86_64&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2010_i686&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_12_i686&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_x86_64&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_x86_64&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_i686&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_i686&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_aarch64&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_aarch64&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_armv7l&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_armv7l&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_ppc64&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_ppc64&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_ppc64le&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_ppc64le&quot;</span><span class="p">,</span>
<span class="s2">&quot;manylinux2014_s390x&quot;</span><span class="p">:</span> <span class="s2">&quot;manylinux_2_17_s390x&quot;</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">manylinux_tag_is_compatible_with_this_system</span><span class="p">(</span><span class="n">tag</span><span class="p">):</span>
<span class="c1"># Normalize and parse the tag</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">LEGACY_ALIASES</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tag</span><span class="p">,</span> <span class="n">tag</span><span class="p">)</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="s2">&quot;manylinux_([0-9]+)_([0-9]+)_(.*)&quot;</span><span class="p">,</span> <span class="n">tag</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">m</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">tag_major_str</span><span class="p">,</span> <span class="n">tag_minor_str</span><span class="p">,</span> <span class="n">tag_arch</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="n">tag_major</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">tag_major_str</span><span class="p">)</span>
<span class="n">tag_minor</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">tag_minor_str</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">system_uses_glibc</span><span class="p">():</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">sys_major</span><span class="p">,</span> <span class="n">sys_minor</span> <span class="o">=</span> <span class="n">get_system_glibc_version</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span><span class="n">sys_major</span><span class="p">,</span> <span class="n">sys_minor</span><span class="p">)</span> <span class="o">&lt;</span> <span class="p">(</span><span class="n">tag_major</span><span class="p">,</span> <span class="n">tag_minor</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">sys_arch</span> <span class="o">=</span> <span class="n">get_system_arch</span><span class="p">()</span>
<span class="k">if</span> <span class="n">sys_arch</span> <span class="o">!=</span> <span class="n">tag_arch</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="c1"># Check for manual override</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">_manylinux</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_manylinux</span><span class="p">,</span> <span class="s2">&quot;manylinux_compatible&quot;</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">_manylinux</span><span class="o">.</span><span class="n">manylinux_compatible</span><span class="p">(</span>
<span class="n">tag_major</span><span class="p">,</span> <span class="n">tag_minor</span><span class="p">,</span> <span class="n">tag_arch</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">result</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">tag_major</span><span class="p">,</span> <span class="n">tag_minor</span><span class="p">)</span> <span class="o">==</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">5</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_manylinux</span><span class="p">,</span> <span class="s2">&quot;manylinux1_compatible&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">_manylinux</span><span class="o">.</span><span class="n">manylinux1_compatible</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">tag_major</span><span class="p">,</span> <span class="n">tag_minor</span><span class="p">)</span> <span class="o">==</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">12</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_manylinux</span><span class="p">,</span> <span class="s2">&quot;manylinux2010_compatible&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">_manylinux</span><span class="o">.</span><span class="n">manylinux2010_compatible</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">True</span>
</pre></div>
</div>
</section>
<section id="package-indexes">
<h3><a class="toc-backref" href="#package-indexes" role="doc-backlink">Package indexes</a></h3>
<p>The exact set of wheel tags accepted by PyPI, or any package index, is
a policy question, and up to the maintainers of that index. But, we
recommend that package indexes accept any wheels whose platform tag
matches the following regexes:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">manylinux1_(x86_64|i686)</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2010_(x86_64|i686)</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux2014_(x86_64|i686|aarch64|armv7l|ppc64|ppc64le|s390x)</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">manylinux_[0-9]+_[0-9]+_(.*)</span></code></li>
</ul>
<p>Package indexes may impose additional requirements; for example, they
might audit uploaded wheels and reject those that contain known
problems, such as a <code class="docutils literal notranslate"><span class="pre">manylinux_2_17</span></code> wheel that references symbols
from later glibc versions, or dependencies on external libraries that
are known not to exist on all systems. Or a package index might decide
to be conservative and reject wheels tagged <code class="docutils literal notranslate"><span class="pre">manylinux_2_999</span></code>, on
the grounds that no-one knows what the Linux distro landscape will
look like when glibc 2.999 is released. We leave the details of any
such checks to the discretion of the package index maintainers.</p>
</section>
</section>
<section id="rejected-alternatives">
<h2><a class="toc-backref" href="#rejected-alternatives" role="doc-backlink">Rejected alternatives</a></h2>
<p><strong>Continuing the manylinux20XX series</strong>: As discussed above, this
leads to much more effort-intensive, slower, and more complex rollouts
of new versions. And while there are two places where it seems at
first to have some compensating benefits, if you look more closely
this turns out not to be the case.</p>
<p>First, this forces us to produce human-readable descriptions of how
Linux distros work, in the text of the PEP. But this is less valuable
than it might seem at first, and can actually be handled better by the
new “perennial” approach anyway.</p>
<p>If you’re trying to build wheels, the main thing you need is a
tutorial on how to use the build images and tooling around them. If
you’re trying to add support for a new build profile or create a
competitor to auditwheel, then your best resources will be the
auditwheel source code and issue tracker, which are always going to be
more detailed, precise, and reliable than a summary spec written in
English and without tests. Documentation like the old manylinux20XX
PEPs does add value! But in both cases, it’s primarily as a secondary
reference to provide overview and context.</p>
<p>And furthermore, the PEP process is poorly suited to maintaining this
kind of reference documentation – there’s a reason we don’t keep the
pip user manual in the PEPs repository! The auditwheel maintainers are
the best situated to understand what kinds of documentation are useful
to their users, and to maintain that documentation over time. For
example, there’s substantial overlap between the different manylinux
versions, and the PEP process currently forces us to handle this by
copy-pasting everything between a growing list of documents; instead,
the auditwheel maintainers might choose to factor out the common parts
into a single piece of shared documentation.</p>
<p>A related concern was that with the perennial approach, it may become
harder for package maintainers to decide which build profile to
target: instead of having to pick between <code class="docutils literal notranslate"><span class="pre">manylinux1</span></code>,
<code class="docutils literal notranslate"><span class="pre">manylinux2010</span></code>, <code class="docutils literal notranslate"><span class="pre">manylinux2014</span></code>, …, they now have a wider array
of options like <code class="docutils literal notranslate"><span class="pre">manylinux_2_5</span></code>, <code class="docutils literal notranslate"><span class="pre">manylinux_2_6</span></code>, …,
<code class="docutils literal notranslate"><span class="pre">manylinux_2_20</span></code>, … But again, we don’t believe this will be a
problem in practice. In either system, most package maintainers won’t
be starting by reading PEPs and trying to implement them from scratch.
If you’re a particularly expert and ambitious package maintainer who
needs to target a new version or new architecture, the perennial
approach gives you additional flexibility. But for regular everyday
maintainers, we expect they’ll start from a tutorial like
packaging.python.org, and by choosing from existing build images. A
tutorial can just as easily recommend <code class="docutils literal notranslate"><span class="pre">manylinux_2_17</span></code> as it can
recommend <code class="docutils literal notranslate"><span class="pre">manylinux2014</span></code>, and we expect the actual set of
pre-provided build images to be identical in both cases. And again, by
maintaining this documentation in the right place, instead of trying
to do it PEPs repository, we expect that we’ll end up with
documentation that’s higher-quality and more fitted to purpose.</p>
<p>Finally, some participants have pointed out that it’s very nice to be
able to look at a wheel and tell definitively whether it meets the
requirements of the spec. With the new “perennial” approach, we can
never say with 100% certainty that a wheel does meet the spec, because
that depends on the Linux distros. As engineers we have a
well-justified dislike for that kind of uncertainty.</p>
<p>However: as demonstrated by the examples above, we can still tell
definitively when a wheel <em>doesn’t</em> meet the spec, which turns out to
be what’s important in practice. And, in practice, with the
manylinux20XX approach, whenever distros change, we actually change
the spec; it takes a bit longer. So even if a wheel was compliant
today, it might be become non-compliant tomorrow. This is frustrating,
but unfortunately this uncertainty is unavoidable if what you care
about is distributing working wheels to users.</p>
<p>So even on these points where the old approach initially seems to have
advantages, we expect the new approach to actually do as well or
better.</p>
<p><strong>Switching to perennial tags, but continuing to write a PEP for each
version</strong>: This was proposed as a kind of hybrid, to try to get some
of the advantages of the perennial tagging system – like easier
rollouts of new versions – while keeping the advantages of the
manylinux20XX scheme, like forcing us to write documentation about
Linux distros, simplifying options for package maintainers, and being
able to definitively tell when a wheel meets the spec. But as
discussed above, on a closer look, it turns out that these advantages
are largely illusory. And this also inherits significant
<em>dis</em>advantages from the manylinux20XX scheme, like creating
indefinite obligations to update a growing list of copy-pasted PEPs.</p>
<p><strong>Making auditwheel normative</strong>: Another possibility that was
considered was to make auditwheel the normative reference on the
definition of manylinux, i.e., a wheel would be compliant if and only
if <code class="docutils literal notranslate"><span class="pre">auditwheel</span> <span class="pre">check</span></code> completed without errors. This was rejected
because the point of packaging PEPs is to define interoperability
between tools, not to bless specific tools.</p>
<p><strong>Adding extra words to the tag string</strong>: Another proposal we
considered was to add extra words to the wheel tag, e.g.
<code class="docutils literal notranslate"><span class="pre">manylinux_glibc_2_17</span></code> instead of <code class="docutils literal notranslate"><span class="pre">manylinux_2_17</span></code>. The motivation
would be to leave the door open to other kinds of versioning
heuristics in the future – for example, we could have
<code class="docutils literal notranslate"><span class="pre">manylinux_glibc_$VERSION</span></code> and <code class="docutils literal notranslate"><span class="pre">manylinux_alpine_$VERSION</span></code>.</p>
<p>But “manylinux” has always been a synonym for “broad compatibility
with mainstream glibc-based distros”; reusing it for unrelated build
profiles like alpine is more confusing than helpful. Also, some early
reviewers who aren’t steeped in the details of packaging found the
word <code class="docutils literal notranslate"><span class="pre">glibc</span></code> actively misleading, jumping to the conclusion that it
meant they needed a system with <em>exactly</em> that glibc version. And tags
like <code class="docutils literal notranslate"><span class="pre">manylinux_$VERSION</span></code> and <code class="docutils literal notranslate"><span class="pre">alpine_$VERSION</span></code> also have the
advantages of parsimony and directness. So we’ll go with that.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0600.rst">https://github.com/python/peps/blob/main/peps/pep-0600.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0600.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#specification">Specification</a><ul>
<li><a class="reference internal" href="#core-definition">Core definition</a></li>
<li><a class="reference internal" href="#legacy-manylinux-tags">Legacy manylinux tags</a></li>
<li><a class="reference internal" href="#package-installers">Package installers</a></li>
<li><a class="reference internal" href="#package-indexes">Package indexes</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rejected-alternatives">Rejected alternatives</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0600.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink