python-peps/pep-0685/index.html

330 lines
26 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 685 Comparison of extra names for optional distribution dependencies | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0685/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 685 Comparison of extra names for optional distribution dependencies | peps.python.org'>
<meta property="og:description" content="This PEP specifies how to normalize distribution extra names when performing comparisons. This prevents tools from either failing to find an extra name, or accidentally matching against an unexpected name.">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0685/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP specifies how to normalize distribution extra names when performing comparisons. This prevents tools from either failing to find an extra name, or accidentally matching against an unexpected name.">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 685</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 685 Comparison of extra names for optional distribution dependencies</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Brett Cannon &lt;brett&#32;&#97;t&#32;python.org&gt;</dd>
<dt class="field-even">PEP-Delegate<span class="colon">:</span></dt>
<dd class="field-even">Paul Moore &lt;p.f.moore&#32;&#97;t&#32;gmail.com&gt;</dd>
<dt class="field-odd">Discussions-To<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/14141">Discourse thread</a></dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Normative proposal accepted for implementation">Accepted</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-even">Topic<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd>
<dt class="field-odd">Created<span class="colon">:</span></dt>
<dd class="field-odd">08-Mar-2022</dd>
<dt class="field-even">Post-History<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/14141" title="Discourse thread">08-Mar-2022</a></dd>
<dt class="field-odd">Resolution<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/55">Discourse message</a></dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#motivation">Motivation</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#specification">Specification</a></li>
<li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li>
<li><a class="reference internal" href="#security-implications">Security Implications</a></li>
<li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li>
<li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li>
<li><a class="reference internal" href="#transition-plan">Transition Plan</a></li>
<li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul>
<li><a class="reference internal" href="#using-setuptools-60-s-normalization">Using setuptools 60s normalization</a></li>
</ul>
</li>
<li><a class="reference internal" href="#open-issues">Open Issues</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<div class="pep-banner canonical-pypa-spec sticky-banner admonition attention">
<p class="admonition-title">Attention</p>
<p>This PEP is a historical document. The up-to-date, canonical specifications are maintained on the <a class="reference external" href="https://packaging.python.org/en/latest/specifications/">PyPA specs page</a>.</p>
<p class="close-button">×</p>
<p>See the <a class="reference external" href="https://www.pypa.io/en/latest/specifications/#handling-fixes-and-other-minor-updates">PyPA specification update process</a> for how to propose changes.</p>
</div>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP specifies how to normalize <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use">distribution extra</a>
names when performing comparisons.
This prevents tools from either failing to find an extra name, or
accidentally matching against an unexpected name.</p>
</section>
<section id="motivation">
<h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2>
<p>The <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use">Provides-Extra</a> core metadata specification states that an extras
name “must be a valid Python identifier”.
<a class="pep reference internal" href="../pep-0508/" title="PEP 508 Dependency specification for Python Software Packages">PEP 508</a> specifies that the value of an <code class="docutils literal notranslate"><span class="pre">extra</span></code> marker may contain a
letter, digit, or any one of <code class="docutils literal notranslate"><span class="pre">.</span></code>, <code class="docutils literal notranslate"><span class="pre">-</span></code>, or <code class="docutils literal notranslate"><span class="pre">_</span></code> after the initial character.
There is no other <a class="reference external" href="https://packaging.python.org/en/latest/specifications/">PyPA specification</a>
which outlines how extra names should be written or normalized for comparison.
Due to the amount of packaging-related code in existence,
it is important to evaluate current practices by the community and
standardize on one that doesnt break most existing code, while being
something tool authors can agree to following.</p>
<p>The issue of there being no consistent standard was brought forward by an
<a class="reference external" href="https://discuss.python.org/t/7614">initial discussion</a>
noting that the extra <code class="docutils literal notranslate"><span class="pre">adhoc-ssl</span></code> was not considered equal to the name
<code class="docutils literal notranslate"><span class="pre">adhoc_ssl</span></code> by pip 22.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p><a class="pep reference internal" href="../pep-0503/" title="PEP 503 Simple Repository API">PEP 503</a> specifies how to normalize distribution names:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">&quot;[-_.]+&quot;</span><span class="p">,</span> <span class="s2">&quot;-&quot;</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
</pre></div>
</div>
<p>This collapses any run of the characters <code class="docutils literal notranslate"><span class="pre">-</span></code>, <code class="docutils literal notranslate"><span class="pre">_</span></code> and <code class="docutils literal notranslate"><span class="pre">.</span></code>
down to a single <code class="docutils literal notranslate"><span class="pre">-</span></code>.
For example, <code class="docutils literal notranslate"><span class="pre">---</span></code> <code class="docutils literal notranslate"><span class="pre">.</span></code> and <code class="docutils literal notranslate"><span class="pre">__</span></code> all get converted to just <code class="docutils literal notranslate"><span class="pre">-</span></code>.
This does <strong>not</strong> produce a valid Python identifier, per
the core metadata 2.2 specification for extra names.</p>
<p><a class="reference external" href="https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330">Setuptools 60 performs normalization</a>
via:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;[^A-Za-z0-9-.]+&#39;</span><span class="p">,</span> <span class="s1">&#39;_&#39;</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
</pre></div>
</div>
<p>The use of an underscore/<code class="docutils literal notranslate"><span class="pre">_</span></code> differs from PEP 503s use of a hyphen/<code class="docutils literal notranslate"><span class="pre">-</span></code>,
and it also normalizes characters outside of those allowed by <a class="pep reference internal" href="../pep-0508/" title="PEP 508 Dependency specification for Python Software Packages">PEP 508</a>.
Runs of <code class="docutils literal notranslate"><span class="pre">.</span></code> and <code class="docutils literal notranslate"><span class="pre">-</span></code>, unlike PEP 503, do <strong>not</strong> get normalized to one <code class="docutils literal notranslate"><span class="pre">_</span></code>,
e.g. <code class="docutils literal notranslate"><span class="pre">..</span></code> stays the same. To note, this is inconsistent with this functions
docstring, which <em>does</em> specify that all non-alphanumeric characters
(which would include <code class="docutils literal notranslate"><span class="pre">-</span></code> and <code class="docutils literal notranslate"><span class="pre">.</span></code>) are normalized and collapsed.</p>
<p>For pip 22, its
“extra normalisation behaviour is quite convoluted and erratic” <a class="reference internal" href="#pip-erratic" id="id1"><span>[pip-erratic]</span></a>
and so its use is not considered.</p>
<div role="list" class="citation-list">
<div class="citation" id="pip-erratic" role="doc-biblioentry">
<dt class="label" id="pip-erratic">[<a href="#id1">pip-erratic</a>]</dt>
<dd>Tzu-ping Chung on Python Discourse &lt;<a class="reference external" href="https://discuss.python.org/t/7614/10">https://discuss.python.org/t/7614/10</a></div>
</div>
</section>
<section id="specification">
<h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2>
<p>When comparing extra names, tools MUST normalize the names being compared
using the semantics outlined in <a class="pep reference internal" href="../pep-0503/#normalized-names" title="PEP 503 Simple Repository API § Normalized Names">PEP 503</a>
for names:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">&quot;[-_.]+&quot;</span><span class="p">,</span> <span class="s2">&quot;-&quot;</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
</pre></div>
</div>
<p>The <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/">core metadata</a> specification will be updated such that the allowed
names for <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use">Provides-Extra</a> matches what <a class="pep reference internal" href="../pep-0508/" title="PEP 508 Dependency specification for Python Software Packages">PEP 508</a> specifies for names.
This will bring extra naming in line with that of the <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#name">Name</a> field.
Because this changes what is considered valid, it will lead to a core
metadata version increase to <code class="docutils literal notranslate"><span class="pre">2.3</span></code>.</p>
<p>For tools writing <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/">core metadata</a>,
they MUST write out extra names in their normalized form.
This applies to the <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use">Provides-Extra</a> field and the
<a class="pep reference internal" href="../pep-0508/#extras" title="PEP 508 Dependency specification for Python Software Packages § Extras">extra marker</a> when used in the <a class="reference external" href="https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use">Requires-Dist</a> field.</p>
<p>Tools generating metadata MUST raise an error if a user specified
two or more extra names which would normalize to the same name.
Tools generating metadata MUST raise an error if an invalid extra
name is provided as appropriate for the specified core metadata version.
If a projects metadata specifies an older core metadata version and
the name would be invalid with newer core metadata versions,
tools reading that metadata SHOULD warn the user.
Tools SHOULD warn users when an invalid extra name is read and SHOULD
ignore the name to avoid ambiguity.
Tools MAY raise an error instead of a warning when reading an
invalid name, if they so desire.</p>
</section>
<section id="backwards-compatibility">
<h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2>
<p>Moving to <a class="pep reference internal" href="../pep-0503/" title="PEP 503 Simple Repository API">PEP 503</a> normalization and <a class="pep reference internal" href="../pep-0508/" title="PEP 508 Dependency specification for Python Software Packages">PEP 508</a> name acceptance
allows for all preexisting, valid names to continue to be valid.</p>
<p>Based on research looking at a collection of wheels on PyPI <a class="reference internal" href="#pypi-results" id="id2"><span>[pypi-results]</span></a>,
the risk of extra name clashes is limited to 73 instances when considering
all extras names on PyPI, valid or not (not just those within a single package)
while <em>only</em> looking at valid names leads to only 3 clashes:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">dev-test</span></code>: <code class="docutils literal notranslate"><span class="pre">dev_test</span></code>, <code class="docutils literal notranslate"><span class="pre">dev-test</span></code>, <code class="docutils literal notranslate"><span class="pre">dev.test</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">dev-lint</span></code>: <code class="docutils literal notranslate"><span class="pre">dev-lint</span></code>, <code class="docutils literal notranslate"><span class="pre">dev.lint</span></code>, <code class="docutils literal notranslate"><span class="pre">dev_lint</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">apache-beam</span></code>: <code class="docutils literal notranslate"><span class="pre">apache-beam</span></code>, <code class="docutils literal notranslate"><span class="pre">apache.beam</span></code></li>
</ul>
<p>By requiring tools writing core metadata to only record the normalized name,
the issue of preexisting, invalid extra names should diminish over time.</p>
<div role="list" class="citation-list">
<div class="citation" id="pypi-results" role="doc-biblioentry">
<dt class="label" id="pypi-results">[<a href="#id2">pypi-results</a>]</dt>
<dd>Paul Moore on Python Discourse <a class="reference external" href="https://discuss.python.org/t/14141/17">https://discuss.python.org/t/14141/17</a></div>
</div>
</section>
<section id="security-implications">
<h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2>
<p>It is possible that for a distribution that has conflicting extra names, a
tool ends up installing dependencies that somehow weaken the security
of the system.
This is only hypothetical and if it were to occur,
it would probably be more of a security concern for the distributions
specifying such extras names rather than the distribution that pulled
them in together.</p>
</section>
<section id="how-to-teach-this">
<h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2>
<p>This should be transparent to users on a day-to-day basis.
It will be up to tools to educate/stop users when they select extra
names which conflict.</p>
</section>
<section id="reference-implementation">
<h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2>
<p>No reference implementation is provided aside from the code above,
but the expectation is the <a class="reference external" href="https://packaging.pypa.io">packaging project</a> will provide a
function in its <code class="docutils literal notranslate"><span class="pre">packaging.utils</span></code> module that will implement extra name
normalization.
It will also implement extra name comparisons appropriately.
Finally, if the project ever gains the ability to write out metadata,
it will also implement this PEP.</p>
</section>
<section id="transition-plan">
<h2><a class="toc-backref" href="#transition-plan" role="doc-backlink">Transition Plan</a></h2>
<p>There is a risk that a build tool will produce core metadata
conforming to version 2.3 and thus this PEP but which is consumed by a
tool that is unaware of this PEP (if that tool chooses to attempt to
read a core metadata version it does not directly support).
In such a case there is a chance that a user may specify an extra
using an non-normalized name which worked previously but which fails
now.</p>
<p>As such, consumers of this PEP should be prioritized more than
producers so that users can be notified that they are specifying extra
names which are not normalized (and thus may break in the future).</p>
</section>
<section id="rejected-ideas">
<h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2>
<section id="using-setuptools-60-s-normalization">
<h3><a class="toc-backref" href="#using-setuptools-60-s-normalization" role="doc-backlink">Using setuptools 60s normalization</a></h3>
<p>Initially, this PEP proposed using setuptools <code class="docutils literal notranslate"><span class="pre">safe_extra()</span></code> for normalization
to try to minimize backwards-compatibility issues.
However, after checking various wheels on PyPI,
it became clear that standardizing <strong>all</strong> naming on <a class="pep reference internal" href="../pep-0508/" title="PEP 508 Dependency specification for Python Software Packages">PEP 508</a> and
<a class="pep reference internal" href="../pep-0503/" title="PEP 503 Simple Repository API">PEP 503</a> semantics was easier and better long-term,
while causing minimal backwards compatibility issues.</p>
</section>
</section>
<section id="open-issues">
<h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2>
<p>N/A</p>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0685.rst">https://github.com/python/peps/blob/main/peps/pep-0685.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0685.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#motivation">Motivation</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#specification">Specification</a></li>
<li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li>
<li><a class="reference internal" href="#security-implications">Security Implications</a></li>
<li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li>
<li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li>
<li><a class="reference internal" href="#transition-plan">Transition Plan</a></li>
<li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul>
<li><a class="reference internal" href="#using-setuptools-60-s-normalization">Using setuptools 60s normalization</a></li>
</ul>
</li>
<li><a class="reference internal" href="#open-issues">Open Issues</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0685.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>