iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/pep-0470/index.html

577 lines
43 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 470 – Removing External Hosting Support on PyPI | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0470/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 470 – Removing External Hosting Support on PyPI | peps.python.org'>
<meta property="og:description" content="This PEP proposes the deprecation and removal of support for hosting files externally to PyPI as well as the deprecation and removal of the functionality added by PEP 438, particularly rel information to classify different types of links and the meta-ta...">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0470/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP proposes the deprecation and removal of support for hosting files externally to PyPI as well as the deprecation and removal of the functionality added by PEP 438, particularly rel information to classify different types of links and the meta-ta...">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 470</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 470 – Removing External Hosting Support on PyPI</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Donald Stufft &lt;donald&#32;&#97;t&#32;stufft.io&gt;</dd>
<dt class="field-even">BDFL-Delegate<span class="colon">:</span></dt>
<dd class="field-even">Paul Moore &lt;p.f.moore&#32;&#97;t&#32;gmail.com&gt;</dd>
<dt class="field-odd">Discussions-To<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/distutils-sig&#64;python.org/">Distutils-SIG list</a></dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP describing or proposing a change to a Python community process, workflow or governance">Process</abbr></dd>
<dt class="field-even">Topic<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd>
<dt class="field-odd">Created<span class="colon">:</span></dt>
<dd class="field-odd">12-May-2014</dd>
<dt class="field-even">Post-History<span class="colon">:</span></dt>
<dd class="field-even">14-May-2014, 05-Jun-2014, 03-Oct-2014, 13-Oct-2014, 26-Aug-2015</dd>
<dt class="field-odd">Replaces<span class="colon">:</span></dt>
<dd class="field-odd"><a class="reference external" href="../pep-0438/">438</a></dd>
<dt class="field-even">Resolution<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/distutils-sig/2015-September/026789.html">Distutils-SIG message</a></dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#key-user-experience-expectations">Key User Experience Expectations</a></li>
<li><a class="reference internal" href="#why-additional-repositories">Why Additional Repositories?</a></li>
<li><a class="reference internal" href="#why-not-pep-438-or-similar">Why Not PEP 438 or Similar?</a></li>
</ul>
</li>
<li><a class="reference internal" href="#multiple-repository-index-support">Multiple Repository/Index Support</a></li>
<li><a class="reference internal" href="#deprecation-and-removal-of-link-spidering">Deprecation and Removal of Link Spidering</a></li>
<li><a class="reference internal" href="#summary-of-changes">Summary of Changes</a><ul>
<li><a class="reference internal" href="#repository-side">Repository side</a></li>
<li><a class="reference internal" href="#client-side">Client side</a></li>
</ul>
</li>
<li><a class="reference internal" href="#impact">Impact</a></li>
<li><a class="reference internal" href="#frequently-asked-questions">Frequently Asked Questions</a><ul>
<li><a class="reference internal" href="#i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do">I can’t host my project on PyPI because of &lt;X&gt;, what should I do?</a></li>
<li><a class="reference internal" href="#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that">My users have a worse experience with this PEP than before, how do I explain that?</a></li>
<li><a class="reference internal" href="#switching-to-a-repository-structure-breaks-my-workflow-or-isn-t-allowed-by-my-host">Switching to a repository structure breaks my workflow or isn’t allowed by my host?</a></li>
<li><a class="reference internal" href="#why-don-t-you-provide-x">Why don’t you provide &lt;X&gt;?</a></li>
<li><a class="reference internal" href="#why-should-i-register-on-pypi-if-i-m-running-my-own-repository-anyways">Why should I register on PyPI if I’m running my own repository anyways?</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rejected-proposals">Rejected Proposals</a><ul>
<li><a class="reference internal" href="#allow-easier-discovery-of-externally-hosted-indexes">Allow easier discovery of externally hosted indexes</a></li>
<li><a class="reference internal" href="#keep-the-current-classification-system-but-adjust-the-options">Keep the current classification system but adjust the options</a></li>
<li><a class="reference internal" href="#implement-this-pep-but-do-not-remove-the-existing-links">Implement this PEP, but Do Not Remove the Existing Links</a></li>
</ul>
</li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP proposes the deprecation and removal of support for hosting files
externally to PyPI as well as the deprecation and removal of the functionality
added by <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a>, particularly rel information to classify different types of
links and the meta-tag to indicate API version.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>Historically PyPI did not have any method of hosting files nor any method of
automatically retrieving installables, it was instead focused on providing a
central registry of names, to prevent naming collisions, and as a means of
discovery for finding projects to use. In the course of time setuptools began
to scrape these human facing pages, as well as pages linked from those pages,
looking for things it could automatically download and install. Eventually this
became the “Simple” API which used a similar URL structure however it
eliminated any of the extraneous links and information to make the API more
efficient. Additionally PyPI grew the ability for a project to upload release
files directly to PyPI enabling PyPI to act as a repository in addition to an
index.</p>
<p>This gives PyPI two equally important roles that it plays in the Python
ecosystem, that of index to enable easy discovery of Python projects and
central repository to enable easy hosting, download, and installation of Python
projects. Due to the history behind PyPI and the very organic growth it has
experienced the lines between these two roles are blurry, and this blurring has
caused confusion for the end users of both of these roles and this has in turn
caused ire between people attempting to use PyPI in different capacities, most
often when end users want to use PyPI as a repository but the author wants to
use PyPI solely as an index.</p>
<p>This confusion comes down to end users of projects not realizing if a project
is hosted on PyPI or if it relies on an external service. This often manifests
itself when the external service is down but PyPI is not. People will see that
PyPI works, and other projects works, but this one specific one does not. They
oftentimes do not realize who they need to contact in order to get this fixed
or what their remediation steps are.</p>
<p><a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> attempted to solve this issue by allowing projects to explicitly
declare if they were using the repository features or not, and if they were
not, it had the installers classify the links it found as either “internal”,
“verifiable external” or “unverifiable external”. <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> was accepted and
implemented in pip 1.4 (released on Jul 23, 2013) with the final transition
implemented in pip 1.5 (released on Jan 2, 2014).</p>
<p><a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> was successful in bringing about more people to utilize PyPI’s
repository features, an altogether good thing given the global CDN powering
PyPI providing speed ups for a lot of people, however it did so by introducing
a new point of confusion and pain for both the end users and the authors.</p>
<p>By moving to using explicit multiple repositories we can make the lines between
these two roles much more explicit and remove the “hidden” surprises caused by
the current implementation of handling people who do not want to use PyPI as a
repository.</p>
<section id="key-user-experience-expectations">
<h3><a class="toc-backref" href="#key-user-experience-expectations" role="doc-backlink">Key User Experience Expectations</a></h3>
<ol class="arabic simple">
<li>Easily allow external hosting to “just work” when appropriately configured
at the system, user or virtual environment level.</li>
<li>Eliminate any and all references to the confusing “verifiable external” and
“unverifiable external” distinction from the user experience (both when
installing and when releasing packages).</li>
<li>The repository aspects of PyPI should become <em>just</em> the default package
hosting location (i.e. the only one that is treated as opt-out rather than
opt-in by most client tools in their default configuration). Aside from that
aspect, hosting on PyPI should not otherwise provide an enhanced user
experience over hosting your own package repository.</li>
<li>Do all of the above while providing default behaviour that is secure against
most attackers below the nation state adversary level.</li>
</ol>
</section>
<section id="why-additional-repositories">
<h3><a class="toc-backref" href="#why-additional-repositories" role="doc-backlink">Why Additional Repositories?</a></h3>
<p>The two common installer tools, pip and easy_install/setuptools, both support
the concept of additional locations to search for files to satisfy the
installation requirements and have done so for many years. This means that
there is no need to “phase” in a new flag or concept and the solution to
installing a project from a repository other than PyPI will function regardless
of how old (within reason) the end user’s installer is. Not only has this
concept existed in the Python tooling for some time, but it is a concept that
exists across languages and even extending to the OS level with OS package
tools almost universally using multiple repository support making it extremely
likely that someone is already familiar with the concept.</p>
<p>Additionally, the multiple repository approach is a concept that is useful
outside of the narrow scope of allowing projects that wish to be included on
the index portion of PyPI but do not wish to utilize the repository portion of
PyPI. This includes places where a company may wish to host a repository that
contains their internal packages or where a project may wish to have multiple
“channels” of releases, such as alpha, beta, release candidate, and final
release. This could also be used for projects wishing to host files which
cannot be uploaded to PyPI, such as multi-gigabyte data files or, currently at
least, Linux Wheels.</p>
</section>
<section id="why-not-pep-438-or-similar">
<h3><a class="toc-backref" href="#why-not-pep-438-or-similar" role="doc-backlink">Why Not PEP 438 or Similar?</a></h3>
<p>While the additional search location support has existed in pip and setuptools
for quite some time support for <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> has only existed in pip since the 1.4
version, and still has yet to be implemented in setuptools. The design of
<a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> did mean that users still benefited for projects which did not require
external files even with older installers, however for projects which <em>did</em>
require external files, users are still silently being given either potentially
unreliable or, even worse, unsafe files to download. This system is also unique
to Python as it arises out of the history of PyPI, this means that it is almost
certain that this concept will be foreign to most, if not all users, until they
encounter it while attempting to use the Python toolchain.</p>
<p>Additionally, the classification system proposed by <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> has, in practice,
turned out to be extremely confusing to end users, so much so that it is a
position of this PEP that the situation as it stands is completely untenable.
The common pattern for a user with this system is to attempt to install a
project possibly get an error message (or maybe not if the project ever
uploaded something to PyPI but later switched without removing old files), see
that the error message suggests <code class="docutils literal notranslate"><span class="pre">--allow-external</span></code>, they reissue the command
adding that flag most likely getting another error message, see that this time
the error message suggests also adding <code class="docutils literal notranslate"><span class="pre">--allow-unverified</span></code>, and again issue
the command a third time, this time finally getting the thing they wish to
install.</p>
<p>This UX failure exists for several reasons.</p>
<ol class="arabic">
<li>If pip can locate files at all for a project on the Simple API it will
simply use that instead of attempting to locate more. This is generally the
right thing to do as attempting to locate more would erase a large part of
the benefit of <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a>. This means that if a project <em>ever</em> uploaded a file
that matches what the user has requested for install that will be used
regardless of how old it is.</li>
<li><a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> makes an implicit assumption that most projects would either upload
themselves to PyPI or would update themselves to directly linking to release
files. While a large number of projects did ultimately decide to upload to
PyPI, some of them did so only because the UX around what <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> was so bad
that they felt forced to do so. More concerning however, is the fact that
very few projects have opted to directly and safely link to files and
instead they still simply link to pages which must be scraped in order to
find the actual files, thus rendering the safe variant
(<code class="docutils literal notranslate"><span class="pre">--allow-external</span></code>) largely useless.</li>
<li>Even if an author wishes to directly link to their files, doing so safely is
non-obvious. It requires the inclusion of a MD5 hash (for historical
reasons) in the hash of the URL. If they do not include this then their
files will be considered “unverified”.</li>
<li><a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> takes a security centric view and disallows any form of a global opt
in for unverified projects. While this is generally a good thing, it creates
extremely verbose and repetitive command invocations such as:<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ pip install --allow-external myproject --allow-unverified myproject myproject
$ pip install --allow-all-external --allow-unverified myproject myproject
</pre></div>
</div>
</li>
</ol>
</section>
</section>
<section id="multiple-repository-index-support">
<h2><a class="toc-backref" href="#multiple-repository-index-support" role="doc-backlink">Multiple Repository/Index Support</a></h2>
<p>Installers SHOULD implement or continue to offer, the ability to point the
installer at multiple URL locations. The exact mechanisms for a user to
indicate they wish to use an additional location is left up to each individual
implementation.</p>
<p>Additionally the mechanism discovering an installation candidate when multiple
repositories are being used is also up to each individual implementation,
however once configured an implementation should not discourage, warn, or
otherwise cast a negative light upon the use of a repository simply because it
is not the default repository.</p>
<p>Currently both pip and setuptools implement multiple repository support by
using the best installation candidate it can find from either repository,
essentially treating it as if it were one large repository.</p>
<p>Installers SHOULD also implement some mechanism for removing or otherwise
disabling use of the default repository. The exact specifics of how that is
achieved is up to each individual implementation.</p>
<p>Installers SHOULD also implement some mechanism for whitelisting and
blacklisting which projects a user wishes to install from a particular
repository. The exact specifics of how that is achieved is up to each
individual implementation.</p>
<p>The <a class="reference external" href="https://packaging.python.org/">Python packaging guide</a> MUST be updated
with a section detailing the options for setting up their own repository so
that any project that wishes to not host on PyPI in the future can reference
that documentation. This should include the suggestion that projects relying on
hosting their own repositories should document in their project description how
to install their project.</p>
</section>
<section id="deprecation-and-removal-of-link-spidering">
<h2><a class="toc-backref" href="#deprecation-and-removal-of-link-spidering" role="doc-backlink">Deprecation and Removal of Link Spidering</a></h2>
<p>A new hosting mode will be added to PyPI. This hosting mode will be called
<code class="docutils literal notranslate"><span class="pre">pypi-only</span></code> and will be in addition to the three that <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> has already
given us which are <code class="docutils literal notranslate"><span class="pre">pypi-explicit</span></code>, <code class="docutils literal notranslate"><span class="pre">pypi-scrape</span></code>, <code class="docutils literal notranslate"><span class="pre">pypi-scrape-crawl</span></code>.
This new hosting mode will modify a project’s simple api page so that it only
lists the files which are directly hosted on PyPI and will not link to anything
else.</p>
<p>Upon acceptance of this PEP and the addition of the <code class="docutils literal notranslate"><span class="pre">pypi-only</span></code> mode, all new
projects will be defaulted to the PyPI only mode and they will be locked to
this mode and unable to change this particular setting.</p>
<p>An email will then be sent out to all of the projects which are hosted only on
PyPI informing them that in one month their project will be automatically
converted to the <code class="docutils literal notranslate"><span class="pre">pypi-only</span></code> mode. A month after these emails have been sent
any of those projects which were emailed, which still are hosted only on PyPI
will have their mode set permanently to <code class="docutils literal notranslate"><span class="pre">pypi-only</span></code>.</p>
<p>At the same time, an email will be sent to projects which rely on hosting
external to PyPI. This email will warn these projects that externally hosted
files have been deprecated on PyPI and that in 3 months from the time of that
email that all external links will be removed from the installer APIs. This
email <strong>MUST</strong> include instructions for converting their projects to be hosted
on PyPI and <strong>MUST</strong> include links to a script or package that will enable them
to enter their PyPI credentials and package name and have it automatically
download and re-host all of their files on PyPI. This email <strong>MUST</strong> also
include instructions for setting up their own index page. This email must also
contain a link to the Terms of Service for PyPI as many users may have signed
up a long time ago and may not recall what those terms are. Finally this email
must also contain a list of the links registered with PyPI where we were able
to detect an installable file was located.</p>
<p>Two months after the initial email, another email must be sent to any projects
still relying on external hosting. This email will include all of the same
information that the first email contained, except that the removal date will
be one month away instead of three.</p>
<p>Finally a month later all projects will be switched to the <code class="docutils literal notranslate"><span class="pre">pypi-only</span></code> mode
and PyPI will be modified to remove the externally linked files functionality.</p>
</section>
<section id="summary-of-changes">
<h2><a class="toc-backref" href="#summary-of-changes" role="doc-backlink">Summary of Changes</a></h2>
<section id="repository-side">
<h3><a class="toc-backref" href="#repository-side" role="doc-backlink">Repository side</a></h3>
<ol class="arabic simple">
<li>Deprecate and remove the hosting modes as defined by <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a>.</li>
<li>Restrict simple API to only list the files that are contained within the
repository.</li>
</ol>
</section>
<section id="client-side">
<h3><a class="toc-backref" href="#client-side" role="doc-backlink">Client side</a></h3>
<ol class="arabic simple">
<li>Implement multiple repository support.</li>
<li>Implement some mechanism for removing/disabling the default repository.</li>
<li>Deprecate / Remove <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a></li>
</ol>
</section>
</section>
<section id="impact">
<h2><a class="toc-backref" href="#impact" role="doc-backlink">Impact</a></h2>
<p>To determine impact, we’ve looked at all projects using a method of searching
PyPI which is similar to what pip and setuptools use and searched for all
files available on PyPI, safely linked from PyPI, unsafely linked from PyPI,
and finally unsafely available outside of PyPI. When the same file was found
in multiple locations it was deduplicated and only counted it in one location
based on the following preferences: PyPI &gt; Safely Off PyPI &gt; Unsafely Off PyPI.
This gives us the broadest possible definition of impact, it means that any
single file for this project may no longer be visible by default, however that
file could be years old, or it could be a binary file while there is a sdist
available on PyPI. This means that the <em>real</em> impact will likely be much
smaller, but in an attempt not to miscount we take the broadest possible
definition.</p>
<p>At the time of this writing there are 65,232 projects hosted on PyPI and of
those, 59 of them rely on external files that are safely hosted outside of PyPI
and 931 of them rely on external files which are unsafely hosted outside of
PyPI. This shows us that 1.5% of projects will be affected in some way by this
change while 98.5% will continue to function as they always have. In addition,
only 5% of the projects affected are using the features provided by <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> to
safely host outside of PyPI while 95% of them are exposing their users to
Remote Code Execution via a Man In The Middle attack.</p>
</section>
<section id="frequently-asked-questions">
<h2><a class="toc-backref" href="#frequently-asked-questions" role="doc-backlink">Frequently Asked Questions</a></h2>
<section id="i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do">
<h3><a class="toc-backref" href="#i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do" role="doc-backlink">I can’t host my project on PyPI because of &lt;X&gt;, what should I do?</a></h3>
<p>First you should decide if &lt;X&gt; is something inherent to PyPI, or if PyPI could
grow a feature to solve &lt;X&gt; for you. If PyPI can add a feature to enable you to
host your project on PyPI then you should propose that feature. However, if &lt;X&gt;
is something inherent to PyPI, such as wanting to maintain control over your
own files, then you should setup your own package repository and instruct your
users in your project’s description to add it to the list of repositories their
installer of choice will use.</p>
</section>
<section id="my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that">
<h3><a class="toc-backref" href="#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that" role="doc-backlink">My users have a worse experience with this PEP than before, how do I explain that?</a></h3>
<p>Part of this answer is going to be specific to each individual project, you’ll
need to explain to your users what caused you to decide to host in your own
repository instead of utilizing one that they already have in their installer’s
default list of repositories. However, part of this answer will also be
explaining that the previous behavior of transparently including external links
was both a security hazard (given that in most cases it allowed a MITM to
execute arbitrary Python code on the end users machine) and a reliability
concern and that <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> attempted to resolve this by making them explicitly
opt in, but that <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> brought along with it a number of serious usability
issues. <a class="pep reference internal" href="../pep-0470/" title="PEP 470 – Removing External Hosting Support on PyPI">PEP 470</a> represents a simplification of the model to a model that many
users will be familiar with, which is common amongst Linux distributions.</p>
</section>
<section id="switching-to-a-repository-structure-breaks-my-workflow-or-isn-t-allowed-by-my-host">
<h3><a class="toc-backref" href="#switching-to-a-repository-structure-breaks-my-workflow-or-isn-t-allowed-by-my-host" role="doc-backlink">Switching to a repository structure breaks my workflow or isn’t allowed by my host?</a></h3>
<p>There are a number of cheap or free hosts that would gladly support what is
required for a repository. In particular you don’t actually need to upload your
files anywhere differently as long as you can generate a host with the correct
structure that points to where your files are actually located. Many of these
hosts provide free HTTPS using a shared domain name, and free HTTPS
certificates can be gotten from <a class="reference external" href="https://www.startssl.com/">StartSSL</a>, or in
the near future <a class="reference external" href="https://letsencrypt.org/">LetsEncrypt</a> or they may be gotten
cheap from any number of providers.</p>
</section>
<section id="why-don-t-you-provide-x">
<h3><a class="toc-backref" href="#why-don-t-you-provide-x" role="doc-backlink">Why don’t you provide &lt;X&gt;?</a></h3>
<p>The answer here will depend on what &lt;X&gt; is, however the answers typically are
one of:</p>
<ul class="simple">
<li>We hadn’t been thought of it and nobody had suggested it before.</li>
<li>We don’t have sufficient experience with &lt;X&gt; to properly design a solution
for it and would welcome a domain expert to help us provide it.</li>
<li>We’re an open source project and nobody has decided to volunteer to design
and implement &lt;X&gt; yet.</li>
</ul>
<p>Additional PEPs to propose additional features are always welcome, however they
would need someone with the time and expertise to accurately design &lt;X&gt;. This
particular PEP is intended to focus on getting us to a point where the
capabilities of PyPI are straightforward with an easily understood baseline
that is similar to existing models such as Linux distribution repositories.</p>
</section>
<section id="why-should-i-register-on-pypi-if-i-m-running-my-own-repository-anyways">
<h3><a class="toc-backref" href="#why-should-i-register-on-pypi-if-i-m-running-my-own-repository-anyways" role="doc-backlink">Why should I register on PyPI if I’m running my own repository anyways?</a></h3>
<p>PyPI serves two critical functions for the Python ecosystem. One of those is as
a central repository for the actual files that get downloaded and installed by
pip or another package manager and it is this function that this PEP is
concerned with and that you’d be replacing if you’re running your own
repository. However, it also provides a central registry of who owns what name
in order to prevent naming collisions, think of it sort of as DNS but for
Python packages. In addition to making sure that names are handed out in a
first-come, first-served manner it also provides a single place for users to go
to look search for and discover new projects. So the simple answer is, you
should still register your project with PyPI to avoid naming collisions and to
make it so people can still easily discover your project.</p>
</section>
</section>
<section id="rejected-proposals">
<h2><a class="toc-backref" href="#rejected-proposals" role="doc-backlink">Rejected Proposals</a></h2>
<section id="allow-easier-discovery-of-externally-hosted-indexes">
<h3><a class="toc-backref" href="#allow-easier-discovery-of-externally-hosted-indexes" role="doc-backlink">Allow easier discovery of externally hosted indexes</a></h3>
<p>A previous version of this PEP included a new feature added to both PyPI and
installers that would allow project authors to enter into PyPI a list of
URLs that would instruct installers to ignore any files uploaded to PyPI and
instead return an error telling the end user about these extra URLs that they
can add to their installer to make the installation work.</p>
<p>This feature has been removed from the scope of the PEP because it proved too
difficult to develop a solution that avoided UX issues similar to those that
caused so many problems with the <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> solution. If needed, a future PEP
could revisit this idea.</p>
</section>
<section id="keep-the-current-classification-system-but-adjust-the-options">
<h3><a class="toc-backref" href="#keep-the-current-classification-system-but-adjust-the-options" role="doc-backlink">Keep the current classification system but adjust the options</a></h3>
<p>This PEP rejects several related proposals which attempt to fix some of the
usability problems with the current system but while still keeping the general
gist of <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a>.</p>
<p>This includes:</p>
<ul class="simple">
<li>Default to allowing safely externally hosted files, but disallow unsafely
hosted.</li>
<li>Default to disallowing safely externally hosted files with only a global flag
to enable them, but disallow unsafely hosted.</li>
<li>Continue on the suggested path of <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> and remove the option to unsafely
host externally but continue to allow the option to safely host externally.</li>
</ul>
<p>These proposals are rejected because:</p>
<ul class="simple">
<li>The classification system introduced in <a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> in an entirely unique concept
to PyPI which is not generically applicable even in the context of Python
packaging. Adding additional concepts comes at a cost.</li>
<li>The classification system itself is non-obvious to explain and to
pre-determine what classification of link a project will require entails
inspecting the project’s <code class="docutils literal notranslate"><span class="pre">/simple/&lt;project&gt;/</span></code> page, and possibly any URLs
linked from that page.</li>
<li>The ability to host externally while still being linked for automatic
discovery is mostly a historic relic which causes a fair amount of pain and
complexity for little reward.</li>
<li>The installer’s ability to optimize or clean up the user interface is limited
due to the nature of the implicit link scraping which would need to be done.
This extends to the <code class="docutils literal notranslate"><span class="pre">--allow-*</span></code> options as well as the inability to
determine if a link is expected to fail or not.</li>
<li>The mechanism paints a very broad brush when enabling an option, while
<a class="pep reference internal" href="../pep-0438/" title="PEP 438 – Transitioning to release-file hosting on PyPI">PEP 438</a> attempts to limit this with per package options. However a project
that has existed for an extended period of time may oftentimes have several
different URLs listed in their simple index. It is not unusual for at least
one of these to no longer be under control of the project. While an
unregistered domain will sit there relatively harmless most of the time, pip
will continue to attempt to install from it on every discovery phase. This
means that an attacker simply needs to look at projects which rely on unsafe
external URLs and register expired domains to attack users.</li>
</ul>
</section>
<section id="implement-this-pep-but-do-not-remove-the-existing-links">
<h3><a class="toc-backref" href="#implement-this-pep-but-do-not-remove-the-existing-links" role="doc-backlink">Implement this PEP, but Do Not Remove the Existing Links</a></h3>
<p>This is essentially the backwards compatible version of this PEP. It attempts
to allow people using older clients, or clients which do not implement this
PEP to continue on as if nothing had changed. This proposal is rejected because
the vast bulk of those scenarios are unsafe uses of the deprecated features. It
is the opinion of this PEP that silently allowing unsafe actions to take place
on behalf of end users is simply not an acceptable solution.</p>
</section>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0470.rst">https://github.com/python/peps/blob/main/peps/pep-0470.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0470.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#key-user-experience-expectations">Key User Experience Expectations</a></li>
<li><a class="reference internal" href="#why-additional-repositories">Why Additional Repositories?</a></li>
<li><a class="reference internal" href="#why-not-pep-438-or-similar">Why Not PEP 438 or Similar?</a></li>
</ul>
</li>
<li><a class="reference internal" href="#multiple-repository-index-support">Multiple Repository/Index Support</a></li>
<li><a class="reference internal" href="#deprecation-and-removal-of-link-spidering">Deprecation and Removal of Link Spidering</a></li>
<li><a class="reference internal" href="#summary-of-changes">Summary of Changes</a><ul>
<li><a class="reference internal" href="#repository-side">Repository side</a></li>
<li><a class="reference internal" href="#client-side">Client side</a></li>
</ul>
</li>
<li><a class="reference internal" href="#impact">Impact</a></li>
<li><a class="reference internal" href="#frequently-asked-questions">Frequently Asked Questions</a><ul>
<li><a class="reference internal" href="#i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do">I can’t host my project on PyPI because of &lt;X&gt;, what should I do?</a></li>
<li><a class="reference internal" href="#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that">My users have a worse experience with this PEP than before, how do I explain that?</a></li>
<li><a class="reference internal" href="#switching-to-a-repository-structure-breaks-my-workflow-or-isn-t-allowed-by-my-host">Switching to a repository structure breaks my workflow or isn’t allowed by my host?</a></li>
<li><a class="reference internal" href="#why-don-t-you-provide-x">Why don’t you provide &lt;X&gt;?</a></li>
<li><a class="reference internal" href="#why-should-i-register-on-pypi-if-i-m-running-my-own-repository-anyways">Why should I register on PyPI if I’m running my own repository anyways?</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rejected-proposals">Rejected Proposals</a><ul>
<li><a class="reference internal" href="#allow-easier-discovery-of-externally-hosted-indexes">Allow easier discovery of externally hosted indexes</a></li>
<li><a class="reference internal" href="#keep-the-current-classification-system-but-adjust-the-options">Keep the current classification system but adjust the options</a></li>
<li><a class="reference internal" href="#implement-this-pep-but-do-not-remove-the-existing-links">Implement this PEP, but Do Not Remove the Existing Links</a></li>
</ul>
</li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0470.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink