PEP 470: use external indexes over link spidering
This commit is contained in:
Nick Coghlan
parent
65da967dc2
commit
08ad02e889
|
|
@ -0,0 +1,377 @@
|
|||
PEP: 470
|
||||
Title: Using Multi Index Support for External to PyPI Package File Hosting
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Donald Stufft <donald@stufft.io>,
|
||||
BDFL-Delegate: Richard Jones <richard@python.org>
|
||||
Discussions-To: distutils-sig@python.org
|
||||
Status: Draft
|
||||
Type: Process
|
||||
Content-Type: text/x-rst
|
||||
Created: 12-May-2014
|
||||
Post-History: 14-May-2014
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes that the official means of having an installer locate and
|
||||
find package files which are hosted externally to PyPI become the use of
|
||||
multi index support instead of the practice of using external links on the
|
||||
simple installer API.
|
||||
|
||||
It is important to remember that this is **not** about forcing anyone to host
|
||||
their files on PyPI. If someone does not wish to do so they will never be under
|
||||
any obligation too. They can still list their project in PyPI as an index, and
|
||||
the tooling will still allow them to host it elsewhere.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
There is a long history documented in PEP 438 that explains why externally
|
||||
hosted files exist today in the state that they do on PyPI. For the sake of
|
||||
brevity I will not duplicate that and instead urge readers to first take a look
|
||||
at PEP 438 for background.
|
||||
|
||||
There are currently two primary ways for a project to make itself available
|
||||
without directly hosting the package files on PyPI. They can either include
|
||||
links to the package files in the simpler installer API or they can publish
|
||||
a custom package index which contains their project.
|
||||
|
||||
|
||||
Custom Additional Index
|
||||
-----------------------
|
||||
|
||||
Each installer which speaks to PyPI offers a mechanism for the user invoking
|
||||
that installer to provide additional custom locations to search for files
|
||||
during the dependency resolution phase. For pip these locations can be
|
||||
configured per invocation, per shell environment, per requirements file, per
|
||||
virtual environment, and per user.
|
||||
|
||||
The use of additional indexes instead of external links on the simple
|
||||
installer API provides a simple clean interface which is consistent with the
|
||||
way most Linux package systems work (apt-get, yum, etc). More importantly it
|
||||
works the same even for projects which are commercial or otherwise have their
|
||||
access restricted in some form (private networks, password, IP ACLs etc)
|
||||
while the external links method only realistically works for projects which
|
||||
do not have their access restricted.
|
||||
|
||||
Compared to the complex rules which a project must be aware of to prevent
|
||||
themselves from being considered unsafely hosted setting up an index is fairly
|
||||
trivial and in the simplest case does not require anything more than a
|
||||
filesystem and a standard web server such as Nginx. Even if using simple
|
||||
static hosting without autoindexing support, it is still straightforward
|
||||
to generate appropriate index pages as static HTML.
|
||||
|
||||
Example Index with Nginx
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Create a root directory for your index, for the purposes of the example
|
||||
I'll assume you've chosen ``/var/www/index.example.com/``.
|
||||
2. Inside of this root directory, create a directory for each project such
|
||||
as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
|
||||
3. Place the package files for each project in their respective folder,
|
||||
creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
|
||||
4. Configure nginx to serve the root directory, ideally with TLS, with the
|
||||
autoindex directive enable (see below for example configuration).
|
||||
|
||||
::
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name index.example.com;
|
||||
|
||||
ssl_certificate /etc/pki/tls/certs/index.example.com.crt;
|
||||
ssl_certificate_key /etc/pki/tls/certs/index.example.com.key;
|
||||
|
||||
root /var/www/index.example.com;
|
||||
|
||||
autoindex on;
|
||||
}
|
||||
|
||||
|
||||
Examples of Additional indexes with pip
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
**Invocation:**
|
||||
|
||||
::
|
||||
$ pip install --extra-index-url https://pypi.example.com/ foobar
|
||||
|
||||
**Shell Environment:**
|
||||
|
||||
::
|
||||
$ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
|
||||
$ pip install foobar
|
||||
|
||||
**Requirements File:**
|
||||
|
||||
::
|
||||
$ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
|
||||
$ pip install -r requirements.txt
|
||||
|
||||
**Virtual Environment:**
|
||||
|
||||
::
|
||||
$ python -m venv myvenv
|
||||
$ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf
|
||||
$ myvenv/bin/pip install foobar
|
||||
|
||||
**User:**
|
||||
|
||||
::
|
||||
$ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf
|
||||
$ pip install foobar
|
||||
|
||||
|
||||
External Links on the Simple Installer API
|
||||
------------------------------------------
|
||||
|
||||
PEP438 proposed a system of classifying file links as either internal,
|
||||
external, or unsafe. It recommended that by default only internal links would
|
||||
be installed by an installer however users could opt into external links on
|
||||
either a global or a per package basis. Additionally they could also opt into
|
||||
unsafe links on a per package basis.
|
||||
|
||||
This system has turned out to be *extremely* unfriendly towards the end users
|
||||
and it is the position of this PEP that the situation has become untenable. The
|
||||
situation as provided by PEP438 requires an end user to be aware not only of
|
||||
the difference between internal, external, and unsafe, but also to be aware of
|
||||
what hosting mode the package they are trying to install is in, what links are
|
||||
available on that project's /simple/ page, whether or not those links have
|
||||
a properly formatted hash fragment, and what links are available from pages
|
||||
linked to from that project's /simple/ page.
|
||||
|
||||
There are a number of common confusion/pain points with this system that I
|
||||
have witnessed:
|
||||
|
||||
* Users unaware what the simple installer api is at all or how an installer
|
||||
locates installable files.
|
||||
* Users unaware that even if the simple api links to a file, if it does
|
||||
not include a ``#md5=...`` fragment that it will be counted as unsafe.
|
||||
* Users unaware that an installer can look at pages linked from the
|
||||
simple api to determine additional links, or that any links found in this
|
||||
fashion are considered unsafe.
|
||||
* Users are unaware and often surprised that PyPI supports hosting your files
|
||||
someplace other than PyPI at all.
|
||||
|
||||
In addition to that, the information that an installer is able to provide
|
||||
when an installation fails is pretty minimal. We are able to detect if there
|
||||
are externally hosted files directly linked from the simple installer api,
|
||||
however we cannot detect if there are files hosted on a linked page without
|
||||
fetching that page and doing so would cause a massive performance hit just to
|
||||
see if there might be a file there so that a better error message could be
|
||||
provided.
|
||||
|
||||
Finally very few projects have properly linked to their external files so that
|
||||
they can be safely downloaded and verified. At the time of this writing there
|
||||
are a total of 65 projects which have files that are only available externally
|
||||
and are safely hosted.
|
||||
|
||||
The end result of all of this, is that with PEP 438, when a user attempts to
|
||||
install a file that is not hosted on PyPI typically the steps they follow are:
|
||||
|
||||
1. First, they attempt to install it normally, using ``pip install foobar``.
|
||||
This fails because the file is not hosted on PyPI and PEP 438 has us default
|
||||
to only hosted on PyPI. If pip detected any externally hosted files or other
|
||||
pages that we *could* have attempted to find other files at it will give an
|
||||
error message suggesting that they try ``--allow-external foobar``.
|
||||
2. They then attempt to install their package using
|
||||
``pip install --allow-external foobar foobar``. If they are lucky foobar is
|
||||
one of the packages which is hosted externally and safely and this will
|
||||
succeed. If they are unlucky they will get a different error message
|
||||
suggesting that they *also* try ``--allow-unverified foobar``.
|
||||
3. They then attempt to install their package using
|
||||
``pip install --allow-external foobar --allow-unverified foobar foobar``
|
||||
and this finally works.
|
||||
|
||||
This is the same basic steps that practically everyone goes through every time
|
||||
they try to install something that is not hosted on PyPI. If they are lucky it'll
|
||||
only take them two steps, but typically it requires three steps. Worse there is
|
||||
no real indication to these people why one package might install after two
|
||||
but most require three. Even worse than that most of them will never get an
|
||||
externally hosted package that does not take three steps, so they will be
|
||||
increasingly annoyed and frustrated at the intermediate step and will likely
|
||||
eventually just start skipping it.
|
||||
|
||||
|
||||
External Index Discovery
|
||||
========================
|
||||
|
||||
One of the problems with using an additional index is one of discovery. Users
|
||||
will not generally be aware that an additional index is required at all much
|
||||
less where that index can be found. Projects can attempt to convey this
|
||||
information using their description on the PyPI page however that excludes
|
||||
people who discover their project organically through ``pip search``.
|
||||
|
||||
To support projects that wish to externally host their files and to enable
|
||||
users to easily discover what additional indexes are required, PyPI will gain
|
||||
the ability for projects to register external index URLs and additionally an
|
||||
associated comment for each. These URLs will be made available on the simple
|
||||
page however they will not be linked or provided in a form that older
|
||||
installers will automatically search them.
|
||||
|
||||
When an installer fetches the simple page for a project, if it finds this
|
||||
additional meta-data and it cannot find any files for that project in it's
|
||||
configured URLs then it should use this data to tell the user how to add one
|
||||
or more of the additional URLs to search in. This message should include any
|
||||
comments that the project has included to enable them to communicate to the
|
||||
user and provide hints as to which URL they might want if some are only
|
||||
useful or compatible with certain platforms or situations.
|
||||
|
||||
This feature *must* be added to PyPI prior to starting the deprecation and
|
||||
removal process for link spidering.
|
||||
|
||||
|
||||
Deprecation and Removal of Link Spidering
|
||||
=========================================
|
||||
|
||||
A new hosting mode will be added to PyPI. This hosting mode will be called
|
||||
``pypi-only`` and will be in addition to the three that PEP438 has already given
|
||||
us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This
|
||||
new hosting mode will modify a project's simple api page so that it only lists
|
||||
the files which are directly hosted on PyPI and will not link to anything else.
|
||||
|
||||
Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
|
||||
projects will by defaulted to the PyPI only mode and they will be locked to
|
||||
this mode and unable to change this particular setting. ``pypi-only`` projects
|
||||
will still be able to register external index URLs as described above - the
|
||||
"pypi-only" refers only to the download links that are published directly on
|
||||
PyPI.
|
||||
|
||||
An email will then be sent out to all of the projects which are hosted only on
|
||||
PyPI informing them that in one month their project will be automatically
|
||||
converted to the ``pypi-only`` mode. A month after these emails have been sent
|
||||
any of those projects which were emailed, which still are hosted only on PyPI
|
||||
will have their mode set to ``pypi-only``.
|
||||
|
||||
After that switch, an email will be sent to projects which rely on hosting
|
||||
external to PyPI. This email will warn these projects that externally hosted
|
||||
files have been deprecated on PyPI and that in 6 months from the time of that
|
||||
email that all external links will be removed from the installer APIs. This
|
||||
email *must* include instructions for converting their projects to be hosted
|
||||
on PyPI and *must* include links to a script or package that will enable them
|
||||
to enter their PyPI credentials and package name and have it automatically
|
||||
download and re-host all of their files on PyPI. This email *must also*
|
||||
include instructions for setting up their own index page and registering that
|
||||
with PyPI.
|
||||
|
||||
Five months after the initial email, another email must be sent to any projects
|
||||
still relying on external hosting. This email will include all of the same
|
||||
information that the first email contained, except that the removal date will
|
||||
be one month away instead of six.
|
||||
|
||||
Finally a month later all projects will be switched to the ``pypa-only`` mode
|
||||
and PyPI will be modified to remove the externally linked files functionality.
|
||||
|
||||
|
||||
Impact
|
||||
======
|
||||
|
||||
============ ======= ========== =======
|
||||
\ PyPI External Total
|
||||
============ ======= ========== =======
|
||||
**Safe** 37779 65 37844
|
||||
**Unsafe** 0 2974 2974
|
||||
**Total** 37779 3039
|
||||
============ ======= ========== =======
|
||||
|
||||
|
||||
Rejected Proposals
|
||||
==================
|
||||
|
||||
Keep the current classification system but adjust the options
|
||||
-------------------------------------------------------------
|
||||
|
||||
This PEP rejects several related proposals which attempt to fix some of the
|
||||
usability problems with the current system but while still keeping the
|
||||
general gist of PEP 438.
|
||||
|
||||
This includes:
|
||||
|
||||
* Default to allowing safely externally hosted files, but disallow unsafely
|
||||
hosted.
|
||||
* Default to disallowing safely externally hosted files with only a global
|
||||
flag to enable them, but disallow unsafely hosted.
|
||||
|
||||
These proposals are rejected because:
|
||||
|
||||
* The classification "system" is complex, hard to explain, and requires an
|
||||
intimate knowledge of how the simple API works in order to be able to reason
|
||||
about which classification is required. This is reflected in the fact that
|
||||
the code to implement it is complicated and hard to understand as well.
|
||||
|
||||
* People are generally surprised that PyPI allows externally linking to files
|
||||
and doesn't require people to host on PyPI. In contrast most of them are
|
||||
familiar with the concept of multiple software repositories such as is in
|
||||
use by many OSs.
|
||||
|
||||
* PyPI is fronted by a globally distributed CDN which has improved the
|
||||
reliability and speed for end users. It is unlikely that any particular
|
||||
external host has something comparable. This can lead to extremely bad
|
||||
performance for end users when the external host is located in different
|
||||
parts of the world or does not generally have good connectivity.
|
||||
|
||||
As a data point, many users reported sub DSL speeds and latency when
|
||||
accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
|
||||
|
||||
* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
|
||||
downtime quickly, thus enabling a quicker response to downtime. Again it is
|
||||
unlikely that any particular external host will have this. This can lead
|
||||
to single packages in a dependency chain being un-installable. This will
|
||||
often confuse users, who often times have no idea that this package relies
|
||||
on an external host, and they cannot figure out why PyPI appears to be up
|
||||
but the installer cannot find a package.
|
||||
|
||||
* PyPI supports mirroring, both for private organizations and public mirrors.
|
||||
The legal terms of uploading to PyPI ensure that mirror operators, both
|
||||
public and private, have the right to distribute the software found on PyPI.
|
||||
However software that is hosted externally does not have this, causing
|
||||
private organizations to need to investigate each package individually and
|
||||
manually to determine if the license allows them to mirror it.
|
||||
|
||||
For public mirrors this essentially means that these externally hosted
|
||||
packages *cannot* be reasonably mirrored. This is particularly troublesome
|
||||
in countries such as China where the bandwidth to outside of China is
|
||||
highly congested making a mirror within China often times a massively better
|
||||
experience.
|
||||
|
||||
* Installers have no method to determine if they should expect any particular
|
||||
URL to be available or not. It is not unusual for the simple API to reference
|
||||
old packages and URLs which have long since stopped working. This causes
|
||||
installers to have to assume that it is OK for any particular URL to not be
|
||||
accessible. This causes problems where an URL is temporarily down or
|
||||
otherwise unavailable (a common cause of this is using a copy of Python
|
||||
linked against a really ancient copy of OpenSSL which is unable to verify
|
||||
the SSL certificate on PyPI) but it *should* be expected to be up. In this
|
||||
case installers will typically silently ignore this URL and later the user
|
||||
will get a confusing error stating that the installer couldn't find any
|
||||
versions instead of getting the real error message indicating that the URL
|
||||
was unavailable.
|
||||
|
||||
* In the long run, global opt in flags like ``--allow-all-external`` will
|
||||
become little annoyances that developers cargo cult around in order to make
|
||||
their installer work. When they run into a project that requires it they
|
||||
will most likely simply add it to their configuration file for that installer
|
||||
and continue on with whatever they were actually trying to do. This will
|
||||
continue until they try to install their requirements on another computer
|
||||
or attempt to deploy to a server where their install will fail again until
|
||||
they add the "make it work" flag in their configuration file.
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue