PEP: 439 Title: Inclusion of pip bootstrap in Python installation Version: $Revision$ Last-Modified: $Date$ Author: Richard Jones BDFL-Delegate: Nick Coghlan Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013 Abstract ======== This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users. This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEP 427 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP. Rationale ========= Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress. Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users. With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software. It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades. Proposal ======== This proposal affects three components of packaging: `the pip bootstrap`_, `setuptools`_ and, thanks to easier package installation, `modifications to publishing packages`_. The pip bootstrap ----------------- The Python installation includes an executable called "pip3" (see PEP 394 for naming rationale etc.) that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal. Once the bootstrap process is complete the "pip3" command is no longer the bootstrap but rather the full pip command. A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also pip is upgradeable outside of the regular Python upgrade timeframe and processes. To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment [1]_ then the pip implementation will be installed into that virtual environment. The bootstrap process will proceed as follows: 1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip3". 2. The user will invoke a pip command, typically "pip3 install ", for example "pip3 install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip3 has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally. Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip3 install" command. This use case will be handled by: 1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually. Some users may have no Internet access suitable for fetching the pip implementation file. Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip3 install ". This installation - since it uses only the bootstrap code - will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files. The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed, and therefore the download would still be vulnerable to active MITM attacks. To mitigate this risk we will use the embedded signature support in the wheel format to validate the downloaded file. Beyond those arguments controlling index location and download options, the "pip3" boostrap command may support further standard pip options for verbosity, quietness and logging. The "--no-install" option to the "pip3" command will not affect the bootstrapping process. setuptools ---------- The deprecation of requiring setuptools for installation is an existing goal of the packaging comminity (TODO ref needed). Currently pip depends upon setuptools functionality, and it is installed by the current pip boostrap. This PEP does not propose installing setuptools during the new bootstrap. It is intended that before Python 3.4 is shipped the functionlity required by pip will be present in Python's standard library as the distlib module, and that pip would be modified to use that functionality when present. TODO PEP reference for distlib Many existing "setup.py" files require setuptools to be installed (because one of the first things they do is import setuptools). It is intended that pip's behaviour will be either: 1. If setuptools is not present it can only install from wheel files and sdists with 2.0+ metadata, or 2. If setuptools is present it can also install from sdists with legacy metadata and eggs By default, installing setuptools when necessary should be automatic so that users are not inconvenienced, but advanced users should be able to ask that it instead be treated as an error if no wheel is available to satisfy an installation request or dependency (so they don't inadvertently install setuptools on their production systems if they don't want to). Modifications to publishing packages ------------------------------------ An additional new Python package is proposed, "pypublish", which will be a tool for publishing packages to PyPI. It would replace the current "python setup.py register" and "python setup.py upload" distutils commands. Again because of the measured Python release cycle and extensive existing Python installations these commands are difficult to bugfix and extend. Additionally it is desired that the "register" and "upload" commands be able to be performed over HTTPS with certificate validation. Since shipping CA certificate keychains with Python is not really feasible (updating the keychain is quite difficult to manage) it is desirable that those commands, and the accompanying keychain, be made installable and upgradeable outside of Python itself. The existing distutils mechanisms for package registration and upload would remain, though with a deprecation warning. Implementation ============== The changes to pip required by this PEP are being tracked in that project's issue tracker [2]_ Risks ===== The key that is used to sign the pip implementation download might be compromised and this PEP currently proposes no mechanism for key revocation. There is a Perl package installer also named "pip". It is quite rare and not commonly used. The Fedora variant of Linux has historically named Python's "pip" as "python-pip" and Perl's "pip" as "perl-pip". This policy has been altered[3] so that future and upgraded Fedora installations will use the name "pip" for Python's "pip". Existing (non-upgraded) installations will still have the old name for the Python "pip", though the potential for confusion is now much reduced. References ========== .. [1] PEP 405, Python Virtual Environments http://www.python.org/dev/peps/pep-0405/ .. [2] pip issue tracking work needed for this PEP https://github.com/pypa/pip/issues/863 .. [3] Fedora's python-pip package does not provide /usr/bin/pip https://bugzilla.redhat.com/show_bug.cgi?id=958377 Acknowledgments =============== Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat issue. Jannis Leidel and Carl Meyer for their thoughts. Marcus Smith for feedback. Marcela Mašláňová for resolving the Fedora issue. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: