python-peps/pep-0453.txt

PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald@stufft.io>,
        Nick Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Martin von Löwis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013


Abstract
========

This PEP proposes that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module (including via the
``pyvenv`` command line utility).

To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.

The PEP also strongly recommends that CPython redistributors and other Python
implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.


Proposal
========

This PEP proposes the inclusion of a ``getpip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.

This PEP does *not* propose making pip (or any dependencies) part of the
standard library. Instead, pip will be a bundled application provided
along with CPython for the convenience of Python users, but subject to
its own development life cycle and able to be upgraded independently of
the core interpreter and standard library.


Rationale
=========

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* assume the user already has a suitable cross-platform package manager
  installed
* duplicate the instructions and tell their users how to install the
  package manager
* completely forgo the use of dependencies to ease installation concerns
  for their users

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on Linux systems with platform package managers. No such assistance is
availabe for Windows and Mac OS X users. The challenges of dealing with
this problem are a regular feature of feedback the core Python developers
receive from professional educators and others introducing new users to
Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing a cross-platform package manager by default it will be easier
for users trying to install these third-party packages as well as easier
for the people distributing them as they should now be able to safely assume
that most users will have the appropriate installation tools available.
This is expected to become more important in the future as the Wheel_
package format (deliberately) does not have a built in "installer" in the
form of ``setup.py`` so users wishing to install from a wheel file will want
an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box instead of
using the general difficulty of installing third-party packages as
justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.


Why pip?
--------

``pip`` has been chosen as the preferred default installer, as it
addresses several design and user experience issues with its predecessor
``easy_install`` (these issues can't readily be fixed in ``easy_install``
itself due to backwards compatibility concerns). ``pip`` is also well suited
to working within the bounds of a single Python runtime installation
(including associated virtual environments), which is a desirable feature
for a tool bundled with CPython.

Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to interoperate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).


Explicit bootstrapping mechanism
================================

An additional module called ``getpip`` will be added to the standard library
whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a single
callable named ``bootstrap()`` as well as offer direct execution via
``python -m getpip``. Options for installing it such as index server,
installation location (``--user``, ``--root``, etc) will also be available
to enable different installation schemes.

It is believed that users will want the most recent versions available to be
installed so that they can take advantage of the new advances in packaging.
Since any particular version of Python has a much longer staying power than
a version of pip in order to satisfy a user's desire to have the most recent
version the bootstrap will (by default) contact PyPI, find the latest
version, download it, and then install it. This process is security
sensitive, difficult to get right, and evolves along with the rest of
packaging.

Instead of attempting to maintain a "mini pip" for the sole purpose of
installing pip, the ``getpip`` module will, as an implementation detail,
include a private copy of pip and its dependencies which will be used to
discover and install pip from PyPI. It is important to stress that this
private copy of pip is *only* an implementation detail and it should *not*
be relied on or assumed to exist.

Not all users will have network access to PyPI whenever they run the
bootstrap. In order to ensure that these users will still be able to
bootstrap pip the bootstrap will fallback to simply installing the included
copy of pip. The pip ``--no-download`` command line option will be supported
to force installation of the bundled version, without even attempting to
contact PyPI.

This presents a balance between giving users the latest version of pip,
saving them from needing to immediately upgrade pip after bootstrapping it,
and allowing the bootstrap to work offline in situations where users might
already have packages downloaded that they wish to install.


Proposed CLI
------------

The proposed CLI is based on a subset of the existing ``pip install``
options::

    Usage:
      python -m getpip [options]

    Download Options:
      --no-download           Install the bundled version, don't attempt to download
      -i, --index-url <url>   Base URL of Python Package Index (default https://pypi.python.org/simple/).
      --proxy <proxy>         Specify a proxy in the form [user:passwd@]proxy.server:port.
      --timeout <sec>         Set the socket timeout (default 15 seconds).
      --cert <path>           Path to alternate CA bundle.

    Installation Options:
      -U, --upgrade           Upgrade pip and dependencies, even if already installed
      --user                  Install using the user scheme.
      --root <dir>            Install everything relative to this alternate root directory.

Additional options (such as verbosity and logging options) may also
be supported.


Proposed module API
-------------------

The proposed ``getpip`` module API is a single ``bootstrap`` function with
parameter names derived directly from the proposed CLI::

    def bootstrap(download=True, upgrade=False, root=None, user=False,
                  index_url=None, cert=None, proxy=None, timeout=15):
        """Bootstrap pip into the current Python installation (or the given
           root directory)"""

The only changes are to replace the ``--no-download`` opt-out option with
the True-by-default ``download`` option and to replace the hyphen in
``index-url`` with an underscore to create a legal Python identifier.


Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain two new options:

* Install pip (the default Python package management utility)?
* Upgrade pip to the latest version (requires network access)?

Both options will be checked by default, with the option to upgrade pip
being available for selection only if the option to install pip is checked.

If both options are checked, then the installer will invoke the following
command with the just installed Python::

    python -m getpip --upgrade

If only the "Install pip" option is checked, then the following command will
be invoked::

    python -m getpip --upgrade --no-download

This ensures that, by default, installing or updating CPython will ensure
that either the latest available version of PyPI is installed (directly from
PyPI if permitted, otherwise whichever is more recent out of an already
installed version and the private copy inside ``getpip``)


Installing from source
----------------------

While the prebuilt binary installers will be updated to run
``python -m getpip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source distribution.

``getpip`` itself will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.

Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``getpip`` invocation also ensures those installing from
a custom source build can easily force an offline installation of pip,
install it from a private index server, or skip installing pip entirely.


Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since it's release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.

To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement. To handle cases where a user does not wish to have pip
bootstrapped into their virtual environment a ``--without-pip`` option will be
added. The ``--no-download`` option will also be supported, to force the
use of the bundled ``pip`` rather than retrieving the latest version from
PyPI.

The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
two new parameters: ``with_pip`` (defaulting to ``False``) and
``bootstrap_options`` (accepting a dictionary of keyword arguments to
pass to ``getpip.bootstrap`` if ``with_pip`` is set,  defaulting to
``None``).

This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.


Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation will be updated to recommend the use of the bootstrapped
`pip` installer. It will give a brief description of the most common
commands and options, but delegate to the externally maintained ``pip``
documentation for the full details.

The existing content of the module installation guide will be retained,
but under a new "Invoking distutils directly" subsection.


Bundling CA certificates with CPython
-------------------------------------

The reference ``getpip`` implementation includes the ``pip`` CA
bundle along with the rest of pip. This means CPython effectively includes
a CA bundle that is used solely for ``getpip``.

This is considered desirable, as it ensures that ``pip`` will behave the
same across all supported versions of Python, even those prior to Python
3.4 that cannot access the system certificate store on Windows.


Automatic installation of setuptools
------------------------------------

``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
``getpip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``pip``). In normal operation, ``python -m getpip``
will then download and install the latest version of ``setuptools`` from
PyPI (as a dependency of ``pip``), while ``python -m getpip --no-download``
will install the private copy.

However, this behaviour is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.

Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``getpip``.


Updating the bundled pip
------------------------

In order to keep up with evolutions in packaging as well as providing users
who are using the offline installation method with as recent version a
possible the ``getpip`` module will be regularly updated to the latest
versions of everything it bootstraps.

After each new pip release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.


Updating the getpip module API and CLI
--------------------------------------

Future security updates for pip and PyPI (for example, automatic
verification of package signatures) may also provide desirable security
enhancements for the ``getpip`` bootstrapping mechanism.

It is desirable that these features be made available in standard library
maintenance releases, not just new feature releases.

Accordingly, a slight relaxation of the usual "no new features in
maintenance releases" rule is proposed for the ``getpip`` module. This
relaxation also indirectly affects the new ``bootstrap_options`` parameter
in the ``venv`` module APIs.

Specifically, new security related flags will be permitted, with the
following restrictions:

- for compatibility with third-party usage of ``getpip`` module (for
  example, with a private index server), any such flag must be *off* by
  default in maintenance releases. It *should* be switched on by
  default in the next feature release.
- the CPython installers and the ``pyvenv`` CLI in the affected maintenance
  release should explicitly opt-in to the enhanced security features when
  automatically bootstrapping ``pip``

This means that maintenance releases of the CPython installers will
benefit from security enhancements by default, while avoiding breaking
customised usage of the bootstrap mechanism.


Feature addition in maintenance releases
========================================

Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases".

It is being proposed in this case as the current bootstrapping issues for
the third-party Python package ecosystem greatly affects the experience of
new users, especially on Python 2 where many Python 3 standard library
improvements are available as backports on PyPI, but are not included in
the Python 2 standard library.

By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this change should aid the vast majority of current Python users, rather
than only those with the freedom to adopt Python 3.4 as soon as it is
released.

This is also a matter of starting as we mean to continue: as noted above,
``getpip`` will have a limited permanent exemption from the "no new
features in maintenance releases" restriction, as it will include (and
rely on) upgraded private copies of ``pip`` and ``setuptools`` even in
maintenance releases, and may offer new security related options itself.


Open Question: Uninstallation
=============================

No changes are currently proposed to the uninstallation process. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.

.. note::

   Perhaps the installer should be updated to clobber everything in
   site-packages and the Scripts directory when uninstalled (treating them
   as "data directories" from Python's point of view), but I would prefer
   not to make this PEP conditional on that change.


Open Question: Script Execution on Windows
==========================================

While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. This PEP proposes that this installer option
be changed to also add the Scripts directory to PATH (either always, or
else as a checked by default suboption).

Without this change, the most reliable way to invoke pip on Windows (without
tinkering manually with PATH) is actually ``py -m pip`` (or ``py -3 -m pip``
to select the Python 3 version if both Python 2 and 3 are installed)
rather than simply calling ``pip``.

Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip`` needed only to select a non-default version in
the parallel installation case (and outside a virtual environment).

While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).


Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
to provide a consistent, user-friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
downstream distributors:

* Ensure that whenever Python is installed pip is also installed.

  * This may take the form of separate packages with dependencies on each
    other so that installing the Python package installs the pip package
    and installing the pip package installs the Python package.

* Do not remove the bundled copy of pip.

  * This is required for offline installation of pip into a virtual
    environment by the ``venv`` module.
  * This is similar to the existing ``virtualenv`` package for which many
    downstream distributors have already made exception to the common
    "debundling" policy.
  * This does mean that if ``pip`` needs to be updated due to a security
    issue, so does the bundled version in the ``getpip`` bootstrap module
  * However, altering the bundled version of pip to remove the embedded
    CA certificate bundle and rely the system CA bundle instead is a
    reasonable change.

* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
  using ``setup.py``.

  * This will ensure that downstream packages can more easily utilize the
    new metadata formats which may not have a ``setup.py``.

* Ensure that all features of this PEP continue to work with any modifications
  made.

  * Online installation of the latest version of pip into a global or virtual
    python environment using ``python -m getpip``.
  * Offline installation of the bundled version of pip into a global or virtual
    python environment using ``python -m getpip``.
  * ``pip install --upgrade pip`` in a global installation should not affect
    any already created virtual environments.
  * ``pip install --upgrade pip`` in a virtual environment should not affect
    the global installation.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.

Other Python implementations are also encouraged to follow these guidelines
where applicable.


Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.


Backwards Compatibility
-----------------------

Except for security enhancements (as noted above), the public API of the
``getpip`` module itself will fall under the typical backwards compatibility
policy of Python for its standard library. The externally developed software
that this PEP bundles does not.

Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.


Security Releases
-----------------

Any security update that affects the ``getpip`` module will be shared prior to
release with the Python Security Response Team (security@python.org). The
PSRT will then decide if the reported issue warrants a security release of
CPython.


Appendix: Rejected Proposals
============================


Implicit bootstrap
------------------

`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.


Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned through the inclusion and history of
``distutils`` in the standard library that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a timeframe that actually
affects users as any new features will not be available to the general
population for *years*.

Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.

There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.

Finally, the approach described in this PEP avoids some technical issues
related to handle CPython maintenance updates when pip has been independently
updated to a more recent version. The proposed pip-based bootstrapping
mechanism handles that automatically, since pip and the system installer
never get into a fight about who owns the pip installation (it is always
managed through pip, either directly, or indirectly via the getpip bootstrap
module).


Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behaviour would be
surprising (as it differs from the default behaviour of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).


.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/


References
==========

.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew  <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: