PEP 650: Specifying Installer Requirements for Python Projects (#1762)
This commit is contained in:
Brett Cannon
GitHub
parent
19af5f8f0d
commit
67ab198c25
6
Makefile
6
Makefile
|
|
@ -7,6 +7,8 @@ PEP2HTML=pep2html.py
|
|||
|
||||
PYTHON=python3
|
||||
|
||||
VENV_DIR=venv
|
||||
|
||||
.SUFFIXES: .txt .html .rst
|
||||
|
||||
.txt.html:
|
||||
|
|
@ -40,8 +42,8 @@ update:
|
|||
git pull https://github.com/python/peps.git
|
||||
|
||||
venv:
|
||||
$(PYTHON) -m venv venv
|
||||
./venv/bin/python -m pip install -U docutils
|
||||
$(PYTHON) -m venv $(VENV_DIR)
|
||||
./$(VENV_DIR)/bin/python -m pip install -U docutils
|
||||
|
||||
package: all rss
|
||||
mkdir -p build/peps
|
||||
|
|
|
|||
|
|
@ -0,0 +1,538 @@
|
|||
PEP: 650
|
||||
Title: Specifying Installer Requirements for Python Projects
|
||||
Author: Vikram Jayanthi <vikramjayanthi@google.com>,
|
||||
Dustin Ingram <di@python.org>,
|
||||
Brett Cannon <brett@python.org>
|
||||
Status: Draft
|
||||
Type: Process
|
||||
Content-Type: text/x-rst
|
||||
Created: 16-Jul-2020
|
||||
Post-History:
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
Python package installers are not completely interoperable with each
|
||||
other. While pip is the most widely used installer and a de-facto
|
||||
standard, other installers such as Poetry_ or Pipenv_ are popular as
|
||||
well due to offering unique features which are optimal for certain
|
||||
workflows.
|
||||
|
||||
While the abundance of installer options is good for end-users with
|
||||
specific needs, the lack of interoperability between them makes them
|
||||
hard to support uniformly. Specifically, the lack of a standard
|
||||
requirements file for declaring dependencies means that each tool must
|
||||
be explicitly used in order to install dependencies specified with
|
||||
their respective format.
|
||||
|
||||
By providing a standardized API that can be used to invoke a
|
||||
compatible installer, we can solve this problem without needing to
|
||||
resolve individual concerns, unique requirements, and
|
||||
incompatibilities between different installers and their lock files.
|
||||
|
||||
Installers that implement the specification can be invoked in a
|
||||
uniform way, allowing users to use their installer of choice as if
|
||||
they were invoking it directly.
|
||||
|
||||
Terminology
|
||||
===========
|
||||
|
||||
Installer interface
|
||||
The interface by which an *installer backend* and a
|
||||
*universal installer* interact.
|
||||
|
||||
Universal installer
|
||||
An installer that can invoke an *installer backend* by calling the
|
||||
optional invocation methods of the *installer interface*.
|
||||
|
||||
Installer backend
|
||||
An installer that implements the *installer interface*, allowing
|
||||
it to be invoked by a *universal installer*. An
|
||||
*installer backend* may also be a *universal installer* as well,
|
||||
but it is not required.
|
||||
|
||||
Dependency group
|
||||
A set of dependencies that are related. For example, a development
|
||||
environment dependency group might include linting and formatting
|
||||
modules while a production dependency group contains dependencies
|
||||
required for deployment.
|
||||
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
This specification allows anyone to invoke and interact with
|
||||
installers that implement the specified interface, allowing for a
|
||||
universally supported layer on top of existing tool-specific
|
||||
installation processes.
|
||||
|
||||
This in turn would enable the use of all installers that implement the
|
||||
specified interface to be used in environments that support a single
|
||||
*universal installer*, as long as that installer implements this
|
||||
specification as well.
|
||||
|
||||
Below, we identify various use cases applicable to stakeholders in the
|
||||
Python community and anyone who interacts with Python package
|
||||
installers. For developers or companies, this PEP would allow for
|
||||
increased functionality and flexibility with Python package
|
||||
installers.
|
||||
|
||||
Providers
|
||||
---------
|
||||
|
||||
Providers are the parties (organization, person, community, etc.) that
|
||||
supply a service or software tool which interacts with Python
|
||||
packaging and consequently Python package installers. Two different
|
||||
types of providers are considered:
|
||||
|
||||
Platform/Infrastructure Providers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Platform providers (cloud environments, application hosting, etc.) and
|
||||
infrastructure service providers need to support package installers
|
||||
for their users to install Python dependencies. Most support only pip,
|
||||
however there is user demand for other Python installers. Most
|
||||
providers do not want to maintain support for more than one installer
|
||||
because of the complexity it adds to their software or service and the
|
||||
resources it takes to do so.
|
||||
|
||||
Via this specification, we can enable the provider-supported
|
||||
*universal installer* to invoke the user-desired *installer backend*
|
||||
without the provider’s platform needing to have specific knowledge of
|
||||
said backend.
|
||||
|
||||
IDE Providers
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Integrated development environments may interact with Python package
|
||||
installation and management. Most only support pip as a Python package
|
||||
installer, and users are required to find work arounds to install
|
||||
their dependencies using other package installers. Similar to the
|
||||
situation with PaaS & IaaS providers, IDE providers do not want to
|
||||
maintain support for N different Python installers. Instead,
|
||||
implementers of the installer interface (*installer backends*) could
|
||||
be invoked by the IDE by it acting as a *universal installer*.
|
||||
|
||||
Developers
|
||||
----------
|
||||
|
||||
Developers are teams, people, or communities that code and use Python
|
||||
package installers and Python packages. Three different types of
|
||||
developers are considered:
|
||||
|
||||
Developers using PaaS & IaaS providers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Most PaaS and IaaS providers only support one Python package
|
||||
installer: pip_. (Some exceptions include Heroku's Python buildpack_,
|
||||
which supports pip and Pipenv_). This dictates the installers that
|
||||
developers can use while working with these providers, which might not
|
||||
be optimal for their application or workflow.
|
||||
|
||||
Installers adopting this PEP to become installer backends would allow
|
||||
users to use third party platforms/infrastructure without having to
|
||||
worry about which Python package installer they are required to use as
|
||||
long as the provider uses a *universal installer*.
|
||||
|
||||
Developers using IDEs
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Most IDEs only support pip or a few Python package installers.
|
||||
Consequently, developers must use workarounds or hacky methods to
|
||||
install their dependencies if they use an unsupported package
|
||||
installer.
|
||||
|
||||
If the IDE uses/provides a *universal installer* it would allow for
|
||||
any *installer backend* that the developer wanted to be used to
|
||||
install dependencies, freeing them of any extra work to install their
|
||||
dependencies in order to integrate into the IDE's workflow more
|
||||
closely.
|
||||
|
||||
Developers working with other developers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Developers want to be able to use the installer of their choice while
|
||||
working with other developers, but currently have to synchronize their
|
||||
installer choice for compatibility of dependency installation. If all
|
||||
preferred installers instead implemented the specified interface, it
|
||||
would allow for cross use of installers, allowing developers to choose
|
||||
an installer regardless of their collaborator’s preference.
|
||||
|
||||
Upgraders & Package Infrastructure Providers
|
||||
--------------------------------------------
|
||||
|
||||
Package upgraders and package infrastructure in CI/CD such as
|
||||
Dependabot_, PyUP_, etc. currently support a few installers. They work
|
||||
by parsing and editing the installer-specific dependency files
|
||||
directly (such as ``requirements.txt`` or ``poetry.lock``) with
|
||||
relevant package information such as upgrades, downgrades, or new
|
||||
hashes. Similar to Platform and IDE providers, most of these providers
|
||||
do not want to support N different Python package installers as that
|
||||
would require supporting N different file types.
|
||||
|
||||
The current system relies on these services/bots to keep up support as
|
||||
new file formats and types are created and existing ones are changed.
|
||||
By implementing this specification we can allow these services/bots to
|
||||
interface through the spec and parse/write changes to dependencies
|
||||
consistently, regardless of which installer is being used. Additionally
|
||||
it would allow for more innovation in the space as it becomes easier
|
||||
to support different installers and gives developers a standardized
|
||||
way of interacting with them.
|
||||
|
||||
Open Source Community
|
||||
---------------------
|
||||
|
||||
Specifying installer requirements and adopting this PEP will reduce
|
||||
the friction between Python package installers and people's workflows.
|
||||
Consequently it will reduce the friction between Python package
|
||||
installers and 3rd party infrastructure/technologies such as PaaS or
|
||||
IDEs. Overall, it will allow for easier development, deployment and
|
||||
maintenance of Python projects as Python package installation becomes
|
||||
simpler and more interoperable.
|
||||
|
||||
Specifying requirements and creating an interface for installers can
|
||||
also increase the pace of innovation around installers. This would
|
||||
allow for installers to experiment and add unique functionality
|
||||
without requiring the rest of the ecosystem to do the same. Support
|
||||
becomes easier and more likely for a new installer regardless of the
|
||||
functionality it adds and the format in which it writes dependencies,
|
||||
while reducing the developer time and resources needed to do so.
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
Similar to how :pep:`517` specifies build systems, the install system
|
||||
information will live in the ``pyproject.toml`` file under the
|
||||
``install-system`` table.
|
||||
|
||||
[install-system]
|
||||
---------------------
|
||||
|
||||
The install-system table is used to store install-system relevant data
|
||||
and information. There are multiple required keys for this table:
|
||||
``requires`` and ``install-backend``. The ``requires`` key holds the
|
||||
minimum requirements for the install system to execute. The
|
||||
``install-backend`` key holds the name of the install backend’s entry
|
||||
point. This will allow the *universal installer* to install the
|
||||
requirements for the *installer backend* itself to execute (not the
|
||||
requirements that the *installer backend* itself will install) as well
|
||||
as invoke the *installer backend*.
|
||||
|
||||
If either of the required keys are missing or empty then the
|
||||
*universal installer* SHOULD raise an error.
|
||||
|
||||
All package names interacting with this interface are assumed to
|
||||
follow :pep:`508`'s "Dependency specification for Python Software
|
||||
Packages" format.
|
||||
|
||||
An example ``install-system`` table::
|
||||
|
||||
#pyproject.toml
|
||||
[install-system]
|
||||
#Eg : pipenv
|
||||
requires = ["pipenv"]
|
||||
install-backend = "pipenv.api:main"
|
||||
|
||||
|
||||
Installer Requirements:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
The requirements specified by the ``requires`` key must be within the
|
||||
constraints specified by :pep:`517`. Specifically, that dependency
|
||||
cycles are not permitted and the *universal installer* SHOULD refuse
|
||||
to install the dependencies if a cycle is detected.
|
||||
|
||||
Additional parameters or tool specific data
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Additional parameters or tool (*installer backend*) data may also be
|
||||
stored in the ``pyproject.toml`` file. This would be in the “tool.*”
|
||||
table as specified by :pep:`518`. For example if the
|
||||
*installer backend* is Poetry and you wanted to specify multiple
|
||||
dependency groups, the tool.poetry tables could look like this:
|
||||
|
||||
::
|
||||
|
||||
[tool.poetry.dev-dependencies]
|
||||
dependencies = "dev"
|
||||
|
||||
[tool.poetry.deploy]
|
||||
dependencies = "deploy"
|
||||
|
||||
|
||||
Installer interface:
|
||||
----------------------------------
|
||||
The *installer interface* contains mandatory and optional hooks.
|
||||
Compliant *installer backends* MUST implement the mandatory hooks and
|
||||
MAY implement the optional hooks. A *universal installer* MAY
|
||||
implement any of the *installer backend* hooks itself, to act as both
|
||||
a *universal installer* and *installer backend*, but this is not
|
||||
required.
|
||||
|
||||
All hooks take ``**kwargs`` arbitrary parameters that a
|
||||
*installer backend* may require that are not already specified,
|
||||
allowing for backwards compatibility. If unexpected parameters are
|
||||
passed to the *installer backend*, it should ignore them.
|
||||
|
||||
The following information is akin to the corresponding section in
|
||||
:pep:`517`. The hooks may be called with keyword arguments, so
|
||||
*installer backends* implementing them should be careful to make sure
|
||||
that their signatures match both the order and the names of the
|
||||
arguments above.
|
||||
|
||||
All hooks MAY print arbitrary informational text to ``stdout`` and
|
||||
``stderr``. They MUST NOT read from ``stdin``, and the
|
||||
*universal installer* MAY close ``stdin`` before invoking the hooks.
|
||||
|
||||
The *universal installer* may capture ``stdout`` and/or ``stderr``
|
||||
from the backend. If the backend detects that an output stream is not
|
||||
a terminal/console (e.g. not ``sys.stdout.isatty()``), it SHOULD
|
||||
ensure that any output it writes to that stream is ``UTF-8`` encoded.
|
||||
The *universal installer* MUST NOT fail if captured output is not
|
||||
valid UTF-8, but it MAY not preserve all the information in that case
|
||||
(e.g. it may decode using the replace error handler in Python). If the
|
||||
output stream is a terminal, the *installer backend* is responsible
|
||||
for presenting its output accurately, as for any program running in a
|
||||
terminal.
|
||||
|
||||
If a hook raises an exception, or causes the process to terminate,
|
||||
then this indicates an error.
|
||||
|
||||
|
||||
|
||||
Mandatory hooks:
|
||||
------------------------------------
|
||||
invoke_install
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Installs the dependencies::
|
||||
|
||||
def invoke_install(
|
||||
path : typing.Union[str, bytes, os.PathLike[str]],
|
||||
*,
|
||||
dependency_group : string = None,
|
||||
**kwargs
|
||||
) -> int:
|
||||
...
|
||||
|
||||
* ``path`` : An absolute path where the *installer backend* should be
|
||||
invoked from (e.g. the directory where ``pyproject.toml`` is
|
||||
located).
|
||||
* ``dependency_group`` : An optional flag specifying a dependency
|
||||
group that the *installer backend* should install. The install will
|
||||
error if the dependency group doesn't exist. A user can find all
|
||||
dependency groups by calling
|
||||
``get_dependencies_to_install().keys()`` if dependency groups are
|
||||
supported by the *installer backend*.
|
||||
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
||||
require that are not already specified, allows for backwards
|
||||
compatibility.
|
||||
|
||||
* Returns : An exit code (int). 0 if successful, any positive integer
|
||||
if unsuccessful.
|
||||
|
||||
The *universal installer* will use the exit code to determine if the
|
||||
installation is successful and SHOULD return the exit code itself.
|
||||
|
||||
Optional hooks:
|
||||
---------------------------------
|
||||
|
||||
invoke_uninstall
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Uninstall the specified dependencies::
|
||||
|
||||
def invoke_uninstall(
|
||||
*,
|
||||
dependency_group : string = None,
|
||||
**kwargs
|
||||
) -> int:
|
||||
...
|
||||
|
||||
* ``dependency_group`` : An optional flag specifying a dependency
|
||||
group that the *installer backend* should uninstall.
|
||||
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
||||
require that are not already specified, allows for backwards
|
||||
compatibility.
|
||||
|
||||
* Returns : An exit code (int). 0 if successful, any positive integer
|
||||
if unsuccessful.
|
||||
|
||||
The *universal installer* MUST invoke the *installer backend* at the
|
||||
same path that the *universal installer* itself was invoked.
|
||||
|
||||
The *universal installer* will use the exit code to determine if the
|
||||
uninstall is successful and SHOULD return the exit code itself.
|
||||
|
||||
get_dependencies_to_install
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Returns the dependencies that would be installed by
|
||||
``invoke_install(...)``. This allows package upgraders
|
||||
(e.g., Dependabot) to retrieve the dependencies attempting to be
|
||||
installed without parsing the dependency file::
|
||||
|
||||
def get_dependencies_to_install(
|
||||
dependency_group : string = None,
|
||||
**kwargs
|
||||
) -> List[str]:
|
||||
...
|
||||
|
||||
* ``dependency_group`` : Specify a dependency group to get the
|
||||
dependencies ``invoke_install(...)`` would install for that
|
||||
dependency group.
|
||||
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
||||
require that are not already specified, allows for backwards
|
||||
compatibility.
|
||||
|
||||
* Returns: A list of dependencies (:pep:`508` strings) to install.
|
||||
|
||||
If the group is specified, the *installer backend* MUST return the
|
||||
dependencies corresponding to the provided dependency group. If the
|
||||
specified group doesn't exist, or dependency groups are not supported
|
||||
by the *installer backend*, the *installer backend* MUST raise an
|
||||
error.
|
||||
|
||||
If the group is not specified, and the *installer backend* provides
|
||||
the concept of a default/unspecified group, the *installer backend*
|
||||
MAY return the dependencies for the default/unspecified group, but
|
||||
otherwise MUST raise an error.
|
||||
|
||||
get_dependency_groups
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Returns the dependency groups available to be installed. This allows
|
||||
*universal installers* to enumerate all dependency groups the
|
||||
*installer backend* is aware of::
|
||||
|
||||
def get_dependency_groups(
|
||||
**kwargs
|
||||
) -> FrozenSet[str]
|
||||
|
||||
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
||||
require that are not already specified, allows for backwards
|
||||
compatibility.
|
||||
|
||||
* Returns: A set of known dependency groups, as strings The empty set
|
||||
represents no dependency groups.
|
||||
|
||||
update_dependencies
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Outputs a dependency file based off of inputted package list::
|
||||
|
||||
def update_dependencies(
|
||||
dependency_specifiers : Iterable[str],
|
||||
*,
|
||||
dependency_group=None,
|
||||
**kwargs
|
||||
) -> int:
|
||||
...
|
||||
|
||||
* ``dependency_specifiers`` : An iterable of dependencies as
|
||||
:pep:`508` strings that are being updated, for example :
|
||||
``["requests==2.8.1", ...]``. Optionally for a specific dependency
|
||||
group.
|
||||
* ``dependency_group`` : The dependency group that the list of
|
||||
packages is for.
|
||||
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
||||
require that are not already specified, allows for backwards
|
||||
compatibility.
|
||||
|
||||
* Returns : An exit code (int). 0 if successful, any positive integer
|
||||
if unsuccessful.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
All hooks take ``**kwargs`` to allow for backwards compatibility and
|
||||
allow for tool specific *installer backend* functionality which
|
||||
requires a user to provide additional information not required by the
|
||||
hook.
|
||||
|
||||
While *installer backends* must be Python packages, what they do when
|
||||
invoked is an implementation detail of that tool. For example, an
|
||||
*installer backend* could act as a wrapper for a platform package
|
||||
manager (e.g., ``apt``).
|
||||
|
||||
|
||||
Backwards Compatibility
|
||||
=======================
|
||||
|
||||
This PEP would have no impact on pre-existing code and functionality
|
||||
as it only adds new functionality to a *universal installer*. Any
|
||||
existing installer should maintain its existing functionality and use
|
||||
cases, therefore having no backwards compatibility issues. Only code
|
||||
aiming to take advantage of this new functionality will have
|
||||
motivation to make changes to their pre existing code.
|
||||
|
||||
|
||||
Security Implications
|
||||
=====================
|
||||
|
||||
A malicious user has no increased ability or easier access to anything
|
||||
with the addition of standardized installer specifications. The
|
||||
installer that could be invoked by a *universal installer* via the
|
||||
interface specified in this PEP would be explicitly declared by the
|
||||
user. If the user has chosen a malicious installer, then invoking it
|
||||
with a *universal installer* is no different than the user invoking
|
||||
the installer directly. A malicious installer being an
|
||||
*installer backend* doesn't give it additional permissions or
|
||||
abilities.
|
||||
|
||||
|
||||
Rejected Ideas
|
||||
==============
|
||||
|
||||
A standardized lock file
|
||||
------------------------
|
||||
|
||||
A standardized lock file would solve a lot of the same problems that
|
||||
specifying installer requirements would. For example, it would allow
|
||||
for PaaS/IaaS to just support one installer that could read the
|
||||
standardized lock file regardless of the installer that created it.
|
||||
The problem with a standardized lock file is the difference in needs
|
||||
between Python package installers as well as a fundamental issue with
|
||||
creating reproducible environments via the lockfile (one of the main
|
||||
benefits).
|
||||
|
||||
Needs and information stored in dependency files between installers
|
||||
differ significantly and are dependent on installer functionality. For
|
||||
example, a Python package installer such as Poetry requires
|
||||
information for all Python versions and platforms and calculates
|
||||
appropriate hashes while pip wouldn't. Additionally, pip would not be
|
||||
able to guarantee recreating the same environment (install the exact
|
||||
same dependencies) as it is outside the scope of its functionality.
|
||||
This makes a standardized lock file harder to implement and makes it
|
||||
seem more appropriate to make lock files tool specific.
|
||||
|
||||
|
||||
Have installer backends support creating virtual environments
|
||||
-------------------------------------------------------------
|
||||
|
||||
Because installer backends will very likely have a concept of virtual
|
||||
environments and how to install into them, it was briefly considered
|
||||
to have them also support creating virtual environments. In the end,
|
||||
though, it was considered an orthogonal idea.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. _Buildpack: https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python
|
||||
.. _Dependabot: https://dependabot.com/
|
||||
.. _pip: https://pip.pypa.io
|
||||
.. _Pipenv: https://pipenv-fork.readthedocs.io/en/latest/
|
||||
.. _Poetry: https://python-poetry.org/
|
||||
.. _PyUP: https://pyup.io/
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document is placed in the public domain or under the
|
||||
CC0-1.0-Universal license, whichever is more permissive.
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
|
||||
Loading…
Reference in New Issue