2021-01-13 20:40:00 -05:00
|
|
|
|
PEP: 650
|
|
|
|
|
Title: Specifying Installer Requirements for Python Projects
|
|
|
|
|
Author: Vikram Jayanthi <vikramjayanthi@google.com>,
|
|
|
|
|
Dustin Ingram <di@python.org>,
|
|
|
|
|
Brett Cannon <brett@python.org>
|
2021-01-14 17:01:16 -05:00
|
|
|
|
Discussions-To: https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Status: Draft
|
|
|
|
|
Type: Process
|
2022-06-14 17:22:20 -04:00
|
|
|
|
Topic: Packaging
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Content-Type: text/x-rst
|
|
|
|
|
Created: 16-Jul-2020
|
2022-03-09 11:04:44 -05:00
|
|
|
|
Post-History: 14-Jan-2021
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
|
========
|
|
|
|
|
|
|
|
|
|
Python package installers are not completely interoperable with each
|
2021-02-03 09:06:23 -05:00
|
|
|
|
other. While pip is the most widely used installer and a de facto
|
2021-01-13 20:40:00 -05:00
|
|
|
|
standard, other installers such as Poetry_ or Pipenv_ are popular as
|
|
|
|
|
well due to offering unique features which are optimal for certain
|
2021-01-22 20:10:23 -05:00
|
|
|
|
workflows and not directly in line with how pip operates.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
While the abundance of installer options is good for end-users with
|
2021-01-22 20:10:23 -05:00
|
|
|
|
specific needs, the lack of interoperability between them makes it
|
|
|
|
|
hard to support all potential installers. Specifically, the lack of a
|
|
|
|
|
standard requirements file for declaring dependencies means that each
|
|
|
|
|
tool must be explicitly used in order to install dependencies
|
|
|
|
|
specified with their respective format. Otherwise tools must emit a
|
|
|
|
|
requirements file which leads to potential information loss for the
|
|
|
|
|
installer as well as an added export step as part of a developer's
|
|
|
|
|
workflow.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
By providing a standardized API that can be used to invoke a
|
|
|
|
|
compatible installer, we can solve this problem without needing to
|
|
|
|
|
resolve individual concerns, unique requirements, and
|
|
|
|
|
incompatibilities between different installers and their lock files.
|
|
|
|
|
|
|
|
|
|
Installers that implement the specification can be invoked in a
|
|
|
|
|
uniform way, allowing users to use their installer of choice as if
|
|
|
|
|
they were invoking it directly.
|
|
|
|
|
|
|
|
|
|
Terminology
|
|
|
|
|
===========
|
|
|
|
|
|
|
|
|
|
Installer interface
|
|
|
|
|
The interface by which an *installer backend* and a
|
|
|
|
|
*universal installer* interact.
|
|
|
|
|
|
|
|
|
|
Universal installer
|
|
|
|
|
An installer that can invoke an *installer backend* by calling the
|
2021-01-22 20:10:23 -05:00
|
|
|
|
optional invocation methods of the *installer interface*. This can
|
2021-02-03 17:35:49 -05:00
|
|
|
|
also be thought of as the installer frontend, à la the build_
|
2021-01-22 20:10:23 -05:00
|
|
|
|
project for :pep:`517`.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
Installer backend
|
|
|
|
|
An installer that implements the *installer interface*, allowing
|
|
|
|
|
it to be invoked by a *universal installer*. An
|
|
|
|
|
*installer backend* may also be a *universal installer* as well,
|
2021-01-22 20:10:23 -05:00
|
|
|
|
but it is not required. In comparison to :pep:`517`, this would
|
|
|
|
|
be Flit_. *Installer backends* may be wrapper packages around
|
|
|
|
|
a backing installer, e.g. Poetry could choose to not support this
|
|
|
|
|
API, but a package could act as a wrapper to invoke Poetry as
|
|
|
|
|
appropriate to use Poetry to perform an installation.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
Dependency group
|
2021-01-22 20:10:23 -05:00
|
|
|
|
A set of dependencies that are related and required to be
|
|
|
|
|
installed simultaneously for some purpose. For example, a
|
|
|
|
|
"test" dependency group could include the dependencies required to
|
|
|
|
|
run the test suite. How dependency groups are specified is up to
|
|
|
|
|
the *installer backend*.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Motivation
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
This specification allows anyone to invoke and interact with
|
2021-01-22 20:10:23 -05:00
|
|
|
|
*installer backends* that implement the specified interface, allowing
|
|
|
|
|
for a universally supported layer on top of existing tool-specific
|
2021-01-13 20:40:00 -05:00
|
|
|
|
installation processes.
|
|
|
|
|
|
|
|
|
|
This in turn would enable the use of all installers that implement the
|
|
|
|
|
specified interface to be used in environments that support a single
|
|
|
|
|
*universal installer*, as long as that installer implements this
|
|
|
|
|
specification as well.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Below, we identify various use-cases applicable to stakeholders in the
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Python community and anyone who interacts with Python package
|
|
|
|
|
installers. For developers or companies, this PEP would allow for
|
|
|
|
|
increased functionality and flexibility with Python package
|
|
|
|
|
installers.
|
|
|
|
|
|
|
|
|
|
Providers
|
|
|
|
|
---------
|
|
|
|
|
|
|
|
|
|
Providers are the parties (organization, person, community, etc.) that
|
|
|
|
|
supply a service or software tool which interacts with Python
|
|
|
|
|
packaging and consequently Python package installers. Two different
|
|
|
|
|
types of providers are considered:
|
|
|
|
|
|
|
|
|
|
Platform/Infrastructure Providers
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Platform providers (cloud environments, application hosting, etc.) and
|
|
|
|
|
infrastructure service providers need to support package installers
|
2021-01-22 20:10:23 -05:00
|
|
|
|
for their users to install Python dependencies. Most only support pip,
|
2021-01-13 20:40:00 -05:00
|
|
|
|
however there is user demand for other Python installers. Most
|
|
|
|
|
providers do not want to maintain support for more than one installer
|
|
|
|
|
because of the complexity it adds to their software or service and the
|
|
|
|
|
resources it takes to do so.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Via this specification, we can enable a provider-supported
|
2021-01-13 20:40:00 -05:00
|
|
|
|
*universal installer* to invoke the user-desired *installer backend*
|
|
|
|
|
without the provider’s platform needing to have specific knowledge of
|
2021-01-22 20:10:23 -05:00
|
|
|
|
said backend. What this means is if Poetry implemented the installer
|
|
|
|
|
backend API proposed by this PEP (or some other package wrapped Poetry
|
|
|
|
|
to provide the API), then platform providers would support Poetry
|
|
|
|
|
implicitly.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
IDE Providers
|
|
|
|
|
^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Integrated development environments may interact with Python package
|
|
|
|
|
installation and management. Most only support pip as a Python package
|
|
|
|
|
installer, and users are required to find work arounds to install
|
|
|
|
|
their dependencies using other package installers. Similar to the
|
|
|
|
|
situation with PaaS & IaaS providers, IDE providers do not want to
|
|
|
|
|
maintain support for N different Python installers. Instead,
|
|
|
|
|
implementers of the installer interface (*installer backends*) could
|
|
|
|
|
be invoked by the IDE by it acting as a *universal installer*.
|
|
|
|
|
|
|
|
|
|
Developers
|
|
|
|
|
----------
|
|
|
|
|
|
|
|
|
|
Developers are teams, people, or communities that code and use Python
|
|
|
|
|
package installers and Python packages. Three different types of
|
|
|
|
|
developers are considered:
|
|
|
|
|
|
|
|
|
|
Developers using PaaS & IaaS providers
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Most PaaS and IaaS providers only support one Python package
|
|
|
|
|
installer: pip_. (Some exceptions include Heroku's Python buildpack_,
|
|
|
|
|
which supports pip and Pipenv_). This dictates the installers that
|
|
|
|
|
developers can use while working with these providers, which might not
|
|
|
|
|
be optimal for their application or workflow.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Installers adopting this PEP to become *installer backends* would allow
|
2021-01-13 20:40:00 -05:00
|
|
|
|
users to use third party platforms/infrastructure without having to
|
|
|
|
|
worry about which Python package installer they are required to use as
|
|
|
|
|
long as the provider uses a *universal installer*.
|
|
|
|
|
|
|
|
|
|
Developers using IDEs
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Most IDEs only support pip or a few Python package installers.
|
|
|
|
|
Consequently, developers must use workarounds or hacky methods to
|
|
|
|
|
install their dependencies if they use an unsupported package
|
|
|
|
|
installer.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
If the IDE uses/provides a *universal installer* it would allow for
|
2021-01-13 20:40:00 -05:00
|
|
|
|
any *installer backend* that the developer wanted to be used to
|
|
|
|
|
install dependencies, freeing them of any extra work to install their
|
|
|
|
|
dependencies in order to integrate into the IDE's workflow more
|
|
|
|
|
closely.
|
|
|
|
|
|
|
|
|
|
Developers working with other developers
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Developers want to be able to use the installer of their choice while
|
|
|
|
|
working with other developers, but currently have to synchronize their
|
|
|
|
|
installer choice for compatibility of dependency installation. If all
|
|
|
|
|
preferred installers instead implemented the specified interface, it
|
|
|
|
|
would allow for cross use of installers, allowing developers to choose
|
|
|
|
|
an installer regardless of their collaborator’s preference.
|
|
|
|
|
|
|
|
|
|
Upgraders & Package Infrastructure Providers
|
|
|
|
|
--------------------------------------------
|
|
|
|
|
|
|
|
|
|
Package upgraders and package infrastructure in CI/CD such as
|
|
|
|
|
Dependabot_, PyUP_, etc. currently support a few installers. They work
|
|
|
|
|
by parsing and editing the installer-specific dependency files
|
|
|
|
|
directly (such as ``requirements.txt`` or ``poetry.lock``) with
|
|
|
|
|
relevant package information such as upgrades, downgrades, or new
|
|
|
|
|
hashes. Similar to Platform and IDE providers, most of these providers
|
|
|
|
|
do not want to support N different Python package installers as that
|
|
|
|
|
would require supporting N different file types.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Currently, these services/bots have to implement support for each
|
|
|
|
|
package installer individually. Inevitably, the most popular
|
|
|
|
|
installers are supported first, and less popular tools are often never
|
|
|
|
|
supported. By implementing this specification, these services/bots can
|
|
|
|
|
support any (compliant) installer, allowing users to select the tool
|
|
|
|
|
of their choice. This will allow for more innovation in the space, as
|
|
|
|
|
platforms and IDEs are no longer forced to prematurely select a
|
|
|
|
|
"winner".
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
Open Source Community
|
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
|
|
Specifying installer requirements and adopting this PEP will reduce
|
|
|
|
|
the friction between Python package installers and people's workflows.
|
2021-02-03 09:06:23 -05:00
|
|
|
|
Consequently, it will reduce the friction between Python package
|
2021-01-13 20:40:00 -05:00
|
|
|
|
installers and 3rd party infrastructure/technologies such as PaaS or
|
|
|
|
|
IDEs. Overall, it will allow for easier development, deployment and
|
|
|
|
|
maintenance of Python projects as Python package installation becomes
|
|
|
|
|
simpler and more interoperable.
|
|
|
|
|
|
|
|
|
|
Specifying requirements and creating an interface for installers can
|
|
|
|
|
also increase the pace of innovation around installers. This would
|
|
|
|
|
allow for installers to experiment and add unique functionality
|
|
|
|
|
without requiring the rest of the ecosystem to do the same. Support
|
|
|
|
|
becomes easier and more likely for a new installer regardless of the
|
|
|
|
|
functionality it adds and the format in which it writes dependencies,
|
|
|
|
|
while reducing the developer time and resources needed to do so.
|
|
|
|
|
|
|
|
|
|
Specification
|
|
|
|
|
=============
|
|
|
|
|
|
|
|
|
|
Similar to how :pep:`517` specifies build systems, the install system
|
|
|
|
|
information will live in the ``pyproject.toml`` file under the
|
|
|
|
|
``install-system`` table.
|
|
|
|
|
|
|
|
|
|
[install-system]
|
2021-01-22 20:10:23 -05:00
|
|
|
|
----------------
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
The install-system table is used to store install-system relevant data
|
|
|
|
|
and information. There are multiple required keys for this table:
|
|
|
|
|
``requires`` and ``install-backend``. The ``requires`` key holds the
|
2021-01-22 20:10:23 -05:00
|
|
|
|
minimum requirements for the *installer backend* to execute and which
|
|
|
|
|
will be installed by the *universal installer*. The ``install-backend``
|
|
|
|
|
key holds the name of the install backend’s entry point. This will
|
|
|
|
|
allow the *universal installer* to install the requirements for the
|
|
|
|
|
*installer backend* itself to execute (not the requirements that the
|
|
|
|
|
*installer backend* itself will install) as well as invoke the
|
|
|
|
|
*installer backend*.
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
If either of the required keys are missing or empty then the
|
|
|
|
|
*universal installer* SHOULD raise an error.
|
|
|
|
|
|
|
|
|
|
All package names interacting with this interface are assumed to
|
|
|
|
|
follow :pep:`508`'s "Dependency specification for Python Software
|
|
|
|
|
Packages" format.
|
|
|
|
|
|
|
|
|
|
An example ``install-system`` table::
|
|
|
|
|
|
|
|
|
|
#pyproject.toml
|
|
|
|
|
[install-system]
|
|
|
|
|
#Eg : pipenv
|
|
|
|
|
requires = ["pipenv"]
|
|
|
|
|
install-backend = "pipenv.api:main"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Installer Requirements:
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
2021-01-13 20:40:00 -05:00
|
|
|
|
The requirements specified by the ``requires`` key must be within the
|
|
|
|
|
constraints specified by :pep:`517`. Specifically, that dependency
|
|
|
|
|
cycles are not permitted and the *universal installer* SHOULD refuse
|
|
|
|
|
to install the dependencies if a cycle is detected.
|
|
|
|
|
|
|
|
|
|
Additional parameters or tool specific data
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Additional parameters or tool (*installer backend*) data may also be
|
|
|
|
|
stored in the ``pyproject.toml`` file. This would be in the “tool.*”
|
2021-02-03 09:06:23 -05:00
|
|
|
|
table as specified by :pep:`518`. For example, if the
|
2021-01-13 20:40:00 -05:00
|
|
|
|
*installer backend* is Poetry and you wanted to specify multiple
|
|
|
|
|
dependency groups, the tool.poetry tables could look like this:
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
[tool.poetry.dev-dependencies]
|
|
|
|
|
dependencies = "dev"
|
|
|
|
|
|
|
|
|
|
[tool.poetry.deploy]
|
|
|
|
|
dependencies = "deploy"
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Data may also be stored in other ways as the installer backend sees
|
|
|
|
|
fit (e.g. separate configuration file).
|
|
|
|
|
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
Installer interface:
|
2021-01-22 20:10:23 -05:00
|
|
|
|
--------------------
|
2021-01-13 20:40:00 -05:00
|
|
|
|
The *installer interface* contains mandatory and optional hooks.
|
|
|
|
|
Compliant *installer backends* MUST implement the mandatory hooks and
|
|
|
|
|
MAY implement the optional hooks. A *universal installer* MAY
|
|
|
|
|
implement any of the *installer backend* hooks itself, to act as both
|
|
|
|
|
a *universal installer* and *installer backend*, but this is not
|
|
|
|
|
required.
|
|
|
|
|
|
|
|
|
|
All hooks take ``**kwargs`` arbitrary parameters that a
|
|
|
|
|
*installer backend* may require that are not already specified,
|
|
|
|
|
allowing for backwards compatibility. If unexpected parameters are
|
|
|
|
|
passed to the *installer backend*, it should ignore them.
|
|
|
|
|
|
|
|
|
|
The following information is akin to the corresponding section in
|
|
|
|
|
:pep:`517`. The hooks may be called with keyword arguments, so
|
|
|
|
|
*installer backends* implementing them should be careful to make sure
|
|
|
|
|
that their signatures match both the order and the names of the
|
|
|
|
|
arguments above.
|
|
|
|
|
|
|
|
|
|
All hooks MAY print arbitrary informational text to ``stdout`` and
|
|
|
|
|
``stderr``. They MUST NOT read from ``stdin``, and the
|
|
|
|
|
*universal installer* MAY close ``stdin`` before invoking the hooks.
|
|
|
|
|
|
|
|
|
|
The *universal installer* may capture ``stdout`` and/or ``stderr``
|
|
|
|
|
from the backend. If the backend detects that an output stream is not
|
|
|
|
|
a terminal/console (e.g. not ``sys.stdout.isatty()``), it SHOULD
|
|
|
|
|
ensure that any output it writes to that stream is ``UTF-8`` encoded.
|
|
|
|
|
The *universal installer* MUST NOT fail if captured output is not
|
|
|
|
|
valid UTF-8, but it MAY not preserve all the information in that case
|
|
|
|
|
(e.g. it may decode using the replace error handler in Python). If the
|
|
|
|
|
output stream is a terminal, the *installer backend* is responsible
|
|
|
|
|
for presenting its output accurately, as for any program running in a
|
|
|
|
|
terminal.
|
|
|
|
|
|
|
|
|
|
If a hook raises an exception, or causes the process to terminate,
|
|
|
|
|
then this indicates an error.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Mandatory hooks:
|
2021-01-22 20:10:23 -05:00
|
|
|
|
----------------
|
2021-01-13 20:40:00 -05:00
|
|
|
|
invoke_install
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Installs the dependencies::
|
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
def invoke_install(
|
|
|
|
|
path: Union[str, bytes, PathLike[str]],
|
2021-01-13 20:40:00 -05:00
|
|
|
|
*,
|
2021-01-20 14:58:34 -05:00
|
|
|
|
dependency_group: str = None,
|
2021-01-13 20:40:00 -05:00
|
|
|
|
**kwargs
|
|
|
|
|
) -> int:
|
|
|
|
|
...
|
|
|
|
|
|
|
|
|
|
* ``path`` : An absolute path where the *installer backend* should be
|
|
|
|
|
invoked from (e.g. the directory where ``pyproject.toml`` is
|
|
|
|
|
located).
|
|
|
|
|
* ``dependency_group`` : An optional flag specifying a dependency
|
|
|
|
|
group that the *installer backend* should install. The install will
|
|
|
|
|
error if the dependency group doesn't exist. A user can find all
|
|
|
|
|
dependency groups by calling
|
2021-02-03 17:35:49 -05:00
|
|
|
|
``get_dependency_groups()`` if dependency groups are
|
2021-01-13 20:40:00 -05:00
|
|
|
|
supported by the *installer backend*.
|
|
|
|
|
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
|
|
|
|
require that are not already specified, allows for backwards
|
|
|
|
|
compatibility.
|
|
|
|
|
|
|
|
|
|
* Returns : An exit code (int). 0 if successful, any positive integer
|
|
|
|
|
if unsuccessful.
|
|
|
|
|
|
|
|
|
|
The *universal installer* will use the exit code to determine if the
|
|
|
|
|
installation is successful and SHOULD return the exit code itself.
|
|
|
|
|
|
|
|
|
|
Optional hooks:
|
2021-01-22 20:10:23 -05:00
|
|
|
|
---------------
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
invoke_uninstall
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^^^
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Uninstall the specified dependencies::
|
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
def invoke_uninstall(
|
|
|
|
|
path: Union[str, bytes, PathLike[str]],
|
2021-01-13 20:40:00 -05:00
|
|
|
|
*,
|
2021-01-20 14:58:34 -05:00
|
|
|
|
dependency_group: str = None,
|
2021-01-13 20:40:00 -05:00
|
|
|
|
**kwargs
|
|
|
|
|
) -> int:
|
|
|
|
|
...
|
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
* ``path`` : An absolute path where the *installer backend* should be
|
|
|
|
|
invoked from (e.g. the directory where ``pyproject.toml`` is
|
|
|
|
|
located).
|
2021-01-13 20:40:00 -05:00
|
|
|
|
* ``dependency_group`` : An optional flag specifying a dependency
|
|
|
|
|
group that the *installer backend* should uninstall.
|
|
|
|
|
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
|
|
|
|
require that are not already specified, allows for backwards
|
|
|
|
|
compatibility.
|
|
|
|
|
|
|
|
|
|
* Returns : An exit code (int). 0 if successful, any positive integer
|
|
|
|
|
if unsuccessful.
|
|
|
|
|
|
|
|
|
|
The *universal installer* MUST invoke the *installer backend* at the
|
|
|
|
|
same path that the *universal installer* itself was invoked.
|
|
|
|
|
|
|
|
|
|
The *universal installer* will use the exit code to determine if the
|
|
|
|
|
uninstall is successful and SHOULD return the exit code itself.
|
|
|
|
|
|
|
|
|
|
get_dependencies_to_install
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Returns the dependencies that would be installed by
|
|
|
|
|
``invoke_install(...)``. This allows package upgraders
|
|
|
|
|
(e.g., Dependabot) to retrieve the dependencies attempting to be
|
|
|
|
|
installed without parsing the dependency file::
|
|
|
|
|
|
|
|
|
|
def get_dependencies_to_install(
|
2021-01-20 14:58:34 -05:00
|
|
|
|
path: Union[str, bytes, PathLike[str]],
|
|
|
|
|
*,
|
|
|
|
|
dependency_group: str = None,
|
2021-01-13 20:40:00 -05:00
|
|
|
|
**kwargs
|
2021-01-22 20:10:23 -05:00
|
|
|
|
) -> Sequence[str]:
|
2021-01-13 20:40:00 -05:00
|
|
|
|
...
|
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
* ``path`` : An absolute path where the *installer backend* should be
|
|
|
|
|
invoked from (e.g. the directory where ``pyproject.toml`` is
|
|
|
|
|
located).
|
2021-01-13 20:40:00 -05:00
|
|
|
|
* ``dependency_group`` : Specify a dependency group to get the
|
|
|
|
|
dependencies ``invoke_install(...)`` would install for that
|
|
|
|
|
dependency group.
|
|
|
|
|
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
|
|
|
|
require that are not already specified, allows for backwards
|
|
|
|
|
compatibility.
|
|
|
|
|
|
|
|
|
|
* Returns: A list of dependencies (:pep:`508` strings) to install.
|
|
|
|
|
|
|
|
|
|
If the group is specified, the *installer backend* MUST return the
|
|
|
|
|
dependencies corresponding to the provided dependency group. If the
|
|
|
|
|
specified group doesn't exist, or dependency groups are not supported
|
|
|
|
|
by the *installer backend*, the *installer backend* MUST raise an
|
|
|
|
|
error.
|
|
|
|
|
|
|
|
|
|
If the group is not specified, and the *installer backend* provides
|
|
|
|
|
the concept of a default/unspecified group, the *installer backend*
|
|
|
|
|
MAY return the dependencies for the default/unspecified group, but
|
|
|
|
|
otherwise MUST raise an error.
|
|
|
|
|
|
|
|
|
|
get_dependency_groups
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Returns the dependency groups available to be installed. This allows
|
|
|
|
|
*universal installers* to enumerate all dependency groups the
|
|
|
|
|
*installer backend* is aware of::
|
|
|
|
|
|
|
|
|
|
def get_dependency_groups(
|
2021-01-20 14:58:34 -05:00
|
|
|
|
path: Union[str, bytes, PathLike[str]],
|
2021-01-13 20:40:00 -05:00
|
|
|
|
**kwargs
|
2021-01-22 20:10:23 -05:00
|
|
|
|
) -> AbstractSet[str]:
|
2021-01-20 14:58:34 -05:00
|
|
|
|
...
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
* ``path`` : An absolute path where the *installer backend* should be
|
|
|
|
|
invoked from (e.g. the directory where ``pyproject.toml`` is
|
|
|
|
|
located).
|
2021-01-13 20:40:00 -05:00
|
|
|
|
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
|
|
|
|
require that are not already specified, allows for backwards
|
|
|
|
|
compatibility.
|
|
|
|
|
|
|
|
|
|
* Returns: A set of known dependency groups, as strings The empty set
|
|
|
|
|
represents no dependency groups.
|
|
|
|
|
|
|
|
|
|
update_dependencies
|
2021-01-22 20:10:23 -05:00
|
|
|
|
^^^^^^^^^^^^^^^^^^^
|
2021-02-03 09:06:23 -05:00
|
|
|
|
Outputs a dependency file based on inputted package list::
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
def update_dependencies(
|
2021-01-20 14:58:34 -05:00
|
|
|
|
path: Union[str, bytes, PathLike[str]],
|
|
|
|
|
dependency_specifiers: Iterable[str],
|
2021-01-13 20:40:00 -05:00
|
|
|
|
*,
|
|
|
|
|
dependency_group=None,
|
|
|
|
|
**kwargs
|
|
|
|
|
) -> int:
|
|
|
|
|
...
|
|
|
|
|
|
2021-01-20 14:58:34 -05:00
|
|
|
|
* ``path`` : An absolute path where the *installer backend* should be
|
|
|
|
|
invoked from (e.g. the directory where ``pyproject.toml`` is
|
|
|
|
|
located).
|
2021-01-13 20:40:00 -05:00
|
|
|
|
* ``dependency_specifiers`` : An iterable of dependencies as
|
|
|
|
|
:pep:`508` strings that are being updated, for example :
|
|
|
|
|
``["requests==2.8.1", ...]``. Optionally for a specific dependency
|
|
|
|
|
group.
|
|
|
|
|
* ``dependency_group`` : The dependency group that the list of
|
|
|
|
|
packages is for.
|
|
|
|
|
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
|
|
|
|
|
require that are not already specified, allows for backwards
|
|
|
|
|
compatibility.
|
|
|
|
|
|
|
|
|
|
* Returns : An exit code (int). 0 if successful, any positive integer
|
|
|
|
|
if unsuccessful.
|
|
|
|
|
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Example
|
|
|
|
|
=======
|
|
|
|
|
|
|
|
|
|
Let's consider implementing an *installer backend* that uses pip and
|
|
|
|
|
its requirements files for *dependency groups*. An implementation may
|
|
|
|
|
(very roughly) look like the following::
|
|
|
|
|
|
|
|
|
|
import subprocess
|
|
|
|
|
import sys
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def invoke_install(path, *, dependency_group=None, **kwargs):
|
2021-02-03 17:35:49 -05:00
|
|
|
|
try:
|
|
|
|
|
return subprocess.run(
|
|
|
|
|
[
|
|
|
|
|
sys.executable,
|
|
|
|
|
"-m",
|
|
|
|
|
"pip",
|
|
|
|
|
"install",
|
|
|
|
|
"-r",
|
|
|
|
|
dependency_group or "requirements.txt",
|
|
|
|
|
],
|
|
|
|
|
cwd=path,
|
|
|
|
|
).returncode
|
|
|
|
|
except subprocess.CalledProcessError as e:
|
|
|
|
|
return e.returncode
|
2021-01-22 20:10:23 -05:00
|
|
|
|
|
|
|
|
|
If we named this package ``pep650pip``, then we could specify in
|
|
|
|
|
``pyproject.toml``::
|
|
|
|
|
|
|
|
|
|
[install-system]
|
|
|
|
|
#Eg : pipenv
|
|
|
|
|
requires = ["pep650pip", "pip"]
|
|
|
|
|
install-backend = "pep650pip:main"
|
|
|
|
|
|
|
|
|
|
|
2021-01-13 20:40:00 -05:00
|
|
|
|
Rationale
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
All hooks take ``**kwargs`` to allow for backwards compatibility and
|
|
|
|
|
allow for tool specific *installer backend* functionality which
|
|
|
|
|
requires a user to provide additional information not required by the
|
|
|
|
|
hook.
|
|
|
|
|
|
|
|
|
|
While *installer backends* must be Python packages, what they do when
|
|
|
|
|
invoked is an implementation detail of that tool. For example, an
|
|
|
|
|
*installer backend* could act as a wrapper for a platform package
|
|
|
|
|
manager (e.g., ``apt``).
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
The interface does not in any way try to specify *how*
|
|
|
|
|
*installer backends* should function. This is on purpose so that
|
|
|
|
|
*installer backends* can be allowed to innovate and solve problem in
|
|
|
|
|
their own way. This also means this PEP takes no stance on OS
|
|
|
|
|
packaging as that would be an *installer backend*'s domain.
|
|
|
|
|
|
|
|
|
|
Defining the API in Python does mean that *some* Python code will
|
|
|
|
|
eventually need to be executed. That does not preclude non-Python
|
|
|
|
|
*installer backends* from being used, though (e.g. mamba_), as they
|
|
|
|
|
could be executed as a subprocess from Python code.
|
|
|
|
|
|
2021-01-13 20:40:00 -05:00
|
|
|
|
|
|
|
|
|
Backwards Compatibility
|
|
|
|
|
=======================
|
|
|
|
|
|
|
|
|
|
This PEP would have no impact on pre-existing code and functionality
|
|
|
|
|
as it only adds new functionality to a *universal installer*. Any
|
|
|
|
|
existing installer should maintain its existing functionality and use
|
|
|
|
|
cases, therefore having no backwards compatibility issues. Only code
|
|
|
|
|
aiming to take advantage of this new functionality will have
|
|
|
|
|
motivation to make changes to their pre existing code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Security Implications
|
|
|
|
|
=====================
|
|
|
|
|
|
|
|
|
|
A malicious user has no increased ability or easier access to anything
|
|
|
|
|
with the addition of standardized installer specifications. The
|
|
|
|
|
installer that could be invoked by a *universal installer* via the
|
|
|
|
|
interface specified in this PEP would be explicitly declared by the
|
|
|
|
|
user. If the user has chosen a malicious installer, then invoking it
|
|
|
|
|
with a *universal installer* is no different than the user invoking
|
|
|
|
|
the installer directly. A malicious installer being an
|
|
|
|
|
*installer backend* doesn't give it additional permissions or
|
|
|
|
|
abilities.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Rejected Ideas
|
|
|
|
|
==============
|
|
|
|
|
|
|
|
|
|
A standardized lock file
|
|
|
|
|
------------------------
|
|
|
|
|
|
|
|
|
|
A standardized lock file would solve a lot of the same problems that
|
|
|
|
|
specifying installer requirements would. For example, it would allow
|
|
|
|
|
for PaaS/IaaS to just support one installer that could read the
|
|
|
|
|
standardized lock file regardless of the installer that created it.
|
|
|
|
|
The problem with a standardized lock file is the difference in needs
|
|
|
|
|
between Python package installers as well as a fundamental issue with
|
|
|
|
|
creating reproducible environments via the lockfile (one of the main
|
|
|
|
|
benefits).
|
|
|
|
|
|
|
|
|
|
Needs and information stored in dependency files between installers
|
|
|
|
|
differ significantly and are dependent on installer functionality. For
|
|
|
|
|
example, a Python package installer such as Poetry requires
|
|
|
|
|
information for all Python versions and platforms and calculates
|
2021-01-22 20:10:23 -05:00
|
|
|
|
appropriate hashes while pip doesn't. Additionally, pip would not be
|
2021-01-13 20:40:00 -05:00
|
|
|
|
able to guarantee recreating the same environment (install the exact
|
|
|
|
|
same dependencies) as it is outside the scope of its functionality.
|
|
|
|
|
This makes a standardized lock file harder to implement and makes it
|
|
|
|
|
seem more appropriate to make lock files tool specific.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Have installer backends support creating virtual environments
|
|
|
|
|
-------------------------------------------------------------
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
Because *installer backends* will very likely have a concept of virtual
|
2021-01-13 20:40:00 -05:00
|
|
|
|
environments and how to install into them, it was briefly considered
|
|
|
|
|
to have them also support creating virtual environments. In the end,
|
|
|
|
|
though, it was considered an orthogonal idea.
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
|
|
|
|
|
Open Issues
|
|
|
|
|
===========
|
|
|
|
|
|
2021-02-05 11:09:26 -05:00
|
|
|
|
Should the ``dependency_group`` argument take an iterable?
|
|
|
|
|
----------------------------------------------------------
|
2021-01-22 20:10:23 -05:00
|
|
|
|
|
|
|
|
|
This would allow for specifying non-overlapping dependency groups in
|
|
|
|
|
a single call, e.g. "docs" and "test" groups which have independent
|
|
|
|
|
dependencies but which a developer may want to install simultaneously
|
|
|
|
|
while doing development.
|
|
|
|
|
|
|
|
|
|
Is the installer backend executed in-process?
|
|
|
|
|
---------------------------------------------
|
|
|
|
|
|
|
|
|
|
If the *installer backend* is executed in-process then it greatly
|
|
|
|
|
simplifies knowing what environment to install for/into, as the live
|
|
|
|
|
Python environment can be queried for appropriate information.
|
|
|
|
|
|
|
|
|
|
Executing out-of-process allows for minimizing potential issues of
|
|
|
|
|
clashes between the environment being installed into and the
|
|
|
|
|
*installer backend* (and potentially *universal installer*).
|
|
|
|
|
|
|
|
|
|
Enforce that results from the proposed interface feed into other parts?
|
|
|
|
|
-----------------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
E.g. the results from ``get_dependencies_to_install()`` and
|
|
|
|
|
``get_dependency_groups()`` can be passed into ``invoke_install()``.
|
|
|
|
|
This would prevent drift between the results of various parts of the
|
|
|
|
|
proposed interface, but it makes more of the interface required
|
|
|
|
|
instead of optional.
|
|
|
|
|
|
|
|
|
|
Raising exceptions instead of exit codes for failure conditions
|
|
|
|
|
---------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
It has been suggested that instead of returning an exit code the API
|
|
|
|
|
should raise exceptions. If you view this PEP as helping to translate
|
|
|
|
|
current installers into *installer backends*, then relying on exit
|
|
|
|
|
codes makes sense. There's is also the point that the APIs have no
|
|
|
|
|
specific return value, so passing along an exit code does not
|
|
|
|
|
interfere with what the functions return.
|
|
|
|
|
|
|
|
|
|
Compare that to raising exceptions in case of an error. That could
|
|
|
|
|
potentially provide a more structured approach to error raising,
|
|
|
|
|
although to be able to capture errors it would require specifying
|
|
|
|
|
exception types as part of the interface.
|
|
|
|
|
|
2021-01-13 20:40:00 -05:00
|
|
|
|
References
|
|
|
|
|
==========
|
|
|
|
|
|
2021-01-22 20:10:23 -05:00
|
|
|
|
.. _build: https://github.com/pypa/build
|
2021-01-13 20:40:00 -05:00
|
|
|
|
.. _Buildpack: https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python
|
|
|
|
|
.. _Dependabot: https://dependabot.com/
|
2021-01-22 20:10:23 -05:00
|
|
|
|
.. _Flit: https://flit.readthedocs.io
|
|
|
|
|
.. _mamba: https://github.com/mamba-org/mamba
|
2021-01-13 20:40:00 -05:00
|
|
|
|
.. _pip: https://pip.pypa.io
|
|
|
|
|
.. _Pipenv: https://pipenv-fork.readthedocs.io/en/latest/
|
|
|
|
|
.. _Poetry: https://python-poetry.org/
|
|
|
|
|
.. _PyUP: https://pyup.io/
|
|
|
|
|
|
|
|
|
|
Copyright
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
This document is placed in the public domain or under the
|
|
|
|
|
CC0-1.0-Universal license, whichever is more permissive.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
..
|
|
|
|
|
Local Variables:
|
|
|
|
|
mode: indented-text
|
|
|
|
|
indent-tabs-mode: nil
|
|
|
|
|
sentence-end-double-space: t
|
|
|
|
|
fill-column: 70
|
|
|
|
|
coding: utf-8
|
|
|
|
|
End:
|