iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update PEP 650 based on reviewer comments (#1778)

This commit is contained in:
Brett Cannon 2021-01-22 17:10:23 -08:00 committed by GitHub
parent 9d08c18a92
commit 6982646dd8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 158 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

207
pep-0650.rst
View File

@ -18,14 +18,17 @@ Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de-facto
standard, other installers such as Poetry_ or Pipenv_ are popular as
well due to offering unique features which are optimal for certain
workflows.
workflows and not directly in line with how pip operates.
While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes them
hard to support uniformly. Specifically, the lack of a standard
requirements file for declaring dependencies means that each tool must
be explicitly used in order to install dependencies specified with
their respective format.
specific needs, the lack of interoperability between them makes it
hard to support all potential installers. Specifically, the lack of a
standard requirements file for declaring dependencies means that each
tool must be explicitly used in order to install dependencies
specified with their respective format. Otherwise tools must emit a
requirements file which leads to potential information loss for the
installer as well as an added export step as part of a developer's
workflow.
By providing a standardized API that can be used to invoke a
compatible installer, we can solve this problem without needing to
@ -45,27 +48,34 @@ Installer interface
Universal installer
An installer that can invoke an *installer backend* by calling the
optional invocation methods of the *installer interface*.
optional invocation methods of the *installer interface*. This can
also be thought of as the installer frontend, ala the build_
project for :pep:`517`.
Installer backend
An installer that implements the *installer interface*, allowing
it to be invoked by a *universal installer*. An
*installer backend* may also be a *universal installer* as well,
but it is not required.
but it is not required. In comparison to :pep:`517`, this would
be Flit_. *Installer backends* may be wrapper packages around
a backing installer, e.g. Poetry could choose to not support this
API, but a package could act as a wrapper to invoke Poetry as
appropriate to use Poetry to perform an installation.
Dependency group
A set of dependencies that are related. For example, a development
environment dependency group might include linting and formatting
modules while a production dependency group contains dependencies
required for deployment.
A set of dependencies that are related and required to be
installed simultaneously for some purpose. For example, a
"test" dependency group could include the dependencies required to
run the test suite. How dependency groups are specified is up to
the *installer backend*.
Motivation
==========
This specification allows anyone to invoke and interact with
installers that implement the specified interface, allowing for a
universally supported layer on top of existing tool-specific
*installer backends* that implement the specified interface, allowing
for a universally supported layer on top of existing tool-specific
installation processes.
This in turn would enable the use of all installers that implement the
@ -73,7 +83,7 @@ specified interface to be used in environments that support a single
*universal installer*, as long as that installer implements this
specification as well.
Below, we identify various use cases applicable to stakeholders in the
Below, we identify various use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package
@ -92,16 +102,19 @@ Platform/Infrastructure Providers
Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers
for their users to install Python dependencies. Most support only pip,
for their users to install Python dependencies. Most only support pip,
however there is user demand for other Python installers. Most
providers do not want to maintain support for more than one installer
because of the complexity it adds to their software or service and the
resources it takes to do so.
Via this specification, we can enable the provider-supported
Via this specification, we can enable a provider-supported
*universal installer* to invoke the user-desired *installer backend*
without the provider’s platform needing to have specific knowledge of
said backend.
said backend. What this means is if Poetry implemented the installer
backend API proposed by this PEP (or some other package wrapped Poetry
to provide the API), then platform providers would support Poetry
implicitly.
IDE Providers
^^^^^^^^^^^^^
@ -131,7 +144,7 @@ which supports pip and Pipenv_). This dictates the installers that
developers can use while working with these providers, which might not
be optimal for their application or workflow.
Installers adopting this PEP to become installer backends would allow
Installers adopting this PEP to become *installer backends* would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a *universal installer*.
@ -144,7 +157,7 @@ Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package
installer.
If the IDE uses/provides a *universal installer* it would allow for
If the IDE uses/provides a *universal installer* it would allow for
any *installer backend* that the developer wanted to be used to
install dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE's workflow more
@ -172,14 +185,14 @@ hashes. Similar to Platform and IDE providers, most of these providers
do not want to support N different Python package installers as that
would require supporting N different file types.
The current system relies on these services/bots to keep up support as
new file formats and types are created and existing ones are changed.
By implementing this specification we can allow these services/bots to
interface through the spec and parse/write changes to dependencies
consistently, regardless of which installer is being used. Additionally
it would allow for more innovation in the space as it becomes easier
to support different installers and gives developers a standardized
way of interacting with them.
Currently, these services/bots have to implement support for each
package installer individually. Inevitably, the most popular
installers are supported first, and less popular tools are often never
supported. By implementing this specification, these services/bots can
support any (compliant) installer, allowing users to select the tool
of their choice. This will allow for more innovation in the space, as
platforms and IDEs are no longer forced to prematurely select a
"winner".
Open Source Community
---------------------
@ -208,17 +221,18 @@ information will live in the ``pyproject.toml`` file under the
``install-system`` table.
[install-system]
---------------------
----------------
The install-system table is used to store install-system relevant data
and information. There are multiple required keys for this table:
``requires`` and ``install-backend``. The ``requires`` key holds the
minimum requirements for the install system to execute. The
``install-backend`` key holds the name of the install backend’s entry
point. This will allow the *universal installer* to install the
requirements for the *installer backend* itself to execute (not the
requirements that the *installer backend* itself will install) as well
as invoke the *installer backend*.
minimum requirements for the *installer backend* to execute and which
will be installed by the *universal installer*. The ``install-backend``
key holds the name of the install backend’s entry point. This will
allow the *universal installer* to install the requirements for the
*installer backend* itself to execute (not the requirements that the
*installer backend* itself will install) as well as invoke the
*installer backend*.
If either of the required keys are missing or empty then the
*universal installer* SHOULD raise an error.
@ -237,7 +251,7 @@ An example ``install-system`` table::
Installer Requirements:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^
The requirements specified by the ``requires`` key must be within the
constraints specified by :pep:`517`. Specifically, that dependency
cycles are not permitted and the *universal installer* SHOULD refuse
@ -259,9 +273,12 @@ dependency groups, the tool.poetry tables could look like this:
[tool.poetry.deploy]
dependencies = "deploy"
Data may also be stored in other ways as the installer backend sees
fit (e.g. separate configuration file).
Installer interface:
----------------------------------
--------------------
The *installer interface* contains mandatory and optional hooks.
Compliant *installer backends* MUST implement the mandatory hooks and
MAY implement the optional hooks. A *universal installer* MAY
@ -301,9 +318,9 @@ then this indicates an error.
Mandatory hooks:
------------------------------------
----------------
invoke_install
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^
Installs the dependencies::
def invoke_install(
@ -321,7 +338,7 @@ Installs the dependencies::
group that the *installer backend* should install. The install will
error if the dependency group doesn't exist. A user can find all
dependency groups by calling
``get_dependencies_to_install().keys()`` if dependency groups are
``get_dependencies_groups()`` if dependency groups are
supported by the *installer backend*.
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
@ -334,10 +351,10 @@ The *universal installer* will use the exit code to determine if the
installation is successful and SHOULD return the exit code itself.
Optional hooks:
---------------------------------
---------------
invoke_uninstall
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^
Uninstall the specified dependencies::
def invoke_uninstall(
@ -367,7 +384,7 @@ The *universal installer* will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.
get_dependencies_to_install
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the dependencies that would be installed by
``invoke_install(...)``. This allows package upgraders
(e.g., Dependabot) to retrieve the dependencies attempting to be
@ -378,7 +395,7 @@ installed without parsing the dependency file::
*,
dependency_group: str = None,
**kwargs
) -> List[str]:
) -> Sequence[str]:
...
* ``path`` : An absolute path where the *installer backend* should be
@ -405,7 +422,7 @@ MAY return the dependencies for the default/unspecified group, but
otherwise MUST raise an error.
get_dependency_groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
Returns the dependency groups available to be installed. This allows
*universal installers* to enumerate all dependency groups the
*installer backend* is aware of::
@ -413,7 +430,7 @@ Returns the dependency groups available to be installed. This allows
def get_dependency_groups(
path: Union[str, bytes, PathLike[str]],
**kwargs
) -> FrozenSet[str]:
) -> AbstractSet[str]:
...
* ``path`` : An absolute path where the *installer backend* should be
@ -427,7 +444,7 @@ Returns the dependency groups available to be installed. This allows
represents no dependency groups.
update_dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
Outputs a dependency file based off of inputted package list::
def update_dependencies(
@ -456,6 +473,37 @@ Outputs a dependency file based off of inputted package list::
if unsuccessful.
Example
=======
Let's consider implementing an *installer backend* that uses pip and
its requirements files for *dependency groups*. An implementation may
(very roughly) look like the following::
import os
import pathlib
import subprocess
import sys
def invoke_install(path, *, dependency_group=None, **kwargs):
file_name = "requirements.txt"
if dependency_group:
file_name = f"{dependency_group}-{file_name}"
requirements_path = pathlib.Path(path) / file_name
return subprocess.call(
[sys.executable, "-m", "pip", "install", "-r", os.fspath(requirements_path)]
)
If we named this package ``pep650pip``, then we could specify in
``pyproject.toml``::
[install-system]
#Eg : pipenv
requires = ["pep650pip", "pip"]
install-backend = "pep650pip:main"
Rationale
=========
@ -469,6 +517,17 @@ invoked is an implementation detail of that tool. For example, an
*installer backend* could act as a wrapper for a platform package
manager (e.g., ``apt``).
The interface does not in any way try to specify *how*
*installer backends* should function. This is on purpose so that
*installer backends* can be allowed to innovate and solve problem in
their own way. This also means this PEP takes no stance on OS
packaging as that would be an *installer backend*'s domain.
Defining the API in Python does mean that *some* Python code will
eventually need to be executed. That does not preclude non-Python
*installer backends* from being used, though (e.g. mamba_), as they
could be executed as a subprocess from Python code.
Backwards Compatibility
=======================
@ -514,7 +573,7 @@ Needs and information stored in dependency files between installers
differ significantly and are dependent on installer functionality. For
example, a Python package installer such as Poetry requires
information for all Python versions and platforms and calculates
appropriate hashes while pip wouldn't. Additionally, pip would not be
appropriate hashes while pip doesn't. Additionally, pip would not be
able to guarantee recreating the same environment (install the exact
same dependencies) as it is outside the scope of its functionality.
This makes a standardized lock file harder to implement and makes it
@ -524,16 +583,66 @@ seem more appropriate to make lock files tool specific.
Have installer backends support creating virtual environments
-------------------------------------------------------------
Because installer backends will very likely have a concept of virtual
Because *installer backends* will very likely have a concept of virtual
environments and how to install into them, it was briefly considered
to have them also support creating virtual environments. In the end,
though, it was considered an orthogonal idea.
Open Issues
===========
Should the `dependency_group` argument take an iterable?
--------------------------------------------------------
This would allow for specifying non-overlapping dependency groups in
a single call, e.g. "docs" and "test" groups which have independent
dependencies but which a developer may want to install simultaneously
while doing development.
Is the installer backend executed in-process?
---------------------------------------------
If the *installer backend* is executed in-process then it greatly
simplifies knowing what environment to install for/into, as the live
Python environment can be queried for appropriate information.
Executing out-of-process allows for minimizing potential issues of
clashes between the environment being installed into and the
*installer backend* (and potentially *universal installer*).
Enforce that results from the proposed interface feed into other parts?
-----------------------------------------------------------------------
E.g. the results from ``get_dependencies_to_install()`` and
``get_dependency_groups()`` can be passed into ``invoke_install()``.
This would prevent drift between the results of various parts of the
proposed interface, but it makes more of the interface required
instead of optional.
Raising exceptions instead of exit codes for failure conditions
---------------------------------------------------------------
It has been suggested that instead of returning an exit code the API
should raise exceptions. If you view this PEP as helping to translate
current installers into *installer backends*, then relying on exit
codes makes sense. There's is also the point that the APIs have no
specific return value, so passing along an exit code does not
interfere with what the functions return.
Compare that to raising exceptions in case of an error. That could
potentially provide a more structured approach to error raising,
although to be able to capture errors it would require specifying
exception types as part of the interface.
References
==========
.. _build: https://github.com/pypa/build
.. _Buildpack: https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python
.. _Dependabot: https://dependabot.com/
.. _Flit: https://flit.readthedocs.io
.. _mamba: https://github.com/mamba-org/mamba
.. _pip: https://pip.pypa.io
.. _Pipenv: https://pipenv-fork.readthedocs.io/en/latest/
.. _Poetry: https://python-poetry.org/