iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/pep-0650.rst

540 lines
20 KiB
ReStructuredText
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

PEP: 650
Title: Specifying Installer Requirements for Python Projects
Author: Vikram Jayanthi <vikramjayanthi@google.com>,
Dustin Ingram <di@python.org>,
Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 16-Jul-2020
Post-History: 2021-01-14
Abstract
========
Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de-facto
standard, other installers such as Poetry_ or Pipenv_ are popular as
well due to offering unique features which are optimal for certain
workflows.
While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes them
hard to support uniformly. Specifically, the lack of a standard
requirements file for declaring dependencies means that each tool must
be explicitly used in order to install dependencies specified with
their respective format.
By providing a standardized API that can be used to invoke a
compatible installer, we can solve this problem without needing to
resolve individual concerns, unique requirements, and
incompatibilities between different installers and their lock files.
Installers that implement the specification can be invoked in a
uniform way, allowing users to use their installer of choice as if
they were invoking it directly.
Terminology
===========
Installer interface
The interface by which an *installer backend* and a
*universal installer* interact.
Universal installer
An installer that can invoke an *installer backend* by calling the
optional invocation methods of the *installer interface*.
Installer backend
An installer that implements the *installer interface*, allowing
it to be invoked by a *universal installer*. An
*installer backend* may also be a *universal installer* as well,
but it is not required.
Dependency group
A set of dependencies that are related. For example, a development
environment dependency group might include linting and formatting
modules while a production dependency group contains dependencies
required for deployment.
Motivation
==========
This specification allows anyone to invoke and interact with
installers that implement the specified interface, allowing for a
universally supported layer on top of existing tool-specific
installation processes.
This in turn would enable the use of all installers that implement the
specified interface to be used in environments that support a single
*universal installer*, as long as that installer implements this
specification as well.
Below, we identify various use cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package
installers.
Providers
---------
Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python
packaging and consequently Python package installers. Two different
types of providers are considered:
Platform/Infrastructure Providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers
for their users to install Python dependencies. Most support only pip,
however there is user demand for other Python installers. Most
providers do not want to maintain support for more than one installer
because of the complexity it adds to their software or service and the
resources it takes to do so.
Via this specification, we can enable the provider-supported
*universal installer* to invoke the user-desired *installer backend*
without the provider’s platform needing to have specific knowledge of
said backend.
IDE Providers
^^^^^^^^^^^^^
Integrated development environments may interact with Python package
installation and management. Most only support pip as a Python package
installer, and users are required to find work arounds to install
their dependencies using other package installers. Similar to the
situation with PaaS & IaaS providers, IDE providers do not want to
maintain support for N different Python installers. Instead,
implementers of the installer interface (*installer backends*) could
be invoked by the IDE by it acting as a *universal installer*.
Developers
----------
Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:
Developers using PaaS & IaaS providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Most PaaS and IaaS providers only support one Python package
installer: pip_. (Some exceptions include Heroku's Python buildpack_,
which supports pip and Pipenv_). This dictates the installers that
developers can use while working with these providers, which might not
be optimal for their application or workflow.
Installers adopting this PEP to become installer backends would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a *universal installer*.
Developers using IDEs
^^^^^^^^^^^^^^^^^^^^^
Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package
installer.
If the IDE uses/provides a *universal installer* it would allow for
any *installer backend* that the developer wanted to be used to
install dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE's workflow more
closely.
Developers working with other developers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaborator’s preference.
Upgraders & Package Infrastructure Providers
--------------------------------------------
Package upgraders and package infrastructure in CI/CD such as
Dependabot_, PyUP_, etc. currently support a few installers. They work
by parsing and editing the installer-specific dependency files
directly (such as ``requirements.txt`` or ``poetry.lock``) with
relevant package information such as upgrades, downgrades, or new
hashes. Similar to Platform and IDE providers, most of these providers
do not want to support N different Python package installers as that
would require supporting N different file types.
The current system relies on these services/bots to keep up support as
new file formats and types are created and existing ones are changed.
By implementing this specification we can allow these services/bots to
interface through the spec and parse/write changes to dependencies
consistently, regardless of which installer is being used. Additionally
it would allow for more innovation in the space as it becomes easier
to support different installers and gives developers a standardized
way of interacting with them.
Open Source Community
---------------------
Specifying installer requirements and adopting this PEP will reduce
the friction between Python package installers and people's workflows.
Consequently it will reduce the friction between Python package
installers and 3rd party infrastructure/technologies such as PaaS or
IDEs. Overall, it will allow for easier development, deployment and
maintenance of Python projects as Python package installation becomes
simpler and more interoperable.
Specifying requirements and creating an interface for installers can
also increase the pace of innovation around installers. This would
allow for installers to experiment and add unique functionality
without requiring the rest of the ecosystem to do the same. Support
becomes easier and more likely for a new installer regardless of the
functionality it adds and the format in which it writes dependencies,
while reducing the developer time and resources needed to do so.
Specification
=============
Similar to how :pep:`517` specifies build systems, the install system
information will live in the ``pyproject.toml`` file under the
``install-system`` table.
[install-system]
---------------------
The install-system table is used to store install-system relevant data
and information. There are multiple required keys for this table:
``requires`` and ``install-backend``. The ``requires`` key holds the
minimum requirements for the install system to execute. The
``install-backend`` key holds the name of the install backend’s entry
point. This will allow the *universal installer* to install the
requirements for the *installer backend* itself to execute (not the
requirements that the *installer backend* itself will install) as well
as invoke the *installer backend*.
If either of the required keys are missing or empty then the
*universal installer* SHOULD raise an error.
All package names interacting with this interface are assumed to
follow :pep:`508`'s "Dependency specification for Python Software
Packages" format.
An example ``install-system`` table::
#pyproject.toml
[install-system]
#Eg : pipenv
requires = ["pipenv"]
install-backend = "pipenv.api:main"
Installer Requirements:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The requirements specified by the ``requires`` key must be within the
constraints specified by :pep:`517`. Specifically, that dependency
cycles are not permitted and the *universal installer* SHOULD refuse
to install the dependencies if a cycle is detected.
Additional parameters or tool specific data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Additional parameters or tool (*installer backend*) data may also be
stored in the ``pyproject.toml`` file. This would be in the “tool.*”
table as specified by :pep:`518`. For example if the
*installer backend* is Poetry and you wanted to specify multiple
dependency groups, the tool.poetry tables could look like this:
::
[tool.poetry.dev-dependencies]
dependencies = "dev"
[tool.poetry.deploy]
dependencies = "deploy"
Installer interface:
----------------------------------
The *installer interface* contains mandatory and optional hooks.
Compliant *installer backends* MUST implement the mandatory hooks and
MAY implement the optional hooks. A *universal installer* MAY
implement any of the *installer backend* hooks itself, to act as both
a *universal installer* and *installer backend*, but this is not
required.
All hooks take ``**kwargs`` arbitrary parameters that a
*installer backend* may require that are not already specified,
allowing for backwards compatibility. If unexpected parameters are
passed to the *installer backend*, it should ignore them.
The following information is akin to the corresponding section in
:pep:`517`. The hooks may be called with keyword arguments, so
*installer backends* implementing them should be careful to make sure
that their signatures match both the order and the names of the
arguments above.
All hooks MAY print arbitrary informational text to ``stdout`` and
``stderr``. They MUST NOT read from ``stdin``, and the
*universal installer* MAY close ``stdin`` before invoking the hooks.
The *universal installer* may capture ``stdout`` and/or ``stderr``
from the backend. If the backend detects that an output stream is not
a terminal/console (e.g. not ``sys.stdout.isatty()``), it SHOULD
ensure that any output it writes to that stream is ``UTF-8`` encoded.
The *universal installer* MUST NOT fail if captured output is not
valid UTF-8, but it MAY not preserve all the information in that case
(e.g. it may decode using the replace error handler in Python). If the
output stream is a terminal, the *installer backend* is responsible
for presenting its output accurately, as for any program running in a
terminal.
If a hook raises an exception, or causes the process to terminate,
then this indicates an error.
Mandatory hooks:
------------------------------------
invoke_install
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Installs the dependencies::
def invoke_install(
path : typing.Union[str, bytes, os.PathLike[str]],
*,
dependency_group : string = None,
**kwargs
) -> int:
...
* ``path`` : An absolute path where the *installer backend* should be
invoked from (e.g. the directory where ``pyproject.toml`` is
located).
* ``dependency_group`` : An optional flag specifying a dependency
group that the *installer backend* should install. The install will
error if the dependency group doesn't exist. A user can find all
dependency groups by calling
``get_dependencies_to_install().keys()`` if dependency groups are
supported by the *installer backend*.
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
compatibility.
* Returns : An exit code (int). 0 if successful, any positive integer
if unsuccessful.
The *universal installer* will use the exit code to determine if the
installation is successful and SHOULD return the exit code itself.
Optional hooks:
---------------------------------
invoke_uninstall
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Uninstall the specified dependencies::
def invoke_uninstall(
*,
dependency_group : string = None,
**kwargs
) -> int:
...
* ``dependency_group`` : An optional flag specifying a dependency
group that the *installer backend* should uninstall.
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
compatibility.
* Returns : An exit code (int). 0 if successful, any positive integer
if unsuccessful.
The *universal installer* MUST invoke the *installer backend* at the
same path that the *universal installer* itself was invoked.
The *universal installer* will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.
get_dependencies_to_install
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the dependencies that would be installed by
``invoke_install(...)``. This allows package upgraders
(e.g., Dependabot) to retrieve the dependencies attempting to be
installed without parsing the dependency file::
def get_dependencies_to_install(
dependency_group : string = None,
**kwargs
) -> List[str]:
...
* ``dependency_group`` : Specify a dependency group to get the
dependencies ``invoke_install(...)`` would install for that
dependency group.
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
compatibility.
* Returns: A list of dependencies (:pep:`508` strings) to install.
If the group is specified, the *installer backend* MUST return the
dependencies corresponding to the provided dependency group. If the
specified group doesn't exist, or dependency groups are not supported
by the *installer backend*, the *installer backend* MUST raise an
error.
If the group is not specified, and the *installer backend* provides
the concept of a default/unspecified group, the *installer backend*
MAY return the dependencies for the default/unspecified group, but
otherwise MUST raise an error.
get_dependency_groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the dependency groups available to be installed. This allows
*universal installers* to enumerate all dependency groups the
*installer backend* is aware of::
def get_dependency_groups(
**kwargs
) -> FrozenSet[str]
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
compatibility.
* Returns: A set of known dependency groups, as strings The empty set
represents no dependency groups.
update_dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Outputs a dependency file based off of inputted package list::
def update_dependencies(
dependency_specifiers : Iterable[str],
*,
dependency_group=None,
**kwargs
) -> int:
...
* ``dependency_specifiers`` : An iterable of dependencies as
:pep:`508` strings that are being updated, for example :
``["requests==2.8.1", ...]``. Optionally for a specific dependency
group.
* ``dependency_group`` : The dependency group that the list of
packages is for.
* ``**kwargs`` : Arbitrary parameters that a *installer backend* may
require that are not already specified, allows for backwards
compatibility.
* Returns : An exit code (int). 0 if successful, any positive integer
if unsuccessful.
Rationale
=========
All hooks take ``**kwargs`` to allow for backwards compatibility and
allow for tool specific *installer backend* functionality which
requires a user to provide additional information not required by the
hook.
While *installer backends* must be Python packages, what they do when
invoked is an implementation detail of that tool. For example, an
*installer backend* could act as a wrapper for a platform package
manager (e.g., ``apt``).
Backwards Compatibility
=======================
This PEP would have no impact on pre-existing code and functionality
as it only adds new functionality to a *universal installer*. Any
existing installer should maintain its existing functionality and use
cases, therefore having no backwards compatibility issues. Only code
aiming to take advantage of this new functionality will have
motivation to make changes to their pre existing code.
Security Implications
=====================
A malicious user has no increased ability or easier access to anything
with the addition of standardized installer specifications. The
installer that could be invoked by a *universal installer* via the
interface specified in this PEP would be explicitly declared by the
user. If the user has chosen a malicious installer, then invoking it
with a *universal installer* is no different than the user invoking
the installer directly. A malicious installer being an
*installer backend* doesn't give it additional permissions or
abilities.
Rejected Ideas
==============
A standardized lock file
------------------------
A standardized lock file would solve a lot of the same problems that
specifying installer requirements would. For example, it would allow
for PaaS/IaaS to just support one installer that could read the
standardized lock file regardless of the installer that created it.
The problem with a standardized lock file is the difference in needs
between Python package installers as well as a fundamental issue with
creating reproducible environments via the lockfile (one of the main
benefits).
Needs and information stored in dependency files between installers
differ significantly and are dependent on installer functionality. For
example, a Python package installer such as Poetry requires
information for all Python versions and platforms and calculates
appropriate hashes while pip wouldn't. Additionally, pip would not be
able to guarantee recreating the same environment (install the exact
same dependencies) as it is outside the scope of its functionality.
This makes a standardized lock file harder to implement and makes it
seem more appropriate to make lock files tool specific.
Have installer backends support creating virtual environments
-------------------------------------------------------------
Because installer backends will very likely have a concept of virtual
environments and how to install into them, it was briefly considered
to have them also support creating virtual environments. In the end,
though, it was considered an orthogonal idea.
References
==========
.. _Buildpack: https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python
.. _Dependabot: https://dependabot.com/
.. _pip: https://pip.pypa.io
.. _Pipenv: https://pipenv-fork.readthedocs.io/en/latest/
.. _Poetry: https://python-poetry.org/
.. _PyUP: https://pyup.io/
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Reference in New Issue View Git Blame Copy Permalink