From 67ab198c25ec6423c9583dd1351906070a7c1c06 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 13 Jan 2021 17:40:00 -0800 Subject: [PATCH] PEP 650: Specifying Installer Requirements for Python Projects (#1762) --- Makefile | 6 +- pep-0650.rst | 538 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 542 insertions(+), 2 deletions(-) create mode 100644 pep-0650.rst diff --git a/Makefile b/Makefile index 213bb7532..a22ee7579 100644 --- a/Makefile +++ b/Makefile @@ -7,6 +7,8 @@ PEP2HTML=pep2html.py PYTHON=python3 +VENV_DIR=venv + .SUFFIXES: .txt .html .rst .txt.html: @@ -40,8 +42,8 @@ update: git pull https://github.com/python/peps.git venv: - $(PYTHON) -m venv venv - ./venv/bin/python -m pip install -U docutils + $(PYTHON) -m venv $(VENV_DIR) + ./$(VENV_DIR)/bin/python -m pip install -U docutils package: all rss mkdir -p build/peps diff --git a/pep-0650.rst b/pep-0650.rst new file mode 100644 index 000000000..662143de9 --- /dev/null +++ b/pep-0650.rst @@ -0,0 +1,538 @@ +PEP: 650 +Title: Specifying Installer Requirements for Python Projects +Author: Vikram Jayanthi , + Dustin Ingram , + Brett Cannon +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 16-Jul-2020 +Post-History: + + +Abstract +======== + +Python package installers are not completely interoperable with each +other. While pip is the most widely used installer and a de-facto +standard, other installers such as Poetry_ or Pipenv_ are popular as +well due to offering unique features which are optimal for certain +workflows. + +While the abundance of installer options is good for end-users with +specific needs, the lack of interoperability between them makes them +hard to support uniformly. Specifically, the lack of a standard +requirements file for declaring dependencies means that each tool must +be explicitly used in order to install dependencies specified with +their respective format. + +By providing a standardized API that can be used to invoke a +compatible installer, we can solve this problem without needing to +resolve individual concerns, unique requirements, and +incompatibilities between different installers and their lock files. + +Installers that implement the specification can be invoked in a +uniform way, allowing users to use their installer of choice as if +they were invoking it directly. + +Terminology +=========== + +Installer interface + The interface by which an *installer backend* and a + *universal installer* interact. + +Universal installer + An installer that can invoke an *installer backend* by calling the + optional invocation methods of the *installer interface*. + +Installer backend + An installer that implements the *installer interface*, allowing + it to be invoked by a *universal installer*. An + *installer backend* may also be a *universal installer* as well, + but it is not required. + +Dependency group + A set of dependencies that are related. For example, a development + environment dependency group might include linting and formatting + modules while a production dependency group contains dependencies + required for deployment. + + +Motivation +========== + +This specification allows anyone to invoke and interact with +installers that implement the specified interface, allowing for a +universally supported layer on top of existing tool-specific +installation processes. + +This in turn would enable the use of all installers that implement the +specified interface to be used in environments that support a single +*universal installer*, as long as that installer implements this +specification as well. + +Below, we identify various use cases applicable to stakeholders in the +Python community and anyone who interacts with Python package +installers. For developers or companies, this PEP would allow for +increased functionality and flexibility with Python package +installers. + +Providers +--------- + +Providers are the parties (organization, person, community, etc.) that +supply a service or software tool which interacts with Python +packaging and consequently Python package installers. Two different +types of providers are considered: + +Platform/Infrastructure Providers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Platform providers (cloud environments, application hosting, etc.) and +infrastructure service providers need to support package installers +for their users to install Python dependencies. Most support only pip, +however there is user demand for other Python installers. Most +providers do not want to maintain support for more than one installer +because of the complexity it adds to their software or service and the +resources it takes to do so. + +Via this specification, we can enable the provider-supported +*universal installer* to invoke the user-desired *installer backend* +without the provider’s platform needing to have specific knowledge of +said backend. + +IDE Providers +^^^^^^^^^^^^^ + +Integrated development environments may interact with Python package +installation and management. Most only support pip as a Python package +installer, and users are required to find work arounds to install +their dependencies using other package installers. Similar to the +situation with PaaS & IaaS providers, IDE providers do not want to +maintain support for N different Python installers. Instead, +implementers of the installer interface (*installer backends*) could +be invoked by the IDE by it acting as a *universal installer*. + +Developers +---------- + +Developers are teams, people, or communities that code and use Python +package installers and Python packages. Three different types of +developers are considered: + +Developers using PaaS & IaaS providers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Most PaaS and IaaS providers only support one Python package +installer: pip_. (Some exceptions include Heroku's Python buildpack_, +which supports pip and Pipenv_). This dictates the installers that +developers can use while working with these providers, which might not +be optimal for their application or workflow. + +Installers adopting this PEP to become installer backends would allow +users to use third party platforms/infrastructure without having to +worry about which Python package installer they are required to use as +long as the provider uses a *universal installer*. + +Developers using IDEs +^^^^^^^^^^^^^^^^^^^^^ + +Most IDEs only support pip or a few Python package installers. +Consequently, developers must use workarounds or hacky methods to +install their dependencies if they use an unsupported package +installer. + +If the IDE uses/provides a *universal installer* it would allow for +any *installer backend* that the developer wanted to be used to +install dependencies, freeing them of any extra work to install their +dependencies in order to integrate into the IDE's workflow more +closely. + +Developers working with other developers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Developers want to be able to use the installer of their choice while +working with other developers, but currently have to synchronize their +installer choice for compatibility of dependency installation. If all +preferred installers instead implemented the specified interface, it +would allow for cross use of installers, allowing developers to choose +an installer regardless of their collaborator’s preference. + +Upgraders & Package Infrastructure Providers +-------------------------------------------- + +Package upgraders and package infrastructure in CI/CD such as +Dependabot_, PyUP_, etc. currently support a few installers. They work +by parsing and editing the installer-specific dependency files +directly (such as ``requirements.txt`` or ``poetry.lock``) with +relevant package information such as upgrades, downgrades, or new +hashes. Similar to Platform and IDE providers, most of these providers +do not want to support N different Python package installers as that +would require supporting N different file types. + +The current system relies on these services/bots to keep up support as +new file formats and types are created and existing ones are changed. +By implementing this specification we can allow these services/bots to +interface through the spec and parse/write changes to dependencies +consistently, regardless of which installer is being used. Additionally +it would allow for more innovation in the space as it becomes easier +to support different installers and gives developers a standardized +way of interacting with them. + +Open Source Community +--------------------- + +Specifying installer requirements and adopting this PEP will reduce +the friction between Python package installers and people's workflows. +Consequently it will reduce the friction between Python package +installers and 3rd party infrastructure/technologies such as PaaS or +IDEs. Overall, it will allow for easier development, deployment and +maintenance of Python projects as Python package installation becomes +simpler and more interoperable. + +Specifying requirements and creating an interface for installers can +also increase the pace of innovation around installers. This would +allow for installers to experiment and add unique functionality +without requiring the rest of the ecosystem to do the same. Support +becomes easier and more likely for a new installer regardless of the +functionality it adds and the format in which it writes dependencies, +while reducing the developer time and resources needed to do so. + +Specification +============= + +Similar to how :pep:`517` specifies build systems, the install system +information will live in the ``pyproject.toml`` file under the +``install-system`` table. + +[install-system] +--------------------- + +The install-system table is used to store install-system relevant data +and information. There are multiple required keys for this table: +``requires`` and ``install-backend``. The ``requires`` key holds the +minimum requirements for the install system to execute. The +``install-backend`` key holds the name of the install backend’s entry +point. This will allow the *universal installer* to install the +requirements for the *installer backend* itself to execute (not the +requirements that the *installer backend* itself will install) as well +as invoke the *installer backend*. + +If either of the required keys are missing or empty then the +*universal installer* SHOULD raise an error. + +All package names interacting with this interface are assumed to +follow :pep:`508`'s "Dependency specification for Python Software +Packages" format. + +An example ``install-system`` table:: + + #pyproject.toml + [install-system] + #Eg : pipenv + requires = ["pipenv"] + install-backend = "pipenv.api:main" + + +Installer Requirements: +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The requirements specified by the ``requires`` key must be within the +constraints specified by :pep:`517`. Specifically, that dependency +cycles are not permitted and the *universal installer* SHOULD refuse +to install the dependencies if a cycle is detected. + +Additional parameters or tool specific data +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Additional parameters or tool (*installer backend*) data may also be +stored in the ``pyproject.toml`` file. This would be in the “tool.*” +table as specified by :pep:`518`. For example if the +*installer backend* is Poetry and you wanted to specify multiple +dependency groups, the tool.poetry tables could look like this: + +:: + + [tool.poetry.dev-dependencies] + dependencies = "dev" + + [tool.poetry.deploy] + dependencies = "deploy" + + +Installer interface: +---------------------------------- +The *installer interface* contains mandatory and optional hooks. +Compliant *installer backends* MUST implement the mandatory hooks and +MAY implement the optional hooks. A *universal installer* MAY +implement any of the *installer backend* hooks itself, to act as both +a *universal installer* and *installer backend*, but this is not +required. + +All hooks take ``**kwargs`` arbitrary parameters that a +*installer backend* may require that are not already specified, +allowing for backwards compatibility. If unexpected parameters are +passed to the *installer backend*, it should ignore them. + +The following information is akin to the corresponding section in +:pep:`517`. The hooks may be called with keyword arguments, so +*installer backends* implementing them should be careful to make sure +that their signatures match both the order and the names of the +arguments above. + +All hooks MAY print arbitrary informational text to ``stdout`` and +``stderr``. They MUST NOT read from ``stdin``, and the +*universal installer* MAY close ``stdin`` before invoking the hooks. + +The *universal installer* may capture ``stdout`` and/or ``stderr`` +from the backend. If the backend detects that an output stream is not +a terminal/console (e.g. not ``sys.stdout.isatty()``), it SHOULD +ensure that any output it writes to that stream is ``UTF-8`` encoded. +The *universal installer* MUST NOT fail if captured output is not +valid UTF-8, but it MAY not preserve all the information in that case +(e.g. it may decode using the replace error handler in Python). If the +output stream is a terminal, the *installer backend* is responsible +for presenting its output accurately, as for any program running in a +terminal. + +If a hook raises an exception, or causes the process to terminate, +then this indicates an error. + + + +Mandatory hooks: +------------------------------------ +invoke_install +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Installs the dependencies:: + + def invoke_install( + path : typing.Union[str, bytes, os.PathLike[str]], + *, + dependency_group : string = None, + **kwargs + ) -> int: + ... + +* ``path`` : An absolute path where the *installer backend* should be + invoked from (e.g. the directory where ``pyproject.toml`` is + located). +* ``dependency_group`` : An optional flag specifying a dependency + group that the *installer backend* should install. The install will + error if the dependency group doesn't exist. A user can find all + dependency groups by calling + ``get_dependencies_to_install().keys()`` if dependency groups are + supported by the *installer backend*. +* ``**kwargs`` : Arbitrary parameters that a *installer backend* may + require that are not already specified, allows for backwards + compatibility. + +* Returns : An exit code (int). 0 if successful, any positive integer + if unsuccessful. + +The *universal installer* will use the exit code to determine if the +installation is successful and SHOULD return the exit code itself. + +Optional hooks: +--------------------------------- + +invoke_uninstall +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Uninstall the specified dependencies:: + + def invoke_uninstall( + *, + dependency_group : string = None, + **kwargs + ) -> int: + ... + +* ``dependency_group`` : An optional flag specifying a dependency + group that the *installer backend* should uninstall. +* ``**kwargs`` : Arbitrary parameters that a *installer backend* may + require that are not already specified, allows for backwards + compatibility. + +* Returns : An exit code (int). 0 if successful, any positive integer + if unsuccessful. + +The *universal installer* MUST invoke the *installer backend* at the +same path that the *universal installer* itself was invoked. + +The *universal installer* will use the exit code to determine if the +uninstall is successful and SHOULD return the exit code itself. + +get_dependencies_to_install +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the dependencies that would be installed by +``invoke_install(...)``. This allows package upgraders +(e.g., Dependabot) to retrieve the dependencies attempting to be +installed without parsing the dependency file:: + + def get_dependencies_to_install( + dependency_group : string = None, + **kwargs + ) -> List[str]: + ... + +* ``dependency_group`` : Specify a dependency group to get the + dependencies ``invoke_install(...)`` would install for that + dependency group. +* ``**kwargs`` : Arbitrary parameters that a *installer backend* may + require that are not already specified, allows for backwards + compatibility. + +* Returns: A list of dependencies (:pep:`508` strings) to install. + +If the group is specified, the *installer backend* MUST return the +dependencies corresponding to the provided dependency group. If the +specified group doesn't exist, or dependency groups are not supported +by the *installer backend*, the *installer backend* MUST raise an +error. + +If the group is not specified, and the *installer backend* provides +the concept of a default/unspecified group, the *installer backend* +MAY return the dependencies for the default/unspecified group, but +otherwise MUST raise an error. + +get_dependency_groups +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the dependency groups available to be installed. This allows +*universal installers* to enumerate all dependency groups the +*installer backend* is aware of:: + + def get_dependency_groups( + **kwargs + ) -> FrozenSet[str] + +* ``**kwargs`` : Arbitrary parameters that a *installer backend* may + require that are not already specified, allows for backwards + compatibility. + +* Returns: A set of known dependency groups, as strings The empty set + represents no dependency groups. + +update_dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Outputs a dependency file based off of inputted package list:: + + def update_dependencies( + dependency_specifiers : Iterable[str], + *, + dependency_group=None, + **kwargs + ) -> int: + ... + +* ``dependency_specifiers`` : An iterable of dependencies as + :pep:`508` strings that are being updated, for example : + ``["requests==2.8.1", ...]``. Optionally for a specific dependency + group. +* ``dependency_group`` : The dependency group that the list of + packages is for. +* ``**kwargs`` : Arbitrary parameters that a *installer backend* may + require that are not already specified, allows for backwards + compatibility. + +* Returns : An exit code (int). 0 if successful, any positive integer + if unsuccessful. + + +Rationale +========= + +All hooks take ``**kwargs`` to allow for backwards compatibility and +allow for tool specific *installer backend* functionality which +requires a user to provide additional information not required by the +hook. + +While *installer backends* must be Python packages, what they do when +invoked is an implementation detail of that tool. For example, an +*installer backend* could act as a wrapper for a platform package +manager (e.g., ``apt``). + + +Backwards Compatibility +======================= + +This PEP would have no impact on pre-existing code and functionality +as it only adds new functionality to a *universal installer*. Any +existing installer should maintain its existing functionality and use +cases, therefore having no backwards compatibility issues. Only code +aiming to take advantage of this new functionality will have +motivation to make changes to their pre existing code. + + +Security Implications +===================== + +A malicious user has no increased ability or easier access to anything +with the addition of standardized installer specifications. The +installer that could be invoked by a *universal installer* via the +interface specified in this PEP would be explicitly declared by the +user. If the user has chosen a malicious installer, then invoking it +with a *universal installer* is no different than the user invoking +the installer directly. A malicious installer being an +*installer backend* doesn't give it additional permissions or +abilities. + + +Rejected Ideas +============== + +A standardized lock file +------------------------ + +A standardized lock file would solve a lot of the same problems that +specifying installer requirements would. For example, it would allow +for PaaS/IaaS to just support one installer that could read the +standardized lock file regardless of the installer that created it. +The problem with a standardized lock file is the difference in needs +between Python package installers as well as a fundamental issue with +creating reproducible environments via the lockfile (one of the main +benefits). + +Needs and information stored in dependency files between installers +differ significantly and are dependent on installer functionality. For +example, a Python package installer such as Poetry requires +information for all Python versions and platforms and calculates +appropriate hashes while pip wouldn't. Additionally, pip would not be +able to guarantee recreating the same environment (install the exact +same dependencies) as it is outside the scope of its functionality. +This makes a standardized lock file harder to implement and makes it +seem more appropriate to make lock files tool specific. + + +Have installer backends support creating virtual environments +------------------------------------------------------------- + +Because installer backends will very likely have a concept of virtual +environments and how to install into them, it was briefly considered +to have them also support creating virtual environments. In the end, +though, it was considered an orthogonal idea. + +References +========== + +.. _Buildpack: https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python +.. _Dependabot: https://dependabot.com/ +.. _pip: https://pip.pypa.io +.. _Pipenv: https://pipenv-fork.readthedocs.io/en/latest/ +.. _Poetry: https://python-poetry.org/ +.. _PyUP: https://pyup.io/ + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: +