python-peps/pep-0665.rst

PEP: 665
Title: Specifying Installation Requirements for Python Projects
Author: Brett Cannon <brett@python.org>,
        Pradyun Gedam <pradyunsg@gmail.com>,
        Tzu-ping Chung <uranusjr@gmail.com>
PEP-Delegate:
Discussions-To: https://discuss.python.org/t/pep-665-specifying-installation-requirements-for-python-projects/9911
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Jul-2021
Post-History: 29-Jul-2021
Resolution:

========
Abstract
========

This PEP specifies a file format to list the Python package
installation requirements for a project. The list of projects is
considered exhaustive for the installation target and thus
*locked down*, not requiring any information beyond the platform being
installed for and the *lock file* listing the required dependencies
to perform a successful installation of dependencies.


==========
Motivation
==========

Thanks to PEP 621, projects have a way to list their direct/top-level
dependencies which they need to have installed. But PEP 621 also
(purposefully) omits two key details that often become important for
projects:

#. A listing of all indirect/transitive dependencies
#. Specifying (at least) specific versions of dependencies for
   reproducible installations

Both needs can be important for various reasons when creating a new
environment. Consider a project which
is an application that is deployed somewhere (either to users as a
desktop app or to a server). Without a complete listing of all
dependencies and the specific versions to use, there can be a skew
between developers of the same project, or developer and user, based on
what versions of a project's dependencies happen to be available at the
time of installation in a new environment. For instance, a dependency may
have v1 as the newest version on Monday when one developer installed the
dependency, while v2 comes out on Wednesday when another developer
installs the same dependency. Now the two developers are working against
two different versions of the same dependency, which can lead to different
outcomes. This is the use-case of developing a desktop or server
application where one might have a ``requirements.txt`` file which
specifies exact versions of various packages.

Another important reason for reproducible installations is for
security purposes. Guaranteeing that the same binary data is
downloaded and installed for all installations of an app makes sure that no
bad actor has somehow changed a dependency's binary data in a malicious
way. A lock file can assist in this guarantee by recording the exact
details of what should be installed and how to verify that those
dependencies have not changed any bytes unexpectedly. This is the use-case
of developing a secure application using a ``requirements.txt`` file which
specifies the hash of all the packages that should be installed.

Tied into this concept of reproducibility is the speed at which an
environment can be recreated. If you created a lock file as part of
your local development, it can be used to speed up recreating that
development environment by minimizing having to query the network or the
scope of the possible resolution of dependencies. This makes recreating
your local development environment faster as the amount of work required
to calculate what dependencies to install has been minimized. This is the
use-case of when you are working on a library or some such project where
the lock file is not committed to version control and the lock file used as
a local cache of installation resolution details, such as an uncommitted
``poetry.lock`` file.

The community itself has also shown a need for lock files based on the
fact that multiple tools have independently created their own lock
file formats:

#. PDM_
#. `pip-tools`_
#. Pipenv_
#. Poetry_
#. Pyflow_

Other programming language communities have also shown the usefulness
of lock files by developing their own solution to this problem. Some
of those communities include:

#. Dart_
#. npm_/Node
#. Rust_


Below, we identify some use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers who are the ultimate consumers of a lock file (this is not
considered exhaustive and is borrowed from PEP 650).


---------
Providers
---------

Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python
packaging. Two different types of providers are considered:


Platform/Infrastructure Providers
=================================

Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers
for their users to install Python dependencies. Most only support
``requirements.txt`` files and a smattering of other file formats for
listing a project's dependencies. Most providers do not want to maintain
support for more than one dependency specification format
because of the complexity it adds to their software or service and the
resources it takes to do so (e.g. not all platform providers have
the staffing to support pip-tools, Poetry, Pipenv, etc.).

This PEP would allow platform providers to declare support for this PEP
and thus only have to support one dependency specification format. What
this would mean is developers could use whatever toolchain they preferred
for development as long as they could emit a file that implemented this
PEP. This then allows developers to not have to align with what their
platform providers supports as long as everyone agrees to implementing
this PEP.


IDE Providers
=============

Integrated development environments may interact with Python package
installation and management. Most only support select few tools, and
users are required to find work arounds to install
their dependencies using other package installers. Similar to the
situation with PaaS & IaaS providers, IDE providers do not want to
maintain support for N different formats. Instead, tools would only
need to be able to read files which implement this PEP to perform various
actions (e.g. list all the dependencies of the open project, which ones
are missing, install dependencies, generate the lock file, etc.).

As an example, the Python extension for VS Code has to have custom support
for each installer tool people may use: pip, Poetry, Pipenv, etc. This is
not only tedious by having to track multiple projects and any changes they
make, but it also locks out newer tools whose popularity isn't great
enough to warrant inclusion in the extension.


----------
Developers
----------

Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:


Developers using PaaS & IaaS providers
======================================

Most PaaS and IaaS providers only support one Python package
installer: ``requirements.txt``. This dictates the installers that
developers can use while working with these providers, which might not
be optimal for their application or workflow.

Developers adopting this PEP would be able to use third party
platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider also supports this PEP.


Developers using IDEs
=====================

Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package
installer.

If the IDE uses/supports this PEP it would allow for
any developer to use whatever tooling they wanted to generate
their lock file while the IDE can use whatever tooling it wants to
performs actions with/on the lock file.


Developers working with other developers
========================================

Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaborator’s preference.


--------------------------------------------
Upgraders & Package Infrastructure Providers
--------------------------------------------

Package upgraders and package infrastructure in CI/CD such as
Dependabot_, PyUP_, etc. currently support a few formats. They work
by parsing and editing the dependency files with
relevant package information such as upgrades, downgrades, or new
hashes. Similar to Platform and IDE providers, most of these providers
do not want to support N different formats.

Currently, these services/bots have to implement support for each
format individually. Inevitably, the most popular
formats are supported first, and less popular tools are often never
supported. By implementing this specification, these services/bots can
support one format, allowing users to select the tool
of their choice to generate the file. This will allow for more innovation
in the space, as platforms and IDEs are no longer forced to prematurely
select a "winner" tool which generates a lock file.


---------------------
Open Source Community
---------------------

Specifying installer requirements and adopting this PEP will reduce
the friction between Python package installers and people's workflows.
Consequently, it will reduce the friction between Python package
installers and 3rd party infrastructure/technologies such as PaaS or
IDEs. Overall, it will allow for easier development, deployment and
maintenance of Python projects as Python package installation becomes
simpler and more interoperable.

Specifying a single file format can also increase the pace of innovation
around installers and the generation of dependency graphs. By
decoupling generating the dependency graph details from installation It
allows for each area to grow and innovate independently. It also allows
more flexibility in tool selection on either end of the dependency graph
and installation ends of this process.


=========
Rationale
=========

To begin, two key terms should be defined. A **locker** is a tool
which *produces* a lock file. An **installer** is a tool which
*consumes* a lock file to install the appropriate dependencies.

The expected information flow to occur if this PEP were accepted, from
the specification of top-level dependencies to all necessary
dependencies being installed in a fresh environment, is:

1. Read top-level dependencies from ``pyproject.toml`` (PEP 621).
2. Generate a lock file via a locker in ``pyproject-lock.d/``.
3. Install the appropriate dependencies based entirely on information
   contained in the lock file via an installer.


-----
Goals
-----

The file format should be *machine-readable*, *machine-writable*, and
*human-readable*. Since the assumption is the vast majority of lock
file will be generated by a locker tool, the format should be easy
to write by a locker. As install tools will be consuming the lock
file, the format also needs to be easily read by an installer. But the
format should also be readable by a person as people will inevitably
be performing audits on lock files. Having a format that does not lend
itself towards being read by people would hinder that. This includes
changes to a lock file being readable in a diff format for auditing
changes. It also means that understanding *why* something is in
the lock file should be comprehensible in a diff to assist in auditing
changes.

The lock file format needs to be general enough to support
*cross-platform and cross-environment* specifications of dependencies.
This allows having a single lock file which can work on a myriad of
platforms and environments when that makes sense. This has been shown
as a necessary feature by the various tools in the Python packaging
ecosystem which already have a lock file format (e.g. Pipenv_,
Poetry_, PDM_). This can be accomplished by *allowing* (but **not**
requiring) lockers to defer marker evaluation to the installer, and
thus permitting the locker to include a wider range of *possible*
dependencies that the installer has to work with.

The lock file also needs to support *reproducible installations*. If
one wants to restrict what the lock file covers to a single platform
to guarantee the exact dependencies and files which will be installed,
that should be doable. This can be critical in security contexts for
projects like SecureDrop_.

When a computation could be performed either in the locker or
installer, the preference is to *perform the computation in the
locker*. This is because the assumption is a locker will be executed
less frequently than an installer.

The installer should be able to resolve what to install based entirely
on platform/environment information and what is contained within the
lock file. There should be
*no need to use network or other file system I/O* in order to resolve
what to install.

The lock file should provide enough flexibility to allow lockers and
installers to innovate. While the lock file specification provides a
*common denominator of functionality*, it should not act as a ceiling
for functionality.


---------
Non-Goals
---------

Because of the expected size of lock files, no effort was put into
making lock files *human-writable*.

This PEP makes no attempt to make this work in any special way for
installers to use a lock file to install into a *preexisting* environment.
The assumption is the installer is installing into a *new/fresh*
environment.


=============
Specification
=============

-------
Details
-------

Lock files MUST use the TOML_ file format thanks to its adoption by
PEP 518 for ``pyproject.toml``. This not only prevents the need to
have another file format in the Python packaging ecosystem, but it
also assists in making lock files human-readable.

Lock files MUST be kept in a directory named ``pyproject-lock.d``.
Lock files MUST end with a ``.toml`` file extension. Projects may have
as many lock files as they want using whatever file name stems they
choose. This PEP prescribes no specific way to automatically select
between multiple lock files and installers SHOULD avoid guessing which
lock file is "best-fitting" (this does not preclude situations where
only a single lock file with a certain name is expected to exist and
will be used by default, e.g. a documentation hosting site always
using a lock file named ``pyproject-lock.d/rftd.toml`` when provided).

The following are the top-level keys of the TOML file data format.


``version``
===========

The version of the lock file being used. The key MUST be specified and
it MUST be set to ``1``. The number MUST always be an integer and it
MUST only increment in future updates to the specification. What
consistitutes a version number increase is left to future PEPs or
standards changes.

Tools reading a lock file whose version they don't support MUST raise
an error.


``[tool]``
==========

Tools may create their own sub-tables under the ``tool`` table. The
rules for this table match those for ``pyproject.toml`` and its
``[tool]`` table from the `build system declaration spec`_.


``[metadata]``
==============

A table containing data applying to the overall lock file.


``metadata.marker``
-------------------

An optional key storing a string containing an environment marker as
specified in the `dependency specifier spec`_.


The locker MAY specify an environment marker which specifies any
restrictions the lock file was generated under (e.g. specific Python
versions supported).

If the installer is installing for an environment which does not
satisfy the specified environment marker, the installer MUST raise an
error as the lock file does not support the environment.


``metadata.tags``
-----------------

An optional array of inline tables representing
`platform compatibility tags`_ that the lock file supports. The locker
MAY specify tables in the array which represent the compatibility the
lock file was generated for.

The tables have the possible keys of:

- ``interpreter``
- ``abi``
- ``platform``

representing the parts of the platform compatibility tags. Each key is
optional in a table. These keys MUST represent a single value, i.e.
the values are exploded and not compressed in wheel tag parlance.

If the environment an installer is installing for does not match
**any** table in the array (missing keys in the table means implicit
support for that part of the compatibility), the installer MUST raise
an error as the lock file does not support the environment.


``metadata.needs``
------------------

An array of strings representing the package specifiers for the
top-level/direct dependencies of the lock file as defined by the
`dependency specifier spec`_ (i.e. the root of the dependency graph
for the lock file).

Lockers MUST only allow specifiers which may be satisfiable by the
lock file and the dependency graph the lock file encodes. Lockers MUST
normalize project names according to the `simple repository API`_.


``[package]``
===============

A table containing arrays of tables for each dependency recorded
in the lock file.

Each key of the table is the name of a package which MUST be
normalized according to the `simple repository API`_. If extras are
specified as part of the project to install, the extras are to be
included in the key name and are to be sorted in lexicographic order.

Within the file, the tables for the projects MUST be
sorted by:

#. Project/key name in lexicographic order
#. Package version, newest/highest to older/lowest according to the
   `version specifiers spec`_
#. Extras via lexicographic order


``package.<name>.version``
--------------------------

A required string of the version of the package as specified by the
`version specifiers spec`_.


``package.<name>.needs``
------------------------

An optional key containing an array of strings following the
`dependency specifier spec`_ which specify what other packages this
package depends on. See ``metadata.needs`` for full details.


``package.<name>.needed-by``
------------------------------

A key containing an array of package names which depend on this
package. The package names MUST match the package name as used in the
``package`` table.

The lack of a ``needed-by`` key infers that the package is a
top-level package listed in ``metadata.needs``.


``package.<name>.code``
-----------------------

An array of tables listing files that are available to satisfy
the installation of the package for the specified version in the
``version`` key.

Each table has a ``type`` key which specifies how the code is stored.
All other keys in the table are dependent on the value set for
``type``. The acceptable values for ``type`` are listed below; all
other possible values are reserved for future use.

Tables in the array MUST be sorted in lexicographic order of the value
of ``type``, then lexicographic order for the value of ``url``.

When recording a table, the fields SHOULD be listed in the order
the fields are listed in this specification for consistency to make
diffs of a lock file easier to read.

For all types other than "wheel", an INSTALLER MAY refuse to install
code to avoid arbitrary code execution during installation.

An installer MUST verify the hash of any specified file.


``type="wheel"``
''''''''''''''''

A `wheel file`_ for the package version.

Supported keys in the table are:

- ``url``: a string of location of the wheel file (use the
  ``file:`` protocol for the local file system)
- ``hash-algorithm``: a string of the algorithm used to generate the
  hash value stored in ``hash-value``
- ``hash-value``: a string of the hash of the file contents
- ``interpreter-tag``: (optional) a string of the interpreter portion
  of the wheel tag as specified by the `platform compatibility tags`_
  spec
- ``abi-tag``: (optional) a string of the ABI portion of the wheel tag
  as specified by the `platform compatibility tags`_ spec
- ``platform-tag``: (optional) a string of the platform portion of the
  wheel tag as specified by the `platform compatibility tags`_ spec

If the keys related to `platform compatibility tags`_ are absent then
the installer MUST infer the tags from the URL's file name. If any of
the `platform compatibility tags`_ are specified by a key in the table
then a locker MUST provide all three related keys. The values of the
keys may be compressed tags.


``type="sdist"``
''''''''''''''''

A `source distribution file`_ (sdist) for the package version.

- ``url``: a string of location of the sdist file (use the
  ``file:`` protocol for the local file system)
- ``hash-algorithm``: a string of the algorithm used to generate the
  hash value stored in ``hash-value``
- ``hash-value``: a string of the hash of the file contents


``type="git"``
''''''''''''''

A Git_ version control repository for the package.

- ``url``: a string of location of the repository (use the
  ``file:`` protocol for the local file system)
- ``commit``: a string of the commit of the repository which
  represents the version of the package

The repository MUST follow the `source distribution file`_ spec
for source trees, otherwise an error is to be raised by the locker.

As the commit ID for a Git repository is a hash of the repository's
contents, there is no hash to verify.


``type="source tree"``
''''''''''''''''''''''

A source tree which can be used to build a wheel.

- ``url``: a string of location of the source tree (use the
  ``file:`` protocol for the local file system)
- ``mime-type``: (optional) a string representing the MIME type of the
   URL
- ``hash-algorithm``: (optional for a local directory) a string of the
  algorithm used to generate the hash value stored in ``hash-value``
- ``hash-value``: (optional for a local directory) a string of the
  hash of the file contents

The collection of files MUST follow the `source distribution file`_
spec for source trees, otherwise an error is to be raised by the
locker.

Installers MAY use the file extension, MIME type from HTTP headers,
etc. to infer whether they support the storage mechanism used for the
source tree. If the MIME type cannot be inferred and it is not
specified via ``mime-type`` then an error MUST be raised.

If the source tree is NOT a local directory, then an installer MUST
verify the hash value. Otherwise if the source tree is a local
directory then the ``hash-algorithm`` and ``hash-value`` keys MUST be
left out. The installer MAY warn the user of the use of a local
directory due to the potential change in code since the lock file
was created.


-------
Example
-------

::

        version = 1

        [tool]
        # Tool-specific table ala PEP 518's `[tool]` table.

        [metadata]
        marker = "python_version>='3.6'"

        needs = ["mousebender"]

        [[package.attrs]]
        version = "21.2.0"
        needed-by = ["mousebender"]

        [[package.attrs.code]]
        type = "wheel"
        url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
        hash-algorithm="sha256"
        hash-value = "149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"

        [[package.mousebender]]
        version = "2.0.0"
        needs = ["attrs>=19.3", "packaging>=20.3"]

        [[package.mousebender.code]]
        type = "sdist"
        url = "https://files.pythonhosted.org/packages/35/bc/db77f8ca1ccf85f5c3324e4f62fc74bf6f6c098da11d7c30ef6d0f43e859/mousebender-2.0.0.tar.gz"
        hash-algorithm = "sha256"
        hash-value = "c5953026378e5dcc7090596dfcbf73aa5a9786842357273b1df974ebd79bd760"

        [[package.mousebender.code]]
        type = "wheel"
        url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
        hash-algorithm = "sha256"
        hash-value = "a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"

        [[package.packaging]]
        version = "20.9"
        needs = ["pyparsing>=2.0.2"]
        needed-by = ["mousebender"]

        [[package.packaging.code]]
        type = "git"
        url = "https://github.com/pypa/packaging.git"
        commit = "53fd698b1620aca027324001bf53c8ffda0c17d1"

        [[package.pyparsing]]
        version = "2.4.7"
        needed-by = ["packaging"]

        [[package.pyparsing.code]]
        type="wheel"
        url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
        hash-algorithm="sha256"
        hash-value="ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
        interpreter-tag = "py2.py3"
        abi-tag = "none"
        platform-tag = "any"


----------------------
Installer Expectations
----------------------

Installers MUST implement the
`direct URL origin of installed distributions spec`_ as all packages
installed from a lock file inherently originate from a URL and not a
search of an index by package name and version.

Installers MUST error out if they encounter something they are unable
to handle (e.g. lack of environment marker support).


Example Flow
============

#. Have the user specify which lock file they would like to use in
   ``pyproject-lock.d`` (e.g. ``dev``, ``prod``)

#. Check if the environment supports what is specified in
   ``metadata.tags``; error out if it doesn't

#. Check if the environment supports what is specified in
   ``metadata.marker``; error out if it doesn't

#. Gather the list of package names from ``metadata.needs``, and for
   each listed package ...

   #. Resolve any markers to find the appropriate package to install
   #. Find the most appropriate code to install for the package
   #. Repeat the above steps for packages listed in the ``needs`` key
      for each package found to install

#. For each project collected to install ...

   #. Gather the specified code for the package
   #. Verify hashes of code
   #. Install the packages (if necessary)


=======================
Backwards Compatibility
=======================

As there is no pre-existing specification regarding lock files, there
are no explicit backwards compatibility concerns.

As for pre-existing tools that have their own lock file, some updating
will be required. Most document the lock file name, but not its
contents, in which case the file name of the lock file(s) is the
important part. For projects which do not commit their lock file to
version control, they will need to update the equivalent of their
``.gitignore`` file. For projects that do commit their lock file to
version control, what file(s) get committed will need an update.

For projects which do document their lock file format like pipenv_,
they will very likely need a new major version release.

Specifically for Poetry_, it has an
`export command <https://python-poetry.org/docs/cli/#export>`_ which
should allow Poetry to support this lock file format even if the
project chose not to adopt this PEP as Poetry's primary lock file
format.


=====================
Security Implications
=====================

A lock file should not introduce security issues but instead help
solve them. By requiring the recording of hashes of code, a lock file
is able to help prevent tampering with code since the hash details
were recorded. A lock file also helps prevent unexpected package
updates being installed which may be malicious.


=================
How to Teach This
=================

Teaching of this PEP will very much be dependent on the lockers and
installers being used for day-to-day use. Conceptually, though, users
could be taught that the ``pyproject-lock.d`` directory contains files
which specify what should be installed for a project to work. The
benefits of consistency and security should be emphasized to help
users realize why they should care about lock files.


========================
Reference Implementation
========================

No proof-of-concept or reference implementation currently exists.


==============
Rejected Ideas
==============

----------------------------
File Formats Other Than TOML
----------------------------

JSON_ was briefly considered, but due to:

#. TOML already being used for ``pyproject.toml``
#. TOML being more human-readable
#. TOML leading to better diffs

the decision was made to go with TOML. There was some concern over
Python's standard library lacking a TOML parser, but most packaging
tools already use a TOML parser thanks to ``pyproject.toml`` so this
issue did not seem to be a showstopper. Some have also argued against
this concern in the past by the fact that if packaging tools abhor
installing dependencies and feel they can't vendor a package then the
packaging ecosystem has much bigger issues to rectify than needing to
depend on a third-party TOML parser.


----------------------------------------
Alternative Name to ``pyproject-lock.d``
----------------------------------------

The name ``__lockfile__`` was briefly considered, but the directory
would not sort next to ``pyproject.toml`` in instances where files
and directories were sorted together in lexicographic order. The
current naming is also more obvious in terms of its relationship
to ``pyproject.toml``.


-----------------------------
Supporting a Single Lock File
-----------------------------

At one point the idea of not using a directory of lock files but a
single lock file which contained all possible lock information was
considered. But it quickly became apparent that trying to devise a
data format which could encompass both a lock file format which could
support multiple environments as well as strict lock outcomes for
reproducible builds would become quite complex and cumbersome.

The idea of supporting a directory of lock files as well as a single
lock file named ``pyproject-lock.toml`` was also considered. But any
possible simplicity from skipping the directory in the case of a
single lock file seemed unnecessary. Trying to define appropriate
logic for what should be the ``pyproject-lock.toml`` file and what
should go into ``pyproject-lock.d`` seemed unnecessarily complicated.


-----------------------------------------------
Using a Flat List Instead of a Dependency Graph
-----------------------------------------------

The first version of this PEP proposed that the lock file have no
concept of a dependency graph. Instead, the lock file would list
exactly what should be installed for a specific platform such that
installers did not have to make any decisions about *what* to install,
only validating that the lock file would work for the target platform.

This idea was eventually rejected due to the number of combinations
of potential PEP 508 environment markers. The decision was made that
trying to have lockers generate all possible combinations when a
project wants to be cross-platform would be too much.


-------------------------------------------------------------------------
Being Concerned About Different Dependencies Per Wheel File For a Project
-------------------------------------------------------------------------

It is technically possible for a project to specify different
dependencies between its various wheel files. Taking that into
consideration would then require the lock file to operate not
per-project but per-file. Luckily, specifying different dependencies
in this way is very rare and frowned upon and so it was deemed not
worth supporting.


-------------------------------
Use Wheel Tags in the File Name
-------------------------------

Instead of having the ``metadata.tags`` field there was a suggestion
of encoding the tags into the file name. But due to the addition of
the ``metadata.marker`` field and what to do when no tags were needed,
the idea was dropped.


-----------------------------------------
Using Semantic Versioning for ``version``
-----------------------------------------

Instead of a monotonically increasing integer, using a float was
considered to attempt to convey semantic versioning. In the end,
though, it was deemed more hassle than it was worth as adding a new
key would likely constitute a "major" version change (only if the
key was entirely optional would it be considered "minor"), and
experience with the `core metadata spec`_ suggests there's a bigger
chance parsing will be relaxed and made more strict which is also a
"major" change. As such, the simplicity of using an integer made
sense.


-------------------------------
Alternative Names for ``needs``
-------------------------------

Some other names for what became ``needs`` were ``installs`` and
``dependencies``. In the end a Python beginner was asked which term
they preferred and they found ``needs`` clearer. Since there wasn't
any reason to disagree with that, the decision was to go with
``needs``.


-------------------------------------
Alternative Names for ``needed-by``
-------------------------------------

Other names that were considered were ``dependents``, ``depended-by``,
, ``supports`` and ``required-by``. In the end, ``needed-by`` made
sense and tied into ``needs``.


--------------------------------------------------
Only Allowing a Single Code Location For a Project
--------------------------------------------------

While reproducibility is serviced better by only allowing a single
code location, it limits usability for situations where one wants to
support multiple platforms with a single lock file (which the community
has shown is desired).


-------------------------------------
Support for Branches and Tags for Git
-------------------------------------

Due to the `direct URL origin of installed distributions spec`_
supporting the specification of branches and tags, it was suggested
that lock files support the same thing. But because branches and tags
can change what commit they point to between locking and installation,
that was viewed as a security concern (Git commit IDs are hashes of
metadata and thus are viewed as immutable).


-----------------
Accepting PEP 650
-----------------

PEP 650 was an earlier attempt at trying to tackle this problem by
specifying an API for installers instead of standardizing on a lock file
format (ala PEP 517). The
`initial response <https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657/>`__
to PEP 650 could be considered mild/lukewarm. People seemed to be
consistently confused over which tools should provide what functionality
to implement the PEP. It also potentially incurred more overhead as
it would require executing Python APIs to perform any actions involving
packaging.

This PEP chose to standardize around an artifact instead of an API
(ala PEP 621). This would allow for more tool integrations as it
removes the need to specifically use Python to do things such as
create a lock file, update it, or even install packages listed in
a lock file. It also allows for easier introspection by forcing
dependency graph details to be written in a human-readable format.
It also allows for easier sharing of knowledge by standardizing what
people need to know more (e.g. tutorials become more portable between
tools when it comes to understanding the artifact they produce). It's
also simply the approach other language communities have taken and seem
to be happy with.

===========
Open Issues
===========

---------------------------------------
Allow for Tool-Specific ``type`` Values
---------------------------------------

It has been suggested to allow for custom ``type`` values in the
``code`` table. They would be prefixed with ``x-`` and followed by
the tool's name and then the type, i.e. ``x-<tool>-<type>``. This
would provide enough flexibility for things such as other version
control systems, innovative container formats, etc. to be officially
usable in a lock file.


-----------------------------------------------
Support Variable Expansion in the ``url`` field
-----------------------------------------------

This could include predefined variables like ``PROJECT_ROOT`` for the
directory containing ``pyproject-lock.d`` so URLs to local directories
and files could be relative to the project itself.

Environment variables could be supported to avoid hardcoding things
such as user credentials for Git.


---------------------------------------------------------------
Don't Require Lock Files Be in a ``pyproject-lock.d`` directory
---------------------------------------------------------------

It has been suggested that since installers may very well allow users
to specify the path to a lock file that having this PEP say that
"MUST be kept in a directory named ``pyproject-lock.d``" is pointless
as it is bound to be broken. As such, the suggestion is to change
"MUST" to "SHOULD".


---------------------------------------------------
Record the Date of When the Lock File was Generated
---------------------------------------------------

Since the modification date is not guaranteed to match when the lock
file was generated, it has been suggested to record the date as part
of the file's metadata. The question, though, is how useful is this
information and can lockers that care put it into their ``[tool]``
table instead of mandating it be set?


--------------------------
Locking Build Dependencies
--------------------------

Thanks to PEP 518, source trees and sdists can specify what build
tools must be installed in order to build a wheel (or sdist in the
case of a source tree). It has been suggested that the lock file also
record such packages so to increase how reproducible an installation
can be.

There is nothing currently in this PEP, though, that prohibits a
locker from recording build tools thanks to ``metadata.needs`` acting
as the entry point for calculating what to install. There is also a
cost in downloading all potential sdists and source trees, reading
their ``pyproject.toml`` files, and then calculating their build
dependencies for locking purposes for which not everyone will want to
pay the cost for.


--------------------------------------------------------------
Recording the ``Requires-Dist`` Input to the Locker's Resolver
--------------------------------------------------------------

While the ``needs`` key allows for recording dependency specifiers,
this PEP does not currently require the ``needs`` key to record the
**exact** ``Requires-Dist`` metadata that was used to calculate the
lock file. It has been suggested that recording the inputs would help
in auditing the outcome of the lock file.

If this were to be done, it would be an key named ``requested`` which
lived along side ``needs`` and would only be specified if it would
differ from what is specified in ``needs``.


---------------------------------------------
Providing ``marker`` and ``tags`` per Package
---------------------------------------------

It has been suggested to allow for the ``marker`` and ``tags`` keys
as found in the ``[metadata]`` table in ``[package]`` tables as well.
This would allow lockers to specify when a version of a package
should be considered. This also has the potential for lowering
computational costs for installers by assisting them in knowing when
to remove a package version from consideration.

This does, though, somewhat duplicate environment markers being kept
in the ``needs`` array.


===============
Acknowledgments
===============

Thanks to Frost Ming of PDM_ and Sébastien Eustace of Poetry_ for
providing input around dynamic install-time resolution of PEP 508
requirements.

Thanks to Kushal Das for making sure reproducible builds stayed a
concern for this PEP.

Thanks to Andrea McInnes for settling the bikeshedding and choosing
the paint colour of ``needs``.


=========
Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


.. _build system declaration spec: https://packaging.python.org/specifications/declaring-build-dependencies/
.. _core metadata spec: https://packaging.python.org/specifications/core-metadata/
.. _Dart: https://dart.dev/
.. _Dependabot: https://dependabot.com/
.. _dependency specifier spec: https://packaging.python.org/specifications/dependency-specifiers/
.. _direct URL origin of installed distributions spec: https://packaging.python.org/specifications/direct-url/
.. _Git: https://git-scm.com/
.. _JSON: https://www.json.org/
.. _npm: https://www.npmjs.com/
.. _PDM: https://pypi.org/project/pdm/
.. _pip-tools: https://pypi.org/project/pip-tools/
.. _Pipenv: https://pypi.org/project/pipenv/
.. _platform compatibility tags: https://packaging.python.org/specifications/platform-compatibility-tags/
.. _Poetry: https://pypi.org/project/poetry/
.. _Pyflow: https://pypi.org/project/pyflow/
.. _PyUP: https://pyup.io/
.. _Rust: https://www.rust-lang.org/
.. _SecureDrop: https://securedrop.org/
.. _simple repository API: https://packaging.python.org/specifications/simple-repository-api/
.. _source distribution file: https://packaging.python.org/specifications/source-distribution-format/
.. _TOML: https://toml.io
.. _version specifiers spec: https://packaging.python.org/specifications/version-specifiers/
.. _wheel file: https://packaging.python.org/specifications/binary-distribution-format/


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: