PEP 751: update based on feedback (#3877)

* PEP 751: update based on feedback

* Fix a section underline
This commit is contained in:
Brett Cannon 2024-07-26 15:52:27 -07:00 committed by GitHub
parent da8cf9fcbe
commit 07fc184eea
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 164 additions and 148 deletions

View File

@ -31,9 +31,9 @@ Currently, no standard exists to:
- Create an immutable record, such as a lock file, of which dependencies were
installed.
Considering there are at least four well-known solutions to this problem in the
community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an
appetite for lock files in general.
Considering there are at least five well-known solutions to this problem in the
community (``pip freeze``, pip-tools_, uv_, Poetry_, and PDM_), there seems to
be an appetite for lock files in general.
Those tools also vary in what locking scenarios they support. For instance,
``pip freeze`` and pip-tools only generate lock files for the current
@ -63,11 +63,11 @@ used to create their lock file.
The file format is designed to be human-readable. This is
so that the contents of the file can be audited by a human to make sure no
undesired dependencies end up being included in the lock file. It is also to
facilitate easy understanding of what would be installed if the lock file
without necessitating running a tool, once again to help with auditing. Finally,
the format is designed so that viewing a diff of the file is easy by centralizing
relevant details.
undesired dependencies end up being included in the lock file. It is also
designed to facilitate easy understanding of what would be installed from the
lock file without necessitating running a tool, once again to help with
auditing. Finally, the format is designed so that viewing a diff of the file is
easy by centralizing relevant details.
The file format is also designed to not require a resolver at install time.
Being able to analyze dependencies in isolation from one another when listed in
@ -108,6 +108,11 @@ This is similar to what ``pip freeze`` and pip-tools_
support, but with more strictness of the exact files as well as incorporating
support to specify the locked files for multiple environments in the same file.
Per-file locking should be used when the installation attempt should fail
outright if there is no explicitly pre-approved set of installation artifacts
for the target platform. For example: locking the deployment dependencies for a
managed web service.
Package Locking
===============
@ -133,6 +138,12 @@ what environments their contributors are working from.
As already mentioned, this approach is supported by PDM_. Poetry_ has
`shown some interest <https://discuss.python.org/t/46593/83>`__.
Per-package locking should be used when the exact set of potential target
platforms is not known when generating the lock file, as it allows installation
tools to choose the most appropriate artifacts for each platform from the
pre-approved set. For example: locking the development dependencies for an open
source project.
=============
Specification
@ -182,7 +193,7 @@ other is disallowed.
- String
- The name of the hash algorithm used for calculating all hash values.
- Only a single hash algorithm is used for the entire file to allow the
``[[package.files]]`` table to be written inline for readability and
``[[packages.files]]`` table to be written inline for readability and
compactness purposes by only listing a single hash value instead of multiple
values based on multiple hash algorithms.
- Specifying a single hash algorithm guarantees that an algorithm that the user
@ -205,30 +216,30 @@ other is disallowed.
representing the direct, top-level dependencies to be installed.
``[[file-lock]]``
=================
``[[file-locks]]``
==================
- Array of tables
- Mutually exclusive with ``[package-lock]``.
- The array's existence implies the use of the per-file locking approach.
- An environment that meets all of the specified criteria in the table will be
considered compatible with the environment that was locked for.
- Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be
- Lockers MUST NOT generate multiple ``[file-locks]`` tables which would be
considered compatible for the same environment.
- In instances where there would be a conflict but the lock is still desired,
either separate lock files can be written or per-package locking can be used.
- Entries in array SHOULD be sorted by ``file-lock.name`` lexicographically.
- Entries in array SHOULD be sorted by ``file-locks.name`` lexicographically.
``file-lock.name``
------------------
``file-locks.name``
-------------------
- String
- A unique name within the array for the environment this table represents.
``[file-lock.marker-values]``
-----------------------------
``[file-locks.marker-values]``
------------------------------
- Optional
- Table of strings
@ -240,17 +251,18 @@ other is disallowed.
updating the file.
``file-lock.wheel-tags``
------------------------
``file-locks.wheel-tags``
-------------------------
- Optional
- Array of strings
- An unordered array of `wheel tags`_ which must be supported by the environment.
- An unordered array of `wheel tags`_ for which all tags must be supported by
the environment.
- The array MAY not be exhaustive to allow for a smaller array as well as to
help prevent multiple ``[[file-lock]]`` tables being compatible with the
help prevent multiple ``[[file-locks]]`` tables being compatible with the
same environment by having one array being a strict subset of another
``file-lock.wheel-tags`` entry in the same file's
``[[file-lock]]`` tables.
``file-locks.wheel-tags`` entry in the same file's
``[[file-locks]]`` tables.
- Lockers SHOULD sort the keys lexicographically to minimize changes when
updating the file.
- Lockers MUST NOT include
@ -263,7 +275,7 @@ other is disallowed.
==================
- Table
- Mutually exclusive with ``[[file-lock]]``.
- Mutually exclusive with ``[[file-locks]]``.
- Signifies the use of the package locking approach.
@ -278,49 +290,49 @@ other is disallowed.
same information.
``[[package]]``
===============
``[[packages]]``
================
- Array of tables
- The array contains all data on the locked package versions.
- Lockers SHOULD record packages in order by ``package.name`` lexicographically
and ``package.version`` by the sort order for `version specifiers`_.
- Lockers SHOULD record packages in order by ``packages.name`` lexicographically
and ``packages.version`` by the sort order for `version specifiers`_.
- Lockers SHOULD record keys in the same order as written in this PEP to
minimmize changes when updating.
minimize changes when updating.
- Designed so that relevant details as to why a package is included are
in one place to make diff reading easier.
``package.name``
----------------
``packages.name``
-----------------
- String
- The `normalized name`_ of the package.
- The `normalized name`_ of the packages.
- Part of what's required to uniquely identify this entry.
``package.version``
-------------------
``packages.version``
--------------------
- String
- The version of the package.
- The version of the packages.
- Part of what's required to uniquely identify this entry.
``package.multiple-entries``
----------------------------
``packages.multiple-entries``
-----------------------------
- Boolean
- If package locking via ``[package-lock]``, then the multiple entries for the
same package MUST be mutually exclusive via ``package.marker`` (this is not
required for per-file locking as the ``package.*.lock`` entries imply mutual
same package MUST be mutually exclusive via ``packages.marker`` (this is not
required for per-file locking as the ``packages.*.lock`` entries imply mutual
exclusivity).
- Aids in auditing by knowing that there are multiple entries for the same
package that may need to be considered.
``package.description``
-----------------------
``packages.description``
------------------------
- Optional
- String
@ -329,21 +341,21 @@ other is disallowed.
purpose.
``package.simple-repo-package-url``
-----------------------------------
``packages.simple-repo-package-url``
------------------------------------
- Optional (although mutually exclusive with
``package.files.simple-repo-package-url``)
``packages.files.simple-repo-package-url``)
- String
- Stores the `project detail`_ URL from the `Simple Repository API`_.
- Useful for generating Packaging URLs (aka PURLs).
- When possible, lockers SHOULD include this or
``package.files.simple-repo-package-url`` to assist with generating
``packages.files.simple-repo-package-url`` to assist with generating
`software bill of materials`_ (aka SBOMs).
``package.marker``
------------------
``packages.marker``
-------------------
- Optional
- String
@ -354,8 +366,8 @@ other is disallowed.
installed.
``package.requires-python``
---------------------------
``packages.requires-python``
----------------------------
- Optional
- String
@ -366,11 +378,11 @@ other is disallowed.
``package-lock.requires-python`` was chosen.
- It should not provide useful information for installers as it would be
captured by ``package-lock.requires-python`` and isn't relevant when
``[[file-lock]]`` is used.
``[[file-locks]]`` is used.
``package.dependents``
----------------------
``packages.dependents``
-----------------------
- Optional
- Array of strings
@ -380,8 +392,8 @@ other is disallowed.
- This does not provide information which influences installers.
``package.dependencies``
------------------------
``packages.dependencies``
-------------------------
- Optional
- Array of strings
@ -389,62 +401,62 @@ other is disallowed.
- Useful in analyzing why a package happens to be listed in the file
for auditing purposes.
- This does not provide information which influences the installer as
``[[file-lock]]`` specifies the exact files to use and ``[package-lock]``
applicability is determined by ``package.marker``.
``[[file-locks]]`` specifies the exact files to use and ``[package-lock]``
applicability is determined by ``packages.marker``.
``package.direct``
------------------
``packages.direct``
-------------------
- Optional (defaults to ``false``)
- Boolean
- Represents whether the installation is via a `direct URL reference`_.
``[[package.files]]``
---------------------
``[[packages.files]]``
----------------------
- Must be specified if ``[package.vcs]`` is not
- Must be specified if ``[packages.vcs]`` is not
- Array of tables
- Tables can be written inline.
- Represents the files to potentially install for the package and version.
- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by
``package.files.name`` key to minimze changes in diffs.
- Entries in ``[[packages.files]]`` SHOULD be lexicographically sorted by
``packages.files.name`` key to minimze changes in diffs.
``package.files.name``
''''''''''''''''''''''
``packages.files.name``
'''''''''''''''''''''''
- String
- The file name.
- Necessary for installers to decide what to install when using package locking.
``package.files.lock``
''''''''''''''''''''''
``packages.files.lock``
'''''''''''''''''''''''
- Required when ``[[file-lock]]`` is used
- Required when ``[[file-locks]]`` is used
- Array of strings
- An array of ``file-lock.name`` values which signify that the file is to be
installed when the corresponding ``[[file-lock]]`` table applies to the
- An array of ``file-locks.name`` values which signify that the file is to be
installed when the corresponding ``[[file-locks]]`` table applies to the
environment.
- There MUST only be a single file with any one ``file-lock.name`` entry per
- There MUST only be a single file with any one ``file-locks.name`` entry per
package, regardless of version.
``package.files.simple-repo-package-url``
'''''''''''''''''''''''''''''''''''''''''
``packages.files.simple-repo-package-url``
''''''''''''''''''''''''''''''''''''''''''
- Optional (although mutually exclusive with
``package.simple-repo-package-url``)
``packages.simple-repo-package-url``)
- String
- The value has the same meaning as ``package.simple-repo-package-url``.
- The value has the same meaning as ``packages.simple-repo-package-url``.
- This key is available per-file to support :pep:`708` when some files override
what's provided by another `Simple Repository API`_ index.
``package.files.origin``
''''''''''''''''''''''''
``packages.files.origin``
'''''''''''''''''''''''''
- Optional
- String
@ -453,8 +465,8 @@ other is disallowed.
for the file if not already downloaded/available.
``package.files.hash``
''''''''''''''''''''''
``packages.files.hash``
'''''''''''''''''''''''
- String
- The hash value of the file contents using the hash algorithm specified by
@ -463,17 +475,17 @@ other is disallowed.
with.
``[package.vcs]``
-----------------
``[packages.vcs]``
------------------
- Must be specified if ``[[package.files]]`` is not (although may be specified
simultaneously with ``[[package.files]]``).
- Must be specified if ``[[packages.files]]`` is not (although may be specified
simultaneously with ``[[packages.files]]``).
- Table representing the version control system containing the package and
version.
``package.vcs.type``
''''''''''''''''''''
``packages.vcs.type``
'''''''''''''''''''''
- String
- The type of version control system used.
@ -482,15 +494,15 @@ other is disallowed.
of the direct URL data structure.
``package.vcs.origin``
''''''''''''''''''''''
``packages.vcs.origin``
'''''''''''''''''''''''
- String
- The URI of where the repository was located when the lock file was generated.
``package.vcs.commit``
''''''''''''''''''''''
``packages.vcs.commit``
'''''''''''''''''''''''
- String
- The commit ID for the repository which represents the package and version.
@ -498,53 +510,53 @@ other is disallowed.
(e.g. no Git tags).
``package.vcs.lock``
''''''''''''''''''''
``packages.vcs.lock``
'''''''''''''''''''''
- Required when ``[[file-lock]]`` is used
- Required when ``[[file-locks]]`` is used
- An array of strings
- An array of ``file-lock.name`` values which signify that the repository at the
specified commit is to be installed when the corresponding ``[[file-lock]]``
- An array of ``file-locks.name`` values which signify that the repository at the
specified commit is to be installed when the corresponding ``[[file-locks]]``
table applies to the environment.
- A name in the array may only appear if no file listed in
``package.files.lock`` contains the name for the same package, regardless of
``packages.files.lock`` contains the name for the same package, regardless of
version.
``package.directory``
---------------------
``packages.directory``
----------------------
- Optional and only valid when ``[package-lock]`` is specified
- String
- A local directory where a source tree for the package and version exists.
- Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to
- Not valid under ``[[file-locks]]`` as this PEP does not make an attempt to
specify a mechanism for verifying file contents have not changed since locking
was performed.
``[[package.build-requires]]``
------------------------------
``[[packages.build-requires]]``
-------------------------------
- Optional
- An array of tables whose structure matches that of ``[[package]]``.
- An array of tables whose structure matches that of ``[[packages]]``.
- Each entry represents a package and version to use when building the
enclosing package and version.
- The array is complete/locked like ``[[package]]`` itself (i.e. installers
follow the same installation procedure for ``[[package.build-requires]]`` as
``[[package]]``)
- The array is complete/locked like ``[[packages]]`` itself (i.e. installers
follow the same installation procedure for ``[[packages.build-requires]]`` as
``[[packages]]``)
- Selection of which entries to use for an environment as the same as
``[[package]]`` itself, albeit only applying when installing the build
``[[packages]]`` itself, albeit only applying when installing the build
back-end and its dependencies.
- This helps with reproducibility of the building of a package by recording
either what was or would have been used if the locker needed to build the
package.
packages.
- If the installer and user choose to install from source and this array is
missing then the installer MAY choose to resolve what to install for building
at install time, otherwise the installer MUST raise an error.
``[package.tool]``
------------------
``[packages.tool]``
-------------------
- Optional
- Table
@ -571,7 +583,7 @@ Expectations for Lockers
- When creating a lock file for ``[package-lock]``, the locker SHOULD read
the metadata of **all** files that end up being listed in
``[[package.files]]`` to make sure all potential metadata cases are covered
``[[packages.files]]`` to make sure all potential metadata cases are covered
- If a locker chooses not to check every file for its metadata, the tool MUST
either provide the user with the option to have all files checked (whether
that is opt-in or out is left up to the tool), or the user is somehow notified
@ -580,9 +592,9 @@ Expectations for Lockers
- Lockers MAY want to provide a way to let users provide the information
necessary to install for multiple environments at once when doing per-file
locking, e.g. supporting a JSON file format which specifies wheel tags and
marker values much like in ``[[file-lock]]`` for which multiple files can be
marker values much like in ``[[file-locks]]`` for which multiple files can be
specified, which could then be directly recorded in the corresponding
``[[file-lock]]`` table (if it allowed for unambiguous per-file locking
``[[file-locks]]`` table (if it allowed for unambiguous per-file locking
environment selection)
.. code-block:: JSON
@ -598,13 +610,13 @@ Expectations for Installers
---------------------------
- Installers MAY support installation of non-binary files
(i.e. source distributions, source trees, and VCS), but are not required to
(i.e. source distributions, source trees, and VCS), but are not required to.
- Installers MUST provide a way to avoid non-binary file installation for
reproducibility and security purposes
reproducibility and security purposes.
- Installers SHOULD make it opt-in to use non-binary file installation to
facilitate a secure-by-default approach
facilitate a secure-by-default approach.
- Under per-file locking, if what to install is ambiguous then the installer
MUST raise an error
MUST raise an error.
Installing for per-file locking
@ -612,27 +624,29 @@ Installing for per-file locking
An example workflow is:
- Iterate through each ``[[file-lock]]`` table to find the one that applies to
the environment being installed for
- If no compatible environment is found an error MUST be raised
- If multiple environments are found to be compatible then an error MUST be raised
- For the compatible environment, iterate through each entry in ``[[package]]``
- For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look
for any files with ``file-lock.name`` listed in ``package.files.lock``
- Iterate through each ``[[file-locks]]`` table to find the one that applies to
the environment being installed for.
- If no compatible environment is found an error MUST be raised.
- If multiple environments are found to be compatible then an error MUST be
raised.
- For the compatible environment, iterate through each entry in
``[[packages]]``.
- For each ``[[packages]]`` entry, iterate through ``[[packages.files]]`` to
look for any files with ``file-locks.name`` listed in ``packages.files.lock``.
- If a file is found with a matching lock name, add it to the list of candidate
files to install and move on to the next ``[[package]]`` entry
- If no file is found then check if ``package.vcs.lock`` contains a match (no
match is also acceptable)
- If a ``[[package.files]]`` contains multiple matching entries an error MUST
be raised due to ambiguity for what is to be installed
- If multiple ``[[package]]`` entries for the same package have matching files
an error MUST be raised due to ambiguity for what is to be installed
files to install and move on to the next ``[[packages]]`` entry.
- If no file is found then check if ``packages.vcs.lock`` contains a match (no
match is also acceptable).
- If a ``[[packages.files]]`` contains multiple matching entries an error MUST
be raised due to ambiguity for what is to be installed.
- If multiple ``[[packages]]`` entries for the same package have matching files
an error MUST be raised due to ambiguity for what is to be installed.
- Find and verify the candidate files and/or VCS entries based on their hash or
commit ID as appropriate
commit ID as appropriate.
- If a source distribution or VCS was selected and
``[[package.build-requires]]`` exists, then repeat the above process as
appropriate to install the build dependencies necessary to build the package
- Install the candidate files
``[[packages.build-requires]]`` exists, then repeat the above process as
appropriate to install the build dependencies necessary to build the package.
- Install the candidate files.
Installing for package locking
@ -641,22 +655,23 @@ Installing for package locking
An example workflow is:
- Verify that the environment is compatible with
``package-lock.requires-python``; if it isn't an error MUST be raised
- Iterate through each entry in ``[package]]``
- For each entry, if there's a ``package.marker`` key, evaluate the expression
``package-lock.requires-python``; if it isn't an error MUST be raised.
- Iterate through each entry in ``[packages]]``.
- For each entry, if there's a ``packages.marker`` key, evaluate the expression.
- If the expression is false, then move on
- Otherwise the package entry must be installed somehow
- Iterate through the files listed in ``[[package.files]]``, looking for the
"best" file to install
- If no file is found, check for ``[package.vcs]``
- If no match is found, an error MUST be raised
- If the expression is false, then move on.
- Otherwise the package entry must be installed somehow.
- Iterate through the files listed in ``[[packages.files]]``, looking for the
"best" file to install.
- If no file is found, check for ``[packages.vcs]``.
- If no match is found, an error MUST be raised.
- Find and verify the selected files and/or VCS entries based on their hash or
commit ID as appropriate
commit ID as appropriate.
- If the match is a source distribution or VCS and
``[[package.build-requires]]`` is provided, repeat the above as appropriate to
build the package
- Install the selected files
``[[packages.build-requires]]`` is provided, repeat the above as appropriate
to build the package.
- Install the selected files.
=======================
@ -690,7 +705,7 @@ within the file in a ``[tool]`` entry or via a side channel external to the lock
file itself.
This PEP does not do anything to prevent a user from installing an incorrect
package. While including many details to help in auditing a package's inclusion,
packages. While including many details to help in auditing a package's inclusion,
there isn't any mechanism to stop e.g. name confusion attacks via typosquatting.
Lockers may be able to provide some UX to help with this (e.g. by providing
download counts for a package).
@ -759,7 +774,7 @@ At one point, to handle the issue of metadata varying between files and thus
require examining every released file for a package and version for accurate
locking results, the idea was floated to introduce a new core metadata version
which would require all metadata for all wheel files be the same for a single
version of a package. Ultimately, though, it was deemed unnecessary as this PEP
version of a packages. Ultimately, though, it was deemed unnecessary as this PEP
will put pressure on people to make files consistent for performance reasons or
to make indexes provide all the metadata separate from the wheel files
themselves. As well, there's no easy enforcement mechanism, and so community
@ -938,5 +953,6 @@ CC0-1.0-Universal license, whichever is more permissive.
.. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/
.. _software bill of materials: https://www.cisa.gov/sbom
.. _TOML: https://toml.io/
.. _uv: https://github.com/astral-sh/uv
.. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/
.. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/