From 8fd0b8a9068994deb56374507e3be7c4bc9972cf Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 30 Oct 2024 15:09:24 -0700 Subject: [PATCH] PEP 751: switch to storing a graph of dependencies (#4096) --- peps/pep-0751.rst | 1332 +++++++++++++++++++++++++-------------------- 1 file changed, 742 insertions(+), 590 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 62eafef83..8daf44365 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -1,7 +1,6 @@ PEP: 751 -Title: A file format to list Python dependencies for installation reproducibility +Title: A file format to record Python dependencies for installation reproducibility Author: Brett Cannon -Discussions-To: https://discuss.python.org/t/59173 Status: Draft Type: Standards Track Topic: Packaging @@ -16,20 +15,17 @@ Abstract This PEP proposes a new file format for dependency specification to enable reproducible installation in a Python environment. The format is designed to be human-readable and machine-generated. Installers consuming the -file should be able to evaluate each package in question in isolation, with no -need for dependency resolution at install-time. +file should be able to calculate what to install without the need for dependency +resolution at install-time. ========== Motivation ========== -Currently, no standard exists to: - -- Specify what top-level dependencies should be installed into a Python - environment. -- Create an immutable record, such as a lock file, of which dependencies were - installed. +Currently, no standard exists to create an immutable record, such as a lock +file, which specifies what direct and indirect dependencies should be installed +into a virtual environment. Considering there are at least five well-known solutions to this problem in the community (``pip freeze``, pip-tools_, uv_, Poetry_, and PDM_), there seems to @@ -38,11 +34,8 @@ be an appetite for lock files in general. Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current environment while PDM and Poetry try to lock for *any* environment to some -degree. And none of them directly support locking to specific files to install -which can be important for some workflows. There's also concerns around the lack -of secure defaults in the face of supply chain attacks (e.g., always including -hashes for files). Finally, not all the formats are easy to audit to determine -what would be installed into an environment ahead of time. +degree. There's also concerns around the lack of secure defaults in the face of +supply chain attacks (e.g., always including hashes for files). The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially @@ -69,88 +62,21 @@ allows for situations such as cloud hosting providers to use their own installer that's optimized for their system which is independent of what locker the user used to create their lock file. -The file format is designed to be human-readable. This is -so that the contents of the file can be audited by a human to make sure no -undesired dependencies end up being included in the lock file. It is also -designed to facilitate easy understanding of what would be installed from the -lock file without necessitating running a tool, once again to help with -auditing. Finally, the format is designed so that viewing a diff of the file is -easy by centralizing relevant details. +The file format is designed to be human-readable. This is so that the contents +of the file can be audited by a human to make sure no undesired dependencies end +up being included in the lock file. -The file format is also designed to not require a resolver at install time. -Being able to analyze dependencies in isolation from one another when listed in -a lock file provides a few benefits. First, it supports auditing by making it -easy to figure out if a certain dependency would be installed for a certain -environment without needing to reference other parts of the file contextually. -It should also lead to faster installs which are much more frequent than -creating a lock file. Finally, the four tools mentioned in the Motivation_ -section either already implement this approach of evaluating dependencies in -isolation or have suggested they could (in -`Poetry's case `__). +The file format is also designed to not require a resolver at install time. This +greatly simplifies installers and thus reasoning about what would be installed +when consuming a lock file. It should also lead to faster installs which are +much more frequent than creating a lock file. - ------------------ -Locking Scenarios ------------------ - -The lock file format is designed to support two locking scenarios. The format -should also be flexible enough that adding support for other locking scenarios -is possible via a separate PEP. - - -Per-file Locking -================ - -*Per-file locking* operates under the premise that one wants to install exactly -the same files in any matching environment. As such, the lock file specifies -what files to install. There can be multiple environments specified in a -single file, each with their own set of files to install. By specifying the -exact files to install, installers avoid performing any resolution to decide what -to install. - -The motivation for this approach to locking is for those who have controlled -environments that they work with. For instance, if you have specific, controlled -development and production environments then you can use per-file locking to -make sure the **same** files are installed in both environments for everyone. -This is similar to what ``pip freeze`` and pip-tools_ -support, but with more strictness of the exact files as well as incorporating -support to specify the locked files for multiple environments in the same file. - -Per-file locking should be used when the installation attempt should fail -outright if there is no explicitly pre-approved set of installation artifacts -for the target platform. For example: locking the deployment dependencies for a -managed web service. - - -Package Locking -=============== - -*Package locking* lists the packages and their versions that *may* apply to any -environment being installed for. The list of packages and their versions are -evaluated individually and independently from any other packages and versions -listed in the file. This allows installation to be linear -- read each package -and version and make an isolated decision as to whether it should be installed. -This avoids requiring the installer to perform a *resolution* (i.e. -determine what to install based on what else is to be installed). - -The motivation of this approach comes from -`PDM lock files `__. By listing the -potential packages and versions that may be installed, what's installed is -controlled in a way that's easy to reason about. This also allows for not -specifying the exact environments that would be supported by the lock file so -there's more flexibility for what environments are compatible with the lock -file. This approach supports scenarios like open-source projects that want to -lock what people should use to build the documentation without knowing upfront -what environments their contributors are working from. - -As already mentioned, this approach is supported by PDM_. Poetry_ has -`shown some interest `__. - -Per-package locking should be used when the exact set of potential target -platforms is not known when generating the lock file, as it allows installation -tools to choose the most appropriate artifacts for each platform from the -pre-approved set. For example: locking the development dependencies for an open -source project. +Finally, the lock file is meant to be flexible enough to meets the various needs +tools have for choosing what to install. That means the lock file records the +dependency graph of what _may_ be installed. This allows tools to enter the +graph at any point and still have reproducible results from that root of the +graph. Flexibility also means supporting different installation scenarios within +the same lock file (e.g., with or without test dependencies). ============= @@ -161,18 +87,15 @@ Specification File Name --------- -A lock file MUST be named :file:`pylock.toml` or match the regular expression -``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple lock files exist. -The use of the ``.toml`` file extension is to make syntax highlighting in -editors easier and to reinforce the fact that the file format is meant to be -human-readable. The prefix and suffix of a named file MUST be lowercase for easy -detection and stripping off to find the name, e.g.:: +A lock file MUST be named :file:`pylock.toml`. The use of the ``.toml`` file +extension is to make syntax highlighting in editors easier and to reinforce the +fact that the file format is meant to be human-readable. - if filename.startswith("pylock.") and filename.endswith(".toml"): - name = filename.removeprefix("pylock.").removesuffix(".toml") - -This PEP has no opinion as to the location of lock files (i.e. in the root or -the subdirectory of a project). +The lock file SHOULD be located in the directory as appropriate for the scope of +the lock file. Locking against a single ``pyproject.toml``, for instance, would +place the ``pylock.toml`` in the same directory. If the lock file covered +multiple projects in a monorepo, then the expectation is the ``pylock.toml`` +file would be in the directory that held all the projects being locked. ----------- @@ -185,14 +108,13 @@ All keys listed below are required unless otherwise noted. If two keys are mutually exclusive to one another, then one of the keys is required while the other is disallowed. -Keys in tables -- including the top-level table -- SHOULD be emitted by -lockers in the order they are listed in this PEP when applicable unless -another sort order is specified to minimize noise in diffs. If the keys are not -explicitly specified in this PEP, then the keys SHOULD be sorted by -lexicographic order. +Keys in tables -- including the top-level table -- SHOULD be emitted by lockers +in the order they are listed in this PEP when applicable unless another sort +order is specified to minimize noise in diffs. If the keys are not explicitly +specified in this PEP, then the keys SHOULD be sorted by lexicographic order. -As well, lockers SHOULD sort arrays in lexicographic order -unless otherwise specified for the same reason. +As well, lockers SHOULD sort arrays in lexicographic order unless otherwise +specified for the same reason. ``version`` @@ -212,10 +134,10 @@ unless otherwise specified for the same reason. - String - The name of the hash algorithm used for calculating all hash values. -- Only a single hash algorithm is used for the entire file to allow the - ``[[packages.files]]`` table to be written inline for readability and - compactness purposes by only listing a single hash value instead of multiple - values based on multiple hash algorithms. +- Only a single hash algorithm is used for the entire file to allow hash values + to be written in inline tables for readability and compactness purposes by + only listing a single hash value instead of multiple values based on multiple + hash algorithms. - Specifying a single hash algorithm guarantees that an algorithm that the user prefers is used consistently throughout the file without having to audit each file hash value separately. @@ -228,201 +150,186 @@ unless otherwise specified for the same reason. raise an error. -``dependencies`` -================ - -- Array of strings -- A listing of the `dependency specifiers`_ that act as the input to the lock file, - representing the direct, top-level dependencies to be installed. - - -``[[file-locks]]`` -================== - -- Array of tables -- Mutually exclusive with ``[package-lock]``. -- The array's existence implies the use of the per-file locking approach. -- An environment that meets all of the specified criteria in the table will be - considered compatible with the environment that was locked for. -- Lockers MUST NOT generate multiple ``[file-locks]`` tables which would be - considered compatible for the same environment. -- In instances where there would be a conflict but the lock is still desired, - either separate lock files can be written or per-package locking can be used. -- Entries in array SHOULD be sorted by ``file-locks.name`` lexicographically. - - -``file-locks.name`` -------------------- - -- String -- A unique name within the array for the environment this table represents. - - -``[file-locks.marker-values]`` ------------------------------- - -- Optional -- Table of strings -- The keys represent the names of `environment markers`_ and the values are the - values for those markers. -- Compatibility is defined by the environment's values matching what is in the - table. - - -``file-locks.wheel-tags`` -------------------------- - -- Optional -- Array of strings -- An unordered array of `wheel tags`_ for which all tags must be supported by - the environment. -- The array MAY not be exhaustive to allow for a smaller array as well as to - help prevent multiple ``[[file-locks]]`` tables being compatible with the - same environment by having one array being a strict subset of another - ``file-locks.wheel-tags`` entry in the same file's - ``[[file-locks]]`` tables. -- Lockers MUST NOT include - `compressed tag sets `__ - or duplicate tags for consistency across lockers and to simplify checking for - compatibility. - - -``[package-lock]`` -================== +``[locker]`` +============ - Table -- Mutually exclusive with ``[[file-locks]]``. -- Signifies the use of the package locking approach. +- Record of the tool that generated the lock file. +- Enough details SHOULD be provided such that the lock + file from the details in this table (provided the same I/O data is available, + e.g., Dependabot if only files from a repository is necessary to run the + command). -``package-lock.requires-python`` --------------------------------- +``locker.name`` +--------------- - String -- Holds the `version specifiers`_ for Python version compatibility for the - overall package locking. -- Provides at-a-glance information to know if the lock file *may* apply to a - version of Python instead of having to scan the entire file to compile the - same information. +- The name of the tool used to create the lock file. +- If the locker is a Python project, its normalized name SHOULD be used. + + +``locker.version`` +------------------ + +- String +- The version of the tool used. + + +``locker.run`` +-------------- + +- Optional +- Inline table +- Records the command used to create the lock file. + + +``locker.run.module`` +''''''''''''''''''''' + +- Optional +- String +- The module name used for running the locker (i.e. what would be passed to + ``python -m``). +- Lockers MUST specify this key if the locker can be executed via ``python -m``. + + +``locker.run.args`` +''''''''''''''''''' + +- Optional +- Array of strings +- If the locker has a CLI, the arguments to pass to the locker. +- All paths MUST be relative to the lock file so that another tool could use + the lock file's location as the current working directory. + + +``[[groups]]`` +============== + +- Array of tables +- A named subset of packages as found in ``[[packages]]``. +- Act as roots into the dependency graph. +- Installers MUST allow the user to select one or more groups by name to + install all relevant packages together. +- Installers SHOULD let the user skip specifying a name if there is only one + entry in the array. + + +``groups.name`` +--------------- + +- String +- The name of the group. + + +``groups.project`` +------------------ + +- Mutually-exclusive with ``requirements`` +- String +- The normalized name of a package to act as the starting point into the + dependency graph. +- Analogous to locking to the ``[project]`` table in ``pyproject.toml``. +- Installers MUST let a user specify any optional features/extras that the + package provides. +- Lockers MUST NOT allow for ambiguity by specifying multiple package versions + of the same package under the same group name when a package is listed in any + ``project`` key. + + +``groups.requirements`` +----------------------- + +- Mutually-exclusive with ``project`` +- Array of tables +- Represents the installation requirements for this group. +- Analogous to a key in ``[dependency-groups]`` in ``pyproject.toml``. +- Lockers MUST make sure that resolving any requirement for any environment does + not lead to ambiguity by having multiple values in ``[[packages]]`` match the + same requirement. +- Values in the array SHOULD be written as inline tables, sorted + lexicographically by ``name``, then by ``feature`` with the lack of that key + sorting first. + + +``groups.requirements.name`` +'''''''''''''''''''''''''''''' + +- String +- Normalized name of the package. + + +``groups.requirements.extras`` +''''''''''''''''''''''''''''''' + +- Optional +- Array of strings +- The names of the extras specified for the requirement + (i.e. what comes between ``[...]``). + + +``groups.requirements.version`` +''''''''''''''''''''''''''''''''' + +- Optional +- String +- The `version specifiers`_ for the requirement. + + +``groups.requirements.marker`` +'''''''''''''''''''''''''''''''' + +- Optional +- String +- The `environment markers`_ for the requirement. ``[[packages]]`` ================ - Array of tables -- The array contains all data on the locked package versions. -- Lockers SHOULD record packages in order by ``packages.name`` lexicographically - , ``packages.version`` by the sort order for `version specifiers`_, and - ``packages.markers`` lexicographically. -- Lockers SHOULD record keys in the same order as written in this PEP to - minimize changes when updating. -- Entries are designed so that relevant details as to why a package is included - are in one place to make diff reading easier. +- The array contains all data on the nodes of the dependency graph. +- Lockers SHOULD record packages in order by ``name`` + lexicographically, ``version`` by its Python `version specifiers`_ + ordering, and then by ``groups`` following Python's sort order for lists of + strings (i.e. item by item, then by length as a tiebreaker). +.. Identification + ``packages.name`` ----------------- - String -- The `normalized name`_ of the packages. -- Part of what's required to uniquely identify this entry. +- The `normalized name`_ of the package. ``packages.version`` -------------------- - String -- The version of the packages. -- Part of what's required to uniquely identify this entry. +- The version of the package. -``packages.multiple-entries`` ------------------------------ +``packages.groups`` +------------------- -- Optional (defaults to ``false``) -- Boolean -- If package locking via ``[package-lock]``, then the multiple entries for the - same package MUST be mutually exclusive via ``packages.marker`` (this is not - required for per-file locking as the ``packages.*.lock`` entries imply mutual - exclusivity). -- Aids in auditing by knowing that there are multiple entries for the same - package that may need to be considered. - - -``packages.description`` ------------------------- - -- Optional -- String -- The package's ``Summary`` from its `core metadata`_. -- Useful to help understand why a package was included in the file based on its - purpose. +- Array of strings +- Associates this table with the ``group.name`` entries of the same names. ``packages.index-url`` ------------------------------------- +---------------------- -- Optional (although mutually exclusive with - ``packages.files.index-url``) +- Optional - String - Stores the `project index`_ URL from the `Simple Repository API`_. - Useful for generating Packaging URLs (aka PURLs). -- When possible, lockers SHOULD include this or - ``packages.files.index-url`` to assist with generating +- When possible, lockers SHOULD include this to assist with generating `software bill of materials`_ (aka SBOMs). -``packages.marker`` -------------------- - -- Optional -- String -- The `environment markers`_ expression which specifies whether this package and - version applies to the environment. -- Only applicable via ``[package-lock]`` and the package locking scenario. -- The lack of this key means this package and version is required to be - installed. - - -``packages.requires-python`` ----------------------------- - -- Optional -- String -- Holds the `version specifiers`_ for Python version compatibility for the - package and version. -- Useful for documenting why this package and version was included in the file. -- Also helps document why the version restriction in - ``package-lock.requires-python`` was chosen. -- It should not provide useful information for installers as it would be - captured by ``package-lock.requires-python`` and isn't relevant when - ``[[file-locks]]`` is used. - - -``packages.dependents`` ------------------------ - -- Optional -- Array of strings -- A record of the packages that depend on this package and version. -- Useful for analyzing why a package happens to be listed in the file - for auditing purposes. -- This does not provide information which influences installers. - - -``packages.dependencies`` -------------------------- - -- Optional -- Array of strings -- A record of the dependencies of the package and version. -- Useful in analyzing why a package happens to be listed in the file - for auditing purposes. -- This does not provide information which influences the installer as - ``[[file-locks]]`` specifies the exact files to use and ``[package-lock]`` - applicability is determined by ``packages.marker``. - - ``packages.direct`` ------------------- @@ -431,166 +338,259 @@ unless otherwise specified for the same reason. - Represents whether the installation is via a `direct URL reference`_. -``[[packages.files]]`` ----------------------- +.. Requirements + +``packages.requires-python`` +---------------------------- + +- String +- Holds the `version specifiers`_ for Python version compatibility for the + package and version. +- The value MUST match what's provided by the package version, if available, via + :ref:`packaging:core-metadata-requires-python`. + + +``[[packages.dependencies]]`` +----------------------------- -- Must be specified if ``[packages.vcs]`` and ``[packages.directory]`` is not - (although may be specified simultaneously with the other options). - Array of tables -- Tables can be written inline. -- Represents the files to potentially install for the package and version. -- Entries in ``[[packages.files]]`` SHOULD be lexicographically sorted by - ``packages.files.name`` key to minimize changes in diffs. +- A record of the dependency requirements of the package and version. +- The values MUST semantically match what's provided by the package version via + :ref:`packaging:core-metadata-requires-dist`. +- Values in the array SHOULD be written as inline tables, sorted + lexicographically by ``name``, then by ``feature`` with the lack of that key + sorting first. -``packages.files.name`` -''''''''''''''''''''''' +``packages.dependencies.name`` +'''''''''''''''''''''''''''''' +See ``groups.requirements.name``. + + +``packages.dependencies.extras`` +'''''''''''''''''''''''''''''''' + +See ``groups.requirements.extras``. + + +``packages.dependencies.version`` +''''''''''''''''''''''''''''''''' + +See ``groups.requirements.version``. + + +``packages.dependencies.marker`` +'''''''''''''''''''''''''''''''' + +See ``groups.requirements.marker``. + + +``packages.dependencies.feature`` +''''''''''''''''''''''''''''''''' + +- Optional - String -- The file name. -- Necessary for installers to decide what to install when using package locking. +- The optional feature/:ref:`packaging:core-metadata-provides-extra` that this + requirement is conditional on. -``packages.files.lock`` -''''''''''''''''''''''' +.. Installing -- Required when ``[[file-locks]]`` is used (does not apply under per-package - locking) -- Array of strings -- An array of ``file-locks.name`` values which signify that the file is to be - installed when the corresponding ``[[file-locks]]`` table applies to the - environment. -- There MUST only be a single file with any one ``file-locks.name`` entry per - package, regardless of version. +``packages.editable`` +--------------------- + +- Optional (defaults to ``false``) +- Boolean +- Specifies whether the package should be installed in editable mode. -``packages.files.index-url`` -'''''''''''''''''''''''''''''''''''''''''' +``[packages.source-tree]`` +-------------------------- -- Optional (although mutually exclusive with - ``packages.index-url``) +- Optional +- Table +- For recording where to find the `source tree`_ for the package version. +- Lockers SHOULD write this table inline. +- Support for source trees by installers is optional. +- If support is provided by an installer it SHOULD be opt-in. +- If multiple source trees are provided, installers MUST prefer either the + ``vcs`` option or a file for security/reproducibility due to their commit or + hash, respectively. + + +``packages.source-tree.vcs`` +'''''''''''''''''''''''''''' + +- Optional - String -- The value has the same meaning as ``packages.index-url``. -- This key is available per-file to support :pep:`708` when some files override - what's provided by another `Simple Repository API`_ index. - - -``packages.files.url`` -'''''''''''''''''''''' - -- Optional (and mutually exclusive with ``packages.path``) -- String -- URL where the file was found when the lock file was generated. -- Useful for documenting where the file was originally found and potentially - where to look for the file if it is not already downloaded/available. -- Installers MUST NOT assume the URL will always work, but installers MAY use - the URL if it happens to work. - - -``packages.path`` -''''''''''''''''' - -- Optional (and mutually exclusive with ``packages.path``) -- String -- File system path to where the file was found when the lock file was generated. -- Path may be relative to the lock file's location or absolute. -- Installers MUST NOT assume the path will always work, but installers MAY use - the path if it happens to work. - - -``packages.files.hash`` -''''''''''''''''''''''' - -- String -- The hash value of the file contents using the hash algorithm specified by - ``hash-algorithm``. -- Used by installers to verify the file contents match what the locker worked - with. - - -``[packages.vcs]`` ------------------- - -- Must be specified if ``[[packages.files]]`` and ``[packages.directory]`` is - not (although may be specified simultaneously with the other options). -- Table representing the version control system containing the package and - version. - - -``packages.vcs.type`` -''''''''''''''''''''' - -- String -- The type of version control system used. +- If specifying a VCS, the type of version control system used. - The valid values are specified by the `registered VCSs `__ of the direct URL data structure. -``packages.vcs.url`` -''''''''''''''''''''''' +``packages.source-tree.path`` +''''''''''''''''''''''''''''' -- Mutually exclusive with ``packages.vcs.path`` +- Required if ``url`` is not set - String -- The URL of where the repository was located when the lock file was generated. +- A path to the source tree, which may be absolute or relative. +- If the path is relative it MUST be relative to the lock file. +- The path may either be to a directory, file archive, or VCS checkout if + ``vcs`` if is specified. -``packages.vcs.path`` -''''''''''''''''''''' +``packages.source-tree.url`` +'''''''''''''''''''''''''''' -- Mutually exclusive with ``packages.vcs.url`` +- Required if ``path`` is not set - String -- The file system path where the repository was located when the lock file was - generated. -- The path may be relative to the lock file or absolute. +- A URL to a file archive containing the source tree, or a VCS checkout if + ``vcs`` is specified. -``packages.vcs.commit`` -''''''''''''''''''''''' +``packages.source-tree.commit`` +''''''''''''''''''''''''''''''' +- Required if ``vcs`` is set - String - The commit ID for the repository which represents the package and version. - The value MUST be immutable for the VCS for security purposes (e.g. no Git tags). -``packages.vcs.lock`` -''''''''''''''''''''' +``packages.source-tree.size`` +''''''''''''''''''''''''''''' -- Required when ``[[file-locks]]`` is used -- An array of strings -- An array of ``file-locks.name`` values which signify that the repository at the - specified commit is to be installed when the corresponding ``[[file-locks]]`` - table applies to the environment. -- A name in the array may only appear if no file listed in - ``packages.files.lock`` contains the name for the same package, regardless of - version. +- Optional +- Integer +- The size in bytes for the source tree if it is a file. +- Installers MUST verify the file size matches this value. -``[packages.directory]`` ------------------------- +``packages.source-tree.hash`` +''''''''''''''''''''''''''''' -- Must be specified if ``[[packages.files]]`` and ``[packages.vcs]`` is not - and doing per-package locking. -- Table representing a source tree found on the local file system. +- Required if ``url`` or ``path`` points to a file +- String +- The hash value of the file contents using the hash algorithm specified by + ``hash-algorithm``. +- Installers MUST verify the hash matches the file. -``packages.directory.path`` -''''''''''''''''''''''''''' +``[packages.sdist]`` +-------------------- + +- Optional +- Table +- The location of a source distribution as specified by + :ref:`packaging:source-distribution-format`. +- Lockers SHOULD write the table inline. +- Support for source distributions by installers is optional. +- If support is provided by an installer it SHOULD be opt-in. + + +``packages.sdist.url`` +'''''''''''''''''''''' + +- Required if ``path`` is not set +- String +- The URL to the file. + + +``packages.sdist.path`` +''''''''''''''''''''''' + +- Required if ``url`` is not set +- String +- A path to the file, which may be absolute or relative. +- If the path is relative it MUST be relative to the lock file. + + +``packages.sdist.upload-time`` +'''''''''''''''''''''''''''''' + +- Optional and only applicable when ``url`` is specified +- Offset date time +- The upload date and time of the file as specified by a valid ISO 8601 + date/time string for the ``.files[]."upload-time"`` field in the JSON + version of :ref:`packaging:simple-repository-api`. + +``packages.sdist.size`` +''''''''''''''''''''''' + +- Optional +- Integer +- The size of the file in bytes. +- Installers MUST verify the file size matches this value. + + +``packages.sdist.hash`` +''''''''''''''''''''''' - String -- A local directory where a source tree for the package and version exists. -- The path MUST use forward slashes as the path separator. -- If the path is relative it is relative to the location of the lock file. +- The hash value of the file contents using the hash algorithm specified by + ``hash-algorithm``. +- Installers MUST verify the hash matches the file. -``packages.directory.editable`` +``[[packages.wheels]]`` +----------------------- + +- Optional +- Array of tables +- For recording the wheel files as specified by + :ref:`packaging:binary-distribution-format` for the package version. +- Lockers SHOULD write the table inline. +- Lockers SHOULD sort the array values lexicographically by ``tag``. + + +``packages.wheels.tags`` +'''''''''''''''''''''''' + +- Array of string +- The uncompressed tag portion of the wheel file: Python, ABI, and platform. +- Lockers MUST make sure the tag values are unique within the + ``packages.wheels`` array. + + +``packages.wheels.build`` +''''''''''''''''''''''''' + +- Optional +- String +- The build tag for the wheel file (if appropriate). + + +``packages.wheels.url`` +''''''''''''''''''''''' + +See ``packages.sdist.url``. + + +``packages.wheels.path`` +'''''''''''''''''''''''' + +See ``packages.sdist.path``. + + +``packages.wheels.upload-time`` ''''''''''''''''''''''''''''''' -- Boolean -- Optional (defaults to ``false``) -- Flag representing whether the source tree should be installed as an editable - install. +See ``packages.sdist.upload-time``. + + +``packages.wheels.size`` +'''''''''''''''''''''''' + +See ``packages.sdist.size``. + + +``packages.wheels.hash`` +'''''''''''''''''''''''' + +See ``packages.sdist.hash``. ``[packages.tool]`` @@ -619,128 +619,117 @@ unless otherwise specified for the same reason. Examples -------- -Per-file locking -================ - -.. code-block:: toml +.. code-block:: TOML version = '1.0' hash-algorithm = 'sha256' - dependencies = ['cattrs', 'numpy'] - [[file-locks]] - name = 'CPython 3.12 on manylinux 2.17 x86-64' - marker-values = {} - wheel-tags = ['cp312-cp312-manylinux_2_17_x86_64', 'py3-none-any'] + [locker] + name = 'mousebender' + version = 'pep' + run = { module = 'mousebender', args = ['lock', '--platform', 'cpython3.12-manylinux2014-x64', '--platform', 'cpython3.12-windows-x64', 'cattrs', 'numpy'] } - [[file-locks]] - name = 'CPython 3.12 on Windows x64' - marker-values = {} - wheel-tags = ['cp312-cp312-win_amd64', 'py3-none-any'] + [[groups]] + name = 'Default' + requirements = [ + { name = 'cattrs' }, + { name = 'numpy' }, + ] [[packages]] name = 'attrs' - version = '23.2.0' - multiple-entries = false - description = 'Classes Without Boilerplate' - requires-python = '>=3.7' - dependents = ['cattrs'] - dependencies = [] + version = '24.2.0' + groups = ['Default'] + index_url = 'https://pypi.org/simple/attrs' direct = false - files = [ - {name = 'attrs-23.2.0-py3-none-any.whl', lock = ['CPython 3.12 on manylinux 2.17 x86-64', 'CPython 3.12 on Windows x64'], url = 'https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl', hash = '99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1'} + requires_python = '>=3.7' + dependencies = [ + { name = 'importlib-metadata', marker = 'python_version < "3.8"' }, + { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'benchmark' }, + { name = 'hypothesis', feature = 'benchmark' }, + { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'benchmark' }, + { name = 'pympler', feature = 'benchmark' }, + { name = 'pytest-codspeed', feature = 'benchmark' }, + { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'benchmark' }, + { name = 'pytest-xdist', extras = ['psutil'], feature = 'benchmark' }, + { name = 'pytest', version = '>=4.3.0', feature = 'benchmark' }, + { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'cov' }, + { name = 'coverage', extras = ['toml'], version = '>=5.3', feature = 'cov' }, + { name = 'hypothesis', feature = 'cov' }, + { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'cov' }, + { name = 'pympler', feature = 'cov' }, + { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'cov' }, + { name = 'pytest-xdist', extras = ['psutil'], feature = 'cov' }, + { name = 'pytest', version = '>=4.3.0', feature = 'cov' }, + { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'dev' }, + { name = 'hypothesis', feature = 'dev' }, + { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'dev' }, + { name = 'pre-commit', feature = 'dev' }, + { name = 'pympler', feature = 'dev' }, + { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'dev' }, + { name = 'pytest-xdist', extras = ['psutil'], feature = 'dev' }, + { name = 'pytest', version = '>=4.3.0', feature = 'dev' }, + { name = 'cogapp', feature = 'docs' }, + { name = 'furo', feature = 'docs' }, + { name = 'myst-parser', feature = 'docs' }, + { name = 'sphinx', feature = 'docs' }, + { name = 'sphinx-notfound-page', feature = 'docs' }, + { name = 'sphinxcontrib-towncrier', feature = 'docs' }, + { name = 'towncrier', version = '<24.7', feature = 'docs' }, + { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'tests' }, + { name = 'hypothesis', feature = 'tests' }, + { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests' }, + { name = 'pympler', feature = 'tests' }, + { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests' }, + { name = 'pytest-xdist', extras = ['psutil'], feature = 'tests' }, + { name = 'pytest', version = '>=4.3.0', feature = 'tests' }, + { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests-mypy' }, + { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests-mypy' } + ] + editable = false + wheels = [ + { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/6a/21/5b6702a7f963e95456c0de2d495f67bf5fd62840ac655dc451586d23d39a/attrs-24.2.0-py3-none-any.whl', hash = '81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2', upload_time = 2024-08-06T14:37:36.958006+00:00, size = 63001 } ] [[packages]] name = 'cattrs' - version = '23.2.3' - multiple-entries = false - description = 'Composable complex class support for attrs and dataclasses.' - requires-python = '>=3.8' - dependents = [] - dependencies = ['attrs'] + version = '24.1.2' + groups = ['Default'] + index_url = 'https://pypi.org/simple/cattrs' direct = false - files = [ - {name = 'cattrs-23.2.3-py3-none-any.whl', lock = ['CPython 3.12 on manylinux 2.17 x86-64', 'CPython 3.12 on Windows x64'], url = 'https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl', hash = '0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108'} + requires_python = '>=3.8' + dependencies = [ + { name = 'attrs', version = '>=23.1.0' }, + { name = 'exceptiongroup', version = '>=1.1.1', marker = 'python_version < "3.11"' }, + { name = 'typing-extensions', version = '!=4.6.3,>=4.1.0', marker = 'python_version < "3.11"' }, + { name = 'pymongo', version = '>=4.4.0', feature = 'bson' }, + { name = 'cbor2', version = '>=5.4.6', feature = 'cbor2' }, + { name = 'msgpack', version = '>=1.0.5', feature = 'msgpack' }, + { name = 'msgspec', version = '>=0.18.5', marker = 'implementation_name == "cpython"', feature = 'msgspec' }, + { name = 'orjson', version = '>=3.9.2', marker = 'implementation_name == "cpython"', feature = 'orjson' }, + { name = 'pyyaml', version = '>=6.0', feature = 'pyyaml' }, + { name = 'tomlkit', version = '>=0.11.8', feature = 'tomlkit' }, + { name = 'ujson', version = '>=5.7.0', feature = 'ujson' } + ] + editable = false + wheels = [ + { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/c8/d5/867e75361fc45f6de75fe277dd085627a9db5ebb511a87f27dc1396b5351/cattrs-24.1.2-py3-none-any.whl', hash = '67c7495b760168d931a10233f979b28dc04daf853b30752246f4f8471c6d68d0', upload_time = 2024-09-22T14:58:34.812643+00:00, size = 66446 } ] [[packages]] name = 'numpy' - version = '2.0.1' - multiple-entries = false - description = 'Fundamental package for array computing in Python' - requires-python = '>=3.9' - dependents = [] - dependencies = [] + version = '2.1.2' + groups = ['Default'] + index_url = 'https://pypi.org/simple/numpy' direct = false - files = [ - {name = 'numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', lock = ['cp312-manylinux_2_17_x86_64'], url = 'https://files.pythonhosted.org/packages/2c/f3/61eeef119beb37decb58e7cb29940f19a1464b8608f2cab8a8616aba75fd/numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a'}, - {name = 'numpy-2.0.1-cp312-cp312-win_amd64.whl', lock = ['cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/b5/59/f6ad30785a6578ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl', hash = 'bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171'} + requires_python = '>=3.10' + dependencies = [ + ] - - -Per-package locking -=================== - -Some values for ``packages.files.url`` left out to make creating this -example more easily as it was done by hand. - -.. code-block:: toml - - version = '1.0' - hash-algorithm = 'sha256' - dependencies = ['cattrs', 'numpy'] - - [package-lock] - requires-python = ">=3.9" - - - [[packages]] - name = 'attrs' - version = '23.2.0' - multiple-entries = false - description = 'Classes Without Boilerplate' - requires-python = '>=3.7' - dependents = ['cattrs'] - dependencies = [] - direct = false - files = [ - {name = 'attrs-23.2.0-py3-none-any.whl', lock = ['cp312-manylinux_2_17_x86_64', 'cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl', hash = '99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1'} - ] - - [[packages]] - name = 'cattrs' - version = '23.2.3' - multiple-entries = false - description = 'Composable complex class support for attrs and dataclasses.' - requires-python = '>=3.8' - dependents = [] - dependencies = ['attrs'] - direct = false - files = [ - {name = 'cattrs-23.2.3-py3-none-any.whl', lock = ['cp312-manylinux_2_17_x86_64', 'cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl', hash = '0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108'} - ] - - [[packages]] - name = 'numpy' - version = '2.0.1' - multiple-entries = false - description = 'Fundamental package for array computing in Python' - requires-python = '>=3.9' - dependents = [] - dependencies = [] - direct = false - files = [ - {name = "numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:6bf4e6f4a2a2e26655717a1983ef6324f2664d7011f6ef7482e8c0b3d51e82ac"}, - {name = "numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7d6fddc5fe258d3328cd8e3d7d3e02234c5d70e01ebe377a6ab92adb14039cb4"}, - {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:5daab361be6ddeb299a918a7c0864fa8618af66019138263247af405018b04e1"}, - {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:ea2326a4dca88e4a274ba3a4405eb6c6467d3ffbd8c7d38632502eaae3820587"}, - {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:529af13c5f4b7a932fb0e1911d3a75da204eff023ee5e0e79c1751564221a5c8"}, - {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a"}, - {name = "numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:cbab9fc9c391700e3e1287666dfd82d8666d10e69a6c4a09ab97574c0b7ee0a7"}, - {name = "numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:99d0d92a5e3613c33a5f01db206a33f8fdf3d71f2912b0de1739894668b7a93b"}, - {name = "numpy-2.0.1-cp312-cp312-win32.whl", hash = "sha256:173a00b9995f73b79eb0191129f2455f1e34c203f559dd118636858cc452a1bf"}, - {name = "numpy-2.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171"}, + editable = false + wheels = [ + { tags = ['cp312-cp312-manylinux2014_x86_64', 'cp312-cp312-manylinux_2_17_x86_64'], url = 'https://files.pythonhosted.org/packages/9b/b4/e3c7e6fab0f77fff6194afa173d1f2342073d91b1d3b4b30b17c3fb4407a/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6d95f286b8244b3649b477ac066c6906fbb2905f8ac19b170e2175d3d799f4df', upload_time = 2024-10-05T18:36:20.729642+00:00, size = 16041825 }, + { tags = ['cp312-cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/4c/79/73735a6a5dad6059c085f240a4e74c9270feccd2bc66e4d31b5ca01d329c/numpy-2.1.2-cp312-cp312-win_amd64.whl', hash = '456e3b11cb79ac9946c822a56346ec80275eaf2950314b249b512896c0d2505e', upload_time = 2024-10-05T18:37:38.159022+00:00, size = 12568254 } ] @@ -748,21 +737,22 @@ example more easily as it was done by hand. Expectations for Lockers ------------------------ -- When creating a lock file for ``[package-lock]``, the locker SHOULD read - the metadata of **all** files that end up being listed in - ``[[packages.files]]`` to make sure all potential metadata cases are covered -- If a locker chooses not to check every file for its metadata, the tool MUST - either provide the user with the option to have all files checked (whether - that is opt-in or out is left up to the tool), or the user is somehow notified - that such a standards-violating shortcut is being taken (whether this is by - documentation or at runtime is left to the tool) +- Lockers MUST make sure that entering the dependency graph via a specific group + will not lead to ambiguity for installers as to which value in + ``[[packages]]`` to install for any environment (this can be controlled for + via ``packages.version`` and ``packages.groups``). +- Lockers SHOULD try to make all logically related groups resolve together + (i.e. no ambiguity if grouped together). +- If a ``groups.project`` would have extras that cause ambiguity or installation + failure due to conflicts between the extras, the locker MAY create + separate ``groups.requirements`` entries instead, otherwise the locker MUST + raise an error. +- Lockers MAY try to lock for multiple environments in a single lock file. +- Lockers MAY try to update a lock file containing ``[tool]`` and + ``[packages.tool]`` for other tools than themselves. - Lockers MAY want to provide a way to let users provide the information - necessary to install for multiple environments at once when doing per-file - locking, e.g. supporting a JSON file format which specifies wheel tags and - marker values much like in ``[[file-locks]]`` for which multiple files can be - specified, which could then be directly recorded in the corresponding - ``[[file-locks]]`` table (if it allowed for unambiguous per-file locking - environment selection) + necessary to lock for other environments, e.g., supporting a JSON + file format which specifies wheel tags and marker values. .. code-block:: JSON @@ -777,79 +767,176 @@ Expectations for Installers --------------------------- - Installers MAY support installation of non-binary files - (i.e. source distributions, source trees, and VCS), but are not required to. + (i.e. source trees and source distributions), but are not required to. - Installers MUST provide a way to avoid non-binary file installation for reproducibility and security purposes. - Installers SHOULD make it opt-in to use non-binary file installation to facilitate a secure-by-default approach. -- Under per-file locking, if what to install is ambiguous then the installer - MUST raise an error. - - -Installing for per-file locking -=============================== - -- If no compatible environment is found an error MUST be raised. -- If multiple environments are found to be compatible then an error MUST be +- If a traversal of the graph leads to any ambiguity as to what package version + to install (i.e. more than one package version qualifies), an error MUST be raised. -- If a ``[[packages.files]]`` contains multiple matching entries an error MUST - be raised due to ambiguity for what is to be installed. -- If multiple ``[[packages]]`` entries for the same package have matching files - an error MUST be raised due to ambiguity for what is to be installed. +- Installers MUST only consider package versions included in any selected + groups (i.e. installers cannot consider packages outside of the groups + selected to install from). +- Installers MUST error out if a package version lacks a way to install into the + chosen environment. +- Installers MUST support installing into an empty environment +- Installers SHOULD support syncing a pre-existing environment to match the lock + file. +- Installers MAY support a way to install into a pre-existing environment that + tries to keep packages not listed in the lock file working. -Example workflow ----------------- +Pseudo-Code +=========== -- Iterate through each ``[[file-locks]]`` table to find the one that applies to - the environment being installed for. -- If no compatible environment is found an error MUST be raised. -- If multiple environments are found to be compatible then an error MUST be - raised. -- For the compatible environment, iterate through each entry in - ``[[packages]]``. -- For each ``[[packages]]`` entry, iterate through ``[[packages.files]]`` to - look for any files with ``file-locks.name`` listed in ``packages.files.lock``. -- If a file is found with a matching lock name, add it to the list of candidate - files to install and move on to the next ``[[packages]]`` entry. -- If no file is found then check if ``packages.vcs.lock`` contains a match (no - match is also acceptable). -- If a ``[[packages.files]]`` contains multiple matching entries an error MUST - be raised due to ambiguity for what is to be installed. -- If multiple ``[[packages]]`` entries for the same package have matching files - an error MUST be raised due to ambiguity for what is to be installed. -- Find and verify the candidate files and/or VCS entries based on their hash or - commit ID as appropriate. -- Install the candidate files. +.. code-block:: Python + + class UnsatisfiableError(Exception): + """Raised when a requirement cannot be satisfied.""" -Installing for package locking -============================== - -- Verify that the environment is compatible with - ``package-lock.requires-python``; if it isn't an error MUST be raised. -- If no way to install a required package is found, an error MUST be raised. + class AmbiguityError(Exception): + """Raised when a requirement has multiple solutions.""" -Example workflow ----------------- + def install_packages(lock_file_contents): + # Hard-coded out of laziness. + packages = choose_packages(lock_file_contents, (GROUP_NAME, frozenset())) -- Verify that the environment is compatible with - ``package-lock.requires-python``; if it isn't an error MUST be raised. -- Iterate through each entry in ``[packages]]``. -- For each entry, if there's a ``packages.marker`` key, evaluate the expression. + for package in packages: + tags = list(packaging.tags.sys_tags()) + for tag in tags: # Prioritize by tag order. + tag_str = str(tag) + for wheel in package["wheels"]: + if tag_str in wheel["tags"]: + break + else: + continue + break + else: + raise UnsatisfiableError( + f"No wheel for {package['name']} {package['version']}" + ) + print(f"Installing {package['name']} {package['version']} ({tag_str})") - - If the expression is false, then move on. - - Otherwise the package entry must be installed somehow. -- Iterate through the files listed in ``[[packages.files]]``, looking for the - "best" file to install. -- If no file is found, check for ``[packages.vcs]``. -- It no VCS is found, check for ``packages.directory``. -- If no match is found, an error MUST be raised. -- Find and verify the selected files and/or VCS entries based on their hash or - commit ID as appropriate. -- Install the selected files. + def choose_packages(lock_file_data, *selected_groups): + """Select the package versions that should be installed based on the requested groups. + + 'selected_groups' is a sequence of two-item tuples, representing a group name and + optionally any requested extras if the group is a project. + """ + group_names = frozenset(operator.itemgetter(0)(group) for group in selected_groups) + available_packages = {} # The packages in the selected groups. + for pkg in lock_file_data["packages"]: + if frozenset(pkg["groups"]) & group_names: + available_packages.setdefault(pkg["name"], []).append(pkg) + selected_packages = {} # The package versions that have been selected. + handled_extras = {} # The extras that have been handled. + requirements = [] # A stack of requirements to satisfy. + + # First, get our starting list of requirements. + for group in selected_groups: + requirements.extend(gather_requirements(lock_file_data, group)) + + # Next, go through the requirements and try to find a **single** package version + # that satisfies each requirement. + while requirements: + req = requirements.pop() + # Ignore requirements whose markers disqualify it. + if not applies_to_env(req): + continue + name = req["name"] + if pkg := selected_packages.get(name): + # Safety check that the cross-section of groups doesn't cause issues. + # It somewhat assumes the locker didn't mess up such that there would be + # ambiguity by what package version was initially selected. + if not version_satisfies(req, pkg): + raise UnsatisfiableError( + f"requirement {req!r} not satisfied by " + f"{selected_packages[req['name']]!r}" + ) + if "extras" not in req: + continue + needed_extras = req["extras"] + if not (extras := handled_extras.set_default(name, set())).difference( + needed_extras + ): + continue + # This isn't optimal as we may tread over the same extras multiple times, + # but eventually the maximum set of extras for the package will be handled + # and thus the above guard will short-circuit adding any more requirements. + extras.update(needed_extras) + else: + # Raises UnsatisfiableError or AmbiguityError if no suitable, single package + # version is found. + pkg = compatible_package_version(req, available_packages[req["name"]]) + selected_packages[name] = pkg + requirements.extend(dependencies(pkg, req)) + + return selected_packages.values() + + + def gather_requirements(locked_file_data, group): + """Return a collection of all requirements for a group.""" + # Hard-coded to support `groups.requirements` out of laziness. + group_name, _extras = group + for group in locked_file_data["groups"]: + if group["name"] == group_name: + return group["requirements"] + else: + raise ValueError(f"Group {group_name!r} not found in lock file") + + + def applies_to_env(requirement): + """Check if the requirement applies to the current environment.""" + try: + markers = requirement["marker"] + except KeyError: + return True + else: + return packaging.markers.Marker(markers).evaluate() + + + def version_satisfies(requirement, package): + """Check if the package version satisfies the requirement.""" + try: + raw_specifier = requirement["version"] + except KeyError: + return True + else: + specifier = packaging.specifiers.SpecifierSet(raw_specifier) + return specifier.contains(package["version"], prereleases=True) + + + def compatible_package_version(requirement, available_packages): + """Return the package version that satisfies the requirement. + + If no package version can satisfy the requirement, raise UnsatisfiableError. If + multiple package versions can satisfy the requirement, raise AmbiguityError. + """ + possible_packages = [ + pkg for pkg in available_packages if version_satisfies(requirement, pkg) + ] + if not possible_packages: + raise UnsatisfiableError(f"No package version satisfies {requirement!r}") + elif len(possible_packages) > 1: + raise AmbiguityError(f"Multiple package versions satisfy {requirement!r}") + return possible_packages[0] + + + def dependencies(package, requirement): + """Return the dependencies of the package. + + The extras from the requirement will extend the base requirements as needed. + """ + applicable_deps = [] + extras = frozenset(requirement.get("extras", [])) + for dep in package["dependencies"]: + if "feature" not in dep or dep["feature"] in extras: + applicable_deps.append(dep) + return applicable_deps ======================= @@ -896,10 +983,10 @@ How to Teach This Users should be informed that when they ask to install some package, that package may have its own dependencies, those dependencies may have dependencies, and so on. Without writing down what gets installed as part of installing the -package they requested, things could change from underneath them (e.g. package +package they requested, things could change from underneath them (e.g., package versions). Changes to the underlying dependencies can lead to accidental breakage of their code. Lock files help deal with that by providing a way to -write down what was installed. +write down what was (and should be) installed. Having what to install written down also helps in collaborating with others. By agreeing to a lock file's contents, everyone ends up with the same packages @@ -917,31 +1004,37 @@ the change is on purpose and not one slipped in by a bad actor. Reference Implementation ======================== -A rough proof-of-concept for per-file locking can be found at -https://github.com/brettcannon/mousebender/tree/pep. An example lock file can -be seen at -https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml. - -For per-package locking, PDM_ indirectly proves the approach works as this PEP -maintains equivalent data as PDM does for its lock files (whose format was -inspired by Poetry_). Some of the details of PDM's approach are covered in -https://frostming.com/en/2024/pdm-lockfile/ and -https://frostming.com/en/2024/pdm-lock-strategy/. +A proof-of-concept implementing most of this PEP for wheels can be found at +https://github.com/brettcannon/mousebender/tree/pep . ============== Rejected Ideas ============== ----------------------------- -Only support package locking ----------------------------- +--------------------------------- +A flat set of packages to install +--------------------------------- -At one point it was suggested to skip per-file locking and only support package -locking as the former was not explicitly supported in the larger Python -ecosystem while the latter was. But because this PEP has taken the position -that security is important and per-file locking is the more secure of the two -options, leaving out per-file locking was never considered. +An earlier version of this PEP proposed to use a flat set of package versions +instead of a graph. The idea was that each package version could be evaluated in +isolation as to whether it applied to an environment for installation. The hope +was that would lend itself to easier auditing as one wouldn't have to worry +about how a package version fit into the graph when looking at e.g., a diff for +a lock file. + +Unfortunately this was deemed not as flexible as using a graph. For instance, +recording the graph +`assists in dependency analysis for tools like GitHub `__. +A graph also makes following how you ended up with dependencies within your lock +file from any point in the graph. It also balances out the implementation costs +a bit more between lockers and installers by alleviating the complexity off of +lockers a bit for only a minor increase in complexity for installers by +involving standard graph-traversing algorithms instead of a linear walk. + +And if the dependency graph is already being recorded for the above benefits, +then recording that same data in a flattened manner is redundant that makes +lock files larger and potentially more unruly. ------------------------------------------------------------------------------------- @@ -1103,18 +1196,76 @@ could propose a solution. Open Issues =========== -N/A +---------------------------------------------- +Specify ``requires-python`` at the file level? +---------------------------------------------- + +The lock file formats from PDM_, Poetry_, and uv_ all specify +``requires-python`` at the top level for the absolute minimum Python version +needed for the lock file. This can be inferred, though, by examining all +``packages.requires-python`` values. The global value might also not be +accurate for all platforms depending on how environment markers influence what +package versions are installed and what their Python version requirements are. + + +--------------------- +Don't pre-parse data? +--------------------- + +This PEP currently takes the viewpoint that if a piece of data is going to be +parsed by installers everytime they run, then trying to pre-parse as much as +possible so the TOML parser can help is a good thing. The thinking is TOML +parsers have a higher chance of being optimized, and so letting them do more +parsing leads to a faster outcome. It should also increase readability by +breaking apart data upfront more. + +But in the case of doing this to wheel file names, some might consider it too +much. The question becomes whether separating out all the parts of a wheel +file name hinders readability because people are used to reading the file names +already, or by clearly separating its parts it actually helps make installers +faster, easier to write, and doesn't hinder readability. + +This all equally applies to requirement specifiers. + + +============== +Deferred Ideas +============== + +---------------- +Per-file locking +---------------- + +An earlier version of this PEP supported two approaches to locking: *per-file* +and **per-package**. The idea for the former approach to locking was that if you +were locking for an a-priori set of environments you could lock to just the +files necessary to install into those environments. The thinking was that by +only listing a subset of files that auditing would be easier. + +Unfortunately there was disagreement on how best to express upfront what the +supported environment requirements would be. Since what this PEP currently +proposes still prevents accidental success of installation into unsupported +environments, this idea has been deferred until such time someone can come up +with a representation that makes sense. + + +-------------------------------- +Allowing for multiple lock files +-------------------------------- + +Before the introduction of ``[[groups]]``, this PEP proposed supporting multiple +lock files that would match the regular expression +``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple +lock files exist. But since ``[[groups]]`` subsumes a lot of the need to support +multiple lock files, this specific feature can be postponed until such time that +a need is shown to support multiple lock files. ================ Acknowledgements ================ -Thanks to everyone who participated in the discussions in -https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/, -especially Alyssa Coghlan who probably caused the biggest structural shifts from -the initial proposal. - +Thanks to everyone who participated in the discussions on discuss.python.org. Also thanks to Randy Döring, Seth Michael Larson, Paul Moore, and Ofek Lev for providing feedback on a draft version of this PEP. @@ -1140,6 +1291,7 @@ CC0-1.0-Universal license, whichever is more permissive. .. _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification .. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ .. _software bill of materials: https://www.cisa.gov/sbom +.. _source tree: https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-trees .. _TOML: https://toml.io/ .. _uv: https://github.com/astral-sh/uv .. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/