iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 665: resolve the (final) open issue around sdists

This commit is contained in:
Brett Cannon 2021-11-25 15:03:28 -08:00
parent 152fe41cf2
commit 6ef86af2ae
1 changed files with 198 additions and 207 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

405
pep-0665.rst
View File

@ -9,7 +9,7 @@ Status: Draft
Type: Standards Track Type: Standards Track
Content-Type: text/x-rst Content-Type: text/x-rst
Created: 29-Jul-2021 Created: 29-Jul-2021
Post-History: 29-Jul-2021, 03-Nov-2021 Post-History: 29-Jul-2021, 03-Nov-2021, 25-Nov-2021
Resolution: Resolution:
======== ========
@ -17,14 +17,15 @@ Abstract
======== ========
This PEP specifies a file format to specify the list of Python package This PEP specifies a file format to specify the list of Python package
installation requirements for an application, and the relation between the installation requirements for an application, and the relation between
specified requirements. The list of requirements is considered the specified requirements. The list of requirements is considered
exhaustive for the installation target, and thus not requiring any exhaustive for the installation target, and thus not requiring any
information beyond the platform being installed for, and the file information beyond the platform being installed for, and the file
itself. The file format is flexible enough to allow installing the itself. The file format is flexible enough to allow installing the
requirements across different platforms, which guarantees requirements across different platforms, which allows for
reproducibility on multiple platforms from the same file. reproducibility on multiple platforms from the same file.
=========== ===========
Terminology Terminology
=========== ===========
@ -38,15 +39,15 @@ import system. The packages on PyPI are an example of this.
An *application* or *app* is an end product that other external code An *application* or *app* is an end product that other external code
does not directly rely on via the import system (i.e. they are does not directly rely on via the import system (i.e. they are
standalone). Desktop applications, command-line tools, etc. are standalone). Desktop applications, command-line tools, etc. are
examples. examples of applications.
A *lock file* records the packages that are to be installed for an A *lock file* records the packages that are to be installed for an
app. Traditionally, the exact version of the package to be installed app. Traditionally, the exact version of the package to be installed
is specified by a lock file, but all specified packages are not always is specified by a lock file, but specified packages are not always
installed on a given platform (according a filtering logic described installed on a given platform (according a filtering logic described
in a later section), which enables the lock file to describe in a later section), which enables the lock file to describe
reproducibility across multiple platforms. Examples of this are reproducibility across multiple platforms. Examples of this are
``package-lock.json`` from npm_, ``Poetry.lock`` from Poetry, etc. ``package-lock.json`` from npm_, ``Poetry.lock`` from Poetry_, etc.
*Locking* is the act of taking the input of the packages an app *Locking* is the act of taking the input of the packages an app
depends on and producting a lock file from that. depends on and producting a lock file from that.
@ -85,18 +86,19 @@ and reproducibility on any one specific platform.
Three, reproducibility is more secure. When you control exactly what Three, reproducibility is more secure. When you control exactly what
files are installed, you can make sure no malicious actor is files are installed, you can make sure no malicious actor is
attempting to slip nefarious code into your application (i.e. some attempting to slip nefarious code into your application (i.e. some
supply chain attacks). By making a lock file which always leads to supply chain attacks). By using a lock file which always leads to
reproducible installs, we can avoid certain risks entirely. reproducible installs, we can avoid certain risks entirely.
Four, relying on wheels provides reproducibility without requiring Four, relying on the `wheel file`_ format provides reproducibility
build tools to support reproducibility as well. Thanks to wheels being without requiring build tools to support reproducibility themselves.
static and not executing code as part of installation, wheels always Thanks to wheels being static and not executing code as part of
lead to a reproducible result. Compare this to source distributions installation, wheels always lead to a reproducible result. Compare
(aka sdists) or source trees which only lead to a reproducible install this to source distributions (aka sdists) or source trees which only
if their build tool supports reproducibility due to inherent code lead to a reproducible install if their build tool supports
execution. Unfortunately the vast majority of build tools do not reproducibility due to inherent code execution. Unfortunately the vast
support reproducible builds, so this PEP helps alleviate that issue majority of build tools do not support reproducible builds, so this
by only relying on wheels. PEP helps alleviate that issue by only supporting wheels as a package
format.
This PEP proposes a standard for a lock file, as the current solutions This PEP proposes a standard for a lock file, as the current solutions
don't meet the outlined goals. Today, the closest we come to a lock don't meet the outlined goals. Today, the closest we come to a lock
@ -119,7 +121,7 @@ Unfortunately, those tools all use differing lock file formats. This
means tooling around these tools much be unique. This impacts tooling means tooling around these tools much be unique. This impacts tooling
such as code editors and hosting providers, which want to be as such as code editors and hosting providers, which want to be as
flexible as possible when it comes to accepting a user's application flexible as possible when it comes to accepting a user's application
code, but also have a limit as to how much development resource they code, but also have a limit as to how much development resources they
can spend to add support for yet another lock file format. A can spend to add support for yet another lock file format. A
standardized format would allow tools to focus their work on a single standardized format would allow tools to focus their work on a single
target, and make sure that workflow decisions made by developers target, and make sure that workflow decisions made by developers
@ -159,23 +161,25 @@ Secure by Design
Viewing the `requirements file format`_ as the closest we have to Viewing the `requirements file format`_ as the closest we have to
a lock file standard, there are a few issues with the file format when a lock file standard, there are a few issues with the file format when
it comes to security. First is that the file format simply does not it comes to security. First is that the file format simply does not
require you specify the exact version of a package. This is why require you to specify the exact version of a package. This is why
tools like `pip-tools`_ exist to help manage that for the user. tools like `pip-tools`_ exist to help manage that users of
requirements files.
Second, you must opt into specifying what files may be installed by Second, you must opt into specifying what files are acceptable to be
using the ``--hash`` argument for a specific dependency. This is also installed by using the ``--hash`` argument for a specific dependency.
optional with pip-tools as it requires specifying the This is also optional with pip-tools as it requires specifying the
``--generate-hashes`` CLI argument. ``--generate-hashes`` CLI argument.
Third, even when you control what files may be installed, it does not Third, even when you control what files may be installed, it does not
prevent other packages from being installed. If a dependency is not prevent other packages from being installed. If a dependency is not
listed in the requirements file, pip will happily go searching for a listed in the requirements file, pip will happily go searching for a
file to meet that need, unless you specify ``--no-deps`` as an file to meet that need. You must specify ``--no-deps`` as an
argument. argument to pip to prevent unintended dependency resolution outside
of the requirements file.
Fourth, the format allows for installing a Fourth, the format allows for installing a
`source distribution file`_ (aka "sdist"). By its very nature, `source distribution file`_ (aka "sdist"). By its very nature,
installing an sdist may imply executing arbitrary Python code, meaning installing an sdist requires executing arbitrary Python code, meaning
that there is no control over what files may be installed. Only by that there is no control over what files may be installed. Only by
specifying ``--only-binary :all:`` can you guarantee pip to only use a specifying ``--only-binary :all:`` can you guarantee pip to only use a
`wheel file`_ for each package. `wheel file`_ for each package.
@ -185,17 +189,20 @@ being proposed, a user should always do the following steps:
#. Use pip-tools and its command ``pip-compile --generate-hashes`` #. Use pip-tools and its command ``pip-compile --generate-hashes``
#. Install the requirements file using #. Install the requirements file using
``pip install --no-deps --only-binary :all:`` ``pip install --require-hashes --no-deps --only-binary :all:``
Critically, all of those flags, and both specificity and exhaustion of Critically, all of those flags, and both the specificity and
what to install that pip-tools provides, are optional. exhaustion of what to install that pip-tools provides, are optional
for requirements files.
As such, the proposal raised in this PEP is secure by design to combat As such, the proposal raised in this PEP is secure by design which
some supply chain attacks. Hashes for files which would be used to combats some supply chain attacks. Hashes for files which would be
install from are **required**. You can **only** install from wheels used to install from are **required**. You can **only** install from
to unambiguously define what files will be placed in the file system. wheels to unambiguously define what files will be placed in the file
Installers **must** have an unambiguous installation from a lock file system. Installers **must** lead to an deterministic installation
for a given platform. from a lock file for a given platform. All of this leads to a
reproducible installation which you can deem trustworthy (when you
have audited the lock file and what it lists).
-------------- --------------
@ -205,17 +212,17 @@ Cross-Platform
Various projects which already have a lock file, like PDM_ and Various projects which already have a lock file, like PDM_ and
Poetry_, provide a lock file which is *cross-platform*. This allows Poetry_, provide a lock file which is *cross-platform*. This allows
for a single lock file to work on multiple platforms while still for a single lock file to work on multiple platforms while still
leading to exact same top-level requirements to be installed leading to the exact same top-level requirements to be installed
everywhere while the installation being consistent/unambiguous on everywhere with the installation being consistent/unambiguous on
each platform. each platform.
As to why this is useful, let's use an example involving PyWeek_ As to why this is useful, let's use an example involving PyWeek_
(a week-long game development competition). We assume you are (a week-long game development competition). Assume you are developing
developing on Linux, while someone you choose to partner with is on Linux, while someone you choose to partner with is using macOS.
using macOS. Now assume the judges are using Windows. How do you make Now assume the judges are using Windows. How do you make sure everyone
sure everyone is using the same top-level dependencies, while allowing is using the same top-level dependencies, while allowing for any
for any platform-specific requirements (e.g. a package requires a platform-specific requirements (e.g. a package requires a helper
helper package under Windows)? package under Windows)?
With a cross-platform lock file, you can make sure that the key With a cross-platform lock file, you can make sure that the key
requirements are met consistently across all platforms. You can then requirements are met consistently across all platforms. You can then
@ -231,7 +238,7 @@ The separation of concerns between a locker and an installer allows
for an installer to have a much simpler operation to perform. As for an installer to have a much simpler operation to perform. As
such, it not only allows for installers to be easier to write, but such, it not only allows for installers to be easier to write, but
facilitates in making sure installers create unambiguous, reproducible facilitates in making sure installers create unambiguous, reproducible
installations. installations correctly.
The installer can also expend less computation/energy in creating the The installer can also expend less computation/energy in creating the
installation. This is beneficial not only for faster installs, but installation. This is beneficial not only for faster installs, but
@ -239,21 +246,9 @@ also from an energy consumption perspective, as installers are
expected to be run more often than lockers. expected to be run more often than lockers.
This has led to a design where the locker must do more work upfront This has led to a design where the locker must do more work upfront
to benefit installers. It also means the complexity of package to the benefit installers. It also means the complexity of package
dependencies is simpler and easier to comprehend to avoid ambiguity. dependencies is simpler and easier to comprehend in a lock files to
avoid ambiguity.
-----------------------
As Flexible as Possible
-----------------------
Realizing that workflows vary greatly between companies, projects, and
even people, this PEP tries to be strict where it's important and
undefined/flexible everywhere else. As such, this PEP is strict where
it is important for reproducibility and compatibilitiy, but does not
specify any restrictions in other situations not covered by this PEP;
if the PEP does not specifically state something then it is assumed to
be up to the locker or installer to decide what is best.
============= =============
@ -265,7 +260,7 @@ Details
------- -------
Lock files MUST use the TOML_ file format. This not only prevents the Lock files MUST use the TOML_ file format. This not only prevents the
need to have another file format in the Python packaging ecosystem, need to have another file format in the Python packaging ecosystem
thanks to its adoption by PEP 518 for ``pyproject.toml``, but also thanks to its adoption by PEP 518 for ``pyproject.toml``, but also
assists in making lock files more human-readable. assists in making lock files more human-readable.
@ -273,11 +268,11 @@ Lock files MUST end their file names with ``.pylock.toml``. The
``.toml`` part unambiguously distinguishes the format of the file, ``.toml`` part unambiguously distinguishes the format of the file,
and helps tools like code editors support the file appropriately. The and helps tools like code editors support the file appropriately. The
``.pylock`` part distinguishes the file from other TOML files the user ``.pylock`` part distinguishes the file from other TOML files the user
has, to make logic easier for tools to create functionalities specific has, to make the logic easier for tools to create functionality
to Python lock files, instead of TOML files in general. specific to Python lock files, instead of TOML files in general.
The following sections are the top-level keys of the TOML file data The following sections are the top-level keys of the TOML file data
format. Any field not listed as required is considered optional. format. Any field not listed as **required** is considered optional.
``version`` ``version``
@ -287,15 +282,16 @@ This field is **required**.
The version of the lock file being used. The key MUST be a string The version of the lock file being used. The key MUST be a string
consisting of a number that follows the same formatting as the consisting of a number that follows the same formatting as the
``Metadata-Version`` key in the `core metadata spec`_. The value MUST ``Metadata-Version`` key in the `core metadata spec`_.
be set to ``"1.0"`` until a future PEP allows for a different value.
The introduction of a new *optional* key SHOULD increase the minor The value MUST be set to ``"1.0"`` until a future PEP allows for a
version. The introduction of a new required key or changing the different value. The introduction of a new *optional* key to the file
format MUST increase the major version. How to handle other scenarios format SHOULD increase the minor version. The introduction of a new
is left as a per-PEP decision. required key or changing the format MUST increase the major version.
How to handle other scenarios is left as a per-PEP decision.
Installers MUST warn the user if the lock file specifies a version Installers MUST warn the user if the lock file specifies a version
whose major version is support but whose minor version is whose major version is supported but whose minor version is
unsupported/unrecognized (e.g. the installer supports ``"1.0"``, but unsupported/unrecognized (e.g. the installer supports ``"1.0"``, but
the lock file specifies ``"1.1"``). the lock file specifies ``"1.1"``).
@ -314,9 +310,8 @@ native timestamp type). It MUST be recorded using the UTC time zone to
avoid ambiguity. avoid ambiguity.
If the SOURCE_DATE_EPOCH_ environment variable is set, it MUST be used If the SOURCE_DATE_EPOCH_ environment variable is set, it MUST be used
as the timestamp by the locker. This facilitates reproducibility of the as the timestamp by the locker. This facilitates reproducibility of
lock file itself. the lock file itself.
``[tool]`` ``[tool]``
@ -341,13 +336,13 @@ A table containing data applying to the overall lock file.
A key storing a string containing an environment marker as A key storing a string containing an environment marker as
specified in the `dependency specifier spec`_. specified in the `dependency specifier spec`_.
The locker MAY specify an environment marker which specifies any The locker MAY specify an environment marker which specifies any
restrictions the lock file was generated under. restrictions the lock file was generated under.
If the installer is installing for an environment which does not If the installer is installing for an environment which does not
satisfy the specified environment marker, the installer MUST raise an satisfy the specified environment marker, the installer MUST raise an
error as the lock file does not support the environment. error as the lock file does not support the target installation
environment.
``metadata.tag`` ``metadata.tag``
@ -356,12 +351,10 @@ error as the lock file does not support the environment.
A key storing a string specifying `platform compatibility tags`_ A key storing a string specifying `platform compatibility tags`_
(i.e. wheel tags). The tag MAY be a compressed tag set. (i.e. wheel tags). The tag MAY be a compressed tag set.
The locker MAY specify a tag (set) which specify which platform(s)
the lock file supports.
If the installer is installing for an environment which does not If the installer is installing for an environment which does not
satisfy the specified tag (set), the installer MUST raise an error satisfy the specified tag (set), the installer MUST raise an error
as the lock file does not support the environment. as the lock file does not support the targeted installation
environment.
``metadata.requires`` ``metadata.requires``
@ -377,7 +370,7 @@ and thus the root of the dependency graph.
``metadata.requires-python`` ``metadata.requires-python``
---------------------------- ----------------------------
A string specifying the support version(s) of Python for this lock A string specifying the supported version(s) of Python for this lock
file. It follows the same format as that specified for the file. It follows the same format as that specified for the
``Requires-Python`` field in the `core metadata spec`_. ``Requires-Python`` field in the `core metadata spec`_.
@ -387,11 +380,11 @@ file. It follows the same format as that specified for the
This array is **required**. This array is **required**.
An array per package and version containing details for the potential An array per package and version containing entries for the potential
(wheel) files to install (as represented by ``_name_`` and (wheel) files to install (as represented by ``_name_`` and
``_version_``, respectively). ``_version_``, respectively).
Lockers must MUST normalize a project's name according to the Lockers MUST normalize a project's name according to the
`simple repository API`_. If extras are specified as part of the `simple repository API`_. If extras are specified as part of the
project to install, the extras are to be included in the key name and project to install, the extras are to be included in the key name and
are to be sorted in lexicographic order. are to be sorted in lexicographic order.
@ -402,23 +395,11 @@ Within the file, the tables for the projects SHOULD be sorted by:
#. Package version, newest/highest to older/lowest according to the #. Package version, newest/highest to older/lowest according to the
`version specifiers spec`_ `version specifiers spec`_
#. Optional dependencies (extras) via lexicographic order #. Optional dependencies (extras) via lexicographic order
#. File name based on the ``filename`` or ``url`` field (discussed #. File name based on the ``filename`` field (discussed
below) below)
All of this is to help minimize diff changes between tool executions. These recommendations are to help minimize diff changes between tool
executions.
``package._name_._version_.url``
--------------------------------
A string representing a URL where to get the file.
The installer MAY support any schemes it wants for URLs
(e.g. ``file:`` as well as ``https:``).
An installer MAY choose to not use the URL to retrieve a file
if a file matching the specified hash can be found using some
alternative means (e.g. on the file system in a cache directory).
``package._name_._version_.filename`` ``package._name_._version_.filename``
@ -432,6 +413,36 @@ file name is required to resolve wheel tags derived from the file
name. It also guarantees that the association of the array entry to name. It also guarantees that the association of the array entry to
the file it is meant for is always clear. the file it is meant for is always clear.
``[package._name_._version_.hashes]``
-------------------------------------
This table is **required**.
A table with keys specifying a hash algorithm and values as the hash
for the file represented by this entry in the
``package._name_._version_`` table.
Lockers SHOULD list hashes in lexicographic order. This is to help
minimize diff sizes and the potential to overlook hash value changes.
An installer MUST only install a file which matches one of the
specified hashes.
``package._name_._version_.url``
--------------------------------
A string representing a URL where to get the file.
The installer MAY support any schemes it wants for URLs
(e.g. ``file:`` as well as ``https:``).
An installer MAY choose to not use the URL to retrieve a file
if a file matching the specified hash can be found using alternative
means (e.g. on the file system in a cache directory).
``package._name_._version_.direct`` ``package._name_._version_.direct``
----------------------------------- -----------------------------------
@ -444,20 +455,12 @@ If the key is true, then the installer MUST follow the
the installation as "direct". the installation as "direct".
``[package._name_._version_.hashes]`` ``package._name_._version_.requires-python``
------------------------------------- --------------------------------------------
This table is **required**. A string specifying the support version(s) of Python for this file. It
follows the same format as that specified for the
A table with keys specifying hash algorithms and values as the hash ``Requires-Python`` field in the `core metadata spec`_.
for the file represented by this entry in the
``package._name_._version_`` table.
Lockers SHOULD list hashes in lexicographic order. This is to help
minimize diff sizes and the potential to overlook hash value changes.
An installer MUST only install a file which matches one of the
specified hashes.
``package._name_._version_.requires`` ``package._name_._version_.requires``
@ -467,14 +470,6 @@ An array of strings following the `dependency specifier spec`_ which
represent the dependencies of this file. represent the dependencies of this file.
``package._name_._version_.requires-python``
--------------------------------------------
A string specifying the support version(s) of Python for this file. It
follows the same format as that specified for the
``Requires-Python`` field in the `core metadata spec`_.
------- -------
Example Example
------- -------
@ -485,35 +480,43 @@ Example
created-at = 2021-10-19T22:33:45.520739+00:00 created-at = 2021-10-19T22:33:45.520739+00:00
[tool] [tool]
# Tool-specific table ala PEP 518's `[tool]` table. # Tool-specific table.
[metadata] [metadata]
requires = ["mousebender"] requires = ["mousebender"]
requires-python = ">=3.6" requires-python = ">=3.6"
[[package.attrs."21.2.0"]] [[package.attrs."21.2.0"]]
url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
filename = "attrs-21.2.0-py2.py3-none-any.whl" filename = "attrs-21.2.0-py2.py3-none-any.whl"
hashes.sha256 = "149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1" hashes.sha256 = "149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"
url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
[[package.attrs."21.2.0"]]
# If attrs had another wheel file (e.g. that was platform-specific),
# it could be listed here.
[[package.mousebender."2.0.0"]] [[package.mousebender."2.0.0"]]
url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
filename = "mousebender-2.0.0-py3-none-any.whl" filename = "mousebender-2.0.0-py3-none-any.whl"
hashes.sha256 = "a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c" hashes.sha256 = "a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"
url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
required-python = ">=3.6"
requires = ["attrs", "packaging"] requires = ["attrs", "packaging"]
[[package.packaging."20.9"]] [[package.packaging."20.9"]]
url = "https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"
filename = "packaging-20.9-py2.py3-none-any.whl" filename = "packaging-20.9-py2.py3-none-any.whl"
hashes.blake-256 = "3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af" hashes.blake-256 = "3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af"
hashes.sha256 = "67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a" hashes.sha256 = "67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
url = "https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"
requires-python = ">=3.6"
requires = ["pyparsing"] requires = ["pyparsing"]
[[package.pyparsing."2.4.7"]] [[package.pyparsing."2.4.7"]]
url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
filename = "pyparsing-2.4.7-py2.py3-none-any.whl" filename = "pyparsing-2.4.7-py2.py3-none-any.whl"
hashes.sha256="ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b" hashes.sha256="ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
direct = true # For demonstration purposes.
requires-python = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
------------------------ ------------------------
@ -523,22 +526,19 @@ Expectations for Lockers
Lockers MUST create lock files for which a topological sort of the Lockers MUST create lock files for which a topological sort of the
packages which qualify for installation on the specified platform packages which qualify for installation on the specified platform
results in a graph for which only a single version of any package results in a graph for which only a single version of any package
is possible and there is at least one compatible file to install for qualifies for installation and there is at least one compatible file
those packages. This equates to a lock file that which is acceptable to install for each package. This leads to a lock file for any
based on ``metadata.marker``, ``metadata.tag``, and supported platform where the only decision an installer can make
``metadata.requires-python`` will have a list of package versions is what the "best-fitting" wheel is to install (which is discussed
after evaluating environment markers and eliminating unsupported below).
files for which the only decision the installer will need to make is
which file to use for the package (which is outlined below).
This means that lockers are expected to utilize ``metadata.marker``, Lockers are expected to utilize ``metadata.marker``, ``metadata.tag``,
``metadata.tag``, and ``metadata.requires-python`` as appropriate and ``metadata.requires-python`` as appropriate as well as environment
as well as environment markers specified via ``requires`` and Python markers specified via ``requires`` and Python version requirements via
version requirements via ``requires-python`` to enforce this result ``requires-python`` to enforce this result for installers. Put another
for installers. Put another way, the information used in the lock way, the information used in the lock file is not expected to be
file is not expected to be pristine/raw from the locker's input and pristine/raw from the locker's input and instead is to be changed as
instead is to be changed as necessary to the benefit of the locker's necessary to the benefit of the locker's goals.
goals.
--------------------------- ---------------------------
@ -549,26 +549,25 @@ The expected algorithm for resolving what to install is:
#. Construct a dependency graph based on the data in the lock file #. Construct a dependency graph based on the data in the lock file
with ``metadata.requires`` as the starting/root point. with ``metadata.requires`` as the starting/root point.
#. Eliminate all (wheel) files that are unsupported by the specified #. Eliminate all files that are unsupported by the specified platform.
platform.
#. Eliminate all irrelevant edges between packages based on marker #. Eliminate all irrelevant edges between packages based on marker
evaluation. evaluation for ``requires``.
#. Raise an error if a package version is still reachable from the #. Raise an error if a package version is still reachable from the
root of the dependency graph but lacks any compatible (wheel) root of the dependency graph but lacks any compatible file.
file.
#. Verify that all packages left only have one version to install, #. Verify that all packages left only have one version to install,
raising an error otherwise. raising an error otherwise.
#. Install the best-fitting wheel file for each package which #. Install the best-fitting wheel file for each package which
remains. remains.
Installers MUST follow a deterministic algorithm for determining what Installers MUST follow a deterministic algorithm determine what the
the "best-fitting wheel file" is. A simple solution for this is to "best-fitting wheel file" is. A simple solution for this is to
rely upon the `packaging project <https://pypi.org/p/packaging/>`__ rely upon the `packaging project <https://pypi.org/p/packaging/>`__
and its ``packaging.tags`` module to determine wheel file precedence. and its ``packaging.tags`` module to determine wheel file precedence.
Installers MUST support installing into an empty environment. Installers MUST support installing into an empty environment.
Installers MAY support installing into an environment that already Installers MAY support installing into an environment that already
contains installed packages (and whatever that would entail). contains installed packages (and whatever that would entail to be
supported).
======================== ========================
@ -594,11 +593,10 @@ Poetry_ has said they would **not** support the PEP as-is because
Recording requirements at the file level, which is on purpose to Recording requirements at the file level, which is on purpose to
better reflect what can occur when it comes to dependencies, better reflect what can occur when it comes to dependencies,
`"is contradictory to the design of Poetry" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-973946104>`__. `"is contradictory to the design of Poetry" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-973946104>`__.
This also excludes export support to a this PEP's lock file from This also excludes export support to a this PEP's lock file as
Poetry as
`"Poetry exports the information present in the poetry.lock file into another format" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-974551351>`__ `"Poetry exports the information present in the poetry.lock file into another format" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-974551351>`__
and sdists and source trees are included in ``Poetry.lock`` files, and sdists and source trees are included in ``Poetry.lock`` files.
thus it is not a clean translation from Poetry's lock file to this Thus it is not a clean translation from Poetry's lock file to this
PEP's lock file format. PEP's lock file format.
@ -620,12 +618,6 @@ For projects which do document their lock file format like pipenv_,
they will very likely need a major version release which changes the they will very likely need a major version release which changes the
lock file format. lock file format.
Specifically for Poetry_, it has an
`export command <https://python-poetry.org/docs/cli/#export>`_ which
should allow Poetry to support this lock file format even if the
project chooses not to adopt this PEP as Poetry's primary lock file
format.
===================== =====================
Security Implications Security Implications
@ -634,8 +626,10 @@ Security Implications
A lock file should not introduce security issues but instead help A lock file should not introduce security issues but instead help
solve them. By requiring the recording of hashes for files, a lock solve them. By requiring the recording of hashes for files, a lock
file is able to help prevent tampering with code since the hash file is able to help prevent tampering with code since the hash
details were recorded. A lock file also helps prevent unexpected details were recorded. Relying on only wheel files means what files
package updates being installed which may be malicious. will be installed can be known ahead of time and is reproducible. A
lock file also helps prevent unexpected package updates being
installed which may in turn be malicious.
================= =================
@ -680,7 +674,7 @@ tools already use a TOML parser thanks to ``pyproject.toml`` so this
issue did not seem to be a showstopper. Some have also argued against issue did not seem to be a showstopper. Some have also argued against
this concern in the past by the fact that if packaging tools abhor this concern in the past by the fact that if packaging tools abhor
installing dependencies and feel they can't vendor a package then the installing dependencies and feel they can't vendor a package then the
packaging ecosystem has much bigger issues to rectify than needing to packaging ecosystem has much bigger issues to rectify than the need to
depend on a third-party TOML parser. depend on a third-party TOML parser.
@ -757,14 +751,14 @@ Accepting PEP 650
----------------- -----------------
PEP 650 was an earlier attempt at trying to tackle this problem by PEP 650 was an earlier attempt at trying to tackle this problem by
specifying an API for installers instead of standardizing on a lock file specifying an API for installers instead of standardizing on a lock
format (ala PEP 517). The file format (ala PEP 517). The
`initial response <https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657/>`__ `initial response <https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657/>`__
to PEP 650 could be considered mild/lukewarm. People seemed to be to PEP 650 could be considered mild/lukewarm. People seemed to be
consistently confused over which tools should provide what functionality consistently confused over which tools should provide what
to implement the PEP. It also potentially incurred more overhead as functionality to implement the PEP. It also potentially incurred more
it would require executing Python APIs to perform any actions involving overhead as it would require executing Python APIs to perform any
packaging. actions involving packaging.
This PEP chooses to standardize around an artifact instead of an API This PEP chooses to standardize around an artifact instead of an API
(ala PEP 621). This would allow for more tool integrations as it (ala PEP 621). This would allow for more tool integrations as it
@ -775,8 +769,10 @@ dependency graph details to be written in a human-readable format.
It also allows for easier sharing of knowledge by standardizing what It also allows for easier sharing of knowledge by standardizing what
people need to know more (e.g. tutorials become more portable between people need to know more (e.g. tutorials become more portable between
tools when it comes to understanding the artifact they produce). It's tools when it comes to understanding the artifact they produce). It's
also simply the approach other language communities have taken and seem also simply the approach other language communities have taken and
to be happy with. seem to be happy with.
Acceptance of this PEP would mean PEP 650 gets rejected.
------------------------------------------------------- -------------------------------------------------------
@ -784,11 +780,10 @@ Specifying Requirements per Package Instead of per File
------------------------------------------------------- -------------------------------------------------------
An earlier draft of this PEP specified dependencies at the package An earlier draft of this PEP specified dependencies at the package
level instead of per (wheel) file. While this has traditionally been level instead of per file. While this has traditionally been how
how packaging systems work, it actually did not reflect accurately packaging systems work, it actually did not reflect accurately how
how things are specified. As such, this PEP was subsequently updated things are specified. As such, this PEP was subsequently updated to
to reflect the granularity that dependencies can truly be specified reflect the granularity that dependencies can truly be specified at.
at.
---------------------------------- ----------------------------------
@ -802,44 +797,40 @@ things such as indexes, etc., it was decided this would best be left
to a separate PEP. to a separate PEP.
===========
Open Issues
===========
------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------
Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format
------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------
For security reproducibility reasons this PEP only considers After `extensive discussion <https://discuss.python.org/t/supporting-sdists-and-source-trees-in-pep-665/11869/>`__,
supporting installation from wheel files. Installing from either an it was decided that this PEP would not support source distributions
sdist or source tree requires arbitrary code execution during (aka sdists) or source trees as an acceptable format for code.
installation, unknown files to be installed, and an unknown set of Introducing sdists and source trees to this PEP would immediately undo
dependencies. Those issues all run counter to guaranteeing users get the reproducibility and security goals due to needing to execute code
the same files for the same platform as well as making sure they are to build the sdist or source tree. It would also greatly increase
receiving the expected files. the complexity for (at least) installers as the dynamic build nature
of sdists and source trees means the installer would need to handle
fully resolving whatever requirements the sdists produced dynamically,
both from a building and installation perspective.
To deal with this issue, people would need to build their own wheels Due to all of this, it was decided it was best to have a separate
from sdists and cache them. Then the lockers would record the hashes discussion about what supporting sdists and source trees **after**
of those wheels and the installers would then be expected to use this PEP is accepted/rejected. As the proposed file format is
those wheels. versioned, introducing sdists and source tree support in a later PEP
is doable.
Another option is to allow sdists (and potentially source trees) be It should be noted, though, that this PEP is **not** stop an
listed as support file formats, but have them marked as insecure in out-of-band solution from being developed to be used in conjunction
the lock file and require the installer force the user to opt into with this PEP. Building wheel files from sdists and shipping them with
using insecure file formats. Unfortunately because sdists which don't code upon deployment so they can be included in the lock file is one
necessarily follow version 2.2 of the `core metadata spec`_ for their option. Another is to use a requirements file *just* for sdists and
``PKG-INFO`` file will have unknown dependencies, breaking the source trees, then relying on a lock file for all wheels.
guarantee that results will be reproducible thanks to potential
arbitrary calculations of those dependencies. And even if an sdist did
follow the latest spec, they could still list their requirements as ===========
dynamic, still making it impossible to statically know what should be Open Issues
installed. As such, installers would either have to have a full ===========
resolver to handle these dynamic cases or only sdists which follow
version 2.2 of the core metadata spec **and** statically specify None.
their dependencies could be listed. But at that point the project is
probably capable of providing wheels, making support for sdists that
much less important/useful.
=============== ===============
@ -854,8 +845,8 @@ Thanks to Kushal Das for making sure reproducible builds stayed a
concern for this PEP. concern for this PEP.
Thanks to Andrea McInnes for initially settling the bikeshedding and Thanks to Andrea McInnes for initially settling the bikeshedding and
choosing the paint colour of ``needs`` (at which point that caused choosing the paint colour of ``needs`` (at which point people ralled
people to rally around the ``requires`` colour). around the ``requires`` colour instead).
========= =========