PEP 665: resolve the (final) open issue around sdists
This commit is contained in:
Brett Cannon
parent
152fe41cf2
commit
6ef86af2ae
405
pep-0665.rst
405
pep-0665.rst
|
|
@ -9,7 +9,7 @@ Status: Draft
|
|||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 29-Jul-2021
|
||||
Post-History: 29-Jul-2021, 03-Nov-2021
|
||||
Post-History: 29-Jul-2021, 03-Nov-2021, 25-Nov-2021
|
||||
Resolution:
|
||||
|
||||
========
|
||||
|
|
@ -17,14 +17,15 @@ Abstract
|
|||
========
|
||||
|
||||
This PEP specifies a file format to specify the list of Python package
|
||||
installation requirements for an application, and the relation between the
|
||||
specified requirements. The list of requirements is considered
|
||||
installation requirements for an application, and the relation between
|
||||
the specified requirements. The list of requirements is considered
|
||||
exhaustive for the installation target, and thus not requiring any
|
||||
information beyond the platform being installed for, and the file
|
||||
itself. The file format is flexible enough to allow installing the
|
||||
requirements across different platforms, which guarantees
|
||||
requirements across different platforms, which allows for
|
||||
reproducibility on multiple platforms from the same file.
|
||||
|
||||
|
||||
===========
|
||||
Terminology
|
||||
===========
|
||||
|
|
@ -38,15 +39,15 @@ import system. The packages on PyPI are an example of this.
|
|||
An *application* or *app* is an end product that other external code
|
||||
does not directly rely on via the import system (i.e. they are
|
||||
standalone). Desktop applications, command-line tools, etc. are
|
||||
examples.
|
||||
examples of applications.
|
||||
|
||||
A *lock file* records the packages that are to be installed for an
|
||||
app. Traditionally, the exact version of the package to be installed
|
||||
is specified by a lock file, but all specified packages are not always
|
||||
is specified by a lock file, but specified packages are not always
|
||||
installed on a given platform (according a filtering logic described
|
||||
in a later section), which enables the lock file to describe
|
||||
reproducibility across multiple platforms. Examples of this are
|
||||
``package-lock.json`` from npm_, ``Poetry.lock`` from Poetry, etc.
|
||||
``package-lock.json`` from npm_, ``Poetry.lock`` from Poetry_, etc.
|
||||
|
||||
*Locking* is the act of taking the input of the packages an app
|
||||
depends on and producting a lock file from that.
|
||||
|
|
@ -85,18 +86,19 @@ and reproducibility on any one specific platform.
|
|||
Three, reproducibility is more secure. When you control exactly what
|
||||
files are installed, you can make sure no malicious actor is
|
||||
attempting to slip nefarious code into your application (i.e. some
|
||||
supply chain attacks). By making a lock file which always leads to
|
||||
supply chain attacks). By using a lock file which always leads to
|
||||
reproducible installs, we can avoid certain risks entirely.
|
||||
|
||||
Four, relying on wheels provides reproducibility without requiring
|
||||
build tools to support reproducibility as well. Thanks to wheels being
|
||||
static and not executing code as part of installation, wheels always
|
||||
lead to a reproducible result. Compare this to source distributions
|
||||
(aka sdists) or source trees which only lead to a reproducible install
|
||||
if their build tool supports reproducibility due to inherent code
|
||||
execution. Unfortunately the vast majority of build tools do not
|
||||
support reproducible builds, so this PEP helps alleviate that issue
|
||||
by only relying on wheels.
|
||||
Four, relying on the `wheel file`_ format provides reproducibility
|
||||
without requiring build tools to support reproducibility themselves.
|
||||
Thanks to wheels being static and not executing code as part of
|
||||
installation, wheels always lead to a reproducible result. Compare
|
||||
this to source distributions (aka sdists) or source trees which only
|
||||
lead to a reproducible install if their build tool supports
|
||||
reproducibility due to inherent code execution. Unfortunately the vast
|
||||
majority of build tools do not support reproducible builds, so this
|
||||
PEP helps alleviate that issue by only supporting wheels as a package
|
||||
format.
|
||||
|
||||
This PEP proposes a standard for a lock file, as the current solutions
|
||||
don't meet the outlined goals. Today, the closest we come to a lock
|
||||
|
|
@ -119,7 +121,7 @@ Unfortunately, those tools all use differing lock file formats. This
|
|||
means tooling around these tools much be unique. This impacts tooling
|
||||
such as code editors and hosting providers, which want to be as
|
||||
flexible as possible when it comes to accepting a user's application
|
||||
code, but also have a limit as to how much development resource they
|
||||
code, but also have a limit as to how much development resources they
|
||||
can spend to add support for yet another lock file format. A
|
||||
standardized format would allow tools to focus their work on a single
|
||||
target, and make sure that workflow decisions made by developers
|
||||
|
|
@ -159,23 +161,25 @@ Secure by Design
|
|||
Viewing the `requirements file format`_ as the closest we have to
|
||||
a lock file standard, there are a few issues with the file format when
|
||||
it comes to security. First is that the file format simply does not
|
||||
require you specify the exact version of a package. This is why
|
||||
tools like `pip-tools`_ exist to help manage that for the user.
|
||||
require you to specify the exact version of a package. This is why
|
||||
tools like `pip-tools`_ exist to help manage that users of
|
||||
requirements files.
|
||||
|
||||
Second, you must opt into specifying what files may be installed by
|
||||
using the ``--hash`` argument for a specific dependency. This is also
|
||||
optional with pip-tools as it requires specifying the
|
||||
Second, you must opt into specifying what files are acceptable to be
|
||||
installed by using the ``--hash`` argument for a specific dependency.
|
||||
This is also optional with pip-tools as it requires specifying the
|
||||
``--generate-hashes`` CLI argument.
|
||||
|
||||
Third, even when you control what files may be installed, it does not
|
||||
prevent other packages from being installed. If a dependency is not
|
||||
listed in the requirements file, pip will happily go searching for a
|
||||
file to meet that need, unless you specify ``--no-deps`` as an
|
||||
argument.
|
||||
file to meet that need. You must specify ``--no-deps`` as an
|
||||
argument to pip to prevent unintended dependency resolution outside
|
||||
of the requirements file.
|
||||
|
||||
Fourth, the format allows for installing a
|
||||
`source distribution file`_ (aka "sdist"). By its very nature,
|
||||
installing an sdist may imply executing arbitrary Python code, meaning
|
||||
installing an sdist requires executing arbitrary Python code, meaning
|
||||
that there is no control over what files may be installed. Only by
|
||||
specifying ``--only-binary :all:`` can you guarantee pip to only use a
|
||||
`wheel file`_ for each package.
|
||||
|
|
@ -185,17 +189,20 @@ being proposed, a user should always do the following steps:
|
|||
|
||||
#. Use pip-tools and its command ``pip-compile --generate-hashes``
|
||||
#. Install the requirements file using
|
||||
``pip install --no-deps --only-binary :all:``
|
||||
``pip install --require-hashes --no-deps --only-binary :all:``
|
||||
|
||||
Critically, all of those flags, and both specificity and exhaustion of
|
||||
what to install that pip-tools provides, are optional.
|
||||
Critically, all of those flags, and both the specificity and
|
||||
exhaustion of what to install that pip-tools provides, are optional
|
||||
for requirements files.
|
||||
|
||||
As such, the proposal raised in this PEP is secure by design to combat
|
||||
some supply chain attacks. Hashes for files which would be used to
|
||||
install from are **required**. You can **only** install from wheels
|
||||
to unambiguously define what files will be placed in the file system.
|
||||
Installers **must** have an unambiguous installation from a lock file
|
||||
for a given platform.
|
||||
As such, the proposal raised in this PEP is secure by design which
|
||||
combats some supply chain attacks. Hashes for files which would be
|
||||
used to install from are **required**. You can **only** install from
|
||||
wheels to unambiguously define what files will be placed in the file
|
||||
system. Installers **must** lead to an deterministic installation
|
||||
from a lock file for a given platform. All of this leads to a
|
||||
reproducible installation which you can deem trustworthy (when you
|
||||
have audited the lock file and what it lists).
|
||||
|
||||
|
||||
--------------
|
||||
|
|
@ -205,17 +212,17 @@ Cross-Platform
|
|||
Various projects which already have a lock file, like PDM_ and
|
||||
Poetry_, provide a lock file which is *cross-platform*. This allows
|
||||
for a single lock file to work on multiple platforms while still
|
||||
leading to exact same top-level requirements to be installed
|
||||
everywhere while the installation being consistent/unambiguous on
|
||||
leading to the exact same top-level requirements to be installed
|
||||
everywhere with the installation being consistent/unambiguous on
|
||||
each platform.
|
||||
|
||||
As to why this is useful, let's use an example involving PyWeek_
|
||||
(a week-long game development competition). We assume you are
|
||||
developing on Linux, while someone you choose to partner with is
|
||||
using macOS. Now assume the judges are using Windows. How do you make
|
||||
sure everyone is using the same top-level dependencies, while allowing
|
||||
for any platform-specific requirements (e.g. a package requires a
|
||||
helper package under Windows)?
|
||||
(a week-long game development competition). Assume you are developing
|
||||
on Linux, while someone you choose to partner with is using macOS.
|
||||
Now assume the judges are using Windows. How do you make sure everyone
|
||||
is using the same top-level dependencies, while allowing for any
|
||||
platform-specific requirements (e.g. a package requires a helper
|
||||
package under Windows)?
|
||||
|
||||
With a cross-platform lock file, you can make sure that the key
|
||||
requirements are met consistently across all platforms. You can then
|
||||
|
|
@ -231,7 +238,7 @@ The separation of concerns between a locker and an installer allows
|
|||
for an installer to have a much simpler operation to perform. As
|
||||
such, it not only allows for installers to be easier to write, but
|
||||
facilitates in making sure installers create unambiguous, reproducible
|
||||
installations.
|
||||
installations correctly.
|
||||
|
||||
The installer can also expend less computation/energy in creating the
|
||||
installation. This is beneficial not only for faster installs, but
|
||||
|
|
@ -239,21 +246,9 @@ also from an energy consumption perspective, as installers are
|
|||
expected to be run more often than lockers.
|
||||
|
||||
This has led to a design where the locker must do more work upfront
|
||||
to benefit installers. It also means the complexity of package
|
||||
dependencies is simpler and easier to comprehend to avoid ambiguity.
|
||||
|
||||
|
||||
-----------------------
|
||||
As Flexible as Possible
|
||||
-----------------------
|
||||
|
||||
Realizing that workflows vary greatly between companies, projects, and
|
||||
even people, this PEP tries to be strict where it's important and
|
||||
undefined/flexible everywhere else. As such, this PEP is strict where
|
||||
it is important for reproducibility and compatibilitiy, but does not
|
||||
specify any restrictions in other situations not covered by this PEP;
|
||||
if the PEP does not specifically state something then it is assumed to
|
||||
be up to the locker or installer to decide what is best.
|
||||
to the benefit installers. It also means the complexity of package
|
||||
dependencies is simpler and easier to comprehend in a lock files to
|
||||
avoid ambiguity.
|
||||
|
||||
|
||||
=============
|
||||
|
|
@ -265,7 +260,7 @@ Details
|
|||
-------
|
||||
|
||||
Lock files MUST use the TOML_ file format. This not only prevents the
|
||||
need to have another file format in the Python packaging ecosystem,
|
||||
need to have another file format in the Python packaging ecosystem
|
||||
thanks to its adoption by PEP 518 for ``pyproject.toml``, but also
|
||||
assists in making lock files more human-readable.
|
||||
|
||||
|
|
@ -273,11 +268,11 @@ Lock files MUST end their file names with ``.pylock.toml``. The
|
|||
``.toml`` part unambiguously distinguishes the format of the file,
|
||||
and helps tools like code editors support the file appropriately. The
|
||||
``.pylock`` part distinguishes the file from other TOML files the user
|
||||
has, to make logic easier for tools to create functionalities specific
|
||||
to Python lock files, instead of TOML files in general.
|
||||
has, to make the logic easier for tools to create functionality
|
||||
specific to Python lock files, instead of TOML files in general.
|
||||
|
||||
The following sections are the top-level keys of the TOML file data
|
||||
format. Any field not listed as required is considered optional.
|
||||
format. Any field not listed as **required** is considered optional.
|
||||
|
||||
|
||||
``version``
|
||||
|
|
@ -287,15 +282,16 @@ This field is **required**.
|
|||
|
||||
The version of the lock file being used. The key MUST be a string
|
||||
consisting of a number that follows the same formatting as the
|
||||
``Metadata-Version`` key in the `core metadata spec`_. The value MUST
|
||||
be set to ``"1.0"`` until a future PEP allows for a different value.
|
||||
The introduction of a new *optional* key SHOULD increase the minor
|
||||
version. The introduction of a new required key or changing the
|
||||
format MUST increase the major version. How to handle other scenarios
|
||||
is left as a per-PEP decision.
|
||||
``Metadata-Version`` key in the `core metadata spec`_.
|
||||
|
||||
The value MUST be set to ``"1.0"`` until a future PEP allows for a
|
||||
different value. The introduction of a new *optional* key to the file
|
||||
format SHOULD increase the minor version. The introduction of a new
|
||||
required key or changing the format MUST increase the major version.
|
||||
How to handle other scenarios is left as a per-PEP decision.
|
||||
|
||||
Installers MUST warn the user if the lock file specifies a version
|
||||
whose major version is support but whose minor version is
|
||||
whose major version is supported but whose minor version is
|
||||
unsupported/unrecognized (e.g. the installer supports ``"1.0"``, but
|
||||
the lock file specifies ``"1.1"``).
|
||||
|
||||
|
|
@ -314,9 +310,8 @@ native timestamp type). It MUST be recorded using the UTC time zone to
|
|||
avoid ambiguity.
|
||||
|
||||
If the SOURCE_DATE_EPOCH_ environment variable is set, it MUST be used
|
||||
as the timestamp by the locker. This facilitates reproducibility of the
|
||||
lock file itself.
|
||||
|
||||
as the timestamp by the locker. This facilitates reproducibility of
|
||||
the lock file itself.
|
||||
|
||||
|
||||
``[tool]``
|
||||
|
|
@ -341,13 +336,13 @@ A table containing data applying to the overall lock file.
|
|||
A key storing a string containing an environment marker as
|
||||
specified in the `dependency specifier spec`_.
|
||||
|
||||
|
||||
The locker MAY specify an environment marker which specifies any
|
||||
restrictions the lock file was generated under.
|
||||
|
||||
If the installer is installing for an environment which does not
|
||||
satisfy the specified environment marker, the installer MUST raise an
|
||||
error as the lock file does not support the environment.
|
||||
error as the lock file does not support the target installation
|
||||
environment.
|
||||
|
||||
|
||||
``metadata.tag``
|
||||
|
|
@ -356,12 +351,10 @@ error as the lock file does not support the environment.
|
|||
A key storing a string specifying `platform compatibility tags`_
|
||||
(i.e. wheel tags). The tag MAY be a compressed tag set.
|
||||
|
||||
The locker MAY specify a tag (set) which specify which platform(s)
|
||||
the lock file supports.
|
||||
|
||||
If the installer is installing for an environment which does not
|
||||
satisfy the specified tag (set), the installer MUST raise an error
|
||||
as the lock file does not support the environment.
|
||||
as the lock file does not support the targeted installation
|
||||
environment.
|
||||
|
||||
|
||||
``metadata.requires``
|
||||
|
|
@ -377,7 +370,7 @@ and thus the root of the dependency graph.
|
|||
``metadata.requires-python``
|
||||
----------------------------
|
||||
|
||||
A string specifying the support version(s) of Python for this lock
|
||||
A string specifying the supported version(s) of Python for this lock
|
||||
file. It follows the same format as that specified for the
|
||||
``Requires-Python`` field in the `core metadata spec`_.
|
||||
|
||||
|
|
@ -387,11 +380,11 @@ file. It follows the same format as that specified for the
|
|||
|
||||
This array is **required**.
|
||||
|
||||
An array per package and version containing details for the potential
|
||||
An array per package and version containing entries for the potential
|
||||
(wheel) files to install (as represented by ``_name_`` and
|
||||
``_version_``, respectively).
|
||||
|
||||
Lockers must MUST normalize a project's name according to the
|
||||
Lockers MUST normalize a project's name according to the
|
||||
`simple repository API`_. If extras are specified as part of the
|
||||
project to install, the extras are to be included in the key name and
|
||||
are to be sorted in lexicographic order.
|
||||
|
|
@ -402,23 +395,11 @@ Within the file, the tables for the projects SHOULD be sorted by:
|
|||
#. Package version, newest/highest to older/lowest according to the
|
||||
`version specifiers spec`_
|
||||
#. Optional dependencies (extras) via lexicographic order
|
||||
#. File name based on the ``filename`` or ``url`` field (discussed
|
||||
#. File name based on the ``filename`` field (discussed
|
||||
below)
|
||||
|
||||
All of this is to help minimize diff changes between tool executions.
|
||||
|
||||
|
||||
``package._name_._version_.url``
|
||||
--------------------------------
|
||||
|
||||
A string representing a URL where to get the file.
|
||||
|
||||
The installer MAY support any schemes it wants for URLs
|
||||
(e.g. ``file:`` as well as ``https:``).
|
||||
|
||||
An installer MAY choose to not use the URL to retrieve a file
|
||||
if a file matching the specified hash can be found using some
|
||||
alternative means (e.g. on the file system in a cache directory).
|
||||
These recommendations are to help minimize diff changes between tool
|
||||
executions.
|
||||
|
||||
|
||||
``package._name_._version_.filename``
|
||||
|
|
@ -432,6 +413,36 @@ file name is required to resolve wheel tags derived from the file
|
|||
name. It also guarantees that the association of the array entry to
|
||||
the file it is meant for is always clear.
|
||||
|
||||
|
||||
``[package._name_._version_.hashes]``
|
||||
-------------------------------------
|
||||
|
||||
This table is **required**.
|
||||
|
||||
A table with keys specifying a hash algorithm and values as the hash
|
||||
for the file represented by this entry in the
|
||||
``package._name_._version_`` table.
|
||||
|
||||
Lockers SHOULD list hashes in lexicographic order. This is to help
|
||||
minimize diff sizes and the potential to overlook hash value changes.
|
||||
|
||||
An installer MUST only install a file which matches one of the
|
||||
specified hashes.
|
||||
|
||||
|
||||
``package._name_._version_.url``
|
||||
--------------------------------
|
||||
|
||||
A string representing a URL where to get the file.
|
||||
|
||||
The installer MAY support any schemes it wants for URLs
|
||||
(e.g. ``file:`` as well as ``https:``).
|
||||
|
||||
An installer MAY choose to not use the URL to retrieve a file
|
||||
if a file matching the specified hash can be found using alternative
|
||||
means (e.g. on the file system in a cache directory).
|
||||
|
||||
|
||||
``package._name_._version_.direct``
|
||||
-----------------------------------
|
||||
|
||||
|
|
@ -444,20 +455,12 @@ If the key is true, then the installer MUST follow the
|
|||
the installation as "direct".
|
||||
|
||||
|
||||
``[package._name_._version_.hashes]``
|
||||
-------------------------------------
|
||||
``package._name_._version_.requires-python``
|
||||
--------------------------------------------
|
||||
|
||||
This table is **required**.
|
||||
|
||||
A table with keys specifying hash algorithms and values as the hash
|
||||
for the file represented by this entry in the
|
||||
``package._name_._version_`` table.
|
||||
|
||||
Lockers SHOULD list hashes in lexicographic order. This is to help
|
||||
minimize diff sizes and the potential to overlook hash value changes.
|
||||
|
||||
An installer MUST only install a file which matches one of the
|
||||
specified hashes.
|
||||
A string specifying the support version(s) of Python for this file. It
|
||||
follows the same format as that specified for the
|
||||
``Requires-Python`` field in the `core metadata spec`_.
|
||||
|
||||
|
||||
``package._name_._version_.requires``
|
||||
|
|
@ -467,14 +470,6 @@ An array of strings following the `dependency specifier spec`_ which
|
|||
represent the dependencies of this file.
|
||||
|
||||
|
||||
``package._name_._version_.requires-python``
|
||||
--------------------------------------------
|
||||
|
||||
A string specifying the support version(s) of Python for this file. It
|
||||
follows the same format as that specified for the
|
||||
``Requires-Python`` field in the `core metadata spec`_.
|
||||
|
||||
|
||||
-------
|
||||
Example
|
||||
-------
|
||||
|
|
@ -485,35 +480,43 @@ Example
|
|||
created-at = 2021-10-19T22:33:45.520739+00:00
|
||||
|
||||
[tool]
|
||||
# Tool-specific table ala PEP 518's `[tool]` table.
|
||||
|
||||
# Tool-specific table.
|
||||
|
||||
[metadata]
|
||||
requires = ["mousebender"]
|
||||
requires-python = ">=3.6"
|
||||
|
||||
[[package.attrs."21.2.0"]]
|
||||
url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
|
||||
filename = "attrs-21.2.0-py2.py3-none-any.whl"
|
||||
hashes.sha256 = "149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"
|
||||
url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
|
||||
requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
|
||||
|
||||
[[package.attrs."21.2.0"]]
|
||||
# If attrs had another wheel file (e.g. that was platform-specific),
|
||||
# it could be listed here.
|
||||
|
||||
[[package.mousebender."2.0.0"]]
|
||||
url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
|
||||
filename = "mousebender-2.0.0-py3-none-any.whl"
|
||||
hashes.sha256 = "a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"
|
||||
url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
|
||||
required-python = ">=3.6"
|
||||
requires = ["attrs", "packaging"]
|
||||
|
||||
[[package.packaging."20.9"]]
|
||||
url = "https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"
|
||||
filename = "packaging-20.9-py2.py3-none-any.whl"
|
||||
hashes.blake-256 = "3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af"
|
||||
hashes.sha256 = "67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
|
||||
url = "https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"
|
||||
requires-python = ">=3.6"
|
||||
requires = ["pyparsing"]
|
||||
|
||||
[[package.pyparsing."2.4.7"]]
|
||||
url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
|
||||
filename = "pyparsing-2.4.7-py2.py3-none-any.whl"
|
||||
hashes.sha256="ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
|
||||
url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
|
||||
direct = true # For demonstration purposes.
|
||||
requires-python = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
|
||||
|
||||
|
||||
------------------------
|
||||
|
|
@ -523,22 +526,19 @@ Expectations for Lockers
|
|||
Lockers MUST create lock files for which a topological sort of the
|
||||
packages which qualify for installation on the specified platform
|
||||
results in a graph for which only a single version of any package
|
||||
is possible and there is at least one compatible file to install for
|
||||
those packages. This equates to a lock file that which is acceptable
|
||||
based on ``metadata.marker``, ``metadata.tag``, and
|
||||
``metadata.requires-python`` will have a list of package versions
|
||||
after evaluating environment markers and eliminating unsupported
|
||||
files for which the only decision the installer will need to make is
|
||||
which file to use for the package (which is outlined below).
|
||||
qualifies for installation and there is at least one compatible file
|
||||
to install for each package. This leads to a lock file for any
|
||||
supported platform where the only decision an installer can make
|
||||
is what the "best-fitting" wheel is to install (which is discussed
|
||||
below).
|
||||
|
||||
This means that lockers are expected to utilize ``metadata.marker``,
|
||||
``metadata.tag``, and ``metadata.requires-python`` as appropriate
|
||||
as well as environment markers specified via ``requires`` and Python
|
||||
version requirements via ``requires-python`` to enforce this result
|
||||
for installers. Put another way, the information used in the lock
|
||||
file is not expected to be pristine/raw from the locker's input and
|
||||
instead is to be changed as necessary to the benefit of the locker's
|
||||
goals.
|
||||
Lockers are expected to utilize ``metadata.marker``, ``metadata.tag``,
|
||||
and ``metadata.requires-python`` as appropriate as well as environment
|
||||
markers specified via ``requires`` and Python version requirements via
|
||||
``requires-python`` to enforce this result for installers. Put another
|
||||
way, the information used in the lock file is not expected to be
|
||||
pristine/raw from the locker's input and instead is to be changed as
|
||||
necessary to the benefit of the locker's goals.
|
||||
|
||||
|
||||
---------------------------
|
||||
|
|
@ -549,26 +549,25 @@ The expected algorithm for resolving what to install is:
|
|||
|
||||
#. Construct a dependency graph based on the data in the lock file
|
||||
with ``metadata.requires`` as the starting/root point.
|
||||
#. Eliminate all (wheel) files that are unsupported by the specified
|
||||
platform.
|
||||
#. Eliminate all files that are unsupported by the specified platform.
|
||||
#. Eliminate all irrelevant edges between packages based on marker
|
||||
evaluation.
|
||||
evaluation for ``requires``.
|
||||
#. Raise an error if a package version is still reachable from the
|
||||
root of the dependency graph but lacks any compatible (wheel)
|
||||
file.
|
||||
root of the dependency graph but lacks any compatible file.
|
||||
#. Verify that all packages left only have one version to install,
|
||||
raising an error otherwise.
|
||||
#. Install the best-fitting wheel file for each package which
|
||||
remains.
|
||||
|
||||
Installers MUST follow a deterministic algorithm for determining what
|
||||
the "best-fitting wheel file" is. A simple solution for this is to
|
||||
Installers MUST follow a deterministic algorithm determine what the
|
||||
"best-fitting wheel file" is. A simple solution for this is to
|
||||
rely upon the `packaging project <https://pypi.org/p/packaging/>`__
|
||||
and its ``packaging.tags`` module to determine wheel file precedence.
|
||||
|
||||
Installers MUST support installing into an empty environment.
|
||||
Installers MAY support installing into an environment that already
|
||||
contains installed packages (and whatever that would entail).
|
||||
contains installed packages (and whatever that would entail to be
|
||||
supported).
|
||||
|
||||
|
||||
========================
|
||||
|
|
@ -594,11 +593,10 @@ Poetry_ has said they would **not** support the PEP as-is because
|
|||
Recording requirements at the file level, which is on purpose to
|
||||
better reflect what can occur when it comes to dependencies,
|
||||
`"is contradictory to the design of Poetry" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-973946104>`__.
|
||||
This also excludes export support to a this PEP's lock file from
|
||||
Poetry as
|
||||
This also excludes export support to a this PEP's lock file as
|
||||
`"Poetry exports the information present in the poetry.lock file into another format" <https://github.com/python-poetry/poetry/issues/4710#issuecomment-974551351>`__
|
||||
and sdists and source trees are included in ``Poetry.lock`` files,
|
||||
thus it is not a clean translation from Poetry's lock file to this
|
||||
and sdists and source trees are included in ``Poetry.lock`` files.
|
||||
Thus it is not a clean translation from Poetry's lock file to this
|
||||
PEP's lock file format.
|
||||
|
||||
|
||||
|
|
@ -620,12 +618,6 @@ For projects which do document their lock file format like pipenv_,
|
|||
they will very likely need a major version release which changes the
|
||||
lock file format.
|
||||
|
||||
Specifically for Poetry_, it has an
|
||||
`export command <https://python-poetry.org/docs/cli/#export>`_ which
|
||||
should allow Poetry to support this lock file format even if the
|
||||
project chooses not to adopt this PEP as Poetry's primary lock file
|
||||
format.
|
||||
|
||||
|
||||
=====================
|
||||
Security Implications
|
||||
|
|
@ -634,8 +626,10 @@ Security Implications
|
|||
A lock file should not introduce security issues but instead help
|
||||
solve them. By requiring the recording of hashes for files, a lock
|
||||
file is able to help prevent tampering with code since the hash
|
||||
details were recorded. A lock file also helps prevent unexpected
|
||||
package updates being installed which may be malicious.
|
||||
details were recorded. Relying on only wheel files means what files
|
||||
will be installed can be known ahead of time and is reproducible. A
|
||||
lock file also helps prevent unexpected package updates being
|
||||
installed which may in turn be malicious.
|
||||
|
||||
|
||||
=================
|
||||
|
|
@ -680,7 +674,7 @@ tools already use a TOML parser thanks to ``pyproject.toml`` so this
|
|||
issue did not seem to be a showstopper. Some have also argued against
|
||||
this concern in the past by the fact that if packaging tools abhor
|
||||
installing dependencies and feel they can't vendor a package then the
|
||||
packaging ecosystem has much bigger issues to rectify than needing to
|
||||
packaging ecosystem has much bigger issues to rectify than the need to
|
||||
depend on a third-party TOML parser.
|
||||
|
||||
|
||||
|
|
@ -757,14 +751,14 @@ Accepting PEP 650
|
|||
-----------------
|
||||
|
||||
PEP 650 was an earlier attempt at trying to tackle this problem by
|
||||
specifying an API for installers instead of standardizing on a lock file
|
||||
format (ala PEP 517). The
|
||||
specifying an API for installers instead of standardizing on a lock
|
||||
file format (ala PEP 517). The
|
||||
`initial response <https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657/>`__
|
||||
to PEP 650 could be considered mild/lukewarm. People seemed to be
|
||||
consistently confused over which tools should provide what functionality
|
||||
to implement the PEP. It also potentially incurred more overhead as
|
||||
it would require executing Python APIs to perform any actions involving
|
||||
packaging.
|
||||
consistently confused over which tools should provide what
|
||||
functionality to implement the PEP. It also potentially incurred more
|
||||
overhead as it would require executing Python APIs to perform any
|
||||
actions involving packaging.
|
||||
|
||||
This PEP chooses to standardize around an artifact instead of an API
|
||||
(ala PEP 621). This would allow for more tool integrations as it
|
||||
|
|
@ -775,8 +769,10 @@ dependency graph details to be written in a human-readable format.
|
|||
It also allows for easier sharing of knowledge by standardizing what
|
||||
people need to know more (e.g. tutorials become more portable between
|
||||
tools when it comes to understanding the artifact they produce). It's
|
||||
also simply the approach other language communities have taken and seem
|
||||
to be happy with.
|
||||
also simply the approach other language communities have taken and
|
||||
seem to be happy with.
|
||||
|
||||
Acceptance of this PEP would mean PEP 650 gets rejected.
|
||||
|
||||
|
||||
-------------------------------------------------------
|
||||
|
|
@ -784,11 +780,10 @@ Specifying Requirements per Package Instead of per File
|
|||
-------------------------------------------------------
|
||||
|
||||
An earlier draft of this PEP specified dependencies at the package
|
||||
level instead of per (wheel) file. While this has traditionally been
|
||||
how packaging systems work, it actually did not reflect accurately
|
||||
how things are specified. As such, this PEP was subsequently updated
|
||||
to reflect the granularity that dependencies can truly be specified
|
||||
at.
|
||||
level instead of per file. While this has traditionally been how
|
||||
packaging systems work, it actually did not reflect accurately how
|
||||
things are specified. As such, this PEP was subsequently updated to
|
||||
reflect the granularity that dependencies can truly be specified at.
|
||||
|
||||
|
||||
----------------------------------
|
||||
|
|
@ -802,44 +797,40 @@ things such as indexes, etc., it was decided this would best be left
|
|||
to a separate PEP.
|
||||
|
||||
|
||||
===========
|
||||
Open Issues
|
||||
===========
|
||||
|
||||
|
||||
-------------------------------------------------------------------------------------
|
||||
Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format
|
||||
-------------------------------------------------------------------------------------
|
||||
|
||||
For security reproducibility reasons this PEP only considers
|
||||
supporting installation from wheel files. Installing from either an
|
||||
sdist or source tree requires arbitrary code execution during
|
||||
installation, unknown files to be installed, and an unknown set of
|
||||
dependencies. Those issues all run counter to guaranteeing users get
|
||||
the same files for the same platform as well as making sure they are
|
||||
receiving the expected files.
|
||||
After `extensive discussion <https://discuss.python.org/t/supporting-sdists-and-source-trees-in-pep-665/11869/>`__,
|
||||
it was decided that this PEP would not support source distributions
|
||||
(aka sdists) or source trees as an acceptable format for code.
|
||||
Introducing sdists and source trees to this PEP would immediately undo
|
||||
the reproducibility and security goals due to needing to execute code
|
||||
to build the sdist or source tree. It would also greatly increase
|
||||
the complexity for (at least) installers as the dynamic build nature
|
||||
of sdists and source trees means the installer would need to handle
|
||||
fully resolving whatever requirements the sdists produced dynamically,
|
||||
both from a building and installation perspective.
|
||||
|
||||
To deal with this issue, people would need to build their own wheels
|
||||
from sdists and cache them. Then the lockers would record the hashes
|
||||
of those wheels and the installers would then be expected to use
|
||||
those wheels.
|
||||
Due to all of this, it was decided it was best to have a separate
|
||||
discussion about what supporting sdists and source trees **after**
|
||||
this PEP is accepted/rejected. As the proposed file format is
|
||||
versioned, introducing sdists and source tree support in a later PEP
|
||||
is doable.
|
||||
|
||||
Another option is to allow sdists (and potentially source trees) be
|
||||
listed as support file formats, but have them marked as insecure in
|
||||
the lock file and require the installer force the user to opt into
|
||||
using insecure file formats. Unfortunately because sdists which don't
|
||||
necessarily follow version 2.2 of the `core metadata spec`_ for their
|
||||
``PKG-INFO`` file will have unknown dependencies, breaking the
|
||||
guarantee that results will be reproducible thanks to potential
|
||||
arbitrary calculations of those dependencies. And even if an sdist did
|
||||
follow the latest spec, they could still list their requirements as
|
||||
dynamic, still making it impossible to statically know what should be
|
||||
installed. As such, installers would either have to have a full
|
||||
resolver to handle these dynamic cases or only sdists which follow
|
||||
version 2.2 of the core metadata spec **and** statically specify
|
||||
their dependencies could be listed. But at that point the project is
|
||||
probably capable of providing wheels, making support for sdists that
|
||||
much less important/useful.
|
||||
It should be noted, though, that this PEP is **not** stop an
|
||||
out-of-band solution from being developed to be used in conjunction
|
||||
with this PEP. Building wheel files from sdists and shipping them with
|
||||
code upon deployment so they can be included in the lock file is one
|
||||
option. Another is to use a requirements file *just* for sdists and
|
||||
source trees, then relying on a lock file for all wheels.
|
||||
|
||||
|
||||
===========
|
||||
Open Issues
|
||||
===========
|
||||
|
||||
None.
|
||||
|
||||
|
||||
===============
|
||||
|
|
@ -854,8 +845,8 @@ Thanks to Kushal Das for making sure reproducible builds stayed a
|
|||
concern for this PEP.
|
||||
|
||||
Thanks to Andrea McInnes for initially settling the bikeshedding and
|
||||
choosing the paint colour of ``needs`` (at which point that caused
|
||||
people to rally around the ``requires`` colour).
|
||||
choosing the paint colour of ``needs`` (at which point people ralled
|
||||
around the ``requires`` colour instead).
|
||||
|
||||
|
||||
=========
|
||||
|
|
|
|||
Loading…
Reference in New Issue