iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 777: Address discussion and feedback (#4056) 

* Add Post-History

* Expand rejected ideas

* Update specification to be more formal in spec

This change updates the specification for `Wheel-Version` to include
more information about the format and tool behavior.

It also clarifies that the move to `whlx` is for all future versions of
the wheel format, and that the `x` in `whlx` is the letter x, not the
major version.

* Add note about accessing wheel metadata

* Add Discussion-To header

* Clarify metadata copying behavior

* Clarify PEP 658 discussion

* Clarify rejected idea about .whl2

* Correct post-history link

* Fix single backticks
This commit is contained in:
Ethan Smith 2024-10-15 15:48:36 -07:00 committed by GitHub
parent b0f655b890
commit 96574b5c13
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 112 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

133
peps/pep-0777.rst
View File

@ -3,11 +3,12 @@ Title: How to Re-invent the Wheel
Author: Ethan Smith <ethan@ethanhs.me>
Sponsor: Barry Warsaw <barry@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 09-Oct-2024
Post-History:
Post-History: `10-Oct-2024 <https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484>`__
Abstract
========
@ -76,22 +77,46 @@ However, resolvers do not currently exclude wheels with an incompatible wheel
version. There is also currently no way for a resolver to check a wheel's
version without downloading the wheel directly. To make wheel version filtering
easy for resolvers, the wheel version **MUST** be included in the relevant
metadata file (currently METADATA). This will allow resolvers to efficiently
check the wheel version using the :pep:`658` metadata API without needing to
download and inspect the ``.dist-info/WHEEL`` file.
metadata file (currently ``METADATA``). This will allow resolvers to
efficiently check the wheel version using the :pep:`658` metadata API without
needing to download and inspect the ``.dist-info/WHEEL`` file.
To accomplish this, a new core metadata field is introduced called
``Wheel-Version``. While this field is optional for metadata included in a
wheel of major version 1, it is a mandatory field for metadata in wheels of major
version 2 or higher. This enforces that future revisions of the wheel
To accomplish this, a new field, ``Wheel-Version``, will be added to the
`Core Metadata Specification <https://packaging.python.org/specifications/core-metadata/>`_.
This field is single use, and must contain the exact same version specified as
the ``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement
file defining metadata about the wheel file. If ``Wheel-Version`` is absent
from the metadata file, then tools **MUST** infer the wheel file major
version as 1.
``Wheel-Version`` **MUST NOT** be included in source distribution metadata
(``PKG-INFO``) files. If a tool encounters ``Wheel-Version`` inside of a source
distribution metadata file, it **SHOULD** raise an error.
``Wheel-Version`` **MAY** be included in the metadata file for wheels of
version 1, but for wheels of version 2 or higher, the metadata file **MUST**
include ``Wheel-Version``. This enforces that future revisions of the wheel
specification can rely on resolvers skipping incompatible wheels by checking
the ``Wheel-Version`` field.
the ``Wheel-Version`` field. Build backends are encouraged to include
``Wheel-Version`` in all wheels that they generate, regardless of version.
The ``Wheel-Version`` field in the metadata file shall contain the exact same entry as the
``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement file
defining metadata about the wheel file. Installers **MUST** verify that these
entries match when installing a wheel. If ``Wheel-Version`` is absent from the
metadata file, then the implied major version of the wheel is 1.
Installers **SHOULD** copy the metadata file in a wheel unmodified during
installation. This prevents the need to update the ``RECORD`` file, which is
an error prone process. Tools reading installed core metadata **SHOULD NOT**
assume that the field is present, as other installation formats may omit it.
When installing a wheel, installers **MUST** take the following steps:
1. Check that the values of ``Wheel-Version`` in both the core metadata file
and wheel metadata file match. If they do not match, installers **MUST**
abort installation. Neither value takes precedence.
2. Check that the installer is compatible with ``Wheel-Version``. If
``Wheel-Version`` is absent, assume the version is 1.0. Warn if minor
version is greater, abort if major version is greater. This procedure is
identital to that in :pep:`427`.
3. Proceed with installation as specified in the
`Binary Distribution Format <https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format>`_
specification.
Resolver Behavior Regarding ``Wheel-Version``
---------------------------------------------
@ -127,13 +152,15 @@ select. It could take upwards of four years before the majority of users are on
updated resolvers, based on current data about PyPI installer usage (See the
:ref:`777-pypi-download-analysis`, for
details). To allow for experimentation and faster adoption of 2.0 wheels,
this PEP proposes a one time change to the file extension of the
wheel file format, from ``.whl`` to ``.whlx``. This resolves the initial
transition issue of 2.0 wheels breaking users on existing installers that do
not implement ``Wheel-Version`` checks. By using a different file extension,
2.0 wheels can immediately be uploaded to PyPI, and users will be able to
experiment with the new features right away. Users on older installers will
simply ignore these new files.
this PEP proposes a change to the file extension of the
wheel file format, from ``.whl`` to ``.whlx`` for all future wheel versions.
Note that ``x`` in ``whlx`` is the letter "x" and does not specify the wheel
major version. The change to extension name resolves the initial transition
issue of 2.0 wheels breaking users on existing installers that do not implement
``Wheel-Version`` checks. By using a different file extension, 2.0 wheels can
immediately be uploaded to PyPI, and users will be able to experiment with the
new features right away. Users on older installers will simply ignore these new
files.
One rejected alternative would be to keep the ``.whl`` extension, but delay the
publishing of wheel 2.0 to PyPI. For more on that, please see Rejected Ideas.
@ -211,6 +238,15 @@ contents. Wheel revisions shouldn't cause package installations to break on
older CPython revisions, as not only would it be frustrating, it would be
incredibly hard to debug for users.
This PEP relies on resolvers being able to efficiently acquire package
metadata, usually through :pep:`658`. This might present a problem for users of
package indices that do not serve :pep:`658` metadata. However, today most
installers fall back on using HTTP range requests to efficiently acquire only
the part of a wheel needed to read the metadata, a feature most storage
providers and servers include. Furthermore, future improvements to wheels
such as compression will make up performance losses due to inspecting files
in the wheel.
The main compatibility limitation of this PEP is for projects that start
publishing solely new wheels alongside a source distribution. If a user on an
older installer tries to install the package, it will fall back to the source
@ -266,6 +302,61 @@ build tag in the middle of the file name makes any extensions ambiguous (i.e.
``foo-0.3-py3-none-any-fancy_new_tag.whl`` would parse as the build tag being
``py3``). This limits changes to information stored in the wheel file name.
Store the Wheel Major Version in the File Extension (``.whl2``)
---------------------------------------------------------------
Storing the wheel major version in the file extension has several nice
advantages. For one, there is no need to introduce the ``Wheel-Version``
metadata field, since installers could simply filter based on file extension.
This would also allow future side-by-side packages. However, changing the
extension for wheels each major version has some downsides. First, the version
stored in the ``WHEEL`` file must match the file extension, and this would need
to be verified by installers. Additionally, many systems associate file type by
file extension (e.g. executable associations, various web caching software),
and these would need to be updated every version that is released. Furthermore,
part of the brittleness of the current wheel specification is that so much
metadata is stored in the filename. Filenames are not well suited to store
structured data. Moving away from encoding information in the filename should
be a goal of future wheel revisions.
Another possibility is to use the file extension to encode the outer container
format (i.e. a ZIP file containing ``.dist-info``) separate from the inner
wheel version. However, this could lead to confusion if the file extension and
inner ``Wheel-Version`` diverge. If an installer raises an error due to an
incompatible wheel 3.0 as obtained from the wheel metadata, some users will
be confused by the difference from the file extension ``.whl2``.
Wheel 2.0 Should Change the Outer Container Format
--------------------------------------------------
Since wheel 2.0 will change the extension of wheel files, it is the best
opportunity to modify the outer container format. Compatibility does not need
to be kept with a different file extension that tools will need to opt-in to
reading. The main use-case for a different exterior compression format would
be better compression. For example, the outer container could be changed into
a `Zstandard <https://facebook.github.io/zstd/>`_ tarfile, ``.tar.zst``, which
would decompress faster and produce smaller wheels. However, there are several
practical issues with this. First, Zstandard is not part of the Python standard
library, so pure-Python packaging tools would need to ship an extension to
unpack these wheels. This could cause some compatibility issues for several
platforms where extension modules are not easy to install. Furthermore, a
future wheel revision could always introduce a new layout of non-metadata files
that uses a ``.tar.zst`` inside the existing ZIP-based format.
Finally, it is not a good idea to change the wheel file format too much at
once. The goal of this PEP is to make evolving the specification easier, and
part of the rationale behind making wheel evolution easier is to avoid "all
at once" changes. Changing the outer file format for wheels would require
re-writing how package metadata is not only discovered, but also installed.
Why not Specify Wheel 2.0 In This PEP?
--------------------------------------
There are *many* features that could be included as part of wheel 2.0, but this
PEP does not cover them. The goal of this PEP is to define a compatibility
story for the wheel file format. Changes that do not pertain to compatibility
for wheel versions do not need to be in this PEP, and should be introducted
in follow-up PEPs defining new wheel features.
Discussion Topics
=================