From 96574b5c13df0dbc4903d861f0fb01769bddbebb Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Tue, 15 Oct 2024 15:48:36 -0700 Subject: [PATCH] PEP 777: Address discussion and feedback (#4056) * Add Post-History * Expand rejected ideas * Update specification to be more formal in spec This change updates the specification for `Wheel-Version` to include more information about the format and tool behavior. It also clarifies that the move to `whlx` is for all future versions of the wheel format, and that the `x` in `whlx` is the letter x, not the major version. * Add note about accessing wheel metadata * Add Discussion-To header * Clarify metadata copying behavior * Clarify PEP 658 discussion * Clarify rejected idea about .whl2 * Correct post-history link * Fix single backticks --- peps/pep-0777.rst | 133 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 112 insertions(+), 21 deletions(-) diff --git a/peps/pep-0777.rst b/peps/pep-0777.rst index e89fe9500..fb5a95762 100644 --- a/peps/pep-0777.rst +++ b/peps/pep-0777.rst @@ -3,11 +3,12 @@ Title: How to Re-invent the Wheel Author: Ethan Smith Sponsor: Barry Warsaw PEP-Delegate: Paul Moore +Discussions-To: https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484 Status: Draft Type: Standards Track Topic: Packaging Created: 09-Oct-2024 -Post-History: +Post-History: `10-Oct-2024 `__ Abstract ======== @@ -76,22 +77,46 @@ However, resolvers do not currently exclude wheels with an incompatible wheel version. There is also currently no way for a resolver to check a wheel's version without downloading the wheel directly. To make wheel version filtering easy for resolvers, the wheel version **MUST** be included in the relevant -metadata file (currently METADATA). This will allow resolvers to efficiently -check the wheel version using the :pep:`658` metadata API without needing to -download and inspect the ``.dist-info/WHEEL`` file. +metadata file (currently ``METADATA``). This will allow resolvers to +efficiently check the wheel version using the :pep:`658` metadata API without +needing to download and inspect the ``.dist-info/WHEEL`` file. -To accomplish this, a new core metadata field is introduced called -``Wheel-Version``. While this field is optional for metadata included in a -wheel of major version 1, it is a mandatory field for metadata in wheels of major -version 2 or higher. This enforces that future revisions of the wheel +To accomplish this, a new field, ``Wheel-Version``, will be added to the +`Core Metadata Specification `_. +This field is single use, and must contain the exact same version specified as +the ``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement +file defining metadata about the wheel file. If ``Wheel-Version`` is absent +from the metadata file, then tools **MUST** infer the wheel file major +version as 1. + +``Wheel-Version`` **MUST NOT** be included in source distribution metadata +(``PKG-INFO``) files. If a tool encounters ``Wheel-Version`` inside of a source +distribution metadata file, it **SHOULD** raise an error. + +``Wheel-Version`` **MAY** be included in the metadata file for wheels of +version 1, but for wheels of version 2 or higher, the metadata file **MUST** +include ``Wheel-Version``. This enforces that future revisions of the wheel specification can rely on resolvers skipping incompatible wheels by checking -the ``Wheel-Version`` field. +the ``Wheel-Version`` field. Build backends are encouraged to include +``Wheel-Version`` in all wheels that they generate, regardless of version. -The ``Wheel-Version`` field in the metadata file shall contain the exact same entry as the -``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement file -defining metadata about the wheel file. Installers **MUST** verify that these -entries match when installing a wheel. If ``Wheel-Version`` is absent from the -metadata file, then the implied major version of the wheel is 1. +Installers **SHOULD** copy the metadata file in a wheel unmodified during +installation. This prevents the need to update the ``RECORD`` file, which is +an error prone process. Tools reading installed core metadata **SHOULD NOT** +assume that the field is present, as other installation formats may omit it. + +When installing a wheel, installers **MUST** take the following steps: + +1. Check that the values of ``Wheel-Version`` in both the core metadata file + and wheel metadata file match. If they do not match, installers **MUST** + abort installation. Neither value takes precedence. +2. Check that the installer is compatible with ``Wheel-Version``. If + ``Wheel-Version`` is absent, assume the version is 1.0. Warn if minor + version is greater, abort if major version is greater. This procedure is + identital to that in :pep:`427`. +3. Proceed with installation as specified in the + `Binary Distribution Format `_ + specification. Resolver Behavior Regarding ``Wheel-Version`` --------------------------------------------- @@ -127,13 +152,15 @@ select. It could take upwards of four years before the majority of users are on updated resolvers, based on current data about PyPI installer usage (See the :ref:`777-pypi-download-analysis`, for details). To allow for experimentation and faster adoption of 2.0 wheels, -this PEP proposes a one time change to the file extension of the -wheel file format, from ``.whl`` to ``.whlx``. This resolves the initial -transition issue of 2.0 wheels breaking users on existing installers that do -not implement ``Wheel-Version`` checks. By using a different file extension, -2.0 wheels can immediately be uploaded to PyPI, and users will be able to -experiment with the new features right away. Users on older installers will -simply ignore these new files. +this PEP proposes a change to the file extension of the +wheel file format, from ``.whl`` to ``.whlx`` for all future wheel versions. +Note that ``x`` in ``whlx`` is the letter "x" and does not specify the wheel +major version. The change to extension name resolves the initial transition +issue of 2.0 wheels breaking users on existing installers that do not implement +``Wheel-Version`` checks. By using a different file extension, 2.0 wheels can +immediately be uploaded to PyPI, and users will be able to experiment with the +new features right away. Users on older installers will simply ignore these new +files. One rejected alternative would be to keep the ``.whl`` extension, but delay the publishing of wheel 2.0 to PyPI. For more on that, please see Rejected Ideas. @@ -211,6 +238,15 @@ contents. Wheel revisions shouldn't cause package installations to break on older CPython revisions, as not only would it be frustrating, it would be incredibly hard to debug for users. +This PEP relies on resolvers being able to efficiently acquire package +metadata, usually through :pep:`658`. This might present a problem for users of +package indices that do not serve :pep:`658` metadata. However, today most +installers fall back on using HTTP range requests to efficiently acquire only +the part of a wheel needed to read the metadata, a feature most storage +providers and servers include. Furthermore, future improvements to wheels +such as compression will make up performance losses due to inspecting files +in the wheel. + The main compatibility limitation of this PEP is for projects that start publishing solely new wheels alongside a source distribution. If a user on an older installer tries to install the package, it will fall back to the source @@ -266,6 +302,61 @@ build tag in the middle of the file name makes any extensions ambiguous (i.e. ``foo-0.3-py3-none-any-fancy_new_tag.whl`` would parse as the build tag being ``py3``). This limits changes to information stored in the wheel file name. +Store the Wheel Major Version in the File Extension (``.whl2``) +--------------------------------------------------------------- +Storing the wheel major version in the file extension has several nice +advantages. For one, there is no need to introduce the ``Wheel-Version`` +metadata field, since installers could simply filter based on file extension. +This would also allow future side-by-side packages. However, changing the +extension for wheels each major version has some downsides. First, the version +stored in the ``WHEEL`` file must match the file extension, and this would need +to be verified by installers. Additionally, many systems associate file type by +file extension (e.g. executable associations, various web caching software), +and these would need to be updated every version that is released. Furthermore, +part of the brittleness of the current wheel specification is that so much +metadata is stored in the filename. Filenames are not well suited to store +structured data. Moving away from encoding information in the filename should +be a goal of future wheel revisions. + +Another possibility is to use the file extension to encode the outer container +format (i.e. a ZIP file containing ``.dist-info``) separate from the inner +wheel version. However, this could lead to confusion if the file extension and +inner ``Wheel-Version`` diverge. If an installer raises an error due to an +incompatible wheel 3.0 as obtained from the wheel metadata, some users will +be confused by the difference from the file extension ``.whl2``. + +Wheel 2.0 Should Change the Outer Container Format +-------------------------------------------------- + +Since wheel 2.0 will change the extension of wheel files, it is the best +opportunity to modify the outer container format. Compatibility does not need +to be kept with a different file extension that tools will need to opt-in to +reading. The main use-case for a different exterior compression format would +be better compression. For example, the outer container could be changed into +a `Zstandard `_ tarfile, ``.tar.zst``, which +would decompress faster and produce smaller wheels. However, there are several +practical issues with this. First, Zstandard is not part of the Python standard +library, so pure-Python packaging tools would need to ship an extension to +unpack these wheels. This could cause some compatibility issues for several +platforms where extension modules are not easy to install. Furthermore, a +future wheel revision could always introduce a new layout of non-metadata files +that uses a ``.tar.zst`` inside the existing ZIP-based format. + +Finally, it is not a good idea to change the wheel file format too much at +once. The goal of this PEP is to make evolving the specification easier, and +part of the rationale behind making wheel evolution easier is to avoid "all +at once" changes. Changing the outer file format for wheels would require +re-writing how package metadata is not only discovered, but also installed. + +Why not Specify Wheel 2.0 In This PEP? +-------------------------------------- + +There are *many* features that could be included as part of wheel 2.0, but this +PEP does not cover them. The goal of this PEP is to define a compatibility +story for the wheel file format. Changes that do not pertain to compatibility +for wheel versions do not need to be in this PEP, and should be introducted +in follow-up PEPs defining new wheel features. + Discussion Topics =================