395 lines
22 KiB
ReStructuredText
395 lines
22 KiB
ReStructuredText
PEP: 777
|
|
Title: How to Re-invent the Wheel
|
|
Author: Ethan Smith <ethan@ethanhs.me>
|
|
Sponsor: Barry Warsaw <barry@python.org>
|
|
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
|
|
Discussions-To: https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Topic: Packaging
|
|
Created: 09-Oct-2024
|
|
Post-History: `10-Oct-2024 <https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484>`__
|
|
|
|
Abstract
|
|
========
|
|
|
|
The current :pep:`wheel 1.0 specification <427>` was written over a decade ago,
|
|
and has been extremely robust to changes in the Python packaging ecosystem.
|
|
Previous efforts to improve the wheel specification
|
|
:pep:`were deferred <491#pep-deferral>` to focus on other packaging
|
|
specifications. Meanwhile, the use of wheels has changed dramatically in the
|
|
last decade. There have been many requests for new wheel features over the
|
|
years; however, a fundamental obstacle to evolving the wheel specification has
|
|
been that there is no defined process for how to handle adding
|
|
backwards-incompatible features to wheels. Therefore, to enable other PEPs to
|
|
describe new enhancements to the wheel specification, **this PEP prescribes**
|
|
**compatibility requirements on future wheel revisions**. This PEP does *not*
|
|
specify a new wheel revision. The specification of a new wheel format
|
|
(“Wheel 2.0”) is left to a future PEP.
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Currently, wheel specification changes that require new installer behavior are backwards incompatible and require a major version increase in
|
|
the wheel metadata format. An increase of the wheel major version has yet to
|
|
happen, partially because such a change has the potential to be
|
|
catastrophically disruptive. Per
|
|
`the wheel specification <https://packaging.python.org/en/latest/specifications/binary-distribution-format/#installing-a-wheel-distribution-1-0-py32-none-any-whl>`_,
|
|
any installer that does not support the new major version must abort at install
|
|
time. This means that if the major version were to be incremented without
|
|
further planning, many users would see installation failures as older installers reject new wheels
|
|
uploaded to public package indices like the Python Package Index (PyPI). It is
|
|
critically important to carefully plan the interactions between build tools,
|
|
package indices, and package installers to avoid incompatibility issues,
|
|
especially considering the long tail of users who are slow to update their
|
|
installers.
|
|
|
|
The backward compatibility concerns have prevented valuable improvements
|
|
to the wheel file format, such as
|
|
`better compression <https://discuss.python.org/t/improving-wheel-compression-by-nesting-data-as-a-second-zip/1747>`_,
|
|
`wheel data format improvements <https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853/7>`_,
|
|
`better information about what is included in a wheel <https://discuss.python.org/t/record-the-top-level-names-of-a-wheel-in-metadata/29494>`_,
|
|
and `JSON formatted metadata in the ".dist-info" folder <https://discuss.python.org/t/is-was-there-a-goal-with-pep-566s-json-encoding-section/12324/3>`_.
|
|
|
|
This PEP describes constraints and behavior for new wheel revisions to preserve
|
|
stability for existing tools that do not support a new major version of the wheel format.
|
|
This ensures that backwards incompatible changes to the wheel specification
|
|
will only affect users and tools that are properly set up to use the newer
|
|
wheels. With a clear path for evolving the wheel specification, future PEPs
|
|
will be able to improve the wheel format without needing to re-define a
|
|
completely new compatibility story.
|
|
|
|
Specification
|
|
=============
|
|
|
|
Add Wheel-Version Metadata Field to Core Metadata
|
|
-------------------------------------------------
|
|
|
|
Currently, the :pep:`wheel 1.0 PEP <427>`, PEP 427, specifies that wheel files
|
|
must contain a ``WHEEL`` metadata file that contains the version of the wheel
|
|
specification that the file conforms to. PEP 427 stipulates that installers
|
|
MUST warn on installation of a wheel with a minor version greater than supported,
|
|
and MUST abort on installation of wheels with a major version that is greater than
|
|
what the installer supports. This ensures that users do not get invalid
|
|
installations from wheels that installers cannot properly install.
|
|
|
|
However, resolvers do not currently exclude wheels with an incompatible wheel
|
|
version. There is also currently no way for a resolver to check a wheel's
|
|
version without downloading the wheel directly. To make wheel version filtering
|
|
easy for resolvers, the wheel version **MUST** be included in the relevant
|
|
metadata file (currently ``METADATA``). This will allow resolvers to
|
|
efficiently check the wheel version using the :pep:`658` metadata API without
|
|
needing to download and inspect the ``.dist-info/WHEEL`` file.
|
|
|
|
To accomplish this, a new field, ``Wheel-Version``, will be added to the
|
|
`Core Metadata Specification <https://packaging.python.org/specifications/core-metadata/>`_.
|
|
This field is single use, and must contain the exact same version specified as
|
|
the ``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement
|
|
file defining metadata about the wheel file. If ``Wheel-Version`` is absent
|
|
from the metadata file, then tools **MUST** infer the wheel file major
|
|
version as 1.
|
|
|
|
``Wheel-Version`` **MUST NOT** be included in source distribution metadata
|
|
(``PKG-INFO``) files. If a tool encounters ``Wheel-Version`` inside of a source
|
|
distribution metadata file, it **SHOULD** raise an error.
|
|
|
|
``Wheel-Version`` **MAY** be included in the metadata file for wheels of
|
|
version 1, but for wheels of version 2 or higher, the metadata file **MUST**
|
|
include ``Wheel-Version``. This enforces that future revisions of the wheel
|
|
specification can rely on resolvers skipping incompatible wheels by checking
|
|
the ``Wheel-Version`` field. Build backends are encouraged to include
|
|
``Wheel-Version`` in all wheels that they generate, regardless of version.
|
|
|
|
Installers **SHOULD** copy the metadata file in a wheel unmodified during
|
|
installation. This prevents the need to update the ``RECORD`` file, which is
|
|
an error prone process. Tools reading installed core metadata **SHOULD NOT**
|
|
assume that the field is present, as other installation formats may omit it.
|
|
|
|
When installing a wheel, installers **MUST** take the following steps:
|
|
|
|
1. Check that the values of ``Wheel-Version`` in both the core metadata file
|
|
and wheel metadata file match. If they do not match, installers **MUST**
|
|
abort installation. Neither value takes precedence.
|
|
2. Check that the installer is compatible with ``Wheel-Version``. If
|
|
``Wheel-Version`` is absent, assume the version is 1.0. Warn if minor
|
|
version is greater, abort if major version is greater. This procedure is
|
|
identital to that in :pep:`427`.
|
|
3. Proceed with installation as specified in the
|
|
`Binary Distribution Format <https://packaging.python.org/en/latest/specifications/binary-distribution-format/#binary-distribution-format>`_
|
|
specification.
|
|
|
|
Resolver Behavior Regarding ``Wheel-Version``
|
|
---------------------------------------------
|
|
|
|
Resolvers, in the process of selecting a wheel to install, **MUST** check a
|
|
candidate wheel's ``Wheel-Version``, and ignore incompatible wheel files.
|
|
Without ignoring these files, older installers might select a wheel that uses
|
|
an unsupported wheel version for that installer, and force the installer to
|
|
abort per :pep:`427`. By skipping incompatible wheel files, users will not see
|
|
installation errors when a project adopts a new wheel major version. As already
|
|
specified in PEP 427, installers **MUST** abort if a user tries to directly
|
|
install a wheel that is incompatible. If, in the process of resolving packages
|
|
found in multiple indices, a resolver comes across two wheels of the same
|
|
distribution and version, resolvers should prioritize the wheel of the highest
|
|
compatible version.
|
|
|
|
While the above protects users from unexpected breakages, users may miss a new
|
|
release of a distribution if their installer does not support the wheel version
|
|
used in the release. Imagine in the future that a package publishes 3.0 wheel
|
|
files. Downstream users won't see that there is a new release available if
|
|
their installers only support 2.x wheels. Therefore, installers **SHOULD** emit
|
|
a warning if, in the process of resolving packages, they come across an incompatible wheel
|
|
and skip it.
|
|
|
|
First Major Version Bump Must Change File Extension
|
|
---------------------------------------------------
|
|
|
|
Unfortunately, existing resolvers do not check the compatibility of wheels
|
|
before selecting them as installation candidates. Until a majority of users
|
|
update to installers that properly check for wheel compatibility, it is unsafe
|
|
to allow publishing wheels of a new major version that existing resolvers might
|
|
select. It could take upwards of four years before the majority of users are on
|
|
updated resolvers, based on current data about PyPI installer usage (See the
|
|
:ref:`777-pypi-download-analysis`, for
|
|
details). To allow for experimentation and faster adoption of 2.0 wheels,
|
|
this PEP proposes a change to the file extension of the
|
|
wheel file format, from ``.whl`` to ``.whlx`` for all future wheel versions.
|
|
Note that ``x`` in ``whlx`` is the letter "x" and does not specify the wheel
|
|
major version. The change to extension name resolves the initial transition
|
|
issue of 2.0 wheels breaking users on existing installers that do not implement
|
|
``Wheel-Version`` checks. By using a different file extension, 2.0 wheels can
|
|
immediately be uploaded to PyPI, and users will be able to experiment with the
|
|
new features right away. Users on older installers will simply ignore these new
|
|
files.
|
|
|
|
One rejected alternative would be to keep the ``.whl`` extension, but delay the
|
|
publishing of wheel 2.0 to PyPI. For more on that, please see Rejected Ideas.
|
|
|
|
Recommended Build Backend Behavior with New Wheel Formats
|
|
---------------------------------------------------------
|
|
|
|
Build backends are recommended to generate the most compatible wheel based on
|
|
features a project uses. For example, if a wheel does not use symbolic links,
|
|
and such a feature was introduced in wheel 5.0, the build backend could
|
|
generate a wheel of version 4.0. On the other hand, some features will want to
|
|
be adopted by default. For example, if wheel 3.0 introduces better compression,
|
|
the build backend may wish to enable this feature by default to improve the
|
|
wheel size and download performance.
|
|
|
|
Limitations on Future Wheel Revisions
|
|
-------------------------------------
|
|
|
|
While it is difficult to know what future features may be planned for the wheel
|
|
format, it is important that certain compatibility promises are maintained.
|
|
|
|
Wheel files, when installed, **MUST** stay compatible with the Python standard
|
|
library's ``importlib.metadata`` for all supported CPython versions. For
|
|
example, replacing ``.dist-info/METADATA`` with a JSON formatted metadata file
|
|
MUST be a multi-major version migration with one version introducing the new
|
|
JSON file alongside the existing email header format, and another future
|
|
version removing the email header format metadata file. The version to remove
|
|
``.dist-info/METADATA`` also **MUST** be adopted only after the last CPython
|
|
release that lacked support for the new file reaches end of life. This ensures
|
|
that code using ``importlib.metadata`` will not break with wheel major version
|
|
revisions.
|
|
|
|
Wheel files **MUST** remain ZIP format files as the outer container format.
|
|
Additionally, the ``.dist-info`` metadata directory **MUST** be placed at the
|
|
root of the archive without any compression, so that unpacking the wheel file
|
|
produces a normal ``.dist-info`` directory holding any metadata for the wheel.
|
|
Future wheel revisions **MAY** modify the layout, compression, and other
|
|
attributes about non-metadata components of a wheel such as data and code. This
|
|
assures that future wheel revisions remain compatible with tools operating on
|
|
package metadata, while allowing for improvements to code storage in the wheel,
|
|
such as adopting compression.
|
|
|
|
Package tooling **MUST NOT** assume that the contents and format of the wheel
|
|
file will remain the same for future wheel major versions beyond the
|
|
limitations above about metadata folder contents and outer container format.
|
|
For example, newer wheel major versions may add or remove filename components,
|
|
such as the build tag or the platform tag. Therefore it is incumbent upon
|
|
tooling to check the metadata for the ``Wheel-Version`` before attempting to
|
|
install a wheel.
|
|
|
|
Finally, future wheel revisions **MUST NOT** use any compression formats not in
|
|
the CPython standard library of at least the latest release. Wheels generated
|
|
using any new compression format should be tagged as requiring at least the
|
|
first released version of CPython to support the new compression format,
|
|
regardless of the Python API compatibility of the code within the wheel.
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
Backwards compatibility is an incredibly important issue for evolving the wheel
|
|
format. If adopting a new wheel revision is painful for downstream users,
|
|
package creators will hesitate to adopt the new standards, and users will be
|
|
stuck with failed CI pipelines and other installation woes.
|
|
|
|
Several choices in the above specification are made so that the adoption of a
|
|
new feature is less painful. For example, today wheels of an incompatible major
|
|
version are still selected by pip as installation candidates, which causes
|
|
installer failures if a project starts publishing 2.0 wheels. To avoid this
|
|
issue, this PEP requires resolvers to filter out wheels with major versions or
|
|
features incompatible with the installer.
|
|
|
|
This PEP also defines constraints on future wheel revisions, with the goal of
|
|
maintaining compatibility with CPython, but allowing evolution of wheel
|
|
contents. Wheel revisions shouldn't cause package installations to break on
|
|
older CPython revisions, as not only would it be frustrating, it would be
|
|
incredibly hard to debug for users.
|
|
|
|
This PEP relies on resolvers being able to efficiently acquire package
|
|
metadata, usually through :pep:`658`. This might present a problem for users of
|
|
package indices that do not serve :pep:`658` metadata. However, today most
|
|
installers fall back on using HTTP range requests to efficiently acquire only
|
|
the part of a wheel needed to read the metadata, a feature most storage
|
|
providers and servers include. Furthermore, future improvements to wheels
|
|
such as compression will make up performance losses due to inspecting files
|
|
in the wheel.
|
|
|
|
The main compatibility limitation of this PEP is for projects that start
|
|
publishing solely new wheels alongside a source distribution. If a user on an
|
|
older installer tries to install the package, it will fall back to the source
|
|
distribution, because the resolver will skip all newer wheels. Users are often
|
|
poorly set up to build projects from source, so this could lead to some failed
|
|
builds users would not see otherwise. There are several approaches to resolving
|
|
this issue, such as allowing dual-publishing for the initial migration, or
|
|
marking source distributions as not intended to be built.
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
The Wheel Format is Perfect and Does not Need to be Changed
|
|
-----------------------------------------------------------
|
|
The wheel format has been around for over 10 years, and in that time, Python
|
|
packages have changed a lot. It is much more common for packages to include
|
|
Rust or C extension modules, increasing the size of packages. Better
|
|
compression, such as lzma or zstd, could save a lot of time and bandwidth for
|
|
PyPI and its users. Compatibility tags cannot express the wide variety of
|
|
hardware used to accelerate Python code today, nor encode shared library
|
|
compatibility information. In order to address these issues, evolution of the
|
|
wheel package format is necessary.
|
|
|
|
Wheel Format Changes Should be Tied to CPython Releases
|
|
-------------------------------------------------------
|
|
I do not believe that tying wheel revisions to CPython
|
|
releases is beneficial. The main benefit of doing so is to make adoption of new
|
|
wheels predictable - users with the latest CPython get the latest package
|
|
format! This choice has several issues however. First, tying the new format
|
|
to the latest CPython makes adoption much slower. Users on LTS versions of
|
|
Linux with older Python installations are free to update their pip in a virtual
|
|
environment, but cannot update the version of Python as easily. While some
|
|
changes to the wheel format must be tied to CPython changes necessarily, such
|
|
as adding new compression formats or changing the metadata format, many changes
|
|
do not need to be tied to the Python version, such as symlinks, enhanced
|
|
compatibility tags, and new formats that use existing compression formats in
|
|
the standard library. Additionally, wheels are used across multiple different
|
|
language implementations, which lag behind the CPython version. It seems unfair
|
|
to prevent their users from using a feature due to the Python version. Lastly,
|
|
while this PEP does not suggest tying the wheel version to CPython releases, a
|
|
future PEP may still do so at any time, so this choice does not need to be made
|
|
in this PEP.
|
|
|
|
Keep Using ``.whl`` as the File Extension
|
|
-----------------------------------------
|
|
While keeping the extension ``.whl`` is appealing for many reasons, it presents
|
|
several problems that are difficult to surmount. First, current installers
|
|
would still pick a new wheel and fail to install the package. Furthermore,
|
|
the file name of a wheel would not be able to change without breaking existing
|
|
installers that expect a set wheel file name format. While the current filename
|
|
specification for wheels is sufficient for current usage, the optional
|
|
build tag in the middle of the file name makes any extensions ambiguous (i.e.
|
|
``foo-0.3-py3-none-any-fancy_new_tag.whl`` would parse as the build tag being
|
|
``py3``). This limits changes to information stored in the wheel file name.
|
|
|
|
Store the Wheel Major Version in the File Extension (``.whl2``)
|
|
---------------------------------------------------------------
|
|
Storing the wheel major version in the file extension has several nice
|
|
advantages. For one, there is no need to introduce the ``Wheel-Version``
|
|
metadata field, since installers could simply filter based on file extension.
|
|
This would also allow future side-by-side packages. However, changing the
|
|
extension for wheels each major version has some downsides. First, the version
|
|
stored in the ``WHEEL`` file must match the file extension, and this would need
|
|
to be verified by installers. Additionally, many systems associate file type by
|
|
file extension (e.g. executable associations, various web caching software),
|
|
and these would need to be updated every version that is released. Furthermore,
|
|
part of the brittleness of the current wheel specification is that so much
|
|
metadata is stored in the filename. Filenames are not well suited to store
|
|
structured data. Moving away from encoding information in the filename should
|
|
be a goal of future wheel revisions.
|
|
|
|
Another possibility is to use the file extension to encode the outer container
|
|
format (i.e. a ZIP file containing ``.dist-info``) separate from the inner
|
|
wheel version. However, this could lead to confusion if the file extension and
|
|
inner ``Wheel-Version`` diverge. If an installer raises an error due to an
|
|
incompatible wheel 3.0 as obtained from the wheel metadata, some users will
|
|
be confused by the difference from the file extension ``.whl2``.
|
|
|
|
Wheel 2.0 Should Change the Outer Container Format
|
|
--------------------------------------------------
|
|
|
|
Since wheel 2.0 will change the extension of wheel files, it is the best
|
|
opportunity to modify the outer container format. Compatibility does not need
|
|
to be kept with a different file extension that tools will need to opt-in to
|
|
reading. The main use-case for a different exterior compression format would
|
|
be better compression. For example, the outer container could be changed into
|
|
a `Zstandard <https://facebook.github.io/zstd/>`_ tarfile, ``.tar.zst``, which
|
|
would decompress faster and produce smaller wheels. However, there are several
|
|
practical issues with this. First, Zstandard is not part of the Python standard
|
|
library, so pure-Python packaging tools would need to ship an extension to
|
|
unpack these wheels. This could cause some compatibility issues for several
|
|
platforms where extension modules are not easy to install. Furthermore, a
|
|
future wheel revision could always introduce a new layout of non-metadata files
|
|
that uses a ``.tar.zst`` inside the existing ZIP-based format.
|
|
|
|
Finally, it is not a good idea to change the wheel file format too much at
|
|
once. The goal of this PEP is to make evolving the specification easier, and
|
|
part of the rationale behind making wheel evolution easier is to avoid "all
|
|
at once" changes. Changing the outer file format for wheels would require
|
|
re-writing how package metadata is not only discovered, but also installed.
|
|
|
|
Why not Specify Wheel 2.0 In This PEP?
|
|
--------------------------------------
|
|
|
|
There are *many* features that could be included as part of wheel 2.0, but this
|
|
PEP does not cover them. The goal of this PEP is to define a compatibility
|
|
story for the wheel file format. Changes that do not pertain to compatibility
|
|
for wheel versions do not need to be in this PEP, and should be introducted
|
|
in follow-up PEPs defining new wheel features.
|
|
|
|
Discussion Topics
|
|
=================
|
|
|
|
Should Indices Support Dual-publishing for the First Migration?
|
|
---------------------------------------------------------------
|
|
Since ``.whl`` and ``.whlx`` will look different in file name, they could be
|
|
uploaded side-by-side to package indices like PyPI. This has some nice
|
|
benefits, like dual-support for older and newer installers, so users who can
|
|
get the latest features, while users who don't upgrade still can install the
|
|
latest version of a package.
|
|
|
|
There are many complications however. Should we allow wheel 2 uploads to
|
|
existing wheel 1-only releases? Should we put any requirements on the
|
|
side-by-side wheels, such as:
|
|
|
|
.. admonition:: Constraints on dual-published wheels
|
|
|
|
A given index may contain identical-content wheels with different wheel
|
|
versions, and installers should prefer the newest-available wheel format,
|
|
with all other factors held constant.
|
|
|
|
Should we only allow uploading both with :pep:`694` allowing "atomic"
|
|
dual-publishing?
|
|
|
|
Acknowledgements
|
|
================
|
|
|
|
The author of this PEP is greatly indebted to the incredibly valuable review,
|
|
advice, and feedback of Barry Warsaw and Michael Sarahan.
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|