python-peps/pep-0427.txt

PEP: 427
Title: The Wheel Binary Package Format 0.1
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth <dholth@fastmail.fm>
Discussions-To: <distutils-sig@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Sep-2012
Post-History:


Abstract
========

This PEP describes a built-package format for Python called "wheel".

A wheel is a ZIP-format archive with a specially formatted file name
and the ``.whl`` extension.  It contains a single distribution nearly
as it would be installed according to PEP 376 with a particular
installation scheme.  A wheel file may be installed by simply
unpacking into site-packages with the standard 'unzip' tool, while
preserving enough information to spread its contents out onto their
final paths at any later time.


Note
====

This draft PEP describes version 0.1 of the "wheel" format. When the PEP
is accepted, the version will be changed to 1.0.  (The major version
is used to indicate potentially backwards-incompatible changes to the
format.)


Rationale
=========

Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv.  This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer.  The wheel
binary package format frees installers from having to know about the
build system, saves time by amortizing compile time over many
installations, and removes the need to install a build system in the
target environment.


Details
=======

Installing a wheel 'distribution-1.0.py32.none.any.whl'
-------------------------------------------------------

Wheel installation notionally consists of two phases:

- Unpack.

  a. Parse ``distribution-1.0.dist-info/WHEEL``.
  b. Check that installer is compatible with Wheel-Version.  Warn if
     minor version is greater, abort if major version is greater.
  c. If Root-Is-Purelib == 'true', unpack archive into purelib
     (site-packages).
  d. Else unpack archive into platlib (site-packages).

- Spread.

  a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
     there is data) ``distribution-1.0.data/``.
  b. Move each subtree of ``distribution-1.0.data/`` onto its
     destination path. Each subdirectory of ``distribution-1.0.data/``
     is a key into a dict of destination directories, such as
     ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
     The initially supported paths are taken from
     ``distutils.command.install``.
  c. If applicable, update scripts starting with ``#!python`` to point
     to the correct interpreter.
  d. Update ``distribution-1.0.dist.info/RECORD`` with the installed
     paths.
  e. Remove empty ``distribution-1.0.data`` directory.
  f. Compile any installed .py to .pyc. (Uninstallers should be smart
     enough to remove .pyc even if it is not mentioned in RECORD.)

Recommended installer features
''''''''''''''''''''''''''''''

Rewrite ``#!python``.
    In wheel, scripts are packaged in
    ``{distribution}-{version}.data/scripts/``.  If the first line of
    a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
    point to the correct interpreter.  Unix installers may need to add
    the +x bit to these files if the archive was created on Windows.

Generate script wrappers.
    In wheel, scripts packaged on Unix systems will certainly not have
    accompanying .exe wrappers.  Windows installers may want to add them
    during install.


File Format
-----------

File name convention
''''''''''''''''''''

The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.

distribution
    Distribution name, e.g. 'django', 'pyramid'.

version
    PEP-386 compliant version, e.g. 1.0.

build tag
    Optional build number.  Must start with a digit.  A tie breaker if
    two wheels have the same version.  Sort as None if unspecified,
    else sort the initial digits as a number, and the remainder
    lexicographically.

language implementation and version tag
    E.g. 'py27', 'py2', 'py3'.

abi tag
    E.g. 'cp33m', 'abi3', 'none'.

platform tag
    E.g. 'linux_x86_64', 'any'.

For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.

The last three components of the filename before the extension are
called "compatibility tags."  The compatibility tags express the
package's basic interpreter requirements and are detailed in PEP 425.


File contents
'''''''''''''

The conents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. ``beaglevote`` and {version} is replaced with
its version, e.g. ``1.0.0``, consist of:

#. ``/``, the root of the archive, contains all files to be installed in
   ``purelib`` or ``platlib`` as specified in ``WHEEL``.  ``purelib`` and
   ``platlib`` are usually both ``site-packages``.
#. ``{distribution}-{version}.dist-info/`` contains metadata.
#. ``{distribution}-{version}.data/`` contains one subdirectory
   for each non-empty install scheme key not already covered, where
   the subdirectory name is an index into a dictionary of install paths
   (e.g. ``data``, ``scripts``, ``include``, ``purelib`, ``platlib``).
#. Python scripts must appear in ``scripts`` and begin with exactly
   ``b'#!python'`` in order to enjoy script wrapper generation and
   ``#!python`` rewriting at install time.  They may have any or no
   extension.
#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.3
   (PEP 426) or greater format metadata.
#. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive
   itself::

       Wheel-Version: 0.1
       Generator: bdist_wheel 0.7
       Root-Is-Purelib: true

#. Wheel-Version is the version number of the Wheel specification.
   Generator is the name and optionally the version of the software
   that produced the archive. Root-Is-Purelib is true if the top level
   directory of the archive should be installed into purelib;
   otherwise the root should be installed into platlib.
#. A wheel installer should warn if Wheel-Version is greater than the
   version it supports, and must fail if Wheel-Version has a greater
   major version than the version it supports.
#. Wheel, being an installation format that is intended to work across
   multiple versions of Python, does not generally include .pyc files.
#. Wheel does not contain setup.py or setup.cfg.

This version of the wheel specification is based on the distutils install
schemes and does not define how to install files to other locations.
The layout offers a superset of the functionality provided by the existing
wininst and egg binary formats.


The .dist-info directory
^^^^^^^^^^^^^^^^^^^^^^^^

#. Wheel .dist-info directories include at a minimum METADATA, WHEEL,
   and RECORD.
#. METADATA is the PEP 426 metadata (Metadata version 1.3 or greater)
#. WHEEL is the wheel metadata, specific to a build of the package.
#. RECORD is a list of (almost) all the files in the wheel and their
   secure hashes.  Unlike PEP 376, every file except RECORD, which
   cannot contain a hash of itself, must include its hash.  The hash
   algorithm must be sha256 or better; specifically, md5 and sha1 are
   not permitted, as signed wheel files rely on the strong hashes in
   RECORD to validate the integrity of the archive.
#. INSTALLER and REQUESTED are not included in the archive.
#. RECORD.jws is used for digital signatures.  It is not mentioned in
   RECORD.
#. RECORD.p7s is allowed as a courtesy to anyone who would prefer to
   use s/mime signatures to secure their wheel files.  It is not
   mentioned in RECORD and it is ignored by the official tools.
#. During extraction, wheel installers verify all the hashes in RECORD
   against the file contents.  Apart from RECORD and its signatures,
   installation will fail if any file in the archive is not both
   mentioned and correctly hashed in RECORD.


The .data directory
^^^^^^^^^^^^^^^^^^^

Any file that is not normally installed inside site-packages goes into
the .data directory, named as the .dist-info directory but with the
.data/ extension::

    distribution-1.0.dist-info/

    distribution-1.0.data/

The .data directory contains subdirectories with the scripts, headers,
documentation and so forth from the distribution.  During installation the
contents of these subdirectories are moved onto their destination paths.


Signed wheel files
------------------

Wheel files include an extended RECORD that enables digital
signatures.  PEP 376's RECORD is altered to include
``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64
encoding with no trailing = characters) as the second column instead
of an md5sum.  All possible entries are hashed, including any
generated files such as .pyc files, but not RECORD. For example::

    file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144
    distribution-1.0.dist-info/RECORD,,

RECORD.jws is not mentioned in RECORD at all.  Every other file in the
archive must have a correct hash in RECORD, or the installation will
fail.

The signature format is derived from the JSON Web Signatures (JWS)
specification.  One or more JSON Web Signature JSON Serialization (JWS-JS)
signatures may be stored in a file RECORD.jws adjacent to RECORD.

A signature-aware installer could be instructed to check for a
particular Ed25519 public key by using an extended "extras" syntax.::

    # request a normal optional feature "extra", and indicate
    # the package should be signed by a particular 
    # urlsafe-b64encode-nopad encoded ed25519 public key:
    package[extra, ed25519=ouBJlTJJ4SJXoy8Bi1KRlewWLU6JW7HUXTgvU1YRuiA]

An application could distribute a requires.txt file with many such lines
for all its dependencies and their public keys.  By installing from this
file an application's users can know they are getting packages from the
same publishers.

Applications that wish to "fail open" for backwards compatibility with
non-signature-aware installers should specify that their package
provides the extra ``ed25519=(key)`` with no associated dependencies.


JSON Web Signatures Extensions
''''''''''''''''''''''''''''''

The Ed25519 algorithm is used as an extension to the JSON Web Signatures
specification.  Wheel uses ``alg="Ed25519"`` in the header.  The key
attribute holds the signature's public JSON Web Key.  In JSON Web Key
/ JSON Private Key the Ed25519 verifying (public) key is called vk and
the signing (private) key is called sk.

Example header::

    {
      "alg": "Ed25519",
      "jwk": {
        "alg": "Ed25519",
        "vk": "tmAYCrSfj8gtJ10v3VkvW7jOndKmQIYE12hgnFu3cvk"
      }
    }

Example payload, always the SHA-256 hash of RECORD::

    { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }

A future version of wheel may include timestamps in the payload or in
the signature.

See

- http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html
- http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html
- http://self-issued.info/docs/draft-ietf-jose-json-web-key.html
- http://self-issued.info/docs/draft-jones-jose-json-private-key.html
- http://ed25519.cr.yp.to/


Comparison to .egg
------------------

#. Wheel is an installation format; egg is importable.  Wheel archives
   do not need to include .pyc and are less tied to a specific Python
   version or implementation. Wheel can install (pure Python) packages
   built with previous versions of Python so you don't always have to
   wait for the packager to catch up.
#. Wheel uses .dist-info directories; egg uses .egg-info.  Wheel is
   compatible with the new world of Python packaging and the new
   concepts it brings.
#. Wheel has a richer file naming convention for today's
   multi-implementation world.  A single wheel archive can indicate
   its compatibility with a number of Python language versions and
   implementations, ABIs, and system architectures.  Historically the
   ABI has been specific to a CPython release, wheel is ready for the
   stable ABI.
#. Wheel is lossless.  The first wheel implementation bdist_wheel
   always generates egg-info, and then converts it to a .whl.  It is
   also possible to convert existing eggs and bdist_wininst
   distributions.
#. Wheel is versioned.  Every wheel file contains the version of the
   wheel specification and the implementation that packaged it.
   Hopefully the next migration can simply be to Wheel 2.0.
#. Wheel is a reference to the other Python.


FAQ
===

Wheel defines a .data directory.  Should I put all my data there?
    This specification does not have an opinion on how you should organize
    your code.  The .data directory is just a place for any files that are
    not normally installed inside ``site-packages`` or on the PYTHONPATH.
    In other words, you may continue to use ``pkgutil.get_data(package,
    resource)`` even though *those* files will usually not be distributed
    in *wheel's* ``.data`` directory.

Why are you using Ed25519 and JWS instead of PGP, S/MIME, or ECDSA?
    Wheel's signing scheme is designed to protect against cryptography
    that is not used.  The system yields a tiny, performant pure-Python
    implementation that can just be included with the reference installer.
    The 32-byte public keys are convienent to share directly in the
    same way you would share a SHA-256 digest.  Since the signatures
    are inside the archive itself, they are more likely to be present
    at install time compared to detached signatures.

    Wheel's signing system is designed to be used more like an md5 sum
    or a secure hash used to verify the integrity of an archive than
    something like PGP or X.509 signatures.  A secure hash can verify
    the integrity of a single archive, but a wheel signing key verifies
    the signer of all packages signed with that key.  Once you know to
    expect a particular signing key, a signature-verifying installer
    protects you from installing anything but intact packages from the
    expected signers.  It makes no difference whether the wrong packages
    come from choosing the wrong package index, disk corruption, or an
    actual attack; if a package is not signed with the expected key,
    with its file contents matching their hashes in RECORD, then it will
    not be installed.

Appendix
========

Example urlsafe-base64-nopad implementation::
   
    # urlsafe-base64-nopad for Python 3
    import base64

    def urlsafe_b64encode_nopad(data):
        return base64.urlsafe_b64encode(data).rstrip(b'=')

    def urlsafe_b64decode_nopad(data):
        pad = b'=' * (4 - (len(data) & 3))
        return base64.urlsafe_b64decode(data + pad)


Copyright
=========

This document has been placed into the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: