python-peps/pep-0440.txt

PEP: 440
Title: Version Identification and Dependency Specification
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: Distutils SIG <distutils-sig@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 18 Mar 2013
Post-History: 30 Mar 2013, 27 May 2013, 20 Jun 2013
Replaces: 386


Abstract
========

This PEP describes a scheme for identifying versions of Python software
distributions, and declaring dependencies on particular versions.

This document addresses several limitations of the previous attempt at a
standardised approach to versioning, as described in PEP 345 and PEP 386.

.. note::

   This PEP was broken out of the metadata 2.0 specification in PEP 426.

   Unlike PEP 426, the notes that remain in this document are intended as
   part of the final specification (except for this one).


Definitions
===========

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.

The following terms are to be interpreted as described in PEP 426:

* "Distributions"
* "Releases"
* "Build tools"
* "Index servers"
* "Publication tools"
* "Installation tools"
* "Automated tools"
* "Projects"


Version scheme
==============

Distributions are identified by a public version identifier which
supports all defined version comparison operations

Distributions may also define a source label, which is not used by
automated tools. Source labels are useful when a project internal
versioning scheme requires translation to create a compliant public
version identifier.

The version scheme is used both to describe the distribution version
provided by a particular distribution archive, as well as to place
constraints on the version of dependencies needed in order to build or
run the software.


Public version identifiers
--------------------------

Public version identifiers MUST comply with the following scheme::

    N[.N]+[{a|b|c|rc}N][.postN][.devN]

Public version identifiers MUST NOT include leading or trailing whitespace.

Public version identifiers MUST be unique within a given distribution.

Installation tools SHOULD ignore any public versions which do not comply with
this scheme. Installation tools MAY warn the user when non-compliant
or ambiguous versions are detected.

Public version identifiers are separated into up to four segments:

* Release segment: ``N[.N]+``
* Pre-release segment: ``{a|b|c|rc}N``
* Post-release segment: ``.postN``
* Development release segment: ``.devN``

Any given release will be a "final release", "pre-release", "post-release" or
"developmental release" as defined in the following sections.

.. note::

   Some hard to read version identifiers are permitted by this scheme in
   order to better accommodate the wide range of versioning practices
   across existing public and private Python projects.

   Accordingly, some of the versioning practices which are technically
   permitted by the PEP are strongly discouraged for new projects. Where
   this is the case, the relevant details are noted in the following
   sections.


Source labels
-------------

Source labels are text strings with minimal defined semantics.

To ensure source labels can be readily incorporated as part of file names
and URLs, and to avoid formatting inconsistences in hexadecimal hash
representations they MUST be limited to the following set of permitted
characters:

* Lowercase ASCII letters (``[a-z]``)
* ASCII digits (``[0-9]``)
* underscores (``_``)
* hyphens (``-``)
* periods (``.``)
* plus signs (``+``)

Source labels MUST start and end with an ASCII letter or digit.

Source labels MUST be unique within each project and MUST NOT match any
defined version for the project.


Final releases
--------------

A version identifier that consists solely of a release segment is
termed a "final release".

The release segment consists of one or more non-negative integer
values, separated by dots::

    N[.N]+

Final releases within a project MUST be numbered in a consistently
increasing fashion, otherwise automated tools will not be able to upgrade
them correctly.

Comparison and ordering of release segments considers the numeric value
of each component of the release segment in turn. When comparing release
segments with different numbers of components, the shorter segment is
padded out with additional zeroes as necessary.

Date based release numbers are declared to be incompatible with this scheme,
as their use is not consistent with the expected API versioning semantics
described below. Accordingly, automated tools SHOULD report an error
when encountering a leading release component greater than or equal
to ``1980``.

While any number of additional components after the first are permitted
under this scheme, the most common variants are to use two components
("major.minor") or three components ("major.minor.micro").

For example::

    0.9
    0.9.1
    0.9.2
    ...
    0.9.10
    0.9.11
    1.0
    1.0.1
    1.1
    2.0
    2.0.1

A release series is any set of final release numbers that start with a
common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all
part of the ``3.3`` release series.

.. note::

   ``X.Y`` and ``X.Y.0`` are not considered distinct release numbers, as
   the release segment comparison rules implicit expand the two component
   form to ``X.Y.0`` when comparing it to any release segment that includes
   three components.


Pre-releases
------------

Some projects use an "alpha, beta, release candidate" pre-release cycle to
support testing by their users prior to a final release.

If used as part of a project's development cycle, these pre-releases are
indicated by including a pre-release segment in the version identifier::

    X.YaN  # Alpha release
    X.YbN  # Beta release
    X.YcN  # Candidate release (alternative notation: X.YrcN)
    X.Y    # Final release

A version identifier that consists solely of a release segment and a
pre-release segment is termed a "pre-release".

The pre-release segment consists of an alphabetical identifier for the
pre-release phase, along with a non-negative integer value. Pre-releases for
a given release are ordered first by phase (alpha, beta, release candidate)
and then by the numerical component within that phase.

Installation tools MAY accept both ``c`` and ``rc`` releases for a common
release segment in order to handle some existing legacy distributions.

Installation tools SHOULD interpret all ``rc`` versions as coming after all
``c`` versions (that is, ``rc1`` indicates a later version than ``c2``).
Installation tools MAY warn the user when such ambiguous versions are
detected, or even reject them entirely.

Build tools, publication tools and index servers SHOULD disallow the creation
of both ``c`` and ``rc`` releases for a common release segment.


Post-releases
-------------

Some projects use post-releases to address minor errors in a final release
that do not affect the distributed software (for example, correcting an error
in the release notes).

If used as part of a project's development cycle, these post-releases are
indicated by including a post-release segment in the version identifier::

    X.Y.postN    # Post-release

A version identifier that includes a post-release segment without a
developmental release segment is termed a "post-release".

The post-release segment consists of the string ``.post``, followed by a
non-negative integer value. Post-releases are ordered by their
numerical component, immediately following the corresponding release,
and ahead of any subsequent release.

.. note::

   The use of post-releases to publish maintenance releases containing
   actual bug fixes is strongly discouraged. In general, it is better
   to use a longer release number and increment the final component
   for each maintenance release.

Post-releases are also permitted for pre-releases::

    X.YaN.postM  # Post-release of an alpha release
    X.YbN.postM  # Post-release of a beta release
    X.YcN.postM  # Post-release of a release candidate

.. note::

   Creating post-releases of pre-releases is strongly discouraged, as
   it makes the version identifier difficult to parse for human readers.
   In general, it is substantially clearer to simply create a new
   pre-release by incrementing the numeric component.


Developmental releases
----------------------

Some projects make regular developmental releases, and system packagers
(especially for Linux distributions) may wish to create early releases
directly from source control which do not conflict with later project
releases.

If used as part of a project's development cycle, these developmental
releases are indicated by including a developmental release segment in the
version identifier::

    X.Y.devN    # Developmental release

A version identifier that includes a developmental release segment is
termed a "developmental release".

The developmental release segment consists of the string ``.dev``,
followed by a non-negative integer value. Developmental releases are ordered
by their numerical component, immediately before the corresponding release
(and before any pre-releases with the same release segment), and following
any previous release (including any post-releases).

Developmental releases are also permitted for pre-releases and
post-releases::

    X.YaN.devM      # Developmental release of an alpha release
    X.YbN.devM      # Developmental release of a beta release
    X.YcN.devM      # Developmental release of a release candidate
    X.Y.postN.devM  # Developmental release of a post-release

.. note::

   Creating developmental releases of pre-releases is strongly
   discouraged, as it makes the version identifier difficult to parse for
   human readers. In general, it is substantially clearer to simply create
   additional pre-releases by incrementing the numeric component.

   Developmental releases of post-releases are also strongly discouraged,
   but they may be appropriate for projects which use the post-release
   notation for full maintenance releases which may include code changes.


Examples of compliant version schemes
-------------------------------------

The standard version scheme is designed to encompass a wide range of
identification practices across public and private Python projects. In
practice, a single project attempting to use the full flexibility offered
by the scheme would create a situation where human users had difficulty
figuring out the relative order of versions, even though the rules above
ensure all compliant tools will order them consistently.

The following examples illustrate a small selection of the different
approaches projects may choose to identify their releases, while still
ensuring that the "latest release" and the "latest stable release" can
be easily determined, both by human users and automated tools.

Simple "major.minor" versioning::

    0.1
    0.2
    0.3
    1.0
    1.1
    ...

Simple "major.minor.micro" versioning::

    1.1.0
    1.1.1
    1.1.2
    1.2.0
    ...

"major.minor" versioning with alpha, beta and candidate
pre-releases::

    0.9
    1.0a1
    1.0a2
    1.0b1
    1.0c1
    1.0
    1.1a1
    ...

"major.minor" versioning with developmental releases, release candidates
and post-releases for minor corrections::

    0.9
    1.0.dev1
    1.0.dev2
    1.0.dev3
    1.0.dev4
    1.0rc1
    1.0rc2
    1.0
    1.0.post1
    1.1.dev1
    ...


Summary of permitted suffixes and relative ordering
---------------------------------------------------

.. note::

   This section is intended primarily for authors of tools that
   automatically process distribution metadata, rather than developers
   of Python distributions deciding on a versioning scheme.

The release segment of version identifiers MUST be sorted in
the same order as Python's tuple sorting when the release segment is
parsed as follows::

    tuple(map(int, release_segment.split(".")))

All release segments involved in the comparison MUST be converted to a
consistent length by padding shorter segments with zeroes as needed.

Within a numeric release (``1.0``, ``2.7.3``), the following suffixes
are permitted and MUST be ordered as shown::

   .devN, aN, bN, cN, rcN, <no suffix>, .postN

Note that `rc` will always sort after `c` (regardless of the numeric
component) although they are semantically equivalent. Tools MAY
reject this case as ambiguous and remain in compliance with the PEP.

Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate
(``1.0c1``, ``1.0rc1``), the following suffixes are permitted and MUST be
ordered as shown::

   .devN, <no suffix>, .postN

Within a post-release (``1.0.post1``), the following suffixes are permitted
and MUST be ordered as shown::

    .devN, <no suffix>

Note that ``devN`` and ``postN`` MUST always be preceded by a dot, even
when used immediately following a numeric version (e.g. ``1.0.dev456``,
``1.0.post1``).

Within a pre-release, post-release or development release segment with a
shared prefix, ordering MUST be by the value of the numeric component.

The following example covers many of the possible combinations::

    1.0.dev456
    1.0a1
    1.0a2.dev456
    1.0a12.dev456
    1.0a12
    1.0b1.dev456
    1.0b2
    1.0b2.post345.dev456
    1.0b2.post345
    1.0c1.dev456
    1.0c1
    1.0
    1.0.post456.dev34
    1.0.post456
    1.1.dev1


Version ordering across different metadata versions
---------------------------------------------------

Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not
specify a standard version identification or ordering scheme. This PEP does
not mandate any particular approach to handling such versions, but
acknowledges that the de facto standard for ordering them is
the scheme used by the ``pkg_resources`` component of ``setuptools``.

Software that automatically processes distribution metadata SHOULD attempt
to normalize non-compliant version identifiers to the standard scheme, and
ignore them if normalization fails. As any normalization scheme will be
implementation specific, this means that projects using non-compliant
version identifiers may not be handled consistently across different
tools, even when correctly publishing the earlier metadata versions.

For distributions currently using non-compliant version identifiers, these
filtering guidelines mean that it should be enough for the project to
simply switch to the use of compliant version identifiers to ensure
consistent handling by automated tools.

Distribution users may wish to explicitly remove non-compliant versions from
any private package indexes they control.

For metadata v1.2 (PEP 345), the version ordering described in this PEP
SHOULD be used in preference to the one defined in PEP 386.


Compatibility with other version schemes
----------------------------------------

Some projects may choose to use a version scheme which requires
translation in order to comply with the public version scheme defined in
this PEP. In such cases, the source label can be used to
record the project specific version as an arbitrary label, while the
translated public version is published in the version field.

This allows automated distribution tools to provide consistently correct
ordering of published releases, while still allowing developers to use
the internal versioning scheme they prefer for their projects.


Semantic versioning
~~~~~~~~~~~~~~~~~~~

`Semantic versioning`_ is a popular version identification scheme that is
more prescriptive than this PEP regarding the significance of different
elements of a release number. Even if a project chooses not to abide by
the details of semantic versioning, the scheme is worth understanding as
it covers many of the issues that can arise when depending on other
distributions, and when publishing a distribution that others rely on.

The "Major.Minor.Patch" (described in this PEP as "major.minor.micro")
aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification)
are fully compatible with the version scheme defined in this PEP, and abiding
by these aspects is encouraged.

Semantic versions containing a hyphen (pre-releases - clause 10) or a
plus sign (builds - clause 11) are *not* compatible with this PEP
and are not permitted in the public version field.

One possible mechanism to translate such semantic versioning based build
labels to compatible public versions is to use the ``.devN`` suffix to
specify the appropriate version order.

.. _Semantic versioning: http://semver.org/


DVCS based version labels
~~~~~~~~~~~~~~~~~~~~~~~~~

Many build tools integrate with distributed version control systems like
Git and Mercurial in order to add an identifying hash to the version
identifier. As hashes cannot be ordered reliably such versions are not
permitted in the public version field.

As with semantic versioning, the public ``.devN`` suffix may be used to
uniquely identify such releases for publication, while the source label is
used to record the original DVCS based version label.


Date based versions
~~~~~~~~~~~~~~~~~~~

As with other incompatible version schemes, date based versions can be
stored in the source label field. Translating them to a compliant
public version is straightforward: use a leading ``"0."`` prefix in the
public version label, with the date based version number as the remaining
components in the release segment.

This has the dual benefit of allowing subsequent migration to version
numbering based on API compatibility, as well as triggering more appropriate
version comparison semantics.


Olson database versioning
~~~~~~~~~~~~~~~~~~~~~~~~~

The ``pytz`` project inherits its versioning scheme from the corresponding
Olson timezone database versioning scheme: the year followed by a lowercase
character indicating the version of the database within that year.

This can be translated to a compliant 3-part version identifier as
``0.<year>.<serial>``, where the serial starts at zero (for the '<year>a'
release) and is incremented with each subsequent database update within the
year.

As with other translated version identifiers, the corresponding Olson
database version would be recorded in the source label field.


Version specifiers
==================

A version specifier consists of a series of version clauses, separated by
commas. For example::

   0.9, ~= 0.9, >= 1.0, != 1.3.4.*, < 2.0

The comparison operator (or lack thereof) determines the kind of version
clause:
    
* No operator: equivalent to ``~=``
* ``~=``: `Compatible release`_ clause
* ``==``: `Version matching`_ clause
* ``!=``: `Version exclusion`_ clause
* ``is``: `Direct reference`_ clause
* ``<=``, ``>=``: `Inclusive ordered comparison`_ clause
* ``<``, ``>``: `Exclusive ordered comparison`_ clause

The comma (",") is equivalent to a logical **and** operator: a candidate
version must match all given version clauses in order to match the
specifier as a whole.

Whitespace between a conditional operator and the following version
identifier is optional, as is the whitespace around the commas.

When multiple candidate versions match a version specifier, the preferred
version SHOULD be the latest version as determined by the consistent
ordering defined by the standard `Version scheme`_. Whether or not
pre-releases are considered as candidate versions SHOULD be handled as
described in `Handling of pre-releases`_.


Compatible release
------------------

A compatible release clause consists of either a version identifier without
any comparison operator or else the compatible release operator ``~=``
and a version identifier. It matches any candidate version that is expected
to be compatible with the specified version.

The specified version identifier must be in the standard format described in
`Version scheme`_.

For a given release identifier ``V.N``, the compatible release clause is
approximately equivalent to the pair of comparison clauses::

    >= V.N, == V.*

For example, the following groups of version clauses are equivalent::

    2.2
    ~= 2.2
    >= 2.2, == 2.*

    1.4.5
    ~= 1.4.5
    >= 1.4.5, == 1.4.*

If a pre-release, post-release or developmental release is named in a
compatible release clause as ``V.N.suffix``, then the suffix is ignored
when determining the required prefix match::

    2.2.post3
    ~= 2.2.post3
    >= 2.2.post3, == 2.*

    1.4.5a4
    ~= 1.4.5a4
    >= 1.4.5a4, == 1.4.*

The padding rules for release segment comparisons means that the assumed
degree of forward compatibility in a compatible release clause can be
controlled by appending additional zeroes to the version specifier::

    2.2.0
    ~= 2.2.0
    >= 2.2.0, == 2.2.*

    1.4.5.0
    ~= 1.4.5.0
    >= 1.4.5.0, == 1.4.5.*


Version matching
----------------

A version matching clause includes the version matching operator ``==``
and a version identifier.

The specified version identifier must be in the standard format described in
`Version scheme`_, but a trailing ``.*`` is permitted as described below.

By default, the version matching operator is based on a strict equality
comparison: the specified version must be exactly the same as the requested
version. The *only* substitution performed is the zero padding of the
release segment to ensure the release segments are compared with the same
length.

Whether or not strict version matching is appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when strict version matches are used
inappropriately.

Prefix matching may be requested instead of strict comparison, by appending
a trailing ``.*`` to the version identifier in the version matching clause.
This means that additional trailing segments will be ignored when
determining whether or not a version identifier matches the clause. If the
version includes only a release segment, than trailing components in the
release segment are also ignored.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::
    
    == 1.1        # Not equal, so 1.1.post1 does not match clause
    == 1.1.post1  # Equal, so 1.1.post1 matches clause
    == 1.1.*      # Same prefix, so 1.1.post1 matches clause

The use of ``==`` (without at least the wildcard suffix) when defining
dependencies for published distributions is strongly discouraged as it
greatly complicates the deployment of security fixes. The strict version
comparison operator is intended primarily for use when defining
dependencies for repeatable *deployments of applications* while using
a shared distribution index.


Version exclusion
-----------------

A version exclusion clause includes the version exclusion operator ``!=``
and a version identifier.

The allowed version identifiers and comparison semantics are the same as
those of the `Version matching`_ operator, except that the sense of any
match is inverted.

For example, given the version ``1.1.post1``, the following clauses would
match or not as shown::
    
    != 1.1        # Not equal, so 1.1.post1 matches clause
    != 1.1.post1  # Equal, so 1.1.post1 does not match clause
    != 1.1.*      # Same prefix, so 1.1.post1 does not match clause


Direct reference
----------------

A direct reference includes the direct reference operator ``is`` and
an explicit URL.

Whether or not direct references are appropriate depends on the specific
use case for the version specifier. Automated tools SHOULD at least issue
warnings and MAY reject them entirely when direct references are used
inappropriately.

Depending on the use case, some appropriate targets for a direct URL
reference are a source archive, an sdist, a wheel binary archive or a
direct reference to a tag or specific commit in an online version control
system. The exact URLs and targets supported will be automated tool
specific.

For example, a local source archive may be referenced directly::

    pip (is file:///localbuilds/pip-1.3.1.zip)

Alternatively, a prebuilt archive may also be referenced::

    pip (is file:///localbuilds/pip-1.3.1-py33-none-any.whl)


All direct URL references that do not refer to a local file URL SHOULD
specify a secure transport mechanism (such as ``https``), include an
expected hash value in the URL for verification purposes, or both. If an
insecure transport is specified without any hash information, with hash
information that the tool doesn't understand, or with a selected hash
algortihm that the tool considers too weak to trust, automated tools
SHOULD at least emit a warning and MAY refuse to rely on the URL.

Index servers MAY place additional restrictions on direct references (such
as requiring all references be to files hosted on the index server itself).

It is RECOMMENDED that only hashes which are unconditionally provided by
the latest version of the standard library's ``hashlib`` module be used
for source archive hashes. At time of writing, that list consists of
``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and
``'sha512'``.

For source archive and wheel references, an expected hash value may be
specified by including a ``<hash-algorithm>=<expected-hash>`` as part of
the URL fragment.

Version control references, the ``VCS+protocol`` scheme SHOULD be
used to identify both the version control system and the secure transport.

To support version control systems that do not support including commit or
tag references directly in the URL, that information may be appended to the
end of the URL using the ``@<tag>`` notation.


Inclusive ordered comparison
----------------------------

An inclusive ordered comparison clause includes a comparison operator and a
version identifier, and will match any version where the comparison is correct
based on the relative position of the candidate version and the specified
version given the consistent ordering defined by the standard
`Version scheme`_.

The inclusive ordered comparison operators are ``<=`` and ``>=``.

As with version matching, the release segment is zero padded as necessary to
ensure the release segments are compared with the same length.


Exclusive ordered comparison
----------------------------

Exclusive ordered comparisons are similar to inclusive ordered comparisons,
except that the comparison operators are ``<`` and ``>`` and the clause
MUST be effectively interpreted as implying the prefix based version
exclusion clause ``!= V.*``.

The exclusive ordered comparison ``> V`` MUST NOT match a post-release
or maintenance release of the given version. Maintenance releases can be
permitted by using the clause ``> V.0``, while both post releases and
maintenance releases can be permitted by using the inclusive ordered
comparison ``>= V.post1``.

The exclusive ordered comparison ``< V`` MUST NOT match a pre-release of
the given version, even if acceptance of pre-releases is enabled as
described in the section below.


Handling of pre-releases
------------------------

Pre-releases of any kind, including developmental releases, are implicitly
excluded from all version specifiers, *unless* they are already present
on the system, explicitly requested by the user, or if the only available
version that satisfies the version specifier is a pre-release.

By default, dependency resolution tools SHOULD:

* accept already installed pre-releases for all version specifiers
* accept remotely available pre-releases for version specifiers where
  there is no final or post release that satisfies the version specifier
* exclude all other pre-releases from consideration

Dependency resolution tools MAY issue a warning if a pre-release is needed
to satisfy a version specifier.

Dependency resolution tools SHOULD also allow users to request the
following alternative behaviours:

* accepting pre-releases for all version specifiers
* excluding pre-releases for all version specifiers (reporting an error or
  warning if a pre-release is already installed locally, or if a
  pre-release is the only way to satisfy a particular specifier)

Dependency resolution tools MAY also allow the above behaviour to be
controlled on a per-distribution basis.

Post-releases and final releases receive no special treatment in version
specifiers - they are always included unless explicitly excluded.


Examples
--------

* ``3.1``: version 3.1 or later, but not version 4.0 or later.
* ``3.1.2``: version 3.1.2 or later, but not version 3.2.0 or later.
* ``3.1a1``: version 3.1a1 or later, but not version 4.0 or later.
* ``== 3.1``: specifically version 3.1 (or 3.1.0), excludes all pre-releases,
  post releases, developmental releases and any 3.1.x maintenance releases.
* ``== 3.1.*``: any version that starts with 3.1. Equivalent to the
  ``3.1.0`` compatible release clause.
* ``3.1.0, != 3.1.3``: version 3.1.0 or later, but not version 3.1.3 and
  not version 3.2.0 or later.


Updating the versioning specification
=====================================

The versioning specification may be updated with clarifications without
requiring a new PEP or a change to the metadata version.

Actually changing the version comparison semantics still requires a new
versioning scheme and metadata version defined in new PEPs.


Summary of differences from \PEP 386
====================================

* Moved the description of version specifiers into the versioning PEP

* Aadded the "source label" concept to better handle projects that wish to
  use a non-compliant versioning scheme internally, especially those based
  on DVCS hashes
  
* Added the "compatible release" clause

* Added the "direct reference" clause

* Added the trailing wildcard syntax for prefix based version matching
  and exclusion

* Changed the top level sort position of the ``.devN`` suffix

* Allowed single value version numbers

* Explicit exclusion of leading or trailing whitespace

* Explicit criterion for the exclusion of date based versions

* Implicitly exclude pre-releases unless they're already present or
  needed to satisfy a dependency

* Treat post releases the same way as unqualified releases

* Discuss ordering and dependencies across metadata versions

The rationale for major changes is given in the following sections.


Adding source labels
--------------------

The new source label support is intended to make it clearer that the
constraints on public version identifiers are there primarily to aid in
the creation of reliable automated dependency analysis tools. Projects
are free to use whatever versioning scheme they like internally, so long
as they are able to translate it to something the dependency analysis tools
will understand.


Changing the version scheme
---------------------------

The key change in the version scheme in this PEP relative to that in
PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
projects already using both development releases and alphas/betas/release
candidates do not want their developmental releases sorted in
between their release candidates and their final releases. There is no
rationale for using ``dev`` releases in that position rather than
merely creating additional release candidates.

The updated sort order also means the sorting of ``dev`` versions is now
consistent between the metadata standard and the pre-existing behaviour
of ``pkg_resources`` (and hence the behaviour of current installation
tools).

Making this change should make it easier for affected existing projects to
migrate to the latest version of the metadata standard.

Another change to the version scheme is to allow single number
versions, similar to those used by non-Python projects like Mozilla
Firefox, Google Chrome and the Fedora Linux distribution. This is actually
expected to be more useful for version specifiers, but it is easier to
allow it for both version specifiers and release numbers, rather than
splitting the two definitions.

The exclusion of leading and trailing whitespace was made explicit after
a couple of projects with version identifiers differing only in a
trailing ``\n`` character were found on PyPI.

The exclusion of major release numbers that look like dates was implied
by the overall text of PEP 386, but not clear in the definition of the
version scheme. This exclusion has been made clear in the definition of
the release component.

`Appendix A` shows detailed results of an analysis of PyPI distribution
version information, as collected on 19th February, 2013. This analysis
compares the behaviour of the explicitly ordered version schemes defined in
this PEP and PEP 386 with the de facto standard defined by the behaviour
of setuptools. These metrics are useful, as the intent of both PEPs is to
follow existing setuptools behaviour as closely as is feasible, while
still throwing exceptions for unorderable versions (rather than trying
to guess an appropriate order as setuptools does).

Overall, the percentage of compatible distributions improves from 97.7%
with PEP 386 to 98.7% with this PEP. While the number of projects affected
in practice was small, some of the affected projects are in widespread use
(such as Pinax and selenium). The surprising ordering discrepancy also
concerned developers and acted as an unnecessary barrier to adoption of
the new metadata standard, even for projects that weren't directly affected.

The data also shows that the pre-release sorting discrepancies are seen
only when analysing *all* versions from PyPI, rather than when analysing
public versions. This is largely due to the fact that PyPI normally reports
only the most recent version for each project (unless maintainers
explicitly configure their project to display additional versions). However,
installers that need to satisfy detailed version constraints often need
to look at all available versions, as they may need to retrieve an older
release.

Even this PEP doesn't completely eliminate the sorting differences relative
to setuptools:

* Sorts differently (after translations): 38 / 28194 (0.13 %)
* Sorts differently (no translations): 2 / 28194 (0.01 %)

The two remaining sort order discrepancies picked up by the analysis are due
to a pair of projects which have PyPI releases ending with a carriage
return, alongside releases with the same version number, only *without* the
trailing carriage return.

The sorting discrepancies after translation relate mainly to differences
in the handling of pre-releases where the standard mechanism is considered
to be an improvement. For example, the existing pkg_resources scheme will
sort "1.1beta1" *after* "1.1b2", whereas the suggested standard translation
for "1.1beta1" is "1.1b1", which sorts *before* "1.1b2". Similarly, the
pkg_resources scheme will sort "-dev-N" pre-releases differently from
"devN" pre-releases when they occur within the same release, while the
scheme in this PEP requires normalizing both representations to ".devN" and
sorting them by the numeric component.


A more opinionated description of the versioning scheme
-------------------------------------------------------

As in PEP 386, the primary focus is on codifying existing practices to make
them more amenable to automation, rather than demanding that existing
projects make non-trivial changes to their workflow. However, the
standard scheme allows significantly more flexibility than is needed
for the vast majority of simple Python packages (which often don't even
need maintenance releases - many users are happy with needing to upgrade to a
new feature release to get bug fixes).

For the benefit of novice developers, and for experienced developers
wishing to better understand the various use cases, the specification
now goes into much greater detail on the components of the defined
version scheme, including examples of how each component may be used
in practice.

The PEP also explicitly guides developers in the direction of
semantic versioning (without requiring it), and discourages the use of
several aspects of the full versioning scheme that have largely been
included in order to cover esoteric corner cases in the practices of
existing projects and in repackaging software for Linux distributions.


Describing version specifiers alongside the versioning scheme
-------------------------------------------------------------

The main reason to even have a standardised version scheme in the first place
is to make it easier to do reliable automated dependency analysis. It makes
more sense to describe the primary use case for version identifiers alongside
their definition.


Changing the interpretation of version specifiers
-------------------------------------------------

The previous interpretation of version specifiers made it very easy to
accidentally download a pre-release version of a dependency. This in
turn made it difficult for developers to publish pre-release versions
of software to the Python Package Index, as even marking the package as
hidden wasn't enough to keep automated tools from downloading it, and also
made it harder for users to obtain the test release manually through the
main PyPI web interface.

The previous interpretation also excluded post-releases from some version
specifiers for no adequately justified reason.

The updated interpretation is intended to make it difficult to accidentally
accept a pre-release version as satisfying a dependency, while still
allowing pre-release versions to be retrieved automatically when that's the
only way to satisfy a dependency.

The "some forward compatibility assumed" default version constraint is
derived from the Ruby community's "pessimistic version constraint"
operator [2]_ to allow projects to take a cautious approach to forward
compatibility promises, while still easily setting a minimum required
version for their dependencies. It is made the default behaviour rather
than needing a separate operator in order to explicitly discourage
overspecification of dependencies by library developers. The explicit
comparison operators remain available to cope with dependencies with
unreliable or non-existent backwards compatibility policies, as well
as for legitimate use cases related to deployment of integrated applications.

The optional explicit spelling of the compatible release clause (``~=``) is
inspired by the Ruby (``~>``) and PHP (``~``) equivalents. It is defined
in order to allow easier conversion to the legacy ``pkg_resources`` version
specifier format (which omits the parentheses, but requires a comparison
operator).

Further improvements are also planned to the handling of parallel
installation of multiple versions of the same library, but these will
depend on updates to the installation database definition along with
improved tools for dynamic path manipulation.

The trailing wildcard syntax to request prefix based version matching was
added to make it possible to sensibly define both compatible release clauses
and the desired pre- and post-release handling semantics for ``<`` and ``>``
ordered comparison clauses.

Direct references are added as an "escape clause" to handle messy real
world situations that don't map neatly to the standard distribution model.
This includes dependencies on unpublished software for internal use, as well
as handling the more complex compatibility issues that may arise when
wrapping third party libraries as C extensions (this is of especial concern
to the scientific community). Index servers are given a lot of freedom to
disallow them, since they're intended primarily as a tool for integrators
rather than publishers.


References
==========

The initial attempt at a standardised version scheme, along with the
justifications for needing such a standard can be found in PEP 386.

.. [1] Version compatibility analysis script:
   http://hg.python.org/peps/file/default/pep-0426/pepsort.py

.. [2] Pessimistic version constraint
   http://docs.rubygems.org/read/chapter/16


Appendix A
==========

Metadata v2.0 guidelines versus setuptools (note that this analysis was
run when this PEP was still embedded as part of PEP 426)::

    $ ./pepsort.py
    Comparing PEP 426 version sort to setuptools.

    Analysing release versions
      Compatible: 24477 / 28194 (86.82 %)
      Compatible with translation: 247 / 28194 (0.88 %)
      Compatible with filtering: 84 / 28194 (0.30 %)
      No compatible versions: 420 / 28194 (1.49 %)
      Sorts differently (after translations): 0 / 28194 (0.00 %)
      Sorts differently (no translations): 0 / 28194 (0.00 %)
      No applicable versions: 2966 / 28194 (10.52 %)

    Analysing public versions
      Compatible: 25600 / 28194 (90.80 %)
      Compatible with translation: 1505 / 28194 (5.34 %)
      Compatible with filtering: 13 / 28194 (0.05 %)
      No compatible versions: 420 / 28194 (1.49 %)
      Sorts differently (after translations): 0 / 28194 (0.00 %)
      Sorts differently (no translations): 0 / 28194 (0.00 %)
      No applicable versions: 656 / 28194 (2.33 %)

    Analysing all versions
      Compatible: 24239 / 28194 (85.97 %)
      Compatible with translation: 2833 / 28194 (10.05 %)
      Compatible with filtering: 513 / 28194 (1.82 %)
      No compatible versions: 320 / 28194 (1.13 %)
      Sorts differently (after translations): 38 / 28194 (0.13 %)
      Sorts differently (no translations): 2 / 28194 (0.01 %)
      No applicable versions: 249 / 28194 (0.88 %)

Metadata v1.2 guidelines versus setuptools::

    $ ./pepsort.py 386
    Comparing PEP 386 version sort to setuptools.

    Analysing release versions
      Compatible: 24244 / 28194 (85.99 %)
      Compatible with translation: 247 / 28194 (0.88 %)
      Compatible with filtering: 84 / 28194 (0.30 %)
      No compatible versions: 648 / 28194 (2.30 %)
      Sorts differently (after translations): 0 / 28194 (0.00 %)
      Sorts differently (no translations): 0 / 28194 (0.00 %)
      No applicable versions: 2971 / 28194 (10.54 %)

    Analysing public versions
      Compatible: 25371 / 28194 (89.99 %)
      Compatible with translation: 1507 / 28194 (5.35 %)
      Compatible with filtering: 12 / 28194 (0.04 %)
      No compatible versions: 648 / 28194 (2.30 %)
      Sorts differently (after translations): 0 / 28194 (0.00 %)
      Sorts differently (no translations): 0 / 28194 (0.00 %)
      No applicable versions: 656 / 28194 (2.33 %)

    Analysing all versions
      Compatible: 23969 / 28194 (85.01 %)
      Compatible with translation: 2789 / 28194 (9.89 %)
      Compatible with filtering: 530 / 28194 (1.88 %)
      No compatible versions: 547 / 28194 (1.94 %)
      Sorts differently (after translations): 96 / 28194 (0.34 %)
      Sorts differently (no translations): 14 / 28194 (0.05 %)
      No applicable versions: 249 / 28194 (0.88 %)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: