Update PEP 440 based on pip proof of concept

pep-0440.txt

@@ -11,7 +11,7 @@ Type: Standards Track
 Content-Type: text/x-rst
 Created: 18 Mar 2013
 Post-History: 30 Mar 2013, 27 May 2013, 20 Jun 2013,
-              21 Dec 2013, 28 Jan 2014
+              21 Dec 2013, 28 Jan 2014, 08 Aug 2014
 Replaces: 386
 
 
@@ -24,13 +24,6 @@ distributions, and declaring dependencies on particular versions.
 This document addresses several limitations of the previous attempt at a
 standardized approach to versioning, as described in PEP 345 and PEP 386.
 
-.. note::
-
-   This PEP was broken out of the metadata 2.0 specification in PEP 426.
-
-   Unlike PEP 426, the notes that remain in this document are intended as
-   part of the final specification (except for this one).
-
 
 Definitions
 ===========
@@ -409,20 +402,22 @@ such as ``1.0+foo0100`` which is already in its normalized form.
 Pre-release separators
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Pre-releases should allow either a ``.`` or a ``-`` separator between the
+Pre-releases should allow a ``.``, ``-``, or ``_`` separator between the
 release segment and the pre-release segment. The normal form for this is
 without a separator. This allows versions such as ``1.1.a1`` or ``1.1-a1``
-which would be normalized to ``1.1a1``.
+which would be normalized to ``1.1a1``. It should also allow a seperator to
+be used between the pre-release signifier and the numeral. This allows versions
+such as ``1.0a.1`` which would be normalized to ``1.0a1``.
 
 
 Pre-release spelling
 ~~~~~~~~~~~~~~~~~~~~
 
-Pre-releases allow the additional spellings of alpha, beta, and rc for a, b,
+Pre-releases allow the additional spellings of ``alpha``, ``beta``, ``rc``,
-and c respectively. This allows versions such as ``1.1alpha1``, ``1.1beta2``,
+``pre``, and ``preview`` for ``a``, ``b``, ``c``, ``c``, and ``c`` respectively.
-or ``1.1rc3`` which normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every
+This allows versions such as ``1.1alpha1``, ``1.1beta2``, or ``1.1rc3`` which
-case the additional spelling should be considered equivalent to their normal
+normalize to ``1.1a1``, ``1.1b2``, and ``1.1c3``. In every case the additional
-forms.
+spelling should be considered equivalent to their normal forms.
 
 
 Implicit pre-release number
@@ -436,10 +431,21 @@ allows versions such as ``1.2a`` which is normalized to ``1.2a0``.
 Post release separators
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-Post releases allow either a ``.`` or a ``-`` separator as well as omitting the
+Post releases allow a ``.``,``-``, or ``_`` separator as well as omitting the
 separator all together. The normal form of this is with the ``.`` separator.
 This allows versions such as ``1.2-post2`` or ``1.2post2`` which normalize to
-``1.2.post2``.
+``1.2.post2``. Like the pre-release seperator this also allows an optional
+separator between the post release signifier and the numeral. This allows
+versions like ``1.2.post-2`` which would normalize to ``1.2.post2``.
+
+
+Post release spelling
+~~~~~~~~~~~~~~~~~~~~~
+
+Post-releases allow the additional spellings of ``rev`` and ``r``. This allows
+versions such as ``1.0-r4`` which normalizes to ``1.0.post4``. As with the
+pre-releases the additional spellings should be considered equivalent to their
+normal forms.
 
 
 Implicit post release number
@@ -450,10 +456,21 @@ to be ``0``. The normal form for this is to include the ``0`` explicitly. This
 allows versions such as ``1.2.post`` which is normalized to ``1.2.post0``.
 
 
+Implicit post releases
+~~~~~~~~~~~~~~~~~~~~~~
+
+Post releases allow omitting the ``post`` signifier all together. When using
+this form the separator MUST be ``-`` and no other form is allowed. This allows
+versions such as ``1.0-1`` to be normalized to ``1.0.post1``. This particular
+normalization MUST NOT be used in conjunction with the implicit post release
+number rule. In other words ``1.0-`` is *not* a valid version and it does *not*
+normalize to ``1.0.post0``.
+
+
 Development release separators
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Development releases allow either a ``.`` or a ``-`` separator as well as
+Development releases allow a ``.``, ``-``, or a ``_`` separator as well as
 omitting the separator all together. The normal form of this is with the ``.``
 separator. This allows versions such as ``1.2-dev2`` or ``1.2dev2`` which
 normalize to ``1.2.dev2``.
@@ -468,6 +485,33 @@ explicitly. This allows versions such as ``1.2.dev`` which is normalized to
 ``1.2.dev0``.
 
 
+Local version segments
+~~~~~~~~~~~~~~~~~~~~~~
+
+With a local version, in addition to the use of ``.`` as a separator of
+segments, the use of ``-`` and ``_`` is also acceptable. The normal form is
+using the ``.`` character. This allows versions such as ``1.0+ubuntu-1`` to be
+normalized to ``1.0+ubuntu.1``.
+
+
+Preceding v character
+~~~~~~~~~~~~~~~~~~~~~
+
+In order to support the common version notation of ``v1.0`` versions may be
+preceded by a single literal ``v`` character. This character MUST be ignored
+for all purposes and should be omitted from all normalized forms of the
+version. The same version with and without the ``v`` is considered equivalent.
+
+
+Leading and Trailing Whitespace
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Leading and trailing whitespace must be silently ignored and removed from all
+normalized forms of a version. This includes ``" "``, ``\t``, ``\n``, ``\r``,
+``\f``, and ``\v``. This allows accidental whitespace to be handled sensibly,
+such as a version like ``1.0\n`` which normalizes to ``1.0``.
+
+
 Examples of compliant version schemes
 -------------------------------------
 
@@ -1098,10 +1142,6 @@ Summary of differences from \PEP 386
 
 * Moved the description of version specifiers into the versioning PEP
 
-* Added the "source label" concept to better handle projects that wish to
-  use a non-compliant versioning scheme internally, especially those based
-  on DVCS hashes
-
 * Added the "direct reference" concept as a standard notation for direct
   references to resources (rather than each tool needing to invent its own)
 
@@ -1123,6 +1163,10 @@ Summary of differences from \PEP 386
 
 * Explicit support for date based versions
 
+* Explicit normalisation rules to improve compatibility with
+  existing version metadata on PyPI where it doesn't introduce
+  ambiguity
+
 * Implicitly exclude pre-releases unless they're already present or
   needed to satisfy a dependency
 
@@ -1133,11 +1177,10 @@ Summary of differences from \PEP 386
 The rationale for major changes is given in the following sections.
 
 
-
 Changing the version scheme
 ---------------------------
 
-The key change in the version scheme in this PEP relative to that in
+One key change in the version scheme in this PEP relative to that in
 PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
 of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
 projects already using both development releases and alphas/betas/release
@@ -1165,51 +1208,17 @@ The exclusion of leading and trailing whitespace was made explicit after
 a couple of projects with version identifiers differing only in a
 trailing ``\n`` character were found on PyPI.
 
+Various other normalisation rules were also added as described in the
+separate section on version normalisation below.
+
 `Appendix A` shows detailed results of an analysis of PyPI distribution
-version information, as collected on 19th February, 2013. This analysis
+version information, as collected on 8th August, 2014. This analysis
-compares the behavior of the explicitly ordered version schemes defined in
+compares the behavior of the explicitly ordered version scheme defined in
-this PEP and PEP 386 with the de facto standard defined by the behavior
+this PEP with the de facto standard defined by the behavior of setuptools.
-of setuptools. These metrics are useful, as the intent of both PEPs is to
+These metrics are useful, as the intent of this PEP is to follow existing
-follow existing setuptools behavior as closely as is feasible, while
+setuptools behavior as closely as is feasible, while still throwing
-still throwing exceptions for unorderable versions (rather than trying
+exceptions for unorderable versions (rather than trying to guess an
-to guess an appropriate order as setuptools does).
+appropriate order as setuptools does).
-
-Overall, the percentage of compatible distributions improves from 97.7%
-with PEP 386 to 98.7% with this PEP. While the number of projects affected
-in practice was small, some of the affected projects are in widespread use
-(such as Pinax and selenium). The surprising ordering discrepancy also
-concerned developers and acted as an unnecessary barrier to adoption of
-the new metadata standard, even for projects that weren't directly affected.
-
-The data also shows that the pre-release sorting discrepancies are seen
-only when analyzing *all* versions from PyPI, rather than when analyzing
-public versions. This is largely due to the fact that PyPI normally reports
-only the most recent version for each project (unless maintainers
-explicitly configure their project to display additional versions). However,
-installers that need to satisfy detailed version constraints often need
-to look at all available versions, as they may need to retrieve an older
-release.
-
-Even this PEP doesn't completely eliminate the sorting differences relative
-to setuptools:
-
-* Sorts differently (after translations): 38 / 28194 (0.13 %)
-* Sorts differently (no translations): 2 / 28194 (0.01 %)
-
-The two remaining sort order discrepancies picked up by the analysis are due
-to a pair of projects which have PyPI releases ending with a carriage
-return, alongside releases with the same version number, only *without* the
-trailing carriage return.
-
-The sorting discrepancies after translation relate mainly to differences
-in the handling of pre-releases where the standard mechanism is considered
-to be an improvement. For example, the existing pkg_resources scheme will
-sort "1.1beta1" *after* "1.1b2", whereas the suggested standard translation
-for "1.1beta1" is "1.1b1", which sorts *before* "1.1b2". Similarly, the
-pkg_resources scheme will sort "-dev-N" pre-releases differently from
-"devN" pre-releases when they occur within the same release, while the
-scheme in this PEP requires normalizing both representations to ".devN" and
-sorting them by the numeric component.
 
 
 A more opinionated description of the versioning scheme
@@ -1381,7 +1390,42 @@ It was chosen instead of the hyphen to prevent
 ``pkg_resources.parse_version`` from parsing it as a prerelease, which is
 important for enabling a successful migration to the new, more structured,
 versioning scheme. The plus was chosen instead of a tilde because of the
-significance of the tilde in Debian's version algorithm.
+significance of the tilde in Debian's version ordering algorithm.
+
+
+Providing explicit version normalization rules
+----------------------------------------------
+
+Historically, the de facto standard for parsing versions in Python has been the
+``pkg_resources.parse_version`` command from the setuptools project. This does
+not attempt to reject *any* version and instead tries to make something
+meaningful, with varying levels of success, out of whatever it is given. It has
+a few simple rules but otherwise it more or less relies largely on string
+comparison.
+
+The normalization rules provided in this PEP exist primarily to either increase
+the compatability with ``pkg_resources.parse_version``, particularly in
+documented use cases such as ``rev``, ``r``, ``pre``, etc or to do something
+more reasonable with versions that already exist on PyPI.
+
+All possible normalization rules were weighed against whether or not they were
+*likely* to cause any ambiguity (e.g. while someone might devise a scheme where
+``v1.0`` and ``1.0`` are considered distinct releases, the likelihood of anyone
+actually doing that, much less on any scale that is noticeable, is fairly low).
+They were also weighed against how ``pkg_resources.parse_version`` treated a
+particular version string, especially with regards to how it was sorted. Finally
+each rule was weighed against the kinds of additional versions it allowed, how
+"ugly" those versions looked, how hard there were to parse (both mentally and
+mechanically) and how much additional compatibility it would bring.
+
+The breadth of possible normalizations were kept to things that could easily
+be implemented as part of the parsing of the version and not pre-parsing
+transformations applied to the versions. This was done to limit the side
+effects of each transformation as simple search and replace style transforms
+increase the likelihood of ambiguous or "junk" versions.
+
+For an extended discussion on the various types of normalizations that were
+considered, please see the proof of concept for PEP 440 within pip [4]_.
 
 
 References
@@ -1399,6 +1443,9 @@ justifications for needing such a standard can be found in PEP 386.
 .. [3] Pessimistic version constraint
    http://docs.rubygems.org/read/chapter/16
 
+.. [4] Proof of Concept: PEP 440 within pip
+    https://github.com/pypa/pip/pull/1894
+
 
 Appendix A
 ==========
@@ -1406,11 +1453,11 @@ Appendix A
 Metadata v2.0 guidelines versus setuptools::
 
     $ invoke check.pep440
-    Total Version Compatibility:              231807/239450 (96.81%)
+    Total Version Compatibility:              245806/250521 (98.12%)
-    Total Sorting Compatibility (Unfiltered): 43095/45505 (94.70%)
+    Total Sorting Compatibility (Unfiltered): 45441/47114 (96.45%)
-    Total Sorting Compatibility (Filtered):   45481/45505 (99.95%)
+    Total Sorting Compatibility (Filtered):   47057/47114 (99.88%)
-    Projects with No Compatible Versions:     802/45505 (1.76%)
+    Projects with No Compatible Versions:     498/47114 (1.06%)
-    Projects with Differing Latest Version:   1163/45505 (2.56%)
+    Projects with Differing Latest Version:   688/47114 (1.46%)
 
 
 Copyright