PEP: 413 Title: Faster evolution of the Python Standard Library Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan Status: Draft Type: Process Content-Type: text/x-rst Created: 2012-02-24 Post-History: 2012-02-24, 2012-02-25 Resolution: TBD Abstract ======== This PEP proposes the adoption of a new date-based versioning scheme for the standard library (distinct from, but coupled to, the existing language versioning scheme) that allows accelerated releases of the Python standard library, while maintaining (or even slowing down) the current rate of change in the core language definition. Like PEP 407, it aims to adjust the current balance between measured change that allows the broader community time to adapt and being able to keep pace with external influences that evolve more rapidly than the current release cycle can handle (this problem is particularly notable for standard library elements that relate to web technologies). However, it's more conservative in its aims than PEP 407, seeking to restrict the increased pace of development to builtin and standard library interfaces, without affecting the rate of change for other elements such as the language syntax and version numbering as well as the CPython binary API and bytecode format. Rationale ========= To quote the PEP 407 abstract: Finding a release cycle for an open-source project is a delicate exercise in managing mutually contradicting constraints: developer manpower, availability of release management volunteers, ease of maintenance for users and third-party packagers, quick availability of new features (and behavioural changes), availability of bug fixes without pulling in new features or behavioural changes. The current release cycle errs on the conservative side. It is adequate for people who value stability over reactivity. This PEP is an attempt to keep the stability that has become a Python trademark, while offering a more fluid release of features, by introducing the notion of long-term support versions. I agree with the PEP 407 authors that the current release cycle of the *standard library* is too slow to effectively cope with the pace of change in some key programming areas (specifically, web protocols and related technologies, including databases, templating and serialisation formats). However, I have written this competing PEP because I believe that the approach proposed in PEP 407 of offering full, potentially binary incompatible releases of CPython every 6 months places too great a burden on the wider Python ecosystem. Under the current CPython release cycle, distributors of key binary extensions will often support Python releases even after the CPython branches enter "security fix only" mode (for example, Twisted currently ships binaries for 2.5, 2.6 and 2.7, NumPy and SciPy suport those 3 along with 3.1 and 3.2, PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit binaries for 2.6 and 2.7, etc). If CPython were to triple (or more) its rate of releases, the developers of those libraries (many of which are even more resource starved than CPython) would face an unpalatable choice: either adopt the faster release cycle themselves (up to 18 simultaneous binary releases for PyGame!), drop older Python versions more quickly, or else tell their users to stick to the CPython LTS releases (thus defeating the entire point of speeding up the CPython release cycle in the first place). Similarly, many support tools for Python (e.g. syntax highlighters) can take quite some time to catch up with language level changes. At a cultural level, the Python community is also accustomed to a certain meaning for Python version numbers - they're linked to deprecation periods, support periods, all sorts of things. PEP 407 proposes that collective knowledge all be swept aside, without offering a compelling rationale for why such a course of action is actually *necessary* (aside from, perhaps, making the lives of the CPython core developers a little easier at the expense of everyone else). But, if we go back to the primary rationale for increasing the pace of change (i.e. more timely support for web protocols and related technologies), we can note that those only require *standard library* changes. That means many (perhaps even most) of the negative effects on the wider community can be avoided by explicitly limiting which parts of CPython are affected by the new release cycle, and allowing other parts to evolve at their current, more sedate, pace. Proposal ======== This PEP proposes the addition of a new ``sys.stdlib_info`` attribute that records a date based standard library version above and beyond the underlying interpreter version:: sys.stdlib_info(year=2012, month=8, micro=0, releaselevel='final', serial=0) This information would also be included in the ``sys.version`` string:: Python 3.3.0 (12.08.0, default:c1a07c8092f7+, Feb 17 2012, 23:03:41) [GCC 4.6.1] When maintenance releases are created, *two* new versions of Python would actually be published on python.org (using the first 3.3 maintenance release, planned for February 2013 as an example):: 3.3.1 + 12.08.1 # Maintenance release 3.3.1 + 13.02.0 # Standard library release A standard library release would just be the corresponding maintenance release, with the following additional, backwards compatible changes: * new features in pure Python modules * new features in C extension modules (subject to PEP 399 compatibility requirements) * new features in language builtins (provided the C ABI remains unaffected) A further 6 months later, the next 3.3 maintenance release would again be accompanied by a new standard library release:: 3.3.2 + 12.08.2 # Maintenance release 3.3.2 + 13.08.1 # Standard library release Again, the standard library release would be binary compatible with the previous language release, merely offering additional features at the Python level. Finally, 18 months after the release of 3.3, a new language release would be made around the same time as the final 3.3 maintenance release:: 3.3.3 + 12.08.3 # Maintenance release 3.4.0 + 14.02.0 # Language release Language releases would then contain all the features that are not permitted in standard library releases: * new language syntax * new deprecation warnings * removal of previously deprecated features * changes to the emitted bytecode * changes to the AST * any other significant changes to the compilation toolchain * changes to the core eval loop * changes to the C ABI The 3.4 release cycle would then follow a similar pattern to that for 3.3:: 3.4.1 + 14.02.1 # Maintenance release 3.4.1 + 14.08.0 # Standard library release 3.4.2 + 14.02.2 # Maintenance release 3.4.2 + 15.02.0 # Standard library release 3.4.3 + 14.02.3 # Maintenance release 3.5.0 + 15.08.0 # Language release User Scenarios ============== The versioning scheme proposed above is based on a number of user scenarios that are likely to be encountered if this scheme is adopted. In each case, the scenario is described for both the status quo (i.e. slow release cycle) the versioning scheme in this PEP and the free wheeling minor version number scheme proposed in PEP 407. To give away the ending, the point of using a separate version number is that for almost all scenarios, the important number is the *language* version, not the standard library version. Most users won't even need to care that the standard library version number exists. In the two identified cases where it matters, providing it as a separate number is actually clearer and more explicit than embedding the two different kinds of number into a single sequence and then tagging some of the numbers in the unified sequence as special. Novice user, downloading Python from python.org in March 2013 ------------------------------------------------------------- **Status quo:** must choose between 3.3 and 2.7 **This PEP:** must choose between 3.3 (13.02), 3.3 (12.08) and 2.7. **PEP 407:** must choose between 3.4, 3.3 (LTS) and 2.7. **Verdict:** explaining the meaning of a Long Term Support release is about as complicated as explaining the meaning of the proposed standard library version numbers. I call this a tie. Novice user, attempting to judge currency of third party documentation ---------------------------------------------------------------------- **Status quo:** minor version differences indicate 18-24 months of language evolution **This PEP:** same as status quo for language core, or just compare the standard library version numbers to get a rough time in months. **PEP 407:** minor version differences indicate 18-24 months of language evolution up to 3.3, then 6 months of language evolution thereafter. **Verdict:** date based numbering schemes with regular release cycles are a *much* better way to give novices a rough handle on the currency of information. Since keeping the minor version implications the same is a gain for *current* Python users, I'm calling this a win twice over for the schemes in this PEP. Novice user, looking for an extension module binary release ----------------------------------------------------------- **Status quo:** look for the binary corresponding to the Python version you are running. **This PEP:** same as status quo. **PEP 407 (full releases):** same as status quo, but corresponding binary version is more likely to be missing (or, if it does exist, has to be found amongst a much larger list of alternatives). **PEP 407 (ABI updates limited to LTS releases):** all binary release pages will need to tell users that Python 3.3, 3.4 and 3.5 all need the 3.3 binary. **Verdict:** I call this a clear win for the scheme in this PEP. Absolutely nothing changes from the current situation, since the standard library version is actually irrelevant in this case (only binary extension compatibility is important). Extension module author, deciding whether or not to make a binary release ------------------------------------------------------------------------- **Status quo:** unless using the PEP 384 stable ABI, a new binary release is needed every time the minor version number changes. **This PEP:** same as status quo. **PEP 407 (full releases):** same as status quo, but becomes a far more frequent occurrence. **PEP 407 (ABI updates limited to LTS releases):** before deciding, must first look up whether the new release is an LTS release or an interim release. If it is an LTS release, then a new build is necessary. **Verdict:** I call this another clear win for the scheme in this PEP. As with the end user facing side of this problem, the standard library version is actually irrelevant in this case. Moving that information out to a separate number avoids creating unnecessary confusion. Python developer, deciding priority of eliminating a Deprecation Warning ------------------------------------------------------------------------ **Status quo:** code that triggers deprecation warnings is not guaranteed to run on a version of Python with a higher minor version number. **This PEP:** same as status quo **PEP 407:** unclear, as the PEP doesn't currently spell this out. Assuming the deprecation cycle is linked to LTS releases, then upgrading to a non-LTS release is safe but upgrading to the next LTS release may require avoiding the deprecated construct. **Verdict:** another clear win for the scheme in this PEP since, once again, the standard library version is irrelevant in this scenario. Alternative interpreter implementor, updating with new features --------------------------------------------------------------- **Status quo:** new Python versions arrive infrequently, but are a mish-mash of standard library updates and core language definition and interpreter changes. **This PEP:** standard library updates, which are easier to integrate, are made available more frequently in a form that is clearly and explicitly compatible with the previous version of the language definition. This means that, once an alternative implementation catches up to Python 3.3, they should have a much easier time incorporating standard library features as they happen (especially pure Python changes), leaving minor version number updates as the only task that requires updates to their core compilation and execution components. **PEP 407 (full releases):** same as status quo, but becomes a far more frequent occurrence. **PEP 407 (language updates limited to LTS releases):** unclear, as the PEP doesn't currently spell out a specific development strategy. Assuming a 3.3 compatibility branch is adopted (as proposed in this PEP), then the outcome would be much the same, but the version number signalling would be slightly less clear (since you would have to look up to see if a particular release was an LTS release or not). **Verdict:** while not as clear cut as some previous scenarios, I'm still calling this one in favour of the scheme in this PEP. Explicit is better than implicit, and the scheme in this PEP makes a clear split between the two different kinds of update rather than adding a separate "LTS" tag to an otherwise ordinary release number. Tagging a particular version as being special is great for communicating with version control systems and associated automated tools, but it's a lousy way to communicate information to other humans. Python developer, deciding their minimum version dependency ----------------------------------------------------------- **Status quo:** look for "version added" or "version changed" markers in the documentation, check against ``sys.version_info`` **This PEP:** look for "version added" or "version changed" markers in the documentation. If written as a bare Python version, such as "3.3", check against ``sys.version_info``. If qualified with a standard library version, such as "3.3 (13.02)", check against ``sys.stdlib_info``. **PEP 407:** same as status quo **Verdict:** the scheme in this PEP actually allows third party libraries to be more explicit about their rate of adoption of standard library features. More conservative projects will likely pin their dependency to the language version and avoid features added in the standard library releases. Faster moving projects could instead declare their dependency on a particular standard library version. However, since PEP 407 does have the advantage of preserving the status quo, I'm calling this one for PEP 407 (albeit with a slim margin). Python developers, attempting to reproduce a tracker issue ---------------------------------------------------------- **Status quo:** if not already provided, ask the reporter which version of Python they're using. This is often done by asking for the first two lines displayed by the interactive prompt or the value of ``sys.version``. **This PEP:** same as the status quo (as ``sys.version`` will be updated to also include the standard library version), but may be needed on additional occasions (where the user knew enough to state their Python version, but that proved to be insufficient to reproduce the fault). **PEP 407:** same as the status quo **Verdict:** another marginal win for PEP 407. The new standard library version *is* an extra piece of information that users may need to pass back to developers when reporting issues with Python libraries (or Python itself, on our own tracker). However, by including it in ``sys.version``, many fault reports will already include it, and it is easy to request if needed. Effects ======= Effect on development cycle --------------------------- Similar to PEP 407, this PEP will break up the delivery of new features into more discrete chunks. Instead of a whole raft of changes landing all at once in a language release, each language release will be limited to 6 months worth of standard library changes, as well as any changes associated with new syntax. If a release date slips by a month or two, the current proposal is to keep the planned standard library version number rather than updating it to reflect the actual release date. Effect on workflow ------------------ This PEP proposes the creation of a single additional branch for use in the normal workflow. After the release of 3.3, the following branches would be in use:: 2.7 # Maintenance branch, no change 3.3 # Maintenance branch, as for 3.2 3.3-compat # New branch, backwards compatible changes default # Language changes, standard library updates that depend on them When working on a new feature, developers will need to decide whether or not it is an acceptable change for a standard library release. If so, then it should be checked in on ``3.3-compat`` and then merged to ``default``. Otherwise it should be checked in directly to ``default``. The "version added" and "version changed" markers for any changes made on the ``3.3-compat`` branch would need to be flagged with both the language version and the standard library version. For example: "3.3 (13.02)". Any changes made directly on the ``default`` branch would just be flagged with "3.4" as usual. The ``3.3-compat`` branch would be closed after the 3.3+13.08 release, as the next release at that time will be a full language release and changes (including standard library changes) should be marked accordingly. Effect on bugfix cycle ---------------------- The effect on the bug fix workflow is essentially the same as that on the workflow for new features - there is one additional branch to pass through before the change reaches the ``default`` branch. If critical bugs are found in a maintenance release, then new maintenance and standard library releases will be created to resolve the problem. The micro release number will be incremented for both the language version and the standard library version. If critical bugs are found in a standard library release that do not affect the associated maintenance release, then only a new standard library release will be created and only the standard library version's micro release number will be incremented. Note that in these circumstances, the standard library release *may* include additional features, rather than just containing the bug fix. It is assumed that anyone that cares about receiving *only* bug fixes without any new features mixed in will already be relying strictly on the maintenance releases rather than using the new standard library releases. Effect on the community ----------------------- PEP 407 has this to say about the effects on the community: People who value stability can just synchronize on the LTS releases which, with the proposed figures, would give a similar support cycle (both in duration and in stability). I believe this statement is just plain wrong. Life isn't that simple. Instead, developers of third party modules and frameworks will come under pressure to support the full pace of the new release cycle with binary updates, teachers and book authors will receive complaints that they're only covering an "old" version of Python ("You're only using 3.3, the latest is 3.5!"), etc. As the minor version number starts climbing 3 times faster than it has in the past, I believe perceptions of language stability would also fall (whether such opinions were justified or not). I believe isolating the increased pace of change to the standard library, and clearly delineating it with a separate date-based version number will greatly reassure the rest of the community that no, we're not suddenly asking them to triple their own rate of development. Instead, we're merely going to ship standard library updates for the next language release in 6-monthly installments rather than delaying them all until the next language definition update, even those changes that are backwards compatible with the previously released version of Python. The community benefits listed in PEP 407 are equally applicable to this PEP, at least as far as the standard library is concerned: People who value reactivity and access to new features (without taking the risk to install alpha versions or Mercurial snapshots) would get much more value from the new release cycle than currently. People who want to contribute new features or improvements would be more motivated to do so, knowing that their contributions will be more quickly available to normal users. If the faster release cycle encourages more people to focus on contributing to the standard library rather than proposing changes to the language definition, I don't see that as a bad thing. Handling News Updates ===================== What's New? ----------- The "What's New" documents would be split out into separate documents for standard library releases and language releases. So, during the 3.3 release cycle, we would see: * What's New in Python 3.3? * What's New in Python 3.3 (13.02)? * What's New in Python 3.3 (13.08)? And then finally, we would see the next language release: * What's New in Python 3.4? For the benefit of users that ignore standard library releases, the 3.4 What's New would link back to the What's New documents for each of the standard library releases in the 3.3 series. NEWS ---- Merge conflicts on the NEWS file are already a hassle. Since this PEP proposes introduction of an additional branch into the normal workflow, resolving this becomes even more critical. While Mercurial phases may help to some degree, it would be good to eliminate the problem entirely. One suggestion from Barry Warsaw is to adopt a non-conflicting separate-files-per-change approach, similar to that used by Twisted [2_]. For this PEP, one possible layout for such an approach (to be adopted following the initial release of 3.3.0+12.8.0 that will use the current NEWS process) might look like:: Misc/ news_entries/ 3.3.1/ # Maintenance branch changes builtins/ extensions/ library/ documentation/ tests/ 3.4.0/ # default branch changes language/ # Only exists for "x.y.0" builtins/ extensions/ library/ documentation/ tests/ 13.02.0/ # 3.3 compatibility branch changes builtins/ extensions/ library/ documentation/ tests/ NEWS # Now autogenerated from news_entries Putting the version information in the directory heirarchy isn't strictly necessary (since the NEWS file generator could figure out from the version history), but does make it easier for *humans* to keep the different versions in order. Other layouts are obviously also possible (for example, having separate "3.3" and "3.4" directories to group entries for each language version and the associated maintenance and standard library releases). Option: Slowing down the language release cycle =============================================== The current release cycle is a compromise between the desire for stability in the core language definition and C extension ABI, and the desire to get new feature (most notably standard library updates) into users hands quickly. With the standard library release cycle decoupled (to some degree) from that of the core language definition, it provides an opportunity to actually *slow down* the rate of change in the language definition. The language moratorium for Python 3.2 effectively slowed that cycle down to *more than 3 years* (3.1: June 2009, 3.3: August 2012) without causing any major complaints. The NEWS file management scheme described above is actually designed to allow us the flexibility to slow down language releases at the same time as standard library releases become more frequent. As simple example, if a full two years was allowed between 3.3 and 3.4, the 3.3 release cycle would be up looking like:: 3.2.4 # Maintenance release 3.3.0 + 12.08.0 # Language release 3.3.1 + 12.08.1 # Maintenance release 3.3.1 + 13.02.0 # Standard library release 3.3.2 + 12.08.2 # Maintenance release 3.3.2 + 13.08.1 # Standard library release 3.3.3 + 12.08.3 # Maintenance release 3.3.3 + 14.02.1 # Standard library release 3.3.4 + 12.08.4 # Maintenance release 3.4.0 + 14.08.0 # Language release The elegance of the proposed NEWS entry layout is that this decision wouldn't need to be made until after the 13.08 standard library release. At that point, the ``3.3-compat`` branch could be kept open (thus adding another standard library release to the cycle), or else it could be closed, committing to the next release being a full language release. The choice between another standard library release or a full language release would then be available every 6 months after that. Future: Further increasing the pace of standard library development =================================================================== A further benefit of the scheme proposed in this PEP is that it almost *fully* decouples the language release cycle from the standard library release cycle. The standard library could be updated every 3 months, or even once a month, without having any flow on effects on the language version numbering or the perceived stability of the core language. While that pace of development isn't practical as long as the binary installer creation for Windows and Mac OS X involves several manual steps and for as long as we don't have separate "-release" trees that only receive versions that have been marked as good by the stable buildbots, it's a useful criterion to keep in mind: what if we want to make standard library releases even *faster* than every 6 months? If the practical issues were ever resolved, then the separate date-based versioning scheme in this PEP could handle it. The approach proposed in PEP 407 could not. Why isn't PEP 384 enough? ========================= PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited subset of the full C ABI that is guaranteed to remain stable. Extensions built against the stable ABI should be able to support all subsequent Python versions with the same binary. This will help new projects to avoid coupling their C extension modules too closely to a specific version of CPython. For existing modules, however, migrating to the stable ABI can involve quite a lot of work (especially for extension modules that define a lot of classes). With limited development resources available, any time spent on such a change is time that could otherwise have been spent working on features that are offer more direct benefits to end users. Why not separate out the standard library entirely? =================================================== Because it's a lot of work for next to no pay-off. CPython without the standard library is useless (the build chain won't even run, let alone the test suite). You can't create a standalone pure Python standard library, because too many "modules" are actually tightly linked in to the internal details of their respective interpreters (e.g. ``weakref``, ``gc``, ``sys``, ``inspect``, ``ast``). Creating a separate development branch that is kept compatible with the previous feature release, and making releases from that branch that are flagged with a separate date-based version number should provide most of the benefits of a separate standard library repository with only a fraction of the pain. Acknowledgements ================ Thanks go to the PEP 407 authors for starting this discussion, as well as to those authors and Larry Hastings for initial discussions of the proposal made in this PEP. References ========== .. [1] PEP 407: New release cycle and introducing long-term support versions http://www.python.org/dev/peps/pep-0407/ .. [2] Twisted's "topfiles" approach to NEWS generation http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: