python-peps/pep-0598.rst

PEP: 598
Title: Introducing incremental feature releases
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 15-Jun-2019
Python-Version: 3.9


Abstract
========

PEP 596 has proposed reducing the feature delivery latency for the Python
standard library and CPython reference interpreter by increasing the frequency
of major CPython releases from every 18-24 months to instead occur every 9-12
months.

Adopting such an approach has several significant practical downsides, as a
CPython major release carries certain expectations (most notably, a 5 year
maintenance lifecycle, support for parallel installation with the previous
major release, and the possibility of breaking chnages to the CPython-specific
ABI, requiring recompilation of all extension modules) that mean faster major
releases would significantly increase the burden of maintaining 3rd party
Python libraries and applications across all actively supported CPython
releases.

This PEP presents a competing proposal to instead *slow down* the frequency of
parallel installable major feature releases that change the filesystem layout
and CPython ABI to a consistent 24 month cycle, but to compensate for this by
introducing the notion of build compatible incremental feature releases, and
then deferring the full feature freeze of a given major release series from the
initial baseline X.Y.0 release to a subsequent X.Y.Z feature complete release
that occurs ~12 months after the initial baseline feature release.

A new ``feature_complete`` attribute on the ``sys.version_info`` structure will
provide a programmatic indicator as to whether or not a release series remains
open to further incremental feature releases. Alternate implementations of
Python would also be free to clear this flag to indicate that their support for
their nominal Python version may still be a work in progress.

The existing cycle and the new cycle would be synchronised on their feature
freeze release dates, so the full Python 3.9.x feature freeze would occur in
October 2021, 24 months after the Python 3.8.0 feature release, with the initial
Python 3.9.0 release taking place in October 2020.


Example Future Release Schedules
================================

Under this proposal, Python 3.9.0a1 would be released in November 2019, shortly
after the Python 3.8.0 feature freeze release in October 2019.

The 3.9.0b1 release would then follow 6 months later in May 2020, with 3.9.0
itself being released in October 2020.

Assuming minor releases of 3.9.x were to occur quarterly, then the overall
release timeline would look like:

* 2019-11: 3.9.0a1
* ... additional alpha releases as determined by the release manager
* 2020-05: 3.9.0b1
* ... additional beta releases as determined by the release manager
* 2020-08: 3.9.0bX (final beta release that locks ABI compatibility)
* 2020-09: 3.9.0rc1
* ... additional release candidates as determined by the release manager
* 2020-10: 3.9.0 (BFR - baseline feature release)
* 2021-01: 3.9.1 (IFR - incremental feature release)
* 2021-04: 3.9.2 (IFR)
* 2021-07: 3.9.3 (IFR)
* 2021-10: 3.9.4 (feature complete release)
* 2022-01: 3.9.5
* 2022-04: 3.9.6
* 2022-07: 3.9.7
* 2022-10: 3.9.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2025-10: 3.9.x branch closed

Feature complete release numbers would typically be written without any
qualifier (as they are today), while the baseline and incremental feature
releases would be expected to have a qualifier appended indicating that they
aren't a traditional CPython release (``3.9.0 (BFR)``, ``3.9.1 (IFR)``, etc).

The Python 3.10 release series would start being published the month after the
first Python 3.9 feature complete release, in parallel with the final 12 months
of routine Python 3.9 maintenance releases:

* 2021-11: 3.10.0a1
* ... additional alpha releases as determined by the release manager
* 2022-05: 3.10.0b1
* ... additional beta releases as determined by the release manager
* 2022-08: 3.10.0bX (final beta release that locks ABI compatibility)
* 2022-09: 3.10.0rc1
* ... additional release candidates as determined by the release manager
* 2022-10: 3.10.0 (BFR)
* 2023-01: 3.10.1 (IFR)
* 2023-04: 3.10.2 (IFR)
* 2023-07: 3.10.3 (IFR)
* 2023-10: 3.10.4
* 2024-01: 3.10.5
* 2024-04: 3.10.6
* 2024-07: 3.10.7
* 2024-10: 3.10.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2027-10: 3.10.x branch closed

In this model, there are always two or three active branches:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance
* 2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
* 2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
* 2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance
* 2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
* ... etc

(Pre-alpha and pre-beta development occurs on the main git branch, all other
development occurs on a release specific branch with changes typically
backported from the main git branch)

TODO: this really needs a diagram to help explain it, so I'll add a picture
once I have one to add.

This is quite similar to the status quo, but with a more consistent cadence,
alternating between major feature release years (2020, 2022, etc) that focus
on the alpha and beta cycle for a new baseline feature release (while continuing
to publish maintenance releases for the previous major release series), and
incremental feature release years (2021, 2023, etc), that focus on making
smaller improvements to the current major release series (while making plans
for the next major release series the following year).


Proposal
========

Excluding alpha and beta releases, CPython currently has 3 different kinds
of release increment:

* Feature release (i.e. X.Y.0 releases)
* Maintenance release (X.Y.Z releases within ~2 years of X.Y.0)
* Source-only security release (subsequent X.Y.Z releases)

Feature freeze takes place at the time of the X.Y.0b1 release.
Build compatibility freeze now takes place at the time of the last beta release
(providing time for projects to upload wheel archives to PyPI prior to the
first release candidate).

This then creates the following periods in the lifecycle of a release series:

* Pre-beta (release series is the CPython development branch)
* Beta (release enters maintenance mode, ABI compatibility mostly locked)
* Maintenance (ABI locked, only bug fixes & docs enhancements accepted)
* Security fix only (no further binary releases, only security fixes accepted)
* End of life (no further releases of any kind)

The proposal in this PEP is that the "Feature release" category be split up into
three different kinds of feature release:

* Baseline feature release (X.Y.0 releases)
* Incremental feature release (any X.Y.Z releases published between a
  baseline feature release and the corresponding feature complete release)
* Feature complete release (a specific X.Y.Z release ~1 year after X.Y.0)
* Maintenance release (X.Y.Z releases within ~1 years of the feature complete release)
* Source-only security release (subsequent ``X.Y.Z`` releases)

This would then introduce a new "Feature releases" phase in the release series
lifecycle:

* Pre-beta (release series is the CPython development branch)
* Beta (release enters feature additions mode, ABI compatibility not yet locked)
* Feature releases (ABI locked, backwards compatible API additions accepted)
* Maintenance (ABI locked, only bug fixes & docs enhancements accepted)
* Security fix only (no further binary releases, only security fixes accepted)
* End of life (no further releases of any kind)

The pre-release beta period would be relaxed to use the incremental feature
release policy for changes, rather than the stricter maintenance release policy.

For governance purposes, baseline feature releases are the only releases that
would qualify as a "feature release" in the PEP 13 sense (incremental feature
releases wouldn't count).


Baseline feature releases and major release series
--------------------------------------------------

Baseline feature releases are essentially just the existing feature releases,
given a new name to help distinguish them from the new incremental feature
releases, and also to help indicate that unlike their predecessors, they are
no longer considered feature complete at release.

Baseline feature releases would continue to define a new major release series,
locking in the following language, build, and installation compatibility
constraints for the remainder of that series:

- Python language grammar
- ``ast`` module AST format
- CPython interpreter opcode format
- ``pyc`` file magic number and filename compatibility tags
- extension module filename compatibility tags
- wheel archive compatibility tags
- default package and module import directories
- default installation filename and directories

Baseline feature releases would also continue to be the only releases where:

- new deprecations, pending deprecations, and other warnings can be introduced
- existing pending deprecations can be converted to full deprecations
- existing warnings can be converted to errors
- other changes requiring "Porting to Python X.Y" entries in the What's New
  document can be introduced

Key characteristics of a major release series:

- an installation within one major release series does not conflict with
  installations of other major release series (i.e. they can be installed in parallel)
- an installation within a major release series can be updated to a later minor
  release within the same series without requiring reinstallation or any other
  changes to previously installed components

Key characteristics of a baseline feature release:

- in a baseline feature release, ``sys.version_info.feature_complete == False``
- in a baseline feature release, ``sys.version_info.minor == 0``
- baseline feature releases may contain higher risk changes to the language and
  interpreter, such as grammar modifications, major refactoring of interpreter
  and standard library internals, or potentially invasive feature additions that
  carry a risk of unintended side effects on other existing functionality
- features introduced in a baseline feature release are the *only* features
  permitted to rely on ``sys.version_info`` as their sole runtime indicator
  of the feature's availability

Key expectations around major release series and baseline feature releases:

- most public projects will only actively test against the *most recent*
  minor release within a release series
- many (most?) public projects will only add a new release series to their test
  matrix *after* the initial baseline feature release has already been published,
  which can make it difficult to resolve issues that require providing new flags
  or APIs to explicitly opt-in to old behaviour after a default behaviour changed
- private projects with known target environments will test against whichever
  minor release version they're actually using
- most private projects will also only consider migrating to a new release
  series *after* the initial baseline feature release has already been published,
  again posing a problem if the resolution of their problems requires an API
  addition


The key motivation of the proposal in this PEP is that the public and private
project behaviours described above aren't *new* expectations: they're
descriptions of the way CPython release series are already handled by the wider
community today. As such, the PEP represents an attempt to adjust our release
policies and processes to better match the way the wider community already
handles them, rather than changing our processes in a way that then means the
wider community needs to adjust to us rather than the other way around.


Incremental feature releases
----------------------------

Incremental feature releases are the key new process addition being proposed by
this PEP. They are subject to the same strict runtime compatibility requirements
as the existing maintenance releases, but would have the following more
relaxed policies around API additions and enhancements:

* new public APIs can be added to any standard library module (including builtins)
* subject to the feature detection requirement below, new optional arguments can
  be added to existing APIs (including builtins)
* new public APIs can be added to the stable C ABI (with appropriate version guards)
* new public APIs can be added to the CPython C API
* with the approval of the release manager, backwards compatible reliability
  improvements can be made to existing APIs and syntactic constructs
* with the approval of the release manager, performance improvements can be
  incorporated for existing APIs and syntactic constructs

The intent of this change in policy is to allow usability improvements for new
(and existing!) language features to be delivered in a more timely fashion,
rather than requiring users to incur the inherent delay and costs of waiting for
and then upgrading to the next major release series.

Key characteristics of an incremental feature release:

- in an incremental feature release, ``sys.version_info.feature_complete == False``
- in an incremental feature release, ``sys.version_info.minor != 0``
- all API additions made in an incremental feature release must support
  efficient runtime feature detection that doesn't rely on either
  ``sys.version_info`` or runtime code object introspection. In most cases, a
  simple ``hasattr`` check on the affected module will serve this purpose, but
  when it doesn't, an alternative approach will need to be implemented as part
  of the feature addition. Prior art in this area includes the
  ``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available``
  set, and the various ``os.supports_*`` sets that the ``os`` module already
  offers for platform dependent capability detection

Key expectations around incremental feature releases:

- "don't break existing installations on upgrade" remains a key requirement
  for all minor releases, even with the more permissive change inclusion policy
- more intrusive changes should still be deferred to the next baseline feature
  release
- public Python projects that start relying on features added in an incremental
  feature release should set their ``Python-Requires`` metadata appropriately
  (projects already do this when necessary - e.g. ``aiohttp`` specifically
  requires 3.5.3 or later due to an issue with ``asyncio.get_event_loop()``
  in earlier versions)

Some standard library modules may also impose their own restrictions on
acceptable changes in incremental feature releases (for example, new hash
algorithms should only ever be added to ``hashlib.algorithms_guaranteed`` in
a baseline feature release - incremental feature releases would only be
permitted to add algorithms to ``hashlib.algorithms_available``)


Feature complete release and subsequent maintenance releases
------------------------------------------------------------

The feature complete release for a given major release series would be
developed under the normal policy for an incremental feature release, but
would have one distinguishing feature:

- in a feature complete release, ``sys.version_info.feature_complete == True``

Any subsequent maintenance and security fix only releases would also have that
flag set, and may informally be referred to as "feature complete releases".
For release series definition purposes though, the feature complete release
is the first one that sets that flag to "True".


Motivation
==========

The motivation for change in this PEP is essentially the same as the motivation
for change in PEP 596: the current 18-24 month gap between feature releases has
a lot of undesirable consequences, especially for the standard library (see
PEP 596 for further articulation of the details).

This PEP's concern with the specific proposal in PEP 596 is that it doubles the
number of actively supported Python branches, increasing the complexity of
compatibility testing matrices for the entire Python community, increasing the
number of binary Python wheels to be uploaded to PyPI when not using the stable
ABI, and just generally having a high chance of inflicting a relatively high
level of additional cost across the entire Python ecosystem.

The view taken in this PEP is that there's an alternative approach that provides
most of the benefits of a faster major release without actually incurring the
associated costs: we can split the current X.Y.0 "feature freeze" into two
parts, such that the baseline X.Y.0 release only imposes a
"runtime compatibility freeze", and the full standard library feature freeze
is deferred until later in the release series lifecycle.


Caveats and Limitations
=======================

This proposal does NOT retroactively apply to Python 3.8 - it is being proposed
for Python 3.9 and later releases only.

Actual release dates may be adjusted up to a month earlier or later at
the discretion of the release manager, based on release team availability, and
the timing of other events (e.g. PyCon US, or the annual core development
sprints). However, part of the goal of this proposal is to provide a consistent
annual cadence for both contributors and end users, so adjustments ideally would
be rare.

This PEP does not dictate a specific cadence for minor releases within a release
series - it just specifies the rouch timelines for transitions between the
release series lifecycle phases (pre-alpha, alpha, beta, feature releases,
bug fixes, security fixes). The number of minor releases within each phase is
determined by the release manager for that series based on how frequently they
and the rest of the release team for that series are prepared to undertake the
associated work.

However, for the sake of the example timelines, the PEP assumes quarterly
minor releases (the cadence used for Python 3.6 and 3.7, splitting the
difference between the twice yearly cadence used for some historical release
series, and the monthly cadence planned for Python 3.8 and 3.9).


Design Discussion
=================

Why this proposal over simply doing more frequent major releases?
-----------------------------------------------------------------

The filesystem layout changes and other inherently incompatible changes involved
in a major version update create additional work for large sections of the
wider Python community.

Decoupling those layout changes from the Python version numbering scheme is also
something that would in and of itself involves making backwards incompatible
changes, as well as adjusting community expectations around which versions will
install over the top of each other, and which can be installed in parallel on
a single system.

We also don't have a straightforward means to communicate to the community
variations in support periods like "Only support major version X.Y until
X.Y+1 is out, but support X.Z until X.Z+2 is out".

So this PEP takes as its starting assumption that the vast majority of Python
users simply *shouldn't need to care* that we're changing our release policy,
and the only folks that should be affected are those that are eagerly waiting
for standard library improvements, and other backwards compatible interpreter
enhancements.


Duration of the feature additions period
----------------------------------------

This PEP proposes that feature additions be limited to 12 months after the
initial major feature release.

The primary motivation for that is specifically to sync up with the Ubuntu LTS
timing, such that the feature complete release for the Python 3.9.x series gets
published in October 2021, ready for inclusion in the Ubuntu 22.04 release.
(other LTS Linux distributions like RHEL, SLES, and Debian don't have a fixed
publishing cadence, so they can more easily tweak their LTS timing a bit to
align with stable versions of their inputs. Canonical deliberately haven't
given themselves that flexibility with their own release cycle).

The 12 month feature addition period then arises from splitting the time
from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x minor feature
release in 2021-10 evenly between pre-release development and subsequent
minor feature releases.

This is an area where this PEP could adopt part of the proposal in PEP 596,
by instead making that split ~9 months of pre-release development, and ~15
months of minor feature releases:

* 2019-11: 3.9.0a1
* ... additional alpha releases as determined by the release manager
* 2020-03: 3.9.0b1
* 2020-04: 3.9.0b2
* 2020-05: 3.9.0b3 (final beta release that locks ABI compatibility)
* 2020-06: 3.9.0rc1
* ... additional release candidates as determined by the release manager
* 2020-07: 3.9.0 (BFR)
* 2020-10: 3.9.1 (IFR)
* 2021-01: 3.9.2 (IFR)
* 2021-04: 3.9.3 (IFR)
* 2021-07: 3.9.4 (IFR)
* 2021-10: 3.9.5
* 2022-01: 3.9.6
* 2022-04: 3.9.7
* 2022-07: 3.9.8
* 2022-10: 3.9.9 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2025-10: 3.9.x branch closed

This approach would mean there were still always two or three active branches,
it's just that proportionally more time would be spent with a branch in the
"feature releases" phase, as compared to the "pre-alpha", "pre-beta", and
"pre-release" phases:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance
* 2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
* 2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
* 2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance
* 2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
* ... etc


Duration of the unreleased pre-alpha period
-------------------------------------------

In the baseline proposal in this PEP, the proposed timelines still include
periods where we go for 18 months without making a release from the main git
branch (e.g. 3.9.0b1 would branch off in 2020-05, and 3.10.0a1 wouldn't be
published until 2021-11). They just allow for a wider variety of changes to
be backported to the most recent maintenance branch for 12 of those months.

The variant of the proposal that moves the beta branch point earlier in the
release series lifecycle would increase that period of no direct releases to
21 months - the only period where releases were made directly from the main
branch would be during the relatively short window between the last incremental
feature release of the previous release series, and the beta branch point a
few months later.

While alternating the annual cadence between "big foundational enhancements"
and "targeted low risk API usability improvements" is a deliberate feature of
this proposal, it still seems strange to wait that long for feedback in the
event that changes *are* made shortly after the previous major version is
branched.

An alternative way of handling this would be to start publishing alpha releases
for the next major feature release during the feature addition period (similar
to the way that PEP 596 proposes to starting publishing Python 3.9.0 alpha
releases during the Python 3.8.0 release candidate period).

However, rather than setting specific timelines for that at a policy level,
it may make sense to leave that decision to invididual release managers, based
on the specific changes that are being proposed for the release they're
managing.


Why not switch directly to full semantic versioning?
----------------------------------------------------

If this were a versioning design document for a new language, it *would* use
semantic versioning: the policies described above for baseline feature releases
would be applied to X.0.0 releases, the policies for incremental feature
releases would be applied to X.Y.0 releases, and the policies for maintenance
releases would be applied to X.Y.Z releases.

The problem for Python specifically is that all the policies and properties for
parallel installation support and ABI compatibility definitions are currently
associated with the first *two* fields of the version number, and it has been
that way for the better part of thirty years.

As a result, it makes sense to split out the policy question of introducing
incremental feature releases in the first place from the technical question of
making the version numbering scheme better match the semantics of the different
release types.

If the proposal in this PEP were to be accepted by the Steering Council for
Python 3.9, then a better time to tackle that technical question would be for
the subsequent October 2022 major feature release, as there are already inherent
compatibility risks associated with the choice of either "Python 4.0" (erroneous
checks for the major version being exactly 3 rather than 3 or greater), or
"Python 3.10" (code incorrectly assuming that the minor version will always
contain exactly one decimal digit).

While the test of this PEP assumes that the release published in 2022 would be
3.10, there are complex pros and cons on both sides of that future choice, and
this PEP does arguably add a potential pro in favour of choosing the
"Python 4.0" option (with the caveat that we would also need to amend the
affected installation layout and compatibility markers to only consider the
major version number, rather than both the major and minor version).

If such a version numbering change were to be proposed and accepted, then the
example 3.10.x timeline given above would instead become the following 4.x
series timeline:

* 2021-11: 4.0.0a1
* ... additional alpha releases as determined by the release manager
* 2022-05: 4.0.0b1
* ... additional beta releases as determined by the release manager
* 2022-08: 4.0.0bX (final beta release that locks ABI compatibility)
* 2022-09: 4.0.0rc1
* ... additional release candidates as determined by the release manager
* 2022-10: 4.0.0 (BFR)
* 2023-01: 4.1.0 (IFR)
* 2023-04: 4.2.0 (IFR)
* 2023-07: 4.3.0 (IFR)
* 2023-10: 4.4.0 (IFR)
* 2024-01: 4.4.1
* 2024-04: 4.4.2
* 2024-07: 4.4.3
* 2024-10: 4.4.4 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2027-10: 4.x branch closed

And the 5 year schedule forecast would look like:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance
* 2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x maintenance
* 2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature releases, 3.9.x maintenance
* 2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance
* 2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y maintenance
* ... etc



Copyright
=========

This document has been placed in the public domain.


..
  Local Variables:
  mode: indented-text
  indent-tabs-mode: nil
  sentence-end-double-space: t
  fill-column: 80
  coding: utf-8
  End: