python-peps/pep-0598.rst

PEP: 598
Title: Introducing incremental feature releases
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 15-Jun-2019
Python-Version: 3.9


Abstract
========

PEP 596 has proposed reducing the feature delivery latency for the Python
standard library and CPython reference interpreter by increasing the frequency
of CPython feature releases from every 18-24 months to instead occur every 9-12
months.

This PEP proposes to instead *reduce* the frequency of new baseline feature
releases (with the associated filesystem layout changes, bytecode format
changes, and C ABI compatibility breaks) to occur only every other year (2020,
2022, 2024, etc), but to combine that change with a new policy and approach that
allows the introduction of backwards compatible features in the initial set of
point releases within a given release series.

In the event that this more complex proposal is *not* accepted (and either a
faster release cycle is adopted, or the status quo continues), the PEP aims to
provide useful design input for any other "Long Term Support branch"
proposals that may be put forward in the future (such as the EL Python draft at
[3_]).


Summary
=======

The proposal to keep the current CPython release compatibility management
process, but go through it more often has significant practical downsides,
as a CPython feature release carries certain expectations (most notably, a 5
year maintenance lifecycle, support for parallel installation with the previous
feature release, and the possibility of breaking changes to the CPython-specific
ABI, requiring recompilation of all extension modules) that mean faster feature
releases in their current form have the potential to significantly increase the
burden of maintaining 3rd party Python libraries and applications across all
actively supported CPython releases.

It's also arguable whether such an approach would noticeably reduce the typical
feature delivery latency in practice for most end users, as the adoption cycle
for new feature releases is typically measured in months or years, so more
frequent releases may just lead to end users updating to every 3rd or 4th
feature release, rather than every 2nd or 3rd feature release (as often happens
today).

This PEP presents a competing proposal to instead *slow down* the frequency of
parallel installable feature releases that change the filesystem layout
and CPython ABI to a consistent 24 month cycle, but to compensate for this by
introducing the notion of build compatible incremental feature releases, and
then deferring the full feature freeze of a given feature release series from
the initial baseline X.Y.0 release to a subsequent X.Y.Z feature complete
release that occurs ~12 months after the initial baseline feature release.

A new ``feature_complete`` attribute on the ``sys.version_info`` structure will
provide a programmatic indicator as to whether or not a release series remains
open to further incremental feature releases. Alternate implementations of
Python would also be free to clear this flag to indicate that their support for
their nominal Python version may still be a work in progress.

For compatibility testing purposes, and to maintain pickle compatibility in
mixed version environments, a new ``sys.feature_limit`` attribute (and
corresponding CPython CLI parameter, ``--feature-limit X.Y.Z``, and environment
variable, ``PYTHONFEATURELIMIT``) will provide a way to limit the runtime
availability of features added in incremental feature releases.

The existing cycle and the new cycle would be synchronised on their feature
freeze release dates, so the full Python 3.9.x feature freeze would occur in
October 2021, 24 months after the Python 3.8.0 feature release, with the initial
Python 3.9.0 release taking place in October 2020.


Example Future Release Schedules
================================

Under this proposal, Python 3.9.0a1 would be released in November 2019, shortly
after the Python 3.8.0 feature complete release in October 2019.

The 3.9.0b1 release would then follow 6 months later in May 2020, with 3.9.0
itself being released in October 2020.

Assuming micro releases of 3.9.x were to occur quarterly, then the overall
release timeline would look like:

* 2019-11: 3.9.0a1
* ... additional alpha releases as determined by the release manager
* 2020-05: 3.9.0b1
* ... additional beta releases as determined by the release manager
* 2020-08: 3.9.0bX (final beta release that locks ABI compatibility)
* 2020-09: 3.9.0rc1
* ... additional release candidates as determined by the release manager
* 2020-10: 3.9.0 (BFR - baseline feature release)
* 2021-01: 3.9.1 (IFR - incremental feature release)
* 2021-04: 3.9.2 (IFR)
* 2021-07: 3.9.3 (IFR)
* 2021-10: 3.9.4 (feature complete release)
* 2022-01: 3.9.5
* 2022-04: 3.9.6
* 2022-07: 3.9.7
* 2022-10: 3.9.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2025-10: 3.9.x branch closed

Feature complete release numbers would typically be written without any
qualifier (as they are today), while the baseline and incremental feature
releases would be expected to have a qualifier appended indicating that they
aren't a traditional CPython release (``3.9.0 (BFR)``, ``3.9.1 (IFR)``, etc).

The Python 3.10 release series would start being published the month after the
first Python 3.9 feature complete release, in parallel with the final 12 months
of routine Python 3.9 maintenance releases:

* 2021-11: 3.10.0a1
* ... additional alpha releases as determined by the release manager
* 2022-05: 3.10.0b1
* ... additional beta releases as determined by the release manager
* 2022-08: 3.10.0bX (final beta release that locks ABI compatibility)
* 2022-09: 3.10.0rc1
* ... additional release candidates as determined by the release manager
* 2022-10: 3.10.0 (BFR)
* 2023-01: 3.10.1 (IFR)
* 2023-04: 3.10.2 (IFR)
* 2023-07: 3.10.3 (IFR)
* 2023-10: 3.10.4
* 2024-01: 3.10.5
* 2024-04: 3.10.6
* 2024-07: 3.10.7
* 2024-10: 3.10.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2027-10: 3.10.x branch closed

In this model, there are always two or three active branches:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance
* 2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
* 2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
* 2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance
* 2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
* ... etc

(Pre-alpha and pre-beta development occurs on the main git branch, all other
development occurs on a release specific branch with changes typically
backported from the main git branch)

TODO: this really needs a diagram to help explain it, so I'll add a picture
once I have one to add.

This is quite similar to the status quo, but with a more consistent cadence,
alternating between baseline feature release years (2020, 2022, etc) that focus
on the alpha and beta cycle for a new baseline feature release (while continuing
to publish maintenance releases for the previous feature release series), and
feature complete release years (2021, 2023, etc), that focus on making
smaller improvements to the current feature release series (while making plans
for the next feature release series the following year).


Proposal
========

Excluding alpha and beta releases, CPython currently has 3 different kinds
of release increment:

* Feature release (i.e. X.Y.0 releases)
* Maintenance release (X.Y.Z releases within ~2 years of X.Y.0)
* Source-only security release (subsequent X.Y.Z releases)

Feature freeze takes place at the time of the X.Y.0b1 release.
Build compatibility freeze now takes place at the time of the last beta release
(providing time for projects to upload wheel archives to PyPI prior to the
first release candidate).

This then creates the following periods in the lifecycle of a release series:

* Pre-beta (release series is the CPython development branch)
* Beta (release enters maintenance mode, ABI compatibility mostly locked)
* Maintenance (ABI locked, only bug fixes & docs enhancements accepted)
* Security fix only (no further binary releases, only security fixes accepted)
* End of life (no further releases of any kind)

The proposal in this PEP is that the "Feature release" category be split up into
three different kinds of feature release:

* Baseline feature release (X.Y.0 releases)
* Incremental feature release (any X.Y.Z releases published between a
  baseline feature release and the corresponding feature complete release)
* Feature complete release (a specific X.Y.Z release ~1 year after X.Y.0)
* Maintenance release (X.Y.Z releases within ~1 years of the feature complete release)
* Source-only security release (subsequent ``X.Y.Z`` releases)

This would then introduce a new "Feature releases" phase in the release series
lifecycle:

* Pre-beta (release series is the CPython development branch)
* Beta (release enters feature additions mode, ABI compatibility not yet locked)
* Feature releases (ABI locked, backwards compatible API additions accepted)
* Maintenance (ABI locked, only bug fixes & docs enhancements accepted)
* Security fix only (no further binary releases, only security fixes accepted)
* End of life (no further releases of any kind)

The pre-release beta period would be relaxed to use the incremental feature
release policy for changes, rather than the stricter maintenance release policy.

For governance purposes, baseline feature releases are the only releases that
would qualify as a "feature release" in the PEP 13 sense (incremental feature
releases wouldn't count).


Baseline feature releases and feature release series
----------------------------------------------------

Baseline feature releases are essentially just the existing feature releases,
given a new name to help distinguish them from the new incremental feature
releases, and also to help indicate that unlike their predecessors, they are
no longer considered feature complete at release.

Baseline feature releases would continue to define a new feature release series,
locking in the following language, build, and installation compatibility
constraints for the remainder of that series:

- Python language grammar
- ``ast`` module AST format
- CPython interpreter opcode format
- ``pyc`` file magic number and filename compatibility tags
- extension module filename compatibility tags
- wheel archive compatibility tags
- default package and module import directories
- default installation filename and directories

Baseline feature releases would also continue to be the only releases where:

- new deprecations, pending deprecations, and other warnings can be introduced
- existing pending deprecations can be converted to full deprecations
- existing warnings can be converted to errors
- other changes requiring "Porting to Python X.Y" entries in the What's New
  document can be introduced

Key characteristics of a feature release series:

- an installation within one feature release series does not conflict with
  installations of other feature release series (i.e. they can be installed in parallel)
- an installation within a feature release series can be updated to a later
  micro release within the same series without requiring reinstallation
  or any other changes to previously installed components

Key characteristics of a baseline feature release:

- in a baseline feature release, ``sys.version_info.feature_complete == False``
- in a baseline feature release, ``sys.version_info.micro == 0``
- baseline feature releases may contain higher risk changes to the language and
  interpreter, such as grammar modifications, major refactoring of interpreter
  and standard library internals, or potentially invasive feature additions that
  carry a risk of unintended side effects on other existing functionality
- features introduced in a baseline feature release are the *only* features
  permitted to rely on ``sys.version_info`` as their sole runtime indicator
  of the feature's availability

Key expectations around feature release series and baseline feature releases:

- most public projects will only actively test against the *most recent*
  micro release within a release series
- many (most?) public projects will only add a new release series to their test
  matrix *after* the initial baseline feature release has already been published,
  which can make it difficult to resolve issues that require providing new flags
  or APIs to explicitly opt-in to old behaviour after a default behaviour changed
- private projects with known target environments will test against whichever
  micro release version they're actually using
- most private projects will also only consider migrating to a new release
  series *after* the initial baseline feature release has already been published,
  again posing a problem if the resolution of their problems requires an API
  addition


The key motivation of the proposal in this PEP is that the public and private
project behaviours described above aren't *new* expectations: they're
descriptions of the way CPython release series are already handled by the wider
community today. As such, the PEP represents an attempt to adjust our release
policies and processes to better match the way the wider community already
handles them, rather than changing our processes in a way that then means the
wider community needs to adjust to us rather than the other way around.


Incremental feature releases
----------------------------

Incremental feature releases are the key new process addition being proposed by
this PEP. They are subject to the same strict runtime compatibility requirements
as the existing maintenance releases, but would have the following more
relaxed policies around API additions and enhancements:

* new public APIs can be added to any standard library module (including builtins)
* subject to the feature detection requirement below, new optional arguments can
  be added to existing APIs (including builtins)
* new public APIs can be added to the stable C ABI (with appropriate version guards)
* new public APIs can be added to the CPython C API
* with the approval of the release manager, backwards compatible reliability
  improvements can be made to existing APIs and syntactic constructs
* with the approval of the release manager, performance improvements can be
  incorporated for existing APIs and syntactic constructs

The intent of this change in policy is to allow usability improvements for new
(and existing!) language features to be delivered in a more timely fashion,
rather than requiring users to incur the inherent delay and costs of waiting for
and then upgrading to the next feature release series.

It is also designed such that the approval to add a feature to the next baseline
feature release can be considered separately from the question of whether or not
to make it available in the next incremental feature release for the current
release series, potentially allowing the first task to be completed by volunteer
contributors, while the latter activity could be handled by paid contributors
(e.g. customers of commercial Python redistributors could potentially request
that their vendor backport a feature, or core developers could offer to
undertake specific backports on a contract basis). (There would be potential
ethical concerns with gating bug fixes this way, but those concerns don't apply
for backports of new features)

Key characteristics of an incremental feature release:

- in an incremental feature release, ``sys.version_info.feature_complete == False``
- in an incremental feature release, ``sys.version_info.micro != 0``
- all API additions made in an incremental feature release must support
  efficient runtime feature detection that doesn't rely on either
  ``sys.version_info`` or runtime code object introspection. In most cases, a
  simple ``hasattr`` check on the affected module will serve this purpose, but
  when it doesn't, an alternative approach will need to be implemented as part
  of the feature addition. Prior art in this area includes the
  ``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available``
  set, and the various ``os.supports_*`` sets that the ``os`` module already
  offers for platform dependent capability detection
- to maintain pickle compatibility in mixed version environments, and to enable
  easier compatibility testing across multiple API versions within the same
  release series, all API additions made in an incremental feature release
  must support the new ``sys.feature_limit`` setting as described in the next
  section

Key expectations around incremental feature releases:

- "don't break existing installations on upgrade" remains a key requirement
  for all micro releases, even with the more permissive change inclusion policy
- more intrusive changes should still be deferred to the next baseline feature
  release
- public Python projects that start relying on features added in an incremental
  feature release should set their ``Python-Requires`` metadata appropriately
  (projects already do this when necessary - e.g. ``aiohttp`` specifically
  requires 3.5.3 or later due to an issue with ``asyncio.get_event_loop()``
  in earlier versions)

Some standard library modules may also impose their own restrictions on
acceptable changes in incremental feature releases (for example, only a
baseline feature release should ever add new hash algorithms to
``hashlib.algorithms_guaranteed`` - incremental feature releases would only be
permitted to add algorithms to ``hashlib.algorithms_available``)


Maintaining interoperability across incremental feature releases
----------------------------------------------------------------

It is a common practice to use Python's ``pickle`` module to exchange
information between Python processes running on different versions of Python.
Between release series, this compatibility is expected to only run one way
(i.e. excluding deprecated APIs, Python "X.Y+1" processes should be able to
read pickle archives produced by Python "X.Y" processes, but the reverse does
not hold, as the newer archives may reference attributes and parameters that
don't exist in the older version).

Within a release series, however, it is expected to hold in both directions,
as the "No new features" policy means that almost all pickle archives created
on Python "X.Y.Z+1" will be readable by Python "X.Y.Z" processes.

Similarly, Python libraries and applications are often only tested against
the latest version in a release series, and this is usually sufficient to keep
code working on earlier releases in that same series.

Allowing feature additions in later "X.Y.Z" releases with no way to turn them
off would pose a problem for these common practices, as a library or application
that works fine when tested on CPython version "X.Y.Z" would fail on earlier
versions if it used a feature newly introduced in "X.Y.Z", and any pickle
archives it creates that rely on those new interfaces may also not be readable
on the older versions.

To help address these problems, a new ``sys.feature_limit`` attribute would be
added, as a structured sequence corresponding to the first 3 fields in
``sys.version_info`` (``major``, ``minor``, ``micro``).

A new CLI option (``--feature-limit X.Y.Z``) and environment variable
(``PYTHONFEATURELIMIT=X.Y.Z``) would be used to set this attribute. The
``PyCoreConfig`` struct would also gain a new field::

    wchar_t *feature_limit;

If the limit is not set explicitly, it would default to the first 3 fields in
``sys.version_info``. If the limit is set to a value outside the lower bound of
``sys.version_info[:2]`` and the upper bound of ``sys.version_info[:3]``, it
will be clamped to those bounds, padding with zeroes if necessary.

For example, given a current version of "3.9.3", nominal limits would be
converted to runtime ``sys.feature_limit`` values as follows::

    3 => (3, 9, 0)
    3.8.1 => (3, 9, 0)
    3.9 => (3, 9, 0)
    3.9.2 => (3, 9, 2)
    <unset> => (3, 9, 3)
    3.9.3 => (3, 9, 3)
    3.9.4 => (3, 9, 3)
    4 => (3, 9, 3)

New APIs backported to an incremental feature release would be expected to
include a guard that deletes the API from the module if the feature limit is
too low::

    def feature_api():
        ...

    _version_feature_api_added = (3, 9, 1)
    if _version_feature_api_added > sys.feature_limit:
        del feature_api

Similarly, new parameters would be expected to include a guard that adjusts the
function signature to match the old one::


    def feature_api(old_param1, old_param2, new_param=default):
        """Updated API docstring"""
        ...

    _version_feature_api_changed = (3, 9, 1)
    if _version_feature_api_changed > sys.feature_limit:
        _new_feature_api = feature_api
        def feature_api(old_param1, old_param2):
            """Legacy API docstring"""
            return _new_feature_api(old_param1, old_param2)


Structuring the guards this way would keep the code structure as similar as
possible between the main development branch and the backport branches, so
future bug fixes can still be backported automatically.

It is expected that convenience functions and/or additional automated tests
would eventually be added to help ensure these backported APIs are guarded
appropriately, but it seems reasonable to wait until specific concrete
examples are available to drive the design of those APIs and automated tests,
rather than designing them solely on the basis of hypothetical examples.


Feature complete release and subsequent maintenance releases
------------------------------------------------------------

The feature complete release for a given feature release series would be
developed under the normal policy for an incremental feature release, but
would have one distinguishing feature:

- in a feature complete release, ``sys.version_info.feature_complete == True``

Any subsequent maintenance and security fix only releases would also have that
flag set, and may informally be referred to as "feature complete releases".
For release series definition purposes though, the feature complete release
is the first one that sets that flag to "True".


Proposed policy adjustment for provisional APIs
-----------------------------------------------

To help improve consistency in management of provisional APIs, this PEP proposes
that provisional APIs be subject to regular backwards compatibility requirements
following the feature complete release for a given release series.

Other aspects of managing provisional APIs would remain as they are today, so as
long as an API remains in the provisional state, regular backwards compatibility
requirements would not apply to that API in baseline and incremental feature
releases.

This policy is expected to provide increased clarity to end users (as even
provisional APIs will become stable for that release series in the feature
complete release), with minimal practical downsides for standard library
maintainers, based on the following analysis of documented API additions and
changes in micro releases of CPython since 3.0.0:

* 21 3.x.1 version added/changed notes
* 30 3.x.2 version added/changed notes
* 18 3.x.3 version added/changed notes
* 11 3.x.4 version added/changed notes
*  1 3.x.5 version added/changed notes
*  0 3.x.6+ version added/changed notes

When post-baseline-release changes need to be made, the majority of them occur
within the first two maintenance releases, which have always occurred within 12
months of the baseline release.

(Note: these counts are not solely for provisional APIs - they cover all APIs
where semantic changes were made after the baseline release that were considered
necessary to cover in the documentation. To avoid double counting changes, the
numbers exclude any change markers from the What's New section)


Motivation
==========

The motivation for change in this PEP is essentially the same as the motivation
for change in PEP 596: the current 18-24 month gap between feature releases has
a lot of undesirable consequences, especially for the standard library (see
PEP 596 for further articulation of the details).

This PEP's concern with the specific proposal in PEP 596 is that it doubles the
number of actively supported Python branches, increasing the complexity of
compatibility testing matrices for the entire Python community, increasing the
number of binary Python wheels to be uploaded to PyPI when not using the stable
ABI, and just generally having a high chance of inflicting a relatively high
level of additional cost across the entire Python ecosystem.

The view taken in this PEP is that there's an alternative approach that provides
most of the benefits of a faster feature release without actually incurring the
associated costs: we can split the current X.Y.0 "feature freeze" into two
parts, such that the baseline X.Y.0 release only imposes a
"runtime compatibility freeze", and the full standard library feature freeze
is deferred until later in the release series lifecycle.


Caveats and Limitations
=======================

This proposal does NOT retroactively apply to Python 3.8 - it is being proposed
for Python 3.9 and later releases only.

Actual release dates may be adjusted up to a month earlier or later at
the discretion of the release manager, based on release team availability, and
the timing of other events (e.g. PyCon US, or the annual core development
sprints). However, part of the goal of this proposal is to provide a consistent
annual cadence for both contributors and end users, so adjustments ideally would
be rare.

This PEP does not dictate a specific cadence for micro releases within a release
series - it just specifies the rough timelines for transitions between the
release series lifecycle phases (pre-alpha, alpha, beta, feature releases,
bug fixes, security fixes). The number of micro releases within each phase is
determined by the release manager for that series based on how frequently they
and the rest of the release team for that series are prepared to undertake the
associated work.

However, for the sake of the example timelines, the PEP assumes quarterly
micro releases (the cadence used for Python 3.6 and 3.7, splitting the
difference between the twice yearly cadence used for some historical release
series, and the monthly cadence planned for Python 3.8 and 3.9).


Design Discussion
=================

Why this proposal over simply doing more frequent baseline feature releases?
----------------------------------------------------------------------------

The filesystem layout changes and other inherently incompatible changes involved
in a baseline feature release create additional work for large sections of the
wider Python community.

Decoupling those layout changes from the Python version numbering scheme is also
something that would in and of itself involve making backwards incompatible
changes, as well as adjusting community expectations around which versions will
install over the top of each other, and which can be installed in parallel on
a single system.

We also don't have a straightforward means to communicate to the community
variations in support periods like "Only support Python version X.Y until
X.Y+1 is out, but support X.Z until X.Z+2 is out".

So this PEP takes as its starting assumption that the vast majority of Python
users simply *shouldn't need to care* that we're changing our release policy,
and the only folks that should be affected are those that are eagerly waiting
for standard library improvements (and other backwards compatible interpreter
enhancements), and those that need to manage mission critical applications in
complex deployment environments.


Implications for Python library development
-------------------------------------------

Many Python libraries (both open source and proprietary) currently adopt the
practice of testing solely against the latest micro release within each feature
release series that the project still supports.

The design assumption in this PEP is that this practice will continue to be
followed during the feature release phase of a release series, with the
expectation being that anyone choosing to adopt a new release series before it
is feature complete will closely track the incremental feature releases.

Libraries that support a previous feature release series are unlikely to adopt
features added in an incremental feature release, and if they do adopt such
a feature, then any associated fallback compatibility strategies should be
implemented in such a way that they're also effective on the earlier releases
in that release series.


Implications for the proposed Scientific Python ecosystem support period
------------------------------------------------------------------------

Based on discussions at SciPy 2019, a NEP is currently being drafted [2_] to
define a common convention across the Scientific Python ecosystem for dropping
support for older Python versions.

While the exact formulation of that policy is still being discussed, the initial
proposal was very simple: support any Python feature release published within
the last 42 months.

For an 18 month feature release cadence, that works out to always supporting at
least the two most recent feature releases, and then dropping support for all
X.Y.z releases around 6 months after X.(Y+2).0 is released. This means there is
a 6 month period roughly every other year where the three most recent feature
releases are supported.

For a 12 month release cadence, it would work out to always supporting at
least the three most recent feature releases, and then dropping support for all
X.Y.z releases around 6 months after X.(Y+3).0 is released. This means that
for half of each year, the four most recent feature releases would be supported.

For a 24 month release cadence, a 42 month support cycle works out to always
supporting at least the most recent feature release, and then dropping support
for all X.Y.z feature releases around 18 months after X.(Y+1).0 is released.
This means there is a 6 month period every other year where only one feature
release is supported (and that period overlaps with the pre-release testing
period for the X.(Y+2).0 baseline feature release).

Importantly for the proposal in this PEP, that support period would abide by
the recommendation that library developers maintain support for the previous
release series until the latest release series has attained feature complete
status: dropping support 18 months after the baseline feature release will be
roughly equivalent to dropping support 6 months after the feature complete
release, without needing to track exactly *which* release marked the series as
feature complete.


Implications for simple deployment environments
-----------------------------------------------

For the purposes of this PEP, a "simple" deployment environment is any use case
where it is straightforward to ensure that all target environments are updated
to a new Python micro version at the same time (or at least in advance of the
rollout of new higher level application versions), and there isn't any
requirement for older Python versions to be able to reliably read pickle streams
generated with the newer Python version, such that any pre-release testing that
occurs need only target a single Python micro version.

The simplest such case would be scripting for personal use, where the testing
and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where the
same Python container is used in the CI pipeline as is used on deployment, and
any application that bundles its own Python runtime, rather than relying on a
pre-existing Python deployment on the target system.

For these use cases, this PEP shouldn't have any significant implications - only
a single micro version needs to be tested, independently of whether that
version is feature complete or not.


Implications for complex deployment environments
------------------------------------------------

For the purposes of this PEP, "complex" deployment environments are use cases
which don't meet the "simple deployment" criterion above: new application
versions are combined with two or more distinct micro versions within
the same release series as part of the deployment process, rather than always
targeting exactly one micro version at a time.

If the proposal in this PEP has the desired effect of reducing feature delivery
latency, then it can be expected that developers using a release series that is
not yet feature complete will actually make use of the new features as they're
made available. This then means that testing against a newer incremental feature
release becomes an even less valid test of compatibility with the baseline
feature release and older incremental feature releases than testing against a
newer maintenance release is for older maintenance releases.

One option for handling such cases is to simply prohibit the use of new Python
versions until the series has reached "feature complete" status. Such a policy
is effectively already adopted by many organisations when it comes to new
feature release series, with acceptance into operational environments occurring
months or years after the original release. If this policy is adopted, then such
organisations could potentially still adopt a new Python version every other
year - it would just be based on the availability of the feature complete
releases, rather than the baseline feature releases.

A less strict alternative to outright prohibition would be to make use of the
proposed ``PYTHONFEATURELIMIT`` setting to enable phased migrations to new
incremental feature releases:

* initially roll out Python X.Y.0 with ``PYTHONFEATURELIMIT=X.Y.0`` set in CI
  and on deployment
* roll out Python X.Y.1 to CI, keeping the ``PYTHONFEATURELIMIT=X.Y.0`` setting
* deploy Python X.Y.1 to production based on successful CI results
* update deployment environments to set ``PYTHONFEATURELIMIT=X.Y.1``
* set ``PYTHONFEATURELIMIT=X.Y.1`` in CI only after all deployment environments
  have been updated
* repeat this process for each new release up to and including the feature
  complete release for the release series
* once the series is feature complete, either continue with this same process
  for consistency's sake, or else stop updating ``PYTHONFEATURELIMIT`` and leave
  it at the feature complete version number


Duration of the feature additions period
----------------------------------------

This PEP proposes that feature additions be limited to 12 months after the
initial baseline feature release.

The primary motivation for that is specifically to sync up with the Ubuntu LTS
timing, such that the feature complete release for the Python 3.9.x series gets
published in October 2021, ready for inclusion in the Ubuntu 22.04 release.
(other LTS Linux distributions like RHEL, SLES, and Debian don't have a fixed
publishing cadence, so they can more easily tweak their LTS timing a bit to
align with stable versions of their inputs. Canonical deliberately haven't
given themselves that flexibility with their own release cycle).

The 12 month feature addition period then arises from splitting the time
from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x incremental
feature release in 2021-10 evenly between pre-release development and subsequent
incremental feature releases.

This is an area where this PEP could adopt part of the proposal in PEP 596,
by instead making that split ~9 months of pre-release development, and ~15
months of incremental feature releases:

* 2019-11: 3.9.0a1
* ... additional alpha releases as determined by the release manager
* 2020-03: 3.9.0b1
* 2020-04: 3.9.0b2
* 2020-05: 3.9.0b3 (final beta release that locks ABI compatibility)
* 2020-06: 3.9.0rc1
* ... additional release candidates as determined by the release manager
* 2020-07: 3.9.0 (BFR)
* 2020-10: 3.9.1 (IFR)
* 2021-01: 3.9.2 (IFR)
* 2021-04: 3.9.3 (IFR)
* 2021-07: 3.9.4 (IFR)
* 2021-10: 3.9.5
* 2022-01: 3.9.6
* 2022-04: 3.9.7
* 2022-07: 3.9.8
* 2022-10: 3.9.9 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2025-10: 3.9.x branch closed

This approach would mean there were still always two or three active branches,
it's just that proportionally more time would be spent with a branch in the
"feature releases" phase, as compared to the "pre-alpha", "pre-beta", and
"pre-release" phases:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance
* 2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
* 2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
* 2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance
* 2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
* ... etc


Duration of the unreleased pre-alpha period
-------------------------------------------

In the baseline proposal in this PEP, the proposed timelines still include
periods where we go for 18 months without making a release from the main git
branch (e.g. 3.9.0b1 would branch off in 2020-05, and 3.10.0a1 wouldn't be
published until 2021-11). They just allow for a wider variety of changes to
be backported to the most recent maintenance branch for 12 of those months.

The variant of the proposal that moves the beta branch point earlier in the
release series lifecycle would increase that period of no direct releases to
21 months - the only period where releases were made directly from the main
branch would be during the relatively short window between the last incremental
feature release of the previous release series, and the beta branch point a
few months later.

While alternating the annual cadence between "big foundational enhancements"
and "targeted low risk API usability improvements" is a deliberate feature of
this proposal, it still seems strange to wait that long for feedback in the
event that changes *are* made shortly after the previous release series is
branched.

An alternative way of handling this would be to start publishing alpha releases
for the next baseline feature release during the feature addition period (similar
to the way that PEP 596 proposes to starting publishing Python 3.9.0 alpha
releases during the Python 3.8.0 release candidate period).

However, rather than setting specific timelines for that at a policy level,
it may make sense to leave that decision to individual release managers, based
on the specific changes that are being proposed for the release they're
managing.


Why not switch directly to full semantic versioning?
----------------------------------------------------

If this were a versioning design document for a new language, it *would* use
semantic versioning: the policies described above for baseline feature releases
would be applied to X.0.0 releases, the policies for incremental feature
releases would be applied to X.Y.0 releases, and the policies for maintenance
releases would be applied to X.Y.Z releases.

The problem for Python specifically is that all the policies and properties for
parallel installation support and ABI compatibility definitions are currently
associated with the first *two* fields of the version number, and it has been
that way for the better part of thirty years.

As a result, it makes sense to split out the policy question of introducing
incremental feature releases in the first place from the technical question of
making the version numbering scheme better match the semantics of the different
release types.

If the proposal in this PEP were to be accepted by the Steering Council for
Python 3.9, then a better time to tackle that technical question would be for
the subsequent October 2022 baseline feature release, as there are already inherent
compatibility risks associated with the choice of either "Python 4.0" (erroneous
checks for the major version being exactly 3 rather than 3 or greater), or
"Python 3.10" (code incorrectly assuming that the minor version will always
contain exactly one decimal digit) [1_].

While the text of this PEP assumes that the release published in 2022 will be
3.10 (as the PEP author personally considers that the more reasonable and most
likely choice), there are complex pros and cons on both sides of that decision,
and this PEP does arguably add a potential pro in favour of choosing the
"Python 4.0" option (with the caveat that we would also need to amend the
affected installation layout and compatibility markers to only consider the
major version number, rather than both the major and minor version).

If such a version numbering change were to be proposed and accepted, then the
example 3.10.x timeline given above would instead become the following 4.x
series timeline:

* 2021-11: 4.0.0a1
* ... additional alpha releases as determined by the release manager
* 2022-05: 4.0.0b1
* ... additional beta releases as determined by the release manager
* 2022-08: 4.0.0bX (final beta release that locks ABI compatibility)
* 2022-09: 4.0.0rc1
* ... additional release candidates as determined by the release manager
* 2022-10: 4.0.0 (BFR)
* 2023-01: 4.1.0 (IFR)
* 2023-04: 4.2.0 (IFR)
* 2023-07: 4.3.0 (IFR)
* 2023-10: 4.4.0 (IFR)
* 2024-01: 4.4.1
* 2024-04: 4.4.2
* 2024-07: 4.4.3
* 2024-10: 4.4.4 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2027-10: 4.x branch closed

And the 5 year schedule forecast would look like:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance
* 2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x maintenance
* 2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature releases, 3.9.x maintenance
* 2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance
* 2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y maintenance
* ... etc

References
==========

.. [1] Anthony Sottile created a pseudo "Python 3.10" to find and fix such issues
       (https://github.com/asottile/python3.10)

.. [2] NEP proposing a standard policy for dropping support of old Python versions
       (https://github.com/numpy/numpy/pull/14086)

.. [3] Draft Extended Lifecycle for Python (ELPython) design concept
       (https://github.com/elpython/elpython-meta/blob/master/README.md)

Copyright
=========

This document has been placed in the public domain.


..
  Local Variables:
  mode: indented-text
  indent-tabs-mode: nil
  sentence-end-double-space: t
  fill-column: 80
  coding: utf-8
  End: