python-peps/pep-0605.rst

PEP: 605
Title: A rolling feature release stream for CPython
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower <steve.dower@python.org>, Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-605-a-rolling-feature-release-stream-for-cpython/2418
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 20-Sep-2019
Python-Version: 3.9
Post-History: 1-Oct-2019


Abstract
========

Rather than proposing more frequent full CPython releases (as PEP 602 does),
or a policy change to allow backwards compatible feature additions later in a
release series (as PEP 598 does), this PEP instead proposes that we create a
rolling stream of production-ready beta releases by amending CPython's
pre-release management process as follows:

* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
  creation all correspond with the first release candidate for an X.Y.0 release
* The X.Y.0 release candidate period is extended from 3 weeks to 2 months
* Beta releases from the master git branch would occur every 2 months
  whenever there is no X.Y.0 release in the release candidate phase
* Routine alpha releases are removed from the process
* Release managers may still choose to mark particular releases as alpha
  releases if they feel such releases are appropriate. If any such alpha
  releases are made, it is expected that they would be in place of one of the
  scheduled beta releases
* The deprecation policy for interfaces that have been included in an X.Y.0
  release or release candidate without being explicitly declared provisional
  remains unchanged (refer to PEP 387 for details)
* Each new alpha or beta release may introduce new deprecation warnings for
  interfaces that appeared in the previous X.Y release series
* Each new alpha or beta release may change or remove interfaces that either did
  not appear in the previous X.Y release series, or emitted a relevant
  deprecation warning in that series
* Additions to the `abi3` stable C ABI are permanent once they have been
  declared to be part of that ABI in at least one beta release
* All other newly added interfaces that have appeared only in alpha or beta
  releases are considered provisional, and may be changed in incompatible ways
  without requiring a preceding deprecation warning

As part of this change, the current "not for production use" guidance given for
beta releases would be amended to state "suitable for production use only in
environments with sufficiently robust compatibility testing and operational
monitoring capabilities".

This PEP also proposes that the frequency of X.Y.0 releases be adjusted to
begin each new release series in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).


Motivation
==========

The current CPython pre-release and release management processes were developed
in an era where automated continuous integration and operational monitoring
systems were still relatively immature. Since that time, many organisations
have adopted deployment models that allow them to incorporate new CPython
feature releases without adding substantially more risk than they incur for any
other code change. Newer deployment models, such as lightweight task specific
application containers, also make it easier to combine an application with a
language runtime in a CI pipeline, and then keep them together until the entire
container image is later replaced by an updated one.

In light of those changes in the wider environment, PEP 602 has proposed
reducing the feature delivery latency for the Python standard library and
CPython reference interpreter by increasing the frequency of CPython feature
releases from every 18-24 months to instead occur every 12 months.

Unfortunately, for many organisations, the cost of adopting a new Python release
doesn't automatically scale down with a reduced number of changes in the release,
as the primary costs aren't associated with resolving any discovered issues;
the primary costs are associated with the *search* for issues. This search may
involve manual testing of software systems, human review of written materials,
and other activities where the time required scales with the size of the
existing system, rather than with the number of changes between the versions of
Python.

For third party library developers, costs are primarily associated with the
*number* of distinct Python versions in widespread usage. This currently tends
to be influenced by a combination of which releases are still actively
maintained by python-dev, and which releases are the latest versions offered
by particular redistributors (with the Debian, Ubuntu LTS, and RHEL/CentOS
system Python versions being particularly popular development targets). In
addition to the basic CI cost of testing against more Python versions, having
more variants in widespread use can make it more difficult to determine when a
fault report is an actual error in the project, or an issue in the reporting
user's environment.

PEP 602 proposes that affected organisations and projects simply switch to
adopting every second or third CPython release, rather than attempting to adopt
every release, but that creates its own set of new problems to be resolved, both
practical (e.g. deprecations would need to cover more than one release if we're
expecting users to routinely skip releases) and cultural (e.g. with a larger
number of versions in active use, there is a much higher chance that open source
library maintainers will receive bug reports that only occur on Python versions
that they're not using themselves).

PEP 598 was an initial attempt by one of the authors of this PEP to propose
an alternative scheme to reduce feature delivery latency by adopting a
semantic versioning style policy that allowed for the incremental delivery of
backwards compatible features within a release series, until that series
reached feature complete status. That variant still had the undesirable
consequence of imposing visible changes on end users that are happy enough
with the current release management model.

This PEP takes the view that both PEP 598 and PEP 602 share a common flaw: they
are attempting to satisfy the needs of two quite distinct audiences within the
constraints of a single release model, which results in conflicting design
requirements, and the need for awkward trade-offs between those conflicting
requirements. The proposal in this PEP aims to avoid that flaw by proposing the
creation of two *distinct* production-ready release streams, with the existing
release stream being largely left alone, while the new release stream is
tailored towards the audience that would most benefit from a reduction in
feature delivery latency.


Aims of this Proposal
=====================

The core of the proposal in this PEP is changing the beta phase of the CPython
pre-release process to produce a rolling stream of releases at a regular
cadence, and to ensure those builds offer a sufficient level of stability as
to be suitable for use in appropriately managed production systems.

By adopting this approach, the proposal aims to provide an improved outcome
for almost all Python users and contributors:

* for users of the new release stream, targeting the beta phase allows for even
  lower feature delivery latency than the annual cadence proposed in PEP 602;
* for core developers working on new features, increased frequency and adoption
  of the beta releases should improve pre-release feedback cycles;
* for users of the established release stream, the increased adoption and
  improved feedback cycles during the pre-release period should result in
  increased feature maturity at the time of its first X.Y.0 release, as well
  as higher levels of ecosystem readiness;
* for Python library maintainers, the rolling stream of beta releases will
  hopefully provide more opportunities to identify and resolve design issues
  before they make it into a full stable release than is offered by the current
  pre-release management process; and
* for developers of alternative Python implementations, the proposed rolling
  beta releases provide an additional incentive for extension module authors
  to migrate from the full CPython ABI to the Python stable ABI, which may
  over time make more of the ecosystem compatible with implementations that
  don't fully emulate the CPython C API.

That said, it is acknowledged that not all the outcomes of this proposal will be
beneficial for all members of the wider Python ecosystem:

* for Python library maintainers, both this PEP and PEP 602 would likely
  result in user pressure to support the faster release cadence. While this PEP
  attempts to mitigate that by marking the faster releases with the "beta" label
  and removing the expectation to publish pre-built wheel archives, and PEP 602
  attempts to mitigate it by keeping the minimum time between full releases at
  12 months, it isn't possible to eliminate this downside completely;
* for third party extension module maintainers, both this PEP and PEP 602 would
  likely result in user pressure to start supporting the stable ABI in order to
  provide wheel archives that work on the new version as soon as it is
  available. Whether that's a net negative or not will depend on how the request
  is presented to them (it could be a positive if the request comes in the form
  of a courteous contribution to their project from a developer interested in
  supporting the rolling beta releases);
* for some users of the established release stream that rely on the
  availability of pre-built wheel archives, switching to adopting a new release
  every 12 months may be an acceptable rate increase, while moving consistently
  to the 24 month end of the historical 18-24 month cadence would be an
  undesirable rate reduction relative to the 18 month cycle used for recent
  releases. Whether this proposal would be a net negative for these users in the
  long run would depend on how many projects migrated over to targeting the
  Python stable ABI and hence produced wheel archives that would still be usable
  on the rolling beta releases.


Proposal
========

The majority of the proposed changes in this PEP only affect the handling of
pre-release versions. The one change affecting full release versions is a
suggested change to their cadence.

Changes to beta release policy, phase duration, and cadence
-----------------------------------------------------------

This PEP proposes that the policy for beta releases be adjusted to be a
combination of the policies for alpha and beta releases:

* as with current beta releases, the stable BuildBot fleet is expected to be
  green prior to preparation and publication of the beta release
* as with current beta releases, the release manager is expected to review
  open release blocker issues prior to preparation and publication of the beta
  release
* as with current beta releases, any additions to the `abi3` stable C ABI would
  be expected to become a permanent part of that ABI unless and until that
  stable ABI version is retired completely (Note: there are no current plans
  to increment the stable ABI version)
* as with current alpha releases, beta releases would *not* be considered
  feature complete for the next X.Y.0 release
* as with current alpha releases, all APIs added since the last CPython feature
  release (other than additions to the stable C ABI) would be considered
  provisional
* as with current alpha releases, beta releases would be prepared and published
  from the master development branch

Rather than being released monthly for a period of a few months while preparing
a new X.Y.0 release, beta releases would instead be consistently published every
two months.

The only time this would not be the case is during the two month release
candidate period for an upcoming X.Y.0 release, as described in the following
section.


Changes to release candidate policy, phase duration, and cadence
----------------------------------------------------------------

Given the proposed changes to the beta release phase, the release candidate
phase would see the following related adjustments:

* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
  creation would all correspond with the creation of X.Y.0rc1 (currently these
  occur across a mixture of X.Y.0b1, the last beta release, and X.Y.0rc1)
* The X.Y.0 release candidate period would be extended from 3 weeks to 2 months
* There would normally be two release candidates issued a month apart, but
  additional candidates may be published at the release manager's discretion
* The final X.Y.0 release would occur between 1 and 4 weeks after the final
  release candidate (depending if additional release candidates were needed
  after the second)
* If the final X.Y.0 release is delayed beyond the August target date, the
  subsequent release series is not affected, and will still be scheduled for
  August (now slightly less than two years later).

In addition to allowing more time for end user feedback on the release
candidate, this adjusted policy also provides additional time for maintainers
of Python projects to build and publish pre-built wheel archives for the new
stable release series, significantly improving the initial user experience of
the X.Y.0 release.


Removal of routine alpha releases
---------------------------------

Due to the proposed policy changes for beta releases and release candidates,
the traditional CPython alpha release period would no longer serve a clear
purpose, and as such would be skipped, with the first pre-release in a new
series being X.Y.0b1.

However, when it seems appropriate to do so, release managers may choose to
declare that a scheduled beta release will be marked as an alpha release
instead. For example, a release manager may choose to do this when major changes
have been made to the release management tooling, when a particularly
large feature has landed that the release manager views as having a high chance
of introducing unintended side effects, or when a new interface has been added
to the stable C ABI that is only useful when it appears as part of that ABI
(e.g. it may be enabling a new capability in the stable ABI that the full ABI
already supports in a different way), and requires further testing and design
feedback before being locked down in a beta release.

In such cases, the beta serial number would be retained, but the release would
be marked as an alpha release instead, and hence the "not for production use"
caveat would apply.

(The serial number would still be reset to 1 when entering the release
candidate phase - only alpha and beta release would share a serial number
sequence)


Changes to management of the CPython stable C ABI
-------------------------------------------------

The CPython stable ABI [5_] makes the commitment that binary extension modules
built against any particular CPython release will continue to work on future
CPython releases that support the same stable ABI version (this version is
currently ``abi3``).

Under the proposed rolling beta release model, this commitment would be extended
to also apply to the beta releases: once an intentional addition to the ``abi3``
stable ABI for the upcoming Python version has been shipped in a beta release,
then it will not be removed from future releases for as long as the ``abi3``
stable ABI remains supported.

Two main mechanisms will be available for obtaining community feedback on
additions to the stable ABI:

* the preferred mechanism will be to add new APIs to the full CPython API first,
  and only promote them to the stable ABI after they have been included in at
  least one published beta release and received relevant user feedback
* for APIs where that approach is unavailable for some reason (e.g. some API
  additions may serve no useful purpose when the full CPython API is available),
  then developers may request that the release manager mark the next release
  as an alpha release, and attempt to obtain further feedback that way


As a slight readability and usability improvement, this PEP also proposes the
introduction of aliases for each major stable ABI version::

    #define Py_LIMITED_API_3_3 0x03030000
    #define Py_LIMITED_API_3_4 0x03040000
    #define Py_LIMITED_API_3_5 0x03050000
    #define Py_LIMITED_API_3_6 0x03060000
    #define Py_LIMITED_API_3_7 0x03070000
    #define Py_LIMITED_API_3_8 0x03080000
    #define Py_LIMITED_API_3_9 0x03090000
    // etc...

These would be used both in extension module code to set the target ABI
version::

    #define Py_LIMITED_API Py_LIMITED_API_3_8

And also in the CPython interpreter implementation to check which symbols should
be made available::

    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9
    // A Python 3.9+ addition to the stable ABI would appear here
    #endif

The documentation for the rolling beta releases and the stable C ABI would make
it clear that extension modules built against the stable ABI in a later beta
release may not load correctly on earlier alpha or beta releases.

The documentation for alpha releases and the stable C ABI would make it clear
that even extension modules built against the stable ABI in an alpha release
release may not load correctly on any other release.


Changes to management of the CPython version-specific ABI
---------------------------------------------------------

The CPython version-specific ABI has long operated under a policy where binary
compatibility only applies within a release series after the ABI has been
declared frozen, and only source compatibility applies between different
release series.

This policy means that extension modules built against CPython pre-releases
prior to the ABI freeze for that release series may not actually load correctly
on the final release.

This is due to the fact that the extension module may be relying on provisional
or previously deprecated interfaces that were changed or removed in a later
alpha or beta release, or it may be due to public structures used by the
extension module changing size due to the addition of new fields.

Historically, adoption of alpha and beta releases has been low enough that this
hasn't really been a problem in practice. However, this PEP proposes to actively
encourage widespread operational use of beta releases, which makes it desirable
to ensure that users of those releases won't inadvertently publish binary
extension modules that cause segfaults for users running the release candidates
and final releases.

To that end, this PEP proposes amending the extension module ``SOABI`` marker
on non-Windows systems to include a new "p" flag for CPython pre-releases, and
only switch back to omitting that flag once the ABI for that particular X.Y.0
version has been frozen on entry to the release candidate stage.

With this change, alpha and beta releases of 3.9.0 would get an SOABI tag of
``cpython-39p``, while all release candidates and final builds (for both 3.9.0
and later 3.9.x releases) would get an unqualified SOABI tag of ``cpython-39``

Debug builds would still add the "d" to the end of the tag, giving
``cpython-39pd`` for debug builds of pre-releases.

On Windows systems, the suffix for tagged ``pyd`` files in pre-release builds
would include "p" as a pre-release marker immediately after the version number,
giving markers like "cp39p-win_amd64".

A proposed reference implementation for this change is available at [4_] (Note:
at time of writing, that implementation had not yet been tested on Windows).


Example Future Release Schedules
================================

Under this proposal, Python 3.9.0b1 would be released in December 2019, two
months after the Python 3.8.0 baseline feature release in October 2019.

The 3.9.0b2 release would then follow 2 months later in February 2020,
continuing through to 3.9.0b9 in April 2021.

3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then
the full release published as 3.9.0 in August 2021.

The cycle would start over again in October 2021, with the publication
of 3.10.0b1.

Assuming maintenance releases of 3.9.x were also to occur every other month
(offset from the 3.10.0 beta releases), the overall release timeline
would look like:

* 2019-12: 3.9.0b1
* ... beta releases every other month
* 2021-04: 3.9.0b9
* 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2021-07: 3.9.0rc2
* 2021-08: 3.9.0
* 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
* 2021-10: 3.10.0b1
* 2021-11: 3.9.2
* 2021-12: 3.10.0b2
* ... beta and maintenance releases every other month
* 2023-04: 3.10.0b10
* 2023-05: 3.9.11
* 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2023-07: 3.10.0rc2, 3.9.12
* 2023-08: 3.10.0
* 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
* 2023-10: 3.11.0b1
* ... etc


(The exact schedule of maintenance releases would be up to the release team -
alternating months with the rolling beta releases is the proposed target)

There are always two or three active maintenance branches in this model,
which preserves the status quo in that respect.


Caveats and Limitations
=======================

Actual release dates may be scheduled up to a month earlier or later at
the discretion of the release manager, based on release team availability, and
the timing of other events (e.g. PyCon US, or the annual core developer
sprints). However, as one goal of the proposal is to provide a consistent
release cadence, adjustments should ideally be rare.

Within a release series, the exact frequency of maintenance releases would
still be up to the release manager and the binary release team; this PEP
only proposes an expected cadence for pre-releases and X.Y.0 releases.

However, for the sake of the example timelines, the PEP assumes maintenance
releases every other month, allowing them to alternate months with the rolling
beta releases.


Design Discussion
=================

Why rolling beta releases over simply doing more frequent X.Y.0 releases?
-------------------------------------------------------------------------

For large parts of Python's user base, *availability* of new CPython feature
releases isn't the limiting factor on their adoption of those new releases
(this effect is visible in such metrics as PyPI download metadata).

As such, any proposal based on speeding up full feature releases needs to strike
a balance between meeting the needs of users who would be adopting each release
as it became available, and those that would now be in a position of adopting
every 2nd, 3rd, or 4th release, rather than being able to migrate to almost
every release at some point within its lifecycle.

This proposal aims to approach the problem from a different angle by defining a
*new* production-ready release stream that is more specifically tailored to the
interests of operating environments that are able to consume new releases as
fast as the CPython core team is prepared to produce them.


Why rolling beta releases rather than rolling alpha releases?
----------------------------------------------------------------

The code quality standards upheld by the CPython code review process and
BuildBot fleet make the "beta" label more suitable than the "alpha" label.

The "production ready for some environments, but subject to change with limited
notice" caveat also aligns well with at least some uses of the "beta" term,
whereas "alpha" almost universally indicates "not yet ready for production".


Why rolling beta releases rather than something like "brisk release cadence"?
-----------------------------------------------------------------------------

Using the "b" initial for the proposed rolling releases is a design constraint
imposed by some of the pragmatic aspects of the way CPython version numbers are
published (specifically, alpha releases, beta releases, and release candidates
are reported in some places using the strings "a", "b", and "c" respectively,
while in others they're reported using the hex digits ``0xA``, ``0xB``, and
``0xC``, and we want to preserve that, while also ensuring that the rolling
releases are all sorted after any alpha releases, and before the first release
candidate).

However, there isn't anything forcing us to say that the "b" stands for "beta".

That means that if we wanted to increase adoption amongst folks that were
only being put off by the "beta" label, then it could make sense to rebrand the
rolling beta releases as "brisk" releases, emphasising the rate of release over
the fact that the release includes APIs and ABIs that aren't necessarily stable
yet.

In the near term, limiting adoption to folks that are comfortable with the
"beta" label would be a good thing, as initial adopters are likely to
encounter unexpected consequences at the level of the wider Python ecosystem,
and would need to take an active part in getting those issues resolved.

Moving away from the "beta" naming would then become an option to keep in mind
for the future (with "brisk releases" as a potential alternative name).


Why rolling beta releases rather than alternating between stable and unstable release series?
---------------------------------------------------------------------------------------------

Rather than using the beta period for rolling releases, another option would be
to alternate between traditional stable releases (for 3.8.x, 3.10.x, etc), and
release series that used the new rolling release cadence (for 3.9.x, 3.11.x,
etc).

This idea suffers from the same core problem as PEP 598 and PEP 602: it imposes
changes on end users that are happy with the status quo without offering them
any clear compensating benefit.

It's also affected by one of the main concerns raised against PEP 598: at least
some core developers and end users strongly prefer that no particular semantics
be assigned to the *value* of any of the numbers in a release version. These
community members instead prefer that all the semantic significance be
associated with the *position* within the release number that is changing.

By contrast, the rolling beta release proposal aims to address that concern by
ensuring that the proposed changes in policy all revolve around whether a
particular release is an alpha release, beta release, release candidate, or
final release.


Why not use Calendar Versioning for the rolling release stream?
---------------------------------------------------------------

Steve Dower's initial write-up of this proposal [1_] suggested the use of
calendar versioning for the rolling release stream (so the first rolling
pre-release after Python 3.8.0 would have been Python 2019.12 rather than
3.9.0b1).

Paul Moore pointed out [2_] two major practical problems with that proposal:

* it isn't going to be clear to users of the calendar-based versions where they
  stand in relation to the traditionally numbered versions
* it breaks ``Python-Requires`` metadata processing in packaging tools with
  no clear way of fixing it reliably (since all calendar versions would appear
  as newer than any standard version)

This PEP aims to address both of those problems by using the established beta
version numbers for the rolling releases.

As an example, consider the following question: "Does Python 2021.12 include
all the new features released in Python 3.9.0?". With calendar versioning on
the rolling releases, that's impossible to answer without consulting a release
calendar to see when 3.9.0rc1 was branched off from the rolling release series.

By contrast, the equivalent question for rolling beta releases is
straightforward to answer: "Does Python 3.10.0b2 include all the new features
released in Python 3.9.0?". Just from formulating the question, the answer is
clearly "Yes, unless they were provisional features that got removed".

The beta numbering approach also avoids other questions raised by the calendar
versioning concept, such as how ``sys.version_info``, ``PY_VERSION_HEX``,
``site-packages`` directory naming, and installed Python binary and extension
module naming would work.


How would users of the rolling beta releases detect API changes?
----------------------------------------------------------------

When adding new features, core developers would be strongly encouraged to
support feature detection and graceful fallback to alternative approaches via
mechanisms that don't rely on either ``sys.version_info`` or runtime code object
introspection.

In most cases, a simple ``hasattr`` check on the affected module will serve this
purpose, but when it doesn't, alternative approaches would be considered as part
of the feature addition. Prior art in this area includes the
``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available`` set,
and the various ``os.supports_*`` sets that the ``os`` module already offers for
platform dependent capability detection.

It would also be possible to add features that need to be explicitly enabled
via a ``__future__`` import when first included in the rolling beta releases,
even if that feature flag was subsequently enabled by default before its first
appearance in an X.Y.0 release candidate.

The rationale behind these approaches is that explicit detection/enabling like
this would make it straightforward for users of the rolling beta release stream
to notice when we remove or change provisional features
(e.g. ``from __future__`` imports break on compile if the feature flag no
longer exists), or to safely fall back on previous functionality.

The interpreter's rich attribute lookup machinery means we can also choose to
add warnings for provisional or deprecated imports and attributes that we don't
have any practical way to add for checks against the value of
``sys.version_info``.


Implications for CPython core development
-----------------------------------------

The major change for CPython core development is the need to keep the master
branch more consistently release ready.

While the main requirement for that would be to keep the stable BuildBot fleet
green, there would also be encouragement to keep the development version of
the documentation up to date for the benefit of users of the rolling beta
releases. This will include providing draft What's New entries for changes as
they are implemented, although the initial versions may be relatively sparse,
and then expanded based on feedback from beta release users.

On the specific topic of the stable ABI, most API designs will be able to go
through a process where they're first introduced as part of the full CPython
API (allowing changes between beta releases), and only promoted to the stable
ABI once developers are confident that the interface is genuinely stable.

It's only in rare cases where an API serves no useful purpose outside the
stable ABI that it may make sense to publish an alpha release containing a
provisional stable ABI addition rather than iterating on the design in the
provisional CPython API instead.


Implications for Python library development
-------------------------------------------

By using the "beta" labelling rather than the incremental feature release
numbering proposed in PEP 598, the hope would be that consumers of these
new rolling releases would realise that they're likely going to need to build
their own wheel archives from source, and will generally be more prone to
encountering library compatibility issues when updating to a new release.

Library authors who actually want to support the beta stream would have the
option of testing against the latest beta release in their pre-merge test
matrices (just as they test against the latest maintenance release of previously
published versions), with the CPython nightly builds offered by some CI
providers used solely in an advisory capacity for early detection of potential
compatibility problems.

Having a rolling beta release stream available may also make it more feasible
for more CI providers to offer a "CPython beta release" testing option. At the
moment, this feature is only available from CI providers that are willing and
able to put the necessary time and effort into creating, testing, and publishing
their own builds from the CPython master branch (e.g. [6_]).


Implications for the proposed Scientific Python ecosystem support period
------------------------------------------------------------------------

Based on discussions at SciPy 2019, NEP (NumPy Enhancement Proposal) 29 has
been drafted [3_] to propose a common convention across the Scientific Python
ecosystem for dropping support for older Python versions.

While the exact formulation of that policy is still being discussed, the initial
proposal is very simple: support any Python feature release published within
the last 42 months.

For an 18 month feature release cadence, that works out to always supporting at
least the two most recent feature releases, and then dropping support for all
X.Y.Z releases around 6 months after X.(Y+2).0 is released. This means there is
a 6 month period roughly every other year where the three most recent feature
releases are supported.

For a 12 month release cadence, it would work out to always supporting at
least the three most recent feature releases, and then dropping support for all
X.Y.Z releases around 6 months after X.(Y+3).0 is released. This means that
for half of each year, the four most recent feature releases would be supported.

For a 24 month release cadence, a 42 month support cycle works out to always
supporting at least the most recent feature release, and then dropping support
for all X.Y.Z releases around 18 months after X.(Y+1).0 is released.
This means there is a 6 month period every other year where only one feature
release is supported. Under the proposal in this PEP, that period would
correspond to the final few rolling beta releases and the release candidate
phase for the upcoming stable feature release.


Release cycle alignment for core development sprints
----------------------------------------------------

With the proposal in this PEP, it is expected that the focus of core
development sprints would shift slightly based on the current location
in the two year cycle.

In release years, the timing of PyCon US is suitable for new contributors to
work on bug fixes and smaller features before the first release candidate goes
out, while the Language Summit and core developer discussions can focus on
plans for the next release series.

The post-release core development sprint in release years will provide an
opportunity to incorporate feedback received on the release, either as part of
the next maintenance release (for bug fixes and feedback on provisional APIs),
or as part of the next release series. These sprints would also likely
correspond with the Steering Council elections for the next release cycle.

In non-release years, the focus for both events would just be on the upcoming
maintenance and beta releases. These less intense years would hopefully provide
an opportunity to tackle various process changes and infrastructure upgrades
without impacting the release candidate preparation process.


Release cycle alignment for prominent Linux distributions
---------------------------------------------------------

Some rolling release Linux distributions (e.g. Arch, Gentoo) may be in a
position to consume the new rolling beta releases proposed in this PEP, but it
is expected that most distributions would continue to use the established
releases.

The specific dates for those releases proposed in this PEP are chosen to align
with the feature freeze schedules for the annual October releases of the Ubuntu
and Fedora Linux distributions.

For both Fedora and Ubuntu, it means that the release candidate phase aligns
with the development period for a distro release, which is the ideal time for
them to test a new version and provide feedback on potential regressions and
compatibility concerns.

For Ubuntu, this also means that their April LTS releases will have benefited
from a full short-term release cycle using the new system Python version, while
still having that CPython release be open to upstream bug fixes for most of the
time until the next Ubuntu LTS release.

The one Linux release cycle alignment that is likely to be consistently poor
with the specific proposal in this PEP is with Debian, as that has been released
in the first half of odd-numbered years since 2005 (roughly 12 months offset
from Ubuntu LTS releases).

With the annual release proposal in PEP 602, both Debian and Ubuntu LTS would
consistently get a system Python version that is around 6 months old, but
would also consistently select different Python versions from each other.

With a two year cadence, and CPython releases in the latter half of the year,
they're likely to select the same version as each other, but one of them will
be choosing a CPython release that is more than 18 months behind the latest beta
releases by the time the Linux distribution ships.

If that situation does occur, and is deemed undesirable (but not sufficiently
undesirable for *Debian* to choose to adjust their release timing), then that's
where the additional complexity of the "incremental feature release" proposal
in PEP 598 may prove worthwhile.

(Moving CPython releases to the same half of the year as the Debian and Ubuntu
LTS releases would potentially help mitigate the problem, but also creates
new problems where a slip in the CPython release schedule could directly affect
the release schedule for a Linux distribution, or else result in a distribution
shipping a Python version that is *more* than 18 months old)


Implications for simple deployment environments
-----------------------------------------------

For the purposes of this PEP, a "simple" deployment environment is any use case
where it is straightforward to ensure that all target environments are updated
to a new Python release at the same time (or at least in advance of the rollout
of new higher level application versions), and any pre-release testing that
occurs need only target a single Python micro version.

The simplest such case would be scripting for personal use, where the testing
and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where the
same Python container is used in the CI pipeline as is used on deployment, and
any application that bundles its own Python runtime, rather than relying on a
pre-existing Python deployment on the target system.

For these use cases, there is a straightforward mechanism to minimise the
impact of this PEP: continue using the stable releases, and ignore the rolling
beta releases.

To actually adopt the rolling beta releases in these environments, the main
challenge will be handling the potential for extension module segfaults if the
CPython ABI changes in an incompatible way between beta releases.

If all extension modules in use target the stable ABI, then there's no problem,
and everything will work just as smoothly as it does on the stable releases.

Alternatively, "rebuild and recache all extension modules" could become a
standard activity undertaken as part of updating to each new beta release.

Finally, it would also be reasonable to just not worry about it until something
actually breaks, and then handle it like any other library compatibility issue
found in a new beta release.

Aside from extension module ABI compatibilty, the other main point of additional
complexity when using the rolling beta releases would be "roll-back"
compatibility for independently versioned features, such as pickle and SQLite,
where use of new or provisional features in the beta stream may create files
that are not readable by the stable release. Applications that use these
kinds of features and also require the ability to reliably roll-back to a
previous stable CPython release would, as today, be advised to avoid adopting
pre-release versions.


Implications for complex deployment environments
------------------------------------------------

For the purposes of this PEP, "complex" deployment environments are use cases
which don't meet the "simple deployment" criteria above. They may involve
multiple distinct versions of Python, use of a personalised build of Python,
or "gatekeepers" who are required to approve use of a new version prior to
deployment.

For example, organisations that install Python on their users' machines as part
of a standard operating environment fall into this category, as do those that
provide a standard build environment. Distributions such as conda-forge or
WinPython that provide collections of consistently built and verified packages
are impacted in similar ways.

These organisations tend to either prefer high stability (for example, all of
those who are happily using the system Python in a stable Linux distribution
like Debian, RHEL/CentOS, or Ubuntu LTS as their preferred Python environment)
or fast turnaround (for example, those who regularly contribute toward the
latest CPython pre-releases).

In some cases, both usage models may exist within the same organisation for
different purposes, such as:

* using a stable Python environment for mission critical systems, but allowing
  data scientists to use the latest available version for ad hoc data anaylsis
* a hardware manufacturer deploying a stable Python version as part of their
  production firmware, but using the latest available version in the development
  and execution of their automated integration tests

Under any release model, each new release of Python generates work for these
organisations. This work may involve legal, security or technical reviews of
Python itself, assessment and verification of impactful changes, reapplication
of patches, recompilation and testing of third-party dependencies, and
only then deployment.

Organisations that can take updates quickly should be able to make use of the
more frequent beta releases. While each update will still require similar
investigative work to what they require today, the volume of work required per
release should be reduced as each release will be more similar to the previous
than it is under the present model. One advantage of the proposed
release-every-2-months model is that organisations can choose their own adoption
cadence from adopting every beta release, to adopting one per quarter, or one
every 6 months, or one every year. Beyond that, it would likely make more sense
to continue using the stable releases instead.

For organisations with stricter evaluations or a preference for stability, the
longer release cycle for stable releases will reduce the annual effort required
to update, the longer release candidate period will allow more time to do
internal testing before the X.Y.0 release, and the greater use by others
during the beta period will provide more confidence in the initial releases.
Meanwhile, the organisation can confidently upgrade through maintenance
releases for a longer time without fear of breaking changes.


Acknowledgements
================

Thanks to Łukasz Langa for creating PEP 602 and prompting this discussion of
possible improvements to the CPython release cadence, and to Kyle Stanley
and h-vetinari for constructive feedback on the initial draft of this PEP.


References
==========

.. [1] Steve Dower's initial "Fast and Stable releases" proposal
       (https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/20)

.. [2] Paul Moore's initial comments on Steve's proposal
       (https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/37)

.. [3] NEP 29 proposes a common policy for dropping support of old Python versions
       (https://numpy.org/neps/nep-0029-deprecation_policy.html)

.. [4] Example implementation for a pre-release SOABI flag
       (https://github.com/ncoghlan/cpython/pull/3)

.. [5] CPython stable ABI documentation
       (https://docs.python.org/3/c-api/stable.html)

.. [6] Travis CI nightly CPython builds
       (https://docs.travis-ci.com/user/languages/python/#nightly-build-support)

Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.

..
  Local Variables:
  mode: indented-text
  indent-tabs-mode: nil
  sentence-end-double-space: t
  fill-column: 80
  coding: utf-8
  End: