887 lines
43 KiB
ReStructuredText
887 lines
43 KiB
ReStructuredText
|
PEP: 605
|
|||
|
Title: A rolling feature release stream for CPython
|
|||
|
Version: $Revision$
|
|||
|
Last-Modified: $Date$
|
|||
|
Author: Steve Dower <steve.dower@python.org>, Nick Coghlan <ncoghlan@gmail.com>
|
|||
|
Discussions-To: Link TBD
|
|||
|
Status: Draft
|
|||
|
Type: Informational
|
|||
|
Content-Type: text/x-rst
|
|||
|
Created: 20-Sep-2019
|
|||
|
Python-Version: 3.9
|
|||
|
Post-History: 30-Sep-2019
|
|||
|
|
|||
|
|
|||
|
Abstract
|
|||
|
========
|
|||
|
|
|||
|
Rather than proposing more frequent full CPython releases (as PEP 602 does),
|
|||
|
or a policy change to allow backwards compatible feature additions later in a
|
|||
|
release series (as PEP 598 does), this PEP instead proposes that we create a
|
|||
|
rolling stream of production-ready beta releases by amending CPython's
|
|||
|
pre-release management process as follows:
|
|||
|
|
|||
|
* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
|
|||
|
creation all correspond with the first release candidate for an X.Y.0 release
|
|||
|
* The X.Y.0 release candidate period is extended from 3 weeks to 2 months
|
|||
|
* Beta releases from the master git branch would occur every 2 months
|
|||
|
whenever there is no X.Y.0 release in the release candidate phase
|
|||
|
* Routine alpha releases are removed from the process
|
|||
|
* Release managers may still choose to mark particular releases as alpha
|
|||
|
releases if they feel such releases are appropriate. If any such alpha
|
|||
|
releases are made, it is expected that they would be in place of one of the
|
|||
|
scheduled beta releases
|
|||
|
* The deprecation policy for interfaces that have been included in an X.Y.0
|
|||
|
release or release candidate without being explicitly declared provisional
|
|||
|
remains unchanged (refer to PEP 387 for details)
|
|||
|
* Each new alpha or beta release may introduce new deprecation warnings for
|
|||
|
interfaces that appeared in the previous X.Y release series
|
|||
|
* Each new alpha or beta release may change or remove interfaces that either did
|
|||
|
not appear in the previous X.Y release series, or emitted a relevant
|
|||
|
deprecation warning in that series
|
|||
|
* Additions to the `abi3` stable C ABI are permanent once they have been
|
|||
|
declared to be part of that ABI in at least one beta release
|
|||
|
* All other newly added interfaces that have appeared only in alpha or beta
|
|||
|
releases are considered provisional, and may be changed in incompatible ways
|
|||
|
without requiring a preceding deprecation warning
|
|||
|
|
|||
|
As part of this change, the current "not for production use" guidance given for
|
|||
|
beta releases would be amended to state "suitable for production use only in
|
|||
|
environments with sufficiently robust compatibility testing and operational
|
|||
|
monitoring capabilities".
|
|||
|
|
|||
|
This PEP also proposes that the frequency of X.Y.0 releases be adjusted to
|
|||
|
begin each new release series in August every two years (starting in 2021,
|
|||
|
around two years after the release of Python 3.8.0).
|
|||
|
|
|||
|
|
|||
|
Motivation
|
|||
|
==========
|
|||
|
|
|||
|
The current CPython pre-release and release management processes were developed
|
|||
|
in an era where automated continuous integration and operational monitoring
|
|||
|
systems were still relatively immature. Since that time, many organisations
|
|||
|
have adopted deployment models that allow them to incorporate new CPython
|
|||
|
feature releases without adding substantially more risk than they incur for any
|
|||
|
other code change. Newer deployment models, such as lightweight task specific
|
|||
|
application containers, also make it easier to combine an application with a
|
|||
|
language runtime in a CI pipeline, and then keep them together until the entire
|
|||
|
container image is later replaced by an updated one.
|
|||
|
|
|||
|
In light of those changes in the wider environment, PEP 602 has proposed
|
|||
|
reducing the feature delivery latency for the Python standard library and
|
|||
|
CPython reference interpreter by increasing the frequency of CPython feature
|
|||
|
releases from every 18-24 months to instead occur every 12 months.
|
|||
|
|
|||
|
Unfortunately, for many organisations, the cost of adopting a new Python release
|
|||
|
doesn't automatically scale down with a reduced number of changes in the release,
|
|||
|
as the primary costs aren't associated with resolving any discovered issues;
|
|||
|
the primary costs are associated with the *search* for issues. This search may
|
|||
|
involve manual testing of software systems, human review of written materials,
|
|||
|
and other activities where the time required scales with the size of the
|
|||
|
existing system, rather than with the number of changes between the versions of
|
|||
|
Python.
|
|||
|
|
|||
|
For third party library developers, costs are primarily associated with the
|
|||
|
*number* of distinct Python versions in widespread usage. This currently tends
|
|||
|
to be influenced by a combination of which releases are still actively
|
|||
|
maintained by python-dev, and which releases are the latest versions offered
|
|||
|
by particular redistributors (with the Debian, Ubuntu LTS, and RHEL/CentOS
|
|||
|
system Python versions being particularly popular development targets). In
|
|||
|
addition to the basic CI cost of testing against more Python versions, having
|
|||
|
more variants in widespread use can make it more difficult to determine when a
|
|||
|
fault report is an actual error in the project, or an issue in the reporting
|
|||
|
user's environment.
|
|||
|
|
|||
|
PEP 602 proposes that affected organisations and projects simply switch to
|
|||
|
adopting every second or third CPython release, rather than attempting to adopt
|
|||
|
every release, but that creates its own set of new problems to be resolved, both
|
|||
|
practical (e.g. deprecations would need to cover more than one release if we're
|
|||
|
expecting users to routinely skip releases) and cultural (e.g. with a larger
|
|||
|
number of versions in active use, there is a much higher chance that open source
|
|||
|
library maintainers will receive bug reports that only occur on Python versions
|
|||
|
that they're not using themselves).
|
|||
|
|
|||
|
PEP 598 was an initial attempt by one of the authors of this PEP to propose
|
|||
|
an alternative scheme to reduce feature delivery latency by adopting a
|
|||
|
semantic versioning style policy that allowed for the incremental delivery of
|
|||
|
backwards compatible features within a release series, until that series
|
|||
|
reached feature complete status. That variant still had the undesirable
|
|||
|
consequence of imposing visible changes on end users that are happy enough
|
|||
|
with the current release management model.
|
|||
|
|
|||
|
This PEP takes the view that both PEP 598 and PEP 602 share a common flaw: they
|
|||
|
are attempting to satisfy the needs of two quite distinct audiences within the
|
|||
|
constraints of a single release model, which results in conflicting design
|
|||
|
requirements, and the need for awkward trade-offs between those conflicting
|
|||
|
requirements. The proposal in this PEP aims to avoid that flaw by proposing the
|
|||
|
creation of two *distinct* production-ready release streams, with the existing
|
|||
|
release stream being largely left alone, while the new release stream is
|
|||
|
tailored towards the audience that would most benefit from a reduction in
|
|||
|
feature delivery latency.
|
|||
|
|
|||
|
|
|||
|
Aims of this Proposal
|
|||
|
=====================
|
|||
|
|
|||
|
The core of the proposal in this PEP is changing the beta phase of the CPython
|
|||
|
pre-release process to produce a rolling stream of releases at a regular
|
|||
|
cadence, and to ensure those builds offer a sufficient level of stability as
|
|||
|
to be suitable for use in appropriately managed production systems.
|
|||
|
|
|||
|
By adopting this approach, the proposal aims to provide an improved outcome
|
|||
|
for almost all Python users and contributors:
|
|||
|
|
|||
|
* for users of the new release stream, targeting the beta phase allows for even
|
|||
|
lower feature delivery latency than the annual cadence proposed in PEP 602;
|
|||
|
* for core developers working on new features, increased frequency and adoption
|
|||
|
of the beta releases should improve pre-release feedback cycles;
|
|||
|
* for users of the established release stream, the increased adoption and
|
|||
|
improved feedback cycles during the pre-release period should result in
|
|||
|
increased feature maturity at the time of its first X.Y.0 release, as well
|
|||
|
as higher levels of ecosystem readiness;
|
|||
|
* for Python library maintainers, the rolling stream of beta releases will
|
|||
|
hopefully provide more opportunities to identify and resolve design issues
|
|||
|
before they make it into a full stable release than is offered by the current
|
|||
|
pre-release management process; and
|
|||
|
* for developers of alternative Python implementations, the proposed rolling
|
|||
|
beta releases provide an additional incentive for extension module authors
|
|||
|
to migrate from the full CPython ABI to the Python stable ABI, which may
|
|||
|
over time make more of the ecosystem compatible with implementations that
|
|||
|
don't fully emulate the CPython C API.
|
|||
|
|
|||
|
That said, it is acknowledged that not all the outcomes of this proposal will be
|
|||
|
beneficial for all members of the wider Python ecosystem:
|
|||
|
|
|||
|
* for Python library maintainers, both this PEP and PEP 602 would likely
|
|||
|
result in user pressure to support the faster release cadence. While this PEP
|
|||
|
attempts to mitigate that by marking the faster releases with the "beta" label
|
|||
|
and removing the expectation to publish pre-built wheel archives, and PEP 602
|
|||
|
attempts to mitigate it by keeping the minimum time between full releases at
|
|||
|
12 months, it isn't possible to eliminate this downside completely;
|
|||
|
* for third party extension module maintainers, both this PEP and PEP 602 would
|
|||
|
likely result in user pressure to start supporting the stable ABI in order to
|
|||
|
provide wheel archives that work on the new version as soon as it is
|
|||
|
available. Whether that's a net negative or not will depend on how the request
|
|||
|
is presented to them (it could be a positive if the request comes in the form
|
|||
|
of a courteous contribution to their project from a developer interested in
|
|||
|
supporting the rolling beta releases);
|
|||
|
* for some users of the established release stream that rely on the
|
|||
|
availability of pre-built wheel archives, switching to adopting a new release
|
|||
|
every 12 months may be an acceptable rate increase, while moving consistently
|
|||
|
to the 24 month end of the historical 18-24 month cadence would be an
|
|||
|
undesirable rate reduction relative to the 18 month cycle used for recent
|
|||
|
releases. Whether this proposal would be a net negative for these users in the
|
|||
|
long run would depend on how many projects migrated over to targeting the
|
|||
|
Python stable ABI and hence produced wheel archives that would still be usable
|
|||
|
on the rolling beta releases.
|
|||
|
|
|||
|
|
|||
|
Proposal
|
|||
|
========
|
|||
|
|
|||
|
The majority of the proposed changes in this PEP only affect the handling of
|
|||
|
pre-release versions. The one change affecting full release versions is a
|
|||
|
suggested change to their cadence.
|
|||
|
|
|||
|
Changes to beta release policy, phase duration, and cadence
|
|||
|
-----------------------------------------------------------
|
|||
|
|
|||
|
This PEP proposes that the policy for beta releases be adjusted to be a
|
|||
|
combination of the policies for alpha and beta releases:
|
|||
|
|
|||
|
* as with current beta releases, the stable BuildBot fleet is expected to be
|
|||
|
green prior to preparation and publication of the beta release
|
|||
|
* as with current beta releases, the release manager is expected to review
|
|||
|
open release blocker issues prior to preparation and publication of the beta
|
|||
|
release
|
|||
|
* as with current beta releases, any additions to the `abi3` stable C ABI would
|
|||
|
be expected to become a permanent part of that ABI unless and until that
|
|||
|
stable ABI version is retired completely (Note: there are no current plans
|
|||
|
to increment the stable ABI version)
|
|||
|
* as with current alpha releases, beta releases would *not* be considered
|
|||
|
feature complete for the next X.Y.0 release
|
|||
|
* as with current alpha releases, all APIs added since the last CPython feature
|
|||
|
release (other than additions to the stable C ABI) would be considered
|
|||
|
provisional
|
|||
|
* as with current alpha releases, beta releases would be prepared and published
|
|||
|
from the master development branch
|
|||
|
|
|||
|
Rather than being released monthly for a period of a few months while preparing
|
|||
|
a new X.Y.0 release, beta releases would instead be consistently published every
|
|||
|
two months.
|
|||
|
|
|||
|
The only time this would not be the case is during the two month release
|
|||
|
candidate period for an upcoming X.Y.0 release, as described in the following
|
|||
|
section.
|
|||
|
|
|||
|
|
|||
|
Changes to release candidate policy, phase duration, and cadence
|
|||
|
----------------------------------------------------------------
|
|||
|
|
|||
|
Given the proposed changes to the beta release phase, the release candidate
|
|||
|
phase would see the following related adjustments:
|
|||
|
|
|||
|
* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
|
|||
|
creation would all correspond with the creation of X.Y.0rc1 (currently these
|
|||
|
occur across a mixture of X.Y.0b1, the last beta release, and X.Y.0rc1)
|
|||
|
* The X.Y.0 release candidate period would be extended from 3 weeks to 2 months
|
|||
|
* There would normally be two release candidates issued a month apart, but
|
|||
|
additional candidates may be published at the release manager's discretion
|
|||
|
* The final X.Y.0 release would occur between 1 and 4 weeks after the final
|
|||
|
release candidate (depending if additional release candidates were needed
|
|||
|
after the second)
|
|||
|
* If the final X.Y.0 release is delayed beyond the August target date, the
|
|||
|
subsequent release series is not affected, and will still be scheduled for
|
|||
|
August (now slightly less than two years later).
|
|||
|
|
|||
|
In addition to allowing more time for end user feedback on the release
|
|||
|
candidate, this adjusted policy also provides additional time for maintainers
|
|||
|
of Python projects to build and publish pre-built wheel archives for the new
|
|||
|
stable release series, significantly improving the initial user experience of
|
|||
|
the X.Y.0 release.
|
|||
|
|
|||
|
|
|||
|
Removal of routine alpha releases
|
|||
|
---------------------------------
|
|||
|
|
|||
|
Due to the proposed policy changes for beta releases and release candidates,
|
|||
|
the traditional CPython alpha release period would no longer serve a clear
|
|||
|
purpose, and as such would be skipped, with the first pre-release in a new
|
|||
|
series being X.Y.0b1.
|
|||
|
|
|||
|
However, when it seems appropriate to do so, release managers may choose to
|
|||
|
declare that a scheduled beta release will be marked as an alpha release
|
|||
|
instead. For example, a release manager may choose to do this when major changes
|
|||
|
have been made to the release management tooling, when a particularly
|
|||
|
large feature has landed that the release manager views as having a high chance
|
|||
|
of introducing unintended side effects, or when a new interface has been added
|
|||
|
to the stable C ABI that is only useful when it appears as part of that ABI
|
|||
|
(e.g. it may be enabling a new capability in the stable ABI that the full ABI
|
|||
|
already supports in a different way), and requires further testing and design
|
|||
|
feedback before being locked down in a beta release.
|
|||
|
|
|||
|
In such cases, the beta serial number would be retained, but the release would
|
|||
|
be marked as an alpha release instead, and hence the "not for production use"
|
|||
|
caveat would apply.
|
|||
|
|
|||
|
(The serial number would still be reset to 1 when entering the release
|
|||
|
candidate phase - only alpha and beta release would share a serial number
|
|||
|
sequence)
|
|||
|
|
|||
|
|
|||
|
Changes to management of the CPython stable C ABI
|
|||
|
-------------------------------------------------
|
|||
|
|
|||
|
The CPython stable ABI [5_] makes the commitment that binary extension modules
|
|||
|
built against any particular CPython release will continue to work on future
|
|||
|
CPython releases that support the same stable ABI version (this version is
|
|||
|
currently ``abi3``).
|
|||
|
|
|||
|
Under the proposed rolling beta release model, this commitment would be extended
|
|||
|
to also apply to the beta releases: once an intentional addition to the ``abi3``
|
|||
|
stable ABI for the upcoming Python version has been shipped in a beta release,
|
|||
|
then it will not be removed from future releases for as long as the ``abi3``
|
|||
|
stable ABI remains supported.
|
|||
|
|
|||
|
Two main mechanisms will be available for obtaining community feedback on
|
|||
|
additions to the stable ABI:
|
|||
|
|
|||
|
* the preferred mechanism will be to add new APIs to the full CPython API first,
|
|||
|
and only promote them to the stable ABI after they have been included in at
|
|||
|
least one published beta release and received relevant user feedback
|
|||
|
* for APIs where that approach is unavailable for some reason (e.g. some API
|
|||
|
additions may serve no useful purpose when the full CPython API is available),
|
|||
|
then developers may request that the release manager mark the next release
|
|||
|
as an alpha release, and attempt to obtain further feedback that way
|
|||
|
|
|||
|
|
|||
|
As a slight readability and usability improvement, this PEP also proposes the
|
|||
|
introduction of aliases for each major stable ABI version::
|
|||
|
|
|||
|
#define Py_LIMITED_API_3_3 0x03030000
|
|||
|
#define Py_LIMITED_API_3_4 0x03040000
|
|||
|
#define Py_LIMITED_API_3_5 0x03050000
|
|||
|
#define Py_LIMITED_API_3_6 0x03060000
|
|||
|
#define Py_LIMITED_API_3_7 0x03070000
|
|||
|
#define Py_LIMITED_API_3_8 0x03080000
|
|||
|
#define Py_LIMITED_API_3_9 0x03090000
|
|||
|
// etc...
|
|||
|
|
|||
|
These would be used both in extension module code to set the target ABI
|
|||
|
version::
|
|||
|
|
|||
|
#define Py_LIMITED_API Py_LIMITED_API_3_8
|
|||
|
|
|||
|
And also in the CPython interpreter implementation to check which symbols should
|
|||
|
be made available::
|
|||
|
|
|||
|
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9
|
|||
|
// A Python 3.9+ addition to the stable ABI would appear here
|
|||
|
#endif
|
|||
|
|
|||
|
The documentation for the rolling beta releases and the stable C ABI would make
|
|||
|
it clear that extension modules built against the stable ABI in a later beta
|
|||
|
release may not load correctly on earlier alpha or beta releases.
|
|||
|
|
|||
|
The documentation for alpha releases and the stable C ABI would make it clear
|
|||
|
that even extension modules built against the stable ABI in an alpha release
|
|||
|
release may not load correctly on any other release.
|
|||
|
|
|||
|
|
|||
|
Changes to management of the CPython version-specific ABI
|
|||
|
---------------------------------------------------------
|
|||
|
|
|||
|
The CPython version-specific ABI has long operated under a policy where binary
|
|||
|
compatibility only applies within a release series after the ABI has been
|
|||
|
declared frozen, and only source compatibility applies between different
|
|||
|
release series.
|
|||
|
|
|||
|
This policy means that extension modules built against CPython pre-releases
|
|||
|
prior to the ABI freeze for that release series may not actually load correctly
|
|||
|
on the final release.
|
|||
|
|
|||
|
This is due to the fact that the extension module may be relying on provisional
|
|||
|
or previously deprecated interfaces that were changed or removed in a later
|
|||
|
alpha or beta release, or it may be due to public structures used by the
|
|||
|
extension module changing size due to the addition of new fields.
|
|||
|
|
|||
|
Historically, adoption of alpha and beta releases has been low enough that this
|
|||
|
hasn't really been a problem in practice. However, this PEP proposes to actively
|
|||
|
encourage widespread operational use of beta releases, which makes it desirable
|
|||
|
to ensure that users of those releases won't inadvertently publish binary
|
|||
|
extension modules that cause segfaults for users running the release candidates
|
|||
|
and final releases.
|
|||
|
|
|||
|
To that end, this PEP proposes amending the extension module ``SOABI`` marker
|
|||
|
on non-Windows systems to include a new "p" flag for CPython pre-releases, and
|
|||
|
only switch back to omitting that flag once the ABI for that particular X.Y.0
|
|||
|
version has been frozen on entry to the release candidate stage.
|
|||
|
|
|||
|
With this change, alpha and beta releases of 3.9.0 would get an SOABI tag of
|
|||
|
``cpython-39p``, while all release candidates and final builds (for both 3.9.0
|
|||
|
and later 3.9.x releases) would get an unqualified SOABI tag of ``cpython-39``
|
|||
|
|
|||
|
Debug builds would still add the "d" to the end of the tag, giving
|
|||
|
``cpython-39pd`` for debug builds of pre-releases.
|
|||
|
|
|||
|
On Windows systems, the suffix for tagged ``pyd`` files in pre-release builds
|
|||
|
would include "p" as a pre-release marker immediately after the version number,
|
|||
|
giving markers like "cp39p-win_amd64".
|
|||
|
|
|||
|
A proposed reference implementation for this change is available at [4_] (Note:
|
|||
|
at time of writing, that implementation had not yet been tested on Windows).
|
|||
|
|
|||
|
|
|||
|
Example Future Release Schedules
|
|||
|
================================
|
|||
|
|
|||
|
Under this proposal, Python 3.9.0b1 would be released in December 2019, two
|
|||
|
months after the Python 3.8.0 baseline feature release in October 2019.
|
|||
|
|
|||
|
The 3.9.0b2 release would then follow 2 months later in February 2020,
|
|||
|
continuing through to 3.9.0b9 in April 2021.
|
|||
|
|
|||
|
3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then
|
|||
|
the full release published as 3.9.0 in August 2021.
|
|||
|
|
|||
|
The cycle would start over again in October 2021, with the publication
|
|||
|
of 3.10.0b1.
|
|||
|
|
|||
|
Assuming maintenance releases of 3.9.x were also to occur every other month
|
|||
|
(offset from the 3.10.0 beta releases), the overall release timeline
|
|||
|
would look like:
|
|||
|
|
|||
|
* 2019-12: 3.9.0b1
|
|||
|
* ... beta releases every other month
|
|||
|
* 2021-04: 3.9.0b9
|
|||
|
* 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
|
|||
|
* 2021-07: 3.9.0rc2
|
|||
|
* 2021-08: 3.9.0
|
|||
|
* 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
|
|||
|
* 2021-10: 3.10.0b1
|
|||
|
* 2021-11: 3.9.2
|
|||
|
* 2021-12: 3.10.0b2
|
|||
|
* ... beta and maintenance releases every other month
|
|||
|
* 2023-04: 3.10.0b10
|
|||
|
* 2023-05: 3.9.11
|
|||
|
* 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
|
|||
|
* 2023-07: 3.10.0rc2, 3.9.12
|
|||
|
* 2023-08: 3.10.0
|
|||
|
* 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
|
|||
|
* 2023-10: 3.11.0b1
|
|||
|
* ... etc
|
|||
|
|
|||
|
|
|||
|
(The exact schedule of maintenance releases would be up to the release team -
|
|||
|
alternating months with the rolling beta releases is the proposed target)
|
|||
|
|
|||
|
There are always two or three active maintenance branches in this model,
|
|||
|
which preserves the status quo in that respect.
|
|||
|
|
|||
|
|
|||
|
Caveats and Limitations
|
|||
|
=======================
|
|||
|
|
|||
|
Actual release dates may be scheduled up to a month earlier or later at
|
|||
|
the discretion of the release manager, based on release team availability, and
|
|||
|
the timing of other events (e.g. PyCon US, or the annual core developer
|
|||
|
sprints). However, as one goal of the proposal is to provide a consistent
|
|||
|
release cadence, adjustments should ideally be rare.
|
|||
|
|
|||
|
Within a release series, the exact frequency of maintenance releases would
|
|||
|
still be up to the release manager and the binary release team; this PEP
|
|||
|
only proposes an expected cadence for pre-releases and X.Y.0 releases.
|
|||
|
|
|||
|
However, for the sake of the example timelines, the PEP assumes maintenance
|
|||
|
releases every other month, allowing them to alternate months with the rolling
|
|||
|
beta releases.
|
|||
|
|
|||
|
|
|||
|
Design Discussion
|
|||
|
=================
|
|||
|
|
|||
|
Why rolling beta releases over simply doing more frequent X.Y.0 releases?
|
|||
|
-------------------------------------------------------------------------
|
|||
|
|
|||
|
For large parts of Python's user base, *availability* of new CPython feature
|
|||
|
releases isn't the limiting factor on their adoption of those new releases
|
|||
|
(this effect is visible in such metrics as PyPI download metadata).
|
|||
|
|
|||
|
As such, any proposal based on speeding up full feature releases needs to strike
|
|||
|
a balance between meeting the needs of users who would be adopting each release
|
|||
|
as it became available, and those that would now be in a position of adopting
|
|||
|
every 2nd, 3rd, or 4th release, rather than being able to migrate to almost
|
|||
|
every release at some point within its lifecycle.
|
|||
|
|
|||
|
This proposal aims to approach the problem from a different angle by defining a
|
|||
|
*new* production-ready release stream that is more specifically tailored to the
|
|||
|
interests of operating environments that are able to consume new releases as
|
|||
|
fast as the CPython core team is prepared to produce them.
|
|||
|
|
|||
|
|
|||
|
Why rolling beta releases rather than rolling alpha releases?
|
|||
|
----------------------------------------------------------------
|
|||
|
|
|||
|
The code quality standards upheld by the CPython code review process and
|
|||
|
BuildBot fleet make the "beta" label more suitable than the "alpha" label.
|
|||
|
|
|||
|
The "production ready for some environments, but subject to change with limited
|
|||
|
notice" caveat also aligns well with at least some uses of the "beta" term,
|
|||
|
whereas "alpha" almost universally indicates "not yet ready for production".
|
|||
|
|
|||
|
|
|||
|
Why rolling beta releases rather than something like "brisk release cadence"?
|
|||
|
-----------------------------------------------------------------------------
|
|||
|
|
|||
|
Using the "b" initial for the proposed rolling releases is a design constraint
|
|||
|
imposed by some of the pragmatic aspects of the way CPython version numbers are
|
|||
|
published (specifically, alpha releases, beta releases, and release candidates
|
|||
|
are reported in some places using the strings "a", "b", and "c" respectively,
|
|||
|
while in others they're reported using the hex digits ``0xA``, ``0xB``, and
|
|||
|
``0xC``, and we want to preserve that, while also ensuring that the rolling
|
|||
|
releases are all sorted after any alpha releases, and before the first release
|
|||
|
candidate).
|
|||
|
|
|||
|
However, there isn't anything forcing us to say that the "b" stands for "beta".
|
|||
|
|
|||
|
That means that if we wanted to increase adoption amongst folks that were
|
|||
|
only being put off by the "beta" label, then it could make sense to rebrand the
|
|||
|
rolling beta releases as "brisk" releases, emphasising the rate of release over
|
|||
|
the fact that the release includes APIs and ABIs that aren't necessarily stable
|
|||
|
yet.
|
|||
|
|
|||
|
In the near term, limiting adoption to folks that are comfortable with the
|
|||
|
"beta" label would be a good thing, as initial adopters are likely to
|
|||
|
encounter unexpected consequences at the level of the wider Python ecosystem,
|
|||
|
and would need to take an active part in getting those issues resolved.
|
|||
|
|
|||
|
Moving away from the "beta" naming would then become an option to keep in mind
|
|||
|
for the future (with "brisk releases" as a potential alternative name).
|
|||
|
|
|||
|
|
|||
|
Why rolling beta releases rather than alternating between stable and unstable release series?
|
|||
|
---------------------------------------------------------------------------------------------
|
|||
|
|
|||
|
Rather than using the beta period for rolling releases, another option would be
|
|||
|
to alternate between traditional stable releases (for 3.8.x, 3.10.x, etc), and
|
|||
|
release series that used the new rolling release cadence (for 3.9.x, 3.11.x,
|
|||
|
etc).
|
|||
|
|
|||
|
This idea suffers from the same core problem as PEP 598 and PEP 602: it imposes
|
|||
|
changes on end users that are happy with the status quo without offering them
|
|||
|
any clear compensating benefit.
|
|||
|
|
|||
|
It's also affected by one of the main concerns raised against PEP 598: at least
|
|||
|
some core developers and end users strongly prefer that no particular semantics
|
|||
|
be assigned to the *value* of any of the numbers in a release version. These
|
|||
|
community members instead prefer that all the semantic significance be
|
|||
|
associated with the *position* within the release number that is changing.
|
|||
|
|
|||
|
By contrast, the rolling beta release proposal aims to address that concern by
|
|||
|
ensuring that the proposed changes in policy all revolve around whether a
|
|||
|
particular release is an alpha release, beta release, release candidate, or
|
|||
|
final release.
|
|||
|
|
|||
|
|
|||
|
Why not use Calendar Versioning for the rolling release stream?
|
|||
|
---------------------------------------------------------------
|
|||
|
|
|||
|
Steve Dower's initial write-up of this proposal [1_] suggested the use of
|
|||
|
calendar versioning for the rolling release stream (so the first rolling
|
|||
|
pre-release after Python 3.8.0 would have been Python 2019.12 rather than
|
|||
|
3.9.0b1).
|
|||
|
|
|||
|
Paul Moore pointed out [2_] two major practical problems with that proposal:
|
|||
|
|
|||
|
* it isn't going to be clear to users of the calendar-based versions where they
|
|||
|
stand in relation to the traditionally numbered versions
|
|||
|
* it breaks ``Python-Requires`` metadata processing in packaging tools with
|
|||
|
no clear way of fixing it reliably (since all calendar versions would appear
|
|||
|
as newer than any standard version)
|
|||
|
|
|||
|
This PEP aims to address both of those problems by using the established beta
|
|||
|
version numbers for the rolling releases.
|
|||
|
|
|||
|
As an example, consider the following question: "Does Python 2021.12 include
|
|||
|
all the new features released in Python 3.9.0?". With calendar versioning on
|
|||
|
the rolling releases, that's impossible to answer without consulting a release
|
|||
|
calendar to see when 3.9.0rc1 was branched off from the rolling release series.
|
|||
|
|
|||
|
By contrast, the equivalent question for rolling beta releases is
|
|||
|
straightforward to answer: "Does Python 3.10.0b2 include all the new features
|
|||
|
released in Python 3.9.0?". Just from formulating the question, the answer is
|
|||
|
clearly "Yes, unless they were provisional features that got removed".
|
|||
|
|
|||
|
The beta numbering approach also avoids other questions raised by the calendar
|
|||
|
versioning concept, such as how ``sys.version_info``, ``PY_VERSION_HEX``,
|
|||
|
``site-packages`` directory naming, and installed Python binary and extension
|
|||
|
module naming would work.
|
|||
|
|
|||
|
|
|||
|
How would users of the rolling beta releases detect API changes?
|
|||
|
----------------------------------------------------------------
|
|||
|
|
|||
|
When adding new features, core developers would be strongly encouraged to
|
|||
|
support feature detection and graceful fallback to alternative approaches via
|
|||
|
mechanisms that don't rely on either ``sys.version_info`` or runtime code object
|
|||
|
introspection.
|
|||
|
|
|||
|
In most cases, a simple ``hasattr`` check on the affected module will serve this
|
|||
|
purpose, but when it doesn't, alternative approaches would be considered as part
|
|||
|
of the feature addition. Prior art in this area includes the
|
|||
|
``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available`` set,
|
|||
|
and the various ``os.supports_*`` sets that the ``os`` module already offers for
|
|||
|
platform dependent capability detection.
|
|||
|
|
|||
|
It would also be possible to add features that need to be explicitly enabled
|
|||
|
via a ``__future__`` import when first included in the rolling beta releases,
|
|||
|
even if that feature flag was subsequently enabled by default before its first
|
|||
|
appearance in an X.Y.0 release candidate.
|
|||
|
|
|||
|
The rationale behind these approaches is that explicit detection/enabling like
|
|||
|
this would make it straightforward for users of the rolling beta release stream
|
|||
|
to notice when we remove or change provisional features
|
|||
|
(e.g. ``from __future__`` imports break on compile if the feature flag no
|
|||
|
longer exists), or to safely fall back on previous functionality.
|
|||
|
|
|||
|
The interpreter's rich attribute lookup machinery means we can also choose to
|
|||
|
add warnings for provisional or deprecated imports and attributes that we don't
|
|||
|
have any practical way to add for checks against the value of
|
|||
|
``sys.version_info``.
|
|||
|
|
|||
|
|
|||
|
Implications for CPython core development
|
|||
|
-----------------------------------------
|
|||
|
|
|||
|
The major change for CPython core development is the need to keep the master
|
|||
|
branch more consistently release ready.
|
|||
|
|
|||
|
While the main requirement for that would be to keep the stable BuildBot fleet
|
|||
|
green, there would also be encouragement to keep the development version of
|
|||
|
the documentation up to date for the benefit of users of the rolling beta
|
|||
|
releases. This will include providing draft What's New entries for changes as
|
|||
|
they are implemented, although the initial versions may be relatively sparse,
|
|||
|
and then expanded based on feedback from beta release users.
|
|||
|
|
|||
|
On the specific topic of the stable ABI, most API designs will be able to go
|
|||
|
through a process where they're first introduced as part of the full CPython
|
|||
|
API (allowing changes between beta releases), and only promoted to the stable
|
|||
|
ABI once developers are confident that the interface is genuinely stable.
|
|||
|
|
|||
|
It's only in rare cases where an API serves no useful purpose outside the
|
|||
|
stable ABI that it may make sense to publish an alpha release containing a
|
|||
|
provisional stable ABI addition rather than iterating on the design in the
|
|||
|
provisional CPython API instead.
|
|||
|
|
|||
|
|
|||
|
Implications for Python library development
|
|||
|
-------------------------------------------
|
|||
|
|
|||
|
By using the "beta" labelling rather than the incremental feature release
|
|||
|
numbering proposed in PEP 598, the hope would be that consumers of these
|
|||
|
new rolling releases would realise that they're likely going to need to build
|
|||
|
their own wheel archives from source, and will generally be more prone to
|
|||
|
encountering library compatibility issues when updating to a new release.
|
|||
|
|
|||
|
Library authors who actually want to support the beta stream would have the
|
|||
|
option of testing against the latest beta release in their pre-merge test
|
|||
|
matrices (just as they test against the latest maintenance release of previously
|
|||
|
published versions), with the CPython nightly builds offered by some CI
|
|||
|
providers used solely in an advisory capacity for early detection of potential
|
|||
|
compatibility problems.
|
|||
|
|
|||
|
Having a rolling beta release stream available may also make it more feasible
|
|||
|
for more CI providers to offer a "CPython beta release" testing option. At the
|
|||
|
moment, this feature is only available from CI providers that are willing and
|
|||
|
able to put the necessary time and effort into creating, testing, and publishing
|
|||
|
their own builds from the CPython master branch (e.g. [6_]).
|
|||
|
|
|||
|
|
|||
|
Implications for the proposed Scientific Python ecosystem support period
|
|||
|
------------------------------------------------------------------------
|
|||
|
|
|||
|
Based on discussions at SciPy 2019, NEP (NumPy Enhancement Proposal) 29 has
|
|||
|
been drafted [3_] to propose a common convention across the Scientific Python
|
|||
|
ecosystem for dropping support for older Python versions.
|
|||
|
|
|||
|
While the exact formulation of that policy is still being discussed, the initial
|
|||
|
proposal is very simple: support any Python feature release published within
|
|||
|
the last 42 months.
|
|||
|
|
|||
|
For an 18 month feature release cadence, that works out to always supporting at
|
|||
|
least the two most recent feature releases, and then dropping support for all
|
|||
|
X.Y.Z releases around 6 months after X.(Y+2).0 is released. This means there is
|
|||
|
a 6 month period roughly every other year where the three most recent feature
|
|||
|
releases are supported.
|
|||
|
|
|||
|
For a 12 month release cadence, it would work out to always supporting at
|
|||
|
least the three most recent feature releases, and then dropping support for all
|
|||
|
X.Y.Z releases around 6 months after X.(Y+3).0 is released. This means that
|
|||
|
for half of each year, the four most recent feature releases would be supported.
|
|||
|
|
|||
|
For a 24 month release cadence, a 42 month support cycle works out to always
|
|||
|
supporting at least the most recent feature release, and then dropping support
|
|||
|
for all X.Y.Z releases around 18 months after X.(Y+1).0 is released.
|
|||
|
This means there is a 6 month period every other year where only one feature
|
|||
|
release is supported. Under the proposal in this PEP, that period would
|
|||
|
correspond to the final few rolling beta releases and the release candidate
|
|||
|
phase for the upcoming stable feature release.
|
|||
|
|
|||
|
|
|||
|
Release cycle alignment for core development sprints
|
|||
|
----------------------------------------------------
|
|||
|
|
|||
|
With the proposal in this PEP, it is expected that the focus of core
|
|||
|
development sprints would shift slightly based on the current location
|
|||
|
in the two year cycle.
|
|||
|
|
|||
|
In release years, the timing of PyCon US is suitable for new contributors to
|
|||
|
work on bug fixes and smaller features before the first release candidate goes
|
|||
|
out, while the Language Summit and core developer discussions can focus on
|
|||
|
plans for the next release series.
|
|||
|
|
|||
|
The post-release core development sprint in release years will provide an
|
|||
|
opportunity to incorporate feedback received on the release, either as part of
|
|||
|
the next maintenance release (for bug fixes and feedback on provisional APIs),
|
|||
|
or as part of the next release series. These sprints would also likely
|
|||
|
correspond with the Steering Council elections for the next release cycle.
|
|||
|
|
|||
|
In non-release years, the focus for both events would just be on the upcoming
|
|||
|
maintenance and beta releases. These less intense years would hopefully provide
|
|||
|
an opportunity to tackle various process changes and infrastructure upgrades
|
|||
|
without impacting the release candidate preparation process.
|
|||
|
|
|||
|
|
|||
|
Release cycle alignment for prominent Linux distributions
|
|||
|
---------------------------------------------------------
|
|||
|
|
|||
|
Some rolling release Linux distributions (e.g. Arch, Gentoo) may be in a
|
|||
|
position to consume the new rolling beta releases proposed in this PEP, but it
|
|||
|
is expected that most distributions would continue to use the established
|
|||
|
releases.
|
|||
|
|
|||
|
The specific dates for those releases proposed in this PEP are chosen to align
|
|||
|
with the feature freeze schedules for the annual October releases of the Ubuntu
|
|||
|
and Fedora Linux distributions.
|
|||
|
|
|||
|
For both Fedora and Ubuntu, it means that the release candidate phase aligns
|
|||
|
with the development period for a distro release, which is the ideal time for
|
|||
|
them to test a new version and provide feedback on potential regressions and
|
|||
|
compatibility concerns.
|
|||
|
|
|||
|
For Ubuntu, this also means that their April LTS releases will have benefited
|
|||
|
from a full short-term release cycle using the new system Python version, while
|
|||
|
still having that CPython release be open to upstream bug fixes for most of the
|
|||
|
time until the next Ubuntu LTS release.
|
|||
|
|
|||
|
The one Linux release cycle alignment that is likely to be consistently poor
|
|||
|
with the specific proposal in this PEP is with Debian, as that has been released
|
|||
|
in the first half of odd-numbered years since 2005 (roughly 12 months offset
|
|||
|
from Ubuntu LTS releases).
|
|||
|
|
|||
|
With the annual release proposal in PEP 602, both Debian and Ubuntu LTS would
|
|||
|
consistently get a system Python version that is around 6 months old, but
|
|||
|
would also consistently select different Python versions from each other.
|
|||
|
|
|||
|
With a two year cadence, and CPython releases in the latter half of the year,
|
|||
|
they're likely to select the same version as each other, but one of them will
|
|||
|
be choosing a CPython release that is more than 18 months behind the latest beta
|
|||
|
releases by the time the Linux distribution ships.
|
|||
|
|
|||
|
If that situation does occur, and is deemed undesirable (but not sufficiently
|
|||
|
undesirable for *Debian* to choose to adjust their release timing), then that's
|
|||
|
where the additional complexity of the "incremental feature release" proposal
|
|||
|
in PEP 598 may prove worthwhile.
|
|||
|
|
|||
|
(Moving CPython releases to the same half of the year as the Debian and Ubuntu
|
|||
|
LTS releases would potentially help mitigate the problem, but also creates
|
|||
|
new problems where a slip in the CPython release schedule could directly affect
|
|||
|
the release schedule for a Linux distribution, or else result in a distribution
|
|||
|
shipping a Python version that is *more* than 18 months old)
|
|||
|
|
|||
|
|
|||
|
Implications for simple deployment environments
|
|||
|
-----------------------------------------------
|
|||
|
|
|||
|
For the purposes of this PEP, a "simple" deployment environment is any use case
|
|||
|
where it is straightforward to ensure that all target environments are updated
|
|||
|
to a new Python release at the same time (or at least in advance of the rollout
|
|||
|
of new higher level application versions), and any pre-release testing that
|
|||
|
occurs need only target a single Python micro version.
|
|||
|
|
|||
|
The simplest such case would be scripting for personal use, where the testing
|
|||
|
and target environments are the exact same environment.
|
|||
|
|
|||
|
Similarly simple environments would be containerised web services, where the
|
|||
|
same Python container is used in the CI pipeline as is used on deployment, and
|
|||
|
any application that bundles its own Python runtime, rather than relying on a
|
|||
|
pre-existing Python deployment on the target system.
|
|||
|
|
|||
|
For these use cases, there is a straightforward mechanism to minimise the
|
|||
|
impact of this PEP: continue using the stable releases, and ignore the rolling
|
|||
|
beta releases.
|
|||
|
|
|||
|
To actually adopt the rolling beta releases in these environments, the main
|
|||
|
challenge will be handling the potential for extension module segfaults if the
|
|||
|
CPython ABI changes in an incompatible way between beta releases.
|
|||
|
|
|||
|
If all extension modules in use target the stable ABI, then there's no problem,
|
|||
|
and everything will work just as smoothly as it does on the stable releases.
|
|||
|
|
|||
|
Alternatively, "rebuild and recache all extension modules" could become a
|
|||
|
standard activity undertaken as part of updating to each new beta release.
|
|||
|
|
|||
|
Finally, it would also be reasonable to just not worry about it until something
|
|||
|
actually breaks, and then handle it like any other library compatibility issue
|
|||
|
found in a new beta release.
|
|||
|
|
|||
|
Aside from extension module ABI compatibilty, the other main point of additional
|
|||
|
complexity when using the rolling beta releases would be "roll-back"
|
|||
|
compatibility for independently versioned features, such as pickle and SQLite,
|
|||
|
where use of new or provisional features in the beta stream may create files
|
|||
|
that are not readable by the stable release. Applications that use these
|
|||
|
kinds of features and also require the ability to reliably roll-back to a
|
|||
|
previous stable CPython release would, as today, be advised to avoid adopting
|
|||
|
pre-release versions.
|
|||
|
|
|||
|
|
|||
|
Implications for complex deployment environments
|
|||
|
------------------------------------------------
|
|||
|
|
|||
|
For the purposes of this PEP, "complex" deployment environments are use cases
|
|||
|
which don't meet the "simple deployment" criteria above. They may involve
|
|||
|
multiple distinct versions of Python, use of a personalised build of Python,
|
|||
|
or "gatekeepers" who are required to approve use of a new version prior to
|
|||
|
deployment.
|
|||
|
|
|||
|
For example, organisations that install Python on their users' machines as part
|
|||
|
of a standard operating environment fall into this category, as do those that
|
|||
|
provide a standard build environment. Distributions such as conda-forge or
|
|||
|
WinPython that provide collections of consistently built and verified packages
|
|||
|
are impacted in similar ways.
|
|||
|
|
|||
|
These organisations tend to either prefer high stability (for example, all of
|
|||
|
those who are happily using the system Python in a stable Linux distribution
|
|||
|
like Debian, RHEL/CentOS, or Ubuntu LTS as their preferred Python environment)
|
|||
|
or fast turnaround (for example, those who regularly contribute toward the
|
|||
|
latest CPython pre-releases).
|
|||
|
|
|||
|
In some cases, both usage models may exist within the same organisation for
|
|||
|
different purposes, such as:
|
|||
|
|
|||
|
* using a stable Python environment for mission critical systems, but allowing
|
|||
|
data scientists to use the latest available version for ad hoc data anaylsis
|
|||
|
* a hardware manufacturer deploying a stable Python version as part of their
|
|||
|
production firmware, but using the latest available version in the development
|
|||
|
and execution of their automated integration tests
|
|||
|
|
|||
|
Under any release model, each new release of Python generates work for these
|
|||
|
organisations. This work may involve legal, security or technical reviews of
|
|||
|
Python itself, assessment and verification of impactful changes, reapplication
|
|||
|
of patches, recompilation and testing of third-party dependencies, and
|
|||
|
only then deployment.
|
|||
|
|
|||
|
Organisations that can take updates quickly should be able to make use of the
|
|||
|
more frequent beta releases. While each update will still require similar
|
|||
|
investigative work to what they require today, the volume of work required per
|
|||
|
release should be reduced as each release will be more similar to the previous
|
|||
|
than it is under the present model. One advantage of the proposed
|
|||
|
release-every-2-months model is that organisations can choose their own adoption
|
|||
|
cadence from adopting every beta release, to adopting one per quarter, or one
|
|||
|
every 6 months, or one every year. Beyond that, it would likely make more sense
|
|||
|
to continue using the stable releases instead.
|
|||
|
|
|||
|
For organisations with stricter evaluations or a preference for stability, the
|
|||
|
longer release cycle for stable releases will reduce the annual effort required
|
|||
|
to update, the longer release candidate period will allow more time to do
|
|||
|
internal testing before the X.Y.0 release, and the greater use by others
|
|||
|
during the beta period will provide more confidence in the initial releases.
|
|||
|
Meanwhile, the organisation can confidently upgrade through maintenance
|
|||
|
releases for a longer time without fear of breaking changes.
|
|||
|
|
|||
|
|
|||
|
Acknowledgements
|
|||
|
================
|
|||
|
|
|||
|
Thanks to Łukasz Langa for creating PEP 602 and prompting this discussion of
|
|||
|
possible improvements to the CPython release cadence, and to Kyle Stanley
|
|||
|
and h-vetinari for constructive feedback on the initial draft of this PEP.
|
|||
|
|
|||
|
|
|||
|
References
|
|||
|
==========
|
|||
|
|
|||
|
.. [1] Steve Dower's initial "Fast and Stable releases" proposal
|
|||
|
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/20)
|
|||
|
|
|||
|
.. [2] Paul Moore's initial comments on Steve's proposal
|
|||
|
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/37)
|
|||
|
|
|||
|
.. [3] NEP 29 proposes a common policy for dropping support of old Python versions
|
|||
|
(https://numpy.org/neps/nep-0029-deprecation_policy.html)
|
|||
|
|
|||
|
.. [4] Example implementation for a pre-release SOABI flag
|
|||
|
(https://github.com/ncoghlan/cpython/pull/3)
|
|||
|
|
|||
|
.. [5] CPython stable ABI documentation
|
|||
|
(https://docs.python.org/3/c-api/stable.html)
|
|||
|
|
|||
|
.. [6] Travis CI nightly CPython builds
|
|||
|
(https://docs.travis-ci.com/user/languages/python/#nightly-build-support)
|
|||
|
|
|||
|
Copyright
|
|||
|
=========
|
|||
|
|
|||
|
This document is placed in the public domain or under the CC0-1.0-Universal
|
|||
|
license, whichever is more permissive.
|
|||
|
|
|||
|
..
|
|||
|
Local Variables:
|
|||
|
mode: indented-text
|
|||
|
indent-tabs-mode: nil
|
|||
|
sentence-end-double-space: t
|
|||
|
fill-column: 80
|
|||
|
coding: utf-8
|
|||
|
End:
|