diff --git a/pep-0598.rst b/pep-0598.rst index d788057e1..c041cbf3c 100644 --- a/pep-0598.rst +++ b/pep-0598.rst @@ -4,7 +4,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828 -Status: Draft +Status: Withdrawn Type: Informational Content-Type: text/x-rst Created: 15-Jun-2019 @@ -14,7 +14,7 @@ Python-Version: 3.9 Abstract ======== -PEP 596 has proposed reducing the feature delivery latency for the Python +PEP 602 proposes reducing the feature delivery latency for the Python standard library and CPython reference interpreter by increasing the frequency of CPython feature releases from every 18-24 months to instead occur every 9-12 months. @@ -26,11 +26,18 @@ changes, and C ABI compatibility breaks) to occur only every other year (2020, allows the introduction of backwards compatible features in the initial set of point releases within a given release series. -In the event that this more complex proposal is *not* accepted (and either a -faster release cycle is adopted, or the status quo continues), the PEP aims to -provide useful design input for any other "Long Term Support branch" -proposals that may be put forward in the future (such as the EL Python draft at -[3_]). + +PEP Withdrawal +============== + +This PEP has been withdrawn in favour of the rolling beta release stream +proposal in PEP 605. + +However, the concerns raised in this PEP are likely to apply to any other +"Long Term Support branch" proposals that allow feature backports to improve +the developer experience of supporting such releases (such as the EL Python +draft at [3_]), so the ideas presented here may provide useful design +suggestions for such proposals. Summary diff --git a/pep-0605.rst b/pep-0605.rst new file mode 100644 index 000000000..580011f20 --- /dev/null +++ b/pep-0605.rst @@ -0,0 +1,886 @@ +PEP: 605 +Title: A rolling feature release stream for CPython +Version: $Revision$ +Last-Modified: $Date$ +Author: Steve Dower , Nick Coghlan +Discussions-To: Link TBD +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 20-Sep-2019 +Python-Version: 3.9 +Post-History: 30-Sep-2019 + + +Abstract +======== + +Rather than proposing more frequent full CPython releases (as PEP 602 does), +or a policy change to allow backwards compatible feature additions later in a +release series (as PEP 598 does), this PEP instead proposes that we create a +rolling stream of production-ready beta releases by amending CPython's +pre-release management process as follows: + +* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch + creation all correspond with the first release candidate for an X.Y.0 release +* The X.Y.0 release candidate period is extended from 3 weeks to 2 months +* Beta releases from the master git branch would occur every 2 months + whenever there is no X.Y.0 release in the release candidate phase +* Routine alpha releases are removed from the process +* Release managers may still choose to mark particular releases as alpha + releases if they feel such releases are appropriate. If any such alpha + releases are made, it is expected that they would be in place of one of the + scheduled beta releases +* The deprecation policy for interfaces that have been included in an X.Y.0 + release or release candidate without being explicitly declared provisional + remains unchanged (refer to PEP 387 for details) +* Each new alpha or beta release may introduce new deprecation warnings for + interfaces that appeared in the previous X.Y release series +* Each new alpha or beta release may change or remove interfaces that either did + not appear in the previous X.Y release series, or emitted a relevant + deprecation warning in that series +* Additions to the `abi3` stable C ABI are permanent once they have been + declared to be part of that ABI in at least one beta release +* All other newly added interfaces that have appeared only in alpha or beta + releases are considered provisional, and may be changed in incompatible ways + without requiring a preceding deprecation warning + +As part of this change, the current "not for production use" guidance given for +beta releases would be amended to state "suitable for production use only in +environments with sufficiently robust compatibility testing and operational +monitoring capabilities". + +This PEP also proposes that the frequency of X.Y.0 releases be adjusted to +begin each new release series in August every two years (starting in 2021, +around two years after the release of Python 3.8.0). + + +Motivation +========== + +The current CPython pre-release and release management processes were developed +in an era where automated continuous integration and operational monitoring +systems were still relatively immature. Since that time, many organisations +have adopted deployment models that allow them to incorporate new CPython +feature releases without adding substantially more risk than they incur for any +other code change. Newer deployment models, such as lightweight task specific +application containers, also make it easier to combine an application with a +language runtime in a CI pipeline, and then keep them together until the entire +container image is later replaced by an updated one. + +In light of those changes in the wider environment, PEP 602 has proposed +reducing the feature delivery latency for the Python standard library and +CPython reference interpreter by increasing the frequency of CPython feature +releases from every 18-24 months to instead occur every 12 months. + +Unfortunately, for many organisations, the cost of adopting a new Python release +doesn't automatically scale down with a reduced number of changes in the release, +as the primary costs aren't associated with resolving any discovered issues; +the primary costs are associated with the *search* for issues. This search may +involve manual testing of software systems, human review of written materials, +and other activities where the time required scales with the size of the +existing system, rather than with the number of changes between the versions of +Python. + +For third party library developers, costs are primarily associated with the +*number* of distinct Python versions in widespread usage. This currently tends +to be influenced by a combination of which releases are still actively +maintained by python-dev, and which releases are the latest versions offered +by particular redistributors (with the Debian, Ubuntu LTS, and RHEL/CentOS +system Python versions being particularly popular development targets). In +addition to the basic CI cost of testing against more Python versions, having +more variants in widespread use can make it more difficult to determine when a +fault report is an actual error in the project, or an issue in the reporting +user's environment. + +PEP 602 proposes that affected organisations and projects simply switch to +adopting every second or third CPython release, rather than attempting to adopt +every release, but that creates its own set of new problems to be resolved, both +practical (e.g. deprecations would need to cover more than one release if we're +expecting users to routinely skip releases) and cultural (e.g. with a larger +number of versions in active use, there is a much higher chance that open source +library maintainers will receive bug reports that only occur on Python versions +that they're not using themselves). + +PEP 598 was an initial attempt by one of the authors of this PEP to propose +an alternative scheme to reduce feature delivery latency by adopting a +semantic versioning style policy that allowed for the incremental delivery of +backwards compatible features within a release series, until that series +reached feature complete status. That variant still had the undesirable +consequence of imposing visible changes on end users that are happy enough +with the current release management model. + +This PEP takes the view that both PEP 598 and PEP 602 share a common flaw: they +are attempting to satisfy the needs of two quite distinct audiences within the +constraints of a single release model, which results in conflicting design +requirements, and the need for awkward trade-offs between those conflicting +requirements. The proposal in this PEP aims to avoid that flaw by proposing the +creation of two *distinct* production-ready release streams, with the existing +release stream being largely left alone, while the new release stream is +tailored towards the audience that would most benefit from a reduction in +feature delivery latency. + + +Aims of this Proposal +===================== + +The core of the proposal in this PEP is changing the beta phase of the CPython +pre-release process to produce a rolling stream of releases at a regular +cadence, and to ensure those builds offer a sufficient level of stability as +to be suitable for use in appropriately managed production systems. + +By adopting this approach, the proposal aims to provide an improved outcome +for almost all Python users and contributors: + +* for users of the new release stream, targeting the beta phase allows for even + lower feature delivery latency than the annual cadence proposed in PEP 602; +* for core developers working on new features, increased frequency and adoption + of the beta releases should improve pre-release feedback cycles; +* for users of the established release stream, the increased adoption and + improved feedback cycles during the pre-release period should result in + increased feature maturity at the time of its first X.Y.0 release, as well + as higher levels of ecosystem readiness; +* for Python library maintainers, the rolling stream of beta releases will + hopefully provide more opportunities to identify and resolve design issues + before they make it into a full stable release than is offered by the current + pre-release management process; and +* for developers of alternative Python implementations, the proposed rolling + beta releases provide an additional incentive for extension module authors + to migrate from the full CPython ABI to the Python stable ABI, which may + over time make more of the ecosystem compatible with implementations that + don't fully emulate the CPython C API. + +That said, it is acknowledged that not all the outcomes of this proposal will be +beneficial for all members of the wider Python ecosystem: + +* for Python library maintainers, both this PEP and PEP 602 would likely + result in user pressure to support the faster release cadence. While this PEP + attempts to mitigate that by marking the faster releases with the "beta" label + and removing the expectation to publish pre-built wheel archives, and PEP 602 + attempts to mitigate it by keeping the minimum time between full releases at + 12 months, it isn't possible to eliminate this downside completely; +* for third party extension module maintainers, both this PEP and PEP 602 would + likely result in user pressure to start supporting the stable ABI in order to + provide wheel archives that work on the new version as soon as it is + available. Whether that's a net negative or not will depend on how the request + is presented to them (it could be a positive if the request comes in the form + of a courteous contribution to their project from a developer interested in + supporting the rolling beta releases); +* for some users of the established release stream that rely on the + availability of pre-built wheel archives, switching to adopting a new release + every 12 months may be an acceptable rate increase, while moving consistently + to the 24 month end of the historical 18-24 month cadence would be an + undesirable rate reduction relative to the 18 month cycle used for recent + releases. Whether this proposal would be a net negative for these users in the + long run would depend on how many projects migrated over to targeting the + Python stable ABI and hence produced wheel archives that would still be usable + on the rolling beta releases. + + +Proposal +======== + +The majority of the proposed changes in this PEP only affect the handling of +pre-release versions. The one change affecting full release versions is a +suggested change to their cadence. + +Changes to beta release policy, phase duration, and cadence +----------------------------------------------------------- + +This PEP proposes that the policy for beta releases be adjusted to be a +combination of the policies for alpha and beta releases: + +* as with current beta releases, the stable BuildBot fleet is expected to be + green prior to preparation and publication of the beta release +* as with current beta releases, the release manager is expected to review + open release blocker issues prior to preparation and publication of the beta + release +* as with current beta releases, any additions to the `abi3` stable C ABI would + be expected to become a permanent part of that ABI unless and until that + stable ABI version is retired completely (Note: there are no current plans + to increment the stable ABI version) +* as with current alpha releases, beta releases would *not* be considered + feature complete for the next X.Y.0 release +* as with current alpha releases, all APIs added since the last CPython feature + release (other than additions to the stable C ABI) would be considered + provisional +* as with current alpha releases, beta releases would be prepared and published + from the master development branch + +Rather than being released monthly for a period of a few months while preparing +a new X.Y.0 release, beta releases would instead be consistently published every +two months. + +The only time this would not be the case is during the two month release +candidate period for an upcoming X.Y.0 release, as described in the following +section. + + +Changes to release candidate policy, phase duration, and cadence +---------------------------------------------------------------- + +Given the proposed changes to the beta release phase, the release candidate +phase would see the following related adjustments: + +* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch + creation would all correspond with the creation of X.Y.0rc1 (currently these + occur across a mixture of X.Y.0b1, the last beta release, and X.Y.0rc1) +* The X.Y.0 release candidate period would be extended from 3 weeks to 2 months +* There would normally be two release candidates issued a month apart, but + additional candidates may be published at the release manager's discretion +* The final X.Y.0 release would occur between 1 and 4 weeks after the final + release candidate (depending if additional release candidates were needed + after the second) +* If the final X.Y.0 release is delayed beyond the August target date, the + subsequent release series is not affected, and will still be scheduled for + August (now slightly less than two years later). + +In addition to allowing more time for end user feedback on the release +candidate, this adjusted policy also provides additional time for maintainers +of Python projects to build and publish pre-built wheel archives for the new +stable release series, significantly improving the initial user experience of +the X.Y.0 release. + + +Removal of routine alpha releases +--------------------------------- + +Due to the proposed policy changes for beta releases and release candidates, +the traditional CPython alpha release period would no longer serve a clear +purpose, and as such would be skipped, with the first pre-release in a new +series being X.Y.0b1. + +However, when it seems appropriate to do so, release managers may choose to +declare that a scheduled beta release will be marked as an alpha release +instead. For example, a release manager may choose to do this when major changes +have been made to the release management tooling, when a particularly +large feature has landed that the release manager views as having a high chance +of introducing unintended side effects, or when a new interface has been added +to the stable C ABI that is only useful when it appears as part of that ABI +(e.g. it may be enabling a new capability in the stable ABI that the full ABI +already supports in a different way), and requires further testing and design +feedback before being locked down in a beta release. + +In such cases, the beta serial number would be retained, but the release would +be marked as an alpha release instead, and hence the "not for production use" +caveat would apply. + +(The serial number would still be reset to 1 when entering the release +candidate phase - only alpha and beta release would share a serial number +sequence) + + +Changes to management of the CPython stable C ABI +------------------------------------------------- + +The CPython stable ABI [5_] makes the commitment that binary extension modules +built against any particular CPython release will continue to work on future +CPython releases that support the same stable ABI version (this version is +currently ``abi3``). + +Under the proposed rolling beta release model, this commitment would be extended +to also apply to the beta releases: once an intentional addition to the ``abi3`` +stable ABI for the upcoming Python version has been shipped in a beta release, +then it will not be removed from future releases for as long as the ``abi3`` +stable ABI remains supported. + +Two main mechanisms will be available for obtaining community feedback on +additions to the stable ABI: + +* the preferred mechanism will be to add new APIs to the full CPython API first, + and only promote them to the stable ABI after they have been included in at + least one published beta release and received relevant user feedback +* for APIs where that approach is unavailable for some reason (e.g. some API + additions may serve no useful purpose when the full CPython API is available), + then developers may request that the release manager mark the next release + as an alpha release, and attempt to obtain further feedback that way + + +As a slight readability and usability improvement, this PEP also proposes the +introduction of aliases for each major stable ABI version:: + + #define Py_LIMITED_API_3_3 0x03030000 + #define Py_LIMITED_API_3_4 0x03040000 + #define Py_LIMITED_API_3_5 0x03050000 + #define Py_LIMITED_API_3_6 0x03060000 + #define Py_LIMITED_API_3_7 0x03070000 + #define Py_LIMITED_API_3_8 0x03080000 + #define Py_LIMITED_API_3_9 0x03090000 + // etc... + +These would be used both in extension module code to set the target ABI +version:: + + #define Py_LIMITED_API Py_LIMITED_API_3_8 + +And also in the CPython interpreter implementation to check which symbols should +be made available:: + + #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9 + // A Python 3.9+ addition to the stable ABI would appear here + #endif + +The documentation for the rolling beta releases and the stable C ABI would make +it clear that extension modules built against the stable ABI in a later beta +release may not load correctly on earlier alpha or beta releases. + +The documentation for alpha releases and the stable C ABI would make it clear +that even extension modules built against the stable ABI in an alpha release +release may not load correctly on any other release. + + +Changes to management of the CPython version-specific ABI +--------------------------------------------------------- + +The CPython version-specific ABI has long operated under a policy where binary +compatibility only applies within a release series after the ABI has been +declared frozen, and only source compatibility applies between different +release series. + +This policy means that extension modules built against CPython pre-releases +prior to the ABI freeze for that release series may not actually load correctly +on the final release. + +This is due to the fact that the extension module may be relying on provisional +or previously deprecated interfaces that were changed or removed in a later +alpha or beta release, or it may be due to public structures used by the +extension module changing size due to the addition of new fields. + +Historically, adoption of alpha and beta releases has been low enough that this +hasn't really been a problem in practice. However, this PEP proposes to actively +encourage widespread operational use of beta releases, which makes it desirable +to ensure that users of those releases won't inadvertently publish binary +extension modules that cause segfaults for users running the release candidates +and final releases. + +To that end, this PEP proposes amending the extension module ``SOABI`` marker +on non-Windows systems to include a new "p" flag for CPython pre-releases, and +only switch back to omitting that flag once the ABI for that particular X.Y.0 +version has been frozen on entry to the release candidate stage. + +With this change, alpha and beta releases of 3.9.0 would get an SOABI tag of +``cpython-39p``, while all release candidates and final builds (for both 3.9.0 +and later 3.9.x releases) would get an unqualified SOABI tag of ``cpython-39`` + +Debug builds would still add the "d" to the end of the tag, giving +``cpython-39pd`` for debug builds of pre-releases. + +On Windows systems, the suffix for tagged ``pyd`` files in pre-release builds +would include "p" as a pre-release marker immediately after the version number, +giving markers like "cp39p-win_amd64". + +A proposed reference implementation for this change is available at [4_] (Note: +at time of writing, that implementation had not yet been tested on Windows). + + +Example Future Release Schedules +================================ + +Under this proposal, Python 3.9.0b1 would be released in December 2019, two +months after the Python 3.8.0 baseline feature release in October 2019. + +The 3.9.0b2 release would then follow 2 months later in February 2020, +continuing through to 3.9.0b9 in April 2021. + +3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then +the full release published as 3.9.0 in August 2021. + +The cycle would start over again in October 2021, with the publication +of 3.10.0b1. + +Assuming maintenance releases of 3.9.x were also to occur every other month +(offset from the 3.10.0 beta releases), the overall release timeline +would look like: + +* 2019-12: 3.9.0b1 +* ... beta releases every other month +* 2021-04: 3.9.0b9 +* 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze) +* 2021-07: 3.9.0rc2 +* 2021-08: 3.9.0 +* 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release) +* 2021-10: 3.10.0b1 +* 2021-11: 3.9.2 +* 2021-12: 3.10.0b2 +* ... beta and maintenance releases every other month +* 2023-04: 3.10.0b10 +* 2023-05: 3.9.11 +* 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze) +* 2023-07: 3.10.0rc2, 3.9.12 +* 2023-08: 3.10.0 +* 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release) +* 2023-10: 3.11.0b1 +* ... etc + + +(The exact schedule of maintenance releases would be up to the release team - +alternating months with the rolling beta releases is the proposed target) + +There are always two or three active maintenance branches in this model, +which preserves the status quo in that respect. + + +Caveats and Limitations +======================= + +Actual release dates may be scheduled up to a month earlier or later at +the discretion of the release manager, based on release team availability, and +the timing of other events (e.g. PyCon US, or the annual core developer +sprints). However, as one goal of the proposal is to provide a consistent +release cadence, adjustments should ideally be rare. + +Within a release series, the exact frequency of maintenance releases would +still be up to the release manager and the binary release team; this PEP +only proposes an expected cadence for pre-releases and X.Y.0 releases. + +However, for the sake of the example timelines, the PEP assumes maintenance +releases every other month, allowing them to alternate months with the rolling +beta releases. + + +Design Discussion +================= + +Why rolling beta releases over simply doing more frequent X.Y.0 releases? +------------------------------------------------------------------------- + +For large parts of Python's user base, *availability* of new CPython feature +releases isn't the limiting factor on their adoption of those new releases +(this effect is visible in such metrics as PyPI download metadata). + +As such, any proposal based on speeding up full feature releases needs to strike +a balance between meeting the needs of users who would be adopting each release +as it became available, and those that would now be in a position of adopting +every 2nd, 3rd, or 4th release, rather than being able to migrate to almost +every release at some point within its lifecycle. + +This proposal aims to approach the problem from a different angle by defining a +*new* production-ready release stream that is more specifically tailored to the +interests of operating environments that are able to consume new releases as +fast as the CPython core team is prepared to produce them. + + +Why rolling beta releases rather than rolling alpha releases? +---------------------------------------------------------------- + +The code quality standards upheld by the CPython code review process and +BuildBot fleet make the "beta" label more suitable than the "alpha" label. + +The "production ready for some environments, but subject to change with limited +notice" caveat also aligns well with at least some uses of the "beta" term, +whereas "alpha" almost universally indicates "not yet ready for production". + + +Why rolling beta releases rather than something like "brisk release cadence"? +----------------------------------------------------------------------------- + +Using the "b" initial for the proposed rolling releases is a design constraint +imposed by some of the pragmatic aspects of the way CPython version numbers are +published (specifically, alpha releases, beta releases, and release candidates +are reported in some places using the strings "a", "b", and "c" respectively, +while in others they're reported using the hex digits ``0xA``, ``0xB``, and +``0xC``, and we want to preserve that, while also ensuring that the rolling +releases are all sorted after any alpha releases, and before the first release +candidate). + +However, there isn't anything forcing us to say that the "b" stands for "beta". + +That means that if we wanted to increase adoption amongst folks that were +only being put off by the "beta" label, then it could make sense to rebrand the +rolling beta releases as "brisk" releases, emphasising the rate of release over +the fact that the release includes APIs and ABIs that aren't necessarily stable +yet. + +In the near term, limiting adoption to folks that are comfortable with the +"beta" label would be a good thing, as initial adopters are likely to +encounter unexpected consequences at the level of the wider Python ecosystem, +and would need to take an active part in getting those issues resolved. + +Moving away from the "beta" naming would then become an option to keep in mind +for the future (with "brisk releases" as a potential alternative name). + + +Why rolling beta releases rather than alternating between stable and unstable release series? +--------------------------------------------------------------------------------------------- + +Rather than using the beta period for rolling releases, another option would be +to alternate between traditional stable releases (for 3.8.x, 3.10.x, etc), and +release series that used the new rolling release cadence (for 3.9.x, 3.11.x, +etc). + +This idea suffers from the same core problem as PEP 598 and PEP 602: it imposes +changes on end users that are happy with the status quo without offering them +any clear compensating benefit. + +It's also affected by one of the main concerns raised against PEP 598: at least +some core developers and end users strongly prefer that no particular semantics +be assigned to the *value* of any of the numbers in a release version. These +community members instead prefer that all the semantic significance be +associated with the *position* within the release number that is changing. + +By contrast, the rolling beta release proposal aims to address that concern by +ensuring that the proposed changes in policy all revolve around whether a +particular release is an alpha release, beta release, release candidate, or +final release. + + +Why not use Calendar Versioning for the rolling release stream? +--------------------------------------------------------------- + +Steve Dower's initial write-up of this proposal [1_] suggested the use of +calendar versioning for the rolling release stream (so the first rolling +pre-release after Python 3.8.0 would have been Python 2019.12 rather than +3.9.0b1). + +Paul Moore pointed out [2_] two major practical problems with that proposal: + +* it isn't going to be clear to users of the calendar-based versions where they + stand in relation to the traditionally numbered versions +* it breaks ``Python-Requires`` metadata processing in packaging tools with + no clear way of fixing it reliably (since all calendar versions would appear + as newer than any standard version) + +This PEP aims to address both of those problems by using the established beta +version numbers for the rolling releases. + +As an example, consider the following question: "Does Python 2021.12 include +all the new features released in Python 3.9.0?". With calendar versioning on +the rolling releases, that's impossible to answer without consulting a release +calendar to see when 3.9.0rc1 was branched off from the rolling release series. + +By contrast, the equivalent question for rolling beta releases is +straightforward to answer: "Does Python 3.10.0b2 include all the new features +released in Python 3.9.0?". Just from formulating the question, the answer is +clearly "Yes, unless they were provisional features that got removed". + +The beta numbering approach also avoids other questions raised by the calendar +versioning concept, such as how ``sys.version_info``, ``PY_VERSION_HEX``, +``site-packages`` directory naming, and installed Python binary and extension +module naming would work. + + +How would users of the rolling beta releases detect API changes? +---------------------------------------------------------------- + +When adding new features, core developers would be strongly encouraged to +support feature detection and graceful fallback to alternative approaches via +mechanisms that don't rely on either ``sys.version_info`` or runtime code object +introspection. + +In most cases, a simple ``hasattr`` check on the affected module will serve this +purpose, but when it doesn't, alternative approaches would be considered as part +of the feature addition. Prior art in this area includes the +``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available`` set, +and the various ``os.supports_*`` sets that the ``os`` module already offers for +platform dependent capability detection. + +It would also be possible to add features that need to be explicitly enabled +via a ``__future__`` import when first included in the rolling beta releases, +even if that feature flag was subsequently enabled by default before its first +appearance in an X.Y.0 release candidate. + +The rationale behind these approaches is that explicit detection/enabling like +this would make it straightforward for users of the rolling beta release stream +to notice when we remove or change provisional features +(e.g. ``from __future__`` imports break on compile if the feature flag no +longer exists), or to safely fall back on previous functionality. + +The interpreter's rich attribute lookup machinery means we can also choose to +add warnings for provisional or deprecated imports and attributes that we don't +have any practical way to add for checks against the value of +``sys.version_info``. + + +Implications for CPython core development +----------------------------------------- + +The major change for CPython core development is the need to keep the master +branch more consistently release ready. + +While the main requirement for that would be to keep the stable BuildBot fleet +green, there would also be encouragement to keep the development version of +the documentation up to date for the benefit of users of the rolling beta +releases. This will include providing draft What's New entries for changes as +they are implemented, although the initial versions may be relatively sparse, +and then expanded based on feedback from beta release users. + +On the specific topic of the stable ABI, most API designs will be able to go +through a process where they're first introduced as part of the full CPython +API (allowing changes between beta releases), and only promoted to the stable +ABI once developers are confident that the interface is genuinely stable. + +It's only in rare cases where an API serves no useful purpose outside the +stable ABI that it may make sense to publish an alpha release containing a +provisional stable ABI addition rather than iterating on the design in the +provisional CPython API instead. + + +Implications for Python library development +------------------------------------------- + +By using the "beta" labelling rather than the incremental feature release +numbering proposed in PEP 598, the hope would be that consumers of these +new rolling releases would realise that they're likely going to need to build +their own wheel archives from source, and will generally be more prone to +encountering library compatibility issues when updating to a new release. + +Library authors who actually want to support the beta stream would have the +option of testing against the latest beta release in their pre-merge test +matrices (just as they test against the latest maintenance release of previously +published versions), with the CPython nightly builds offered by some CI +providers used solely in an advisory capacity for early detection of potential +compatibility problems. + +Having a rolling beta release stream available may also make it more feasible +for more CI providers to offer a "CPython beta release" testing option. At the +moment, this feature is only available from CI providers that are willing and +able to put the necessary time and effort into creating, testing, and publishing +their own builds from the CPython master branch (e.g. [6_]). + + +Implications for the proposed Scientific Python ecosystem support period +------------------------------------------------------------------------ + +Based on discussions at SciPy 2019, NEP (NumPy Enhancement Proposal) 29 has +been drafted [3_] to propose a common convention across the Scientific Python +ecosystem for dropping support for older Python versions. + +While the exact formulation of that policy is still being discussed, the initial +proposal is very simple: support any Python feature release published within +the last 42 months. + +For an 18 month feature release cadence, that works out to always supporting at +least the two most recent feature releases, and then dropping support for all +X.Y.Z releases around 6 months after X.(Y+2).0 is released. This means there is +a 6 month period roughly every other year where the three most recent feature +releases are supported. + +For a 12 month release cadence, it would work out to always supporting at +least the three most recent feature releases, and then dropping support for all +X.Y.Z releases around 6 months after X.(Y+3).0 is released. This means that +for half of each year, the four most recent feature releases would be supported. + +For a 24 month release cadence, a 42 month support cycle works out to always +supporting at least the most recent feature release, and then dropping support +for all X.Y.Z releases around 18 months after X.(Y+1).0 is released. +This means there is a 6 month period every other year where only one feature +release is supported. Under the proposal in this PEP, that period would +correspond to the final few rolling beta releases and the release candidate +phase for the upcoming stable feature release. + + +Release cycle alignment for core development sprints +---------------------------------------------------- + +With the proposal in this PEP, it is expected that the focus of core +development sprints would shift slightly based on the current location +in the two year cycle. + +In release years, the timing of PyCon US is suitable for new contributors to +work on bug fixes and smaller features before the first release candidate goes +out, while the Language Summit and core developer discussions can focus on +plans for the next release series. + +The post-release core development sprint in release years will provide an +opportunity to incorporate feedback received on the release, either as part of +the next maintenance release (for bug fixes and feedback on provisional APIs), +or as part of the next release series. These sprints would also likely +correspond with the Steering Council elections for the next release cycle. + +In non-release years, the focus for both events would just be on the upcoming +maintenance and beta releases. These less intense years would hopefully provide +an opportunity to tackle various process changes and infrastructure upgrades +without impacting the release candidate preparation process. + + +Release cycle alignment for prominent Linux distributions +--------------------------------------------------------- + +Some rolling release Linux distributions (e.g. Arch, Gentoo) may be in a +position to consume the new rolling beta releases proposed in this PEP, but it +is expected that most distributions would continue to use the established +releases. + +The specific dates for those releases proposed in this PEP are chosen to align +with the feature freeze schedules for the annual October releases of the Ubuntu +and Fedora Linux distributions. + +For both Fedora and Ubuntu, it means that the release candidate phase aligns +with the development period for a distro release, which is the ideal time for +them to test a new version and provide feedback on potential regressions and +compatibility concerns. + +For Ubuntu, this also means that their April LTS releases will have benefited +from a full short-term release cycle using the new system Python version, while +still having that CPython release be open to upstream bug fixes for most of the +time until the next Ubuntu LTS release. + +The one Linux release cycle alignment that is likely to be consistently poor +with the specific proposal in this PEP is with Debian, as that has been released +in the first half of odd-numbered years since 2005 (roughly 12 months offset +from Ubuntu LTS releases). + +With the annual release proposal in PEP 602, both Debian and Ubuntu LTS would +consistently get a system Python version that is around 6 months old, but +would also consistently select different Python versions from each other. + +With a two year cadence, and CPython releases in the latter half of the year, +they're likely to select the same version as each other, but one of them will +be choosing a CPython release that is more than 18 months behind the latest beta +releases by the time the Linux distribution ships. + +If that situation does occur, and is deemed undesirable (but not sufficiently +undesirable for *Debian* to choose to adjust their release timing), then that's +where the additional complexity of the "incremental feature release" proposal +in PEP 598 may prove worthwhile. + +(Moving CPython releases to the same half of the year as the Debian and Ubuntu +LTS releases would potentially help mitigate the problem, but also creates +new problems where a slip in the CPython release schedule could directly affect +the release schedule for a Linux distribution, or else result in a distribution +shipping a Python version that is *more* than 18 months old) + + +Implications for simple deployment environments +----------------------------------------------- + +For the purposes of this PEP, a "simple" deployment environment is any use case +where it is straightforward to ensure that all target environments are updated +to a new Python release at the same time (or at least in advance of the rollout +of new higher level application versions), and any pre-release testing that +occurs need only target a single Python micro version. + +The simplest such case would be scripting for personal use, where the testing +and target environments are the exact same environment. + +Similarly simple environments would be containerised web services, where the +same Python container is used in the CI pipeline as is used on deployment, and +any application that bundles its own Python runtime, rather than relying on a +pre-existing Python deployment on the target system. + +For these use cases, there is a straightforward mechanism to minimise the +impact of this PEP: continue using the stable releases, and ignore the rolling +beta releases. + +To actually adopt the rolling beta releases in these environments, the main +challenge will be handling the potential for extension module segfaults if the +CPython ABI changes in an incompatible way between beta releases. + +If all extension modules in use target the stable ABI, then there's no problem, +and everything will work just as smoothly as it does on the stable releases. + +Alternatively, "rebuild and recache all extension modules" could become a +standard activity undertaken as part of updating to each new beta release. + +Finally, it would also be reasonable to just not worry about it until something +actually breaks, and then handle it like any other library compatibility issue +found in a new beta release. + +Aside from extension module ABI compatibilty, the other main point of additional +complexity when using the rolling beta releases would be "roll-back" +compatibility for independently versioned features, such as pickle and SQLite, +where use of new or provisional features in the beta stream may create files +that are not readable by the stable release. Applications that use these +kinds of features and also require the ability to reliably roll-back to a +previous stable CPython release would, as today, be advised to avoid adopting +pre-release versions. + + +Implications for complex deployment environments +------------------------------------------------ + +For the purposes of this PEP, "complex" deployment environments are use cases +which don't meet the "simple deployment" criteria above. They may involve +multiple distinct versions of Python, use of a personalised build of Python, +or "gatekeepers" who are required to approve use of a new version prior to +deployment. + +For example, organisations that install Python on their users' machines as part +of a standard operating environment fall into this category, as do those that +provide a standard build environment. Distributions such as conda-forge or +WinPython that provide collections of consistently built and verified packages +are impacted in similar ways. + +These organisations tend to either prefer high stability (for example, all of +those who are happily using the system Python in a stable Linux distribution +like Debian, RHEL/CentOS, or Ubuntu LTS as their preferred Python environment) +or fast turnaround (for example, those who regularly contribute toward the +latest CPython pre-releases). + +In some cases, both usage models may exist within the same organisation for +different purposes, such as: + +* using a stable Python environment for mission critical systems, but allowing + data scientists to use the latest available version for ad hoc data anaylsis +* a hardware manufacturer deploying a stable Python version as part of their + production firmware, but using the latest available version in the development + and execution of their automated integration tests + +Under any release model, each new release of Python generates work for these +organisations. This work may involve legal, security or technical reviews of +Python itself, assessment and verification of impactful changes, reapplication +of patches, recompilation and testing of third-party dependencies, and +only then deployment. + +Organisations that can take updates quickly should be able to make use of the +more frequent beta releases. While each update will still require similar +investigative work to what they require today, the volume of work required per +release should be reduced as each release will be more similar to the previous +than it is under the present model. One advantage of the proposed +release-every-2-months model is that organisations can choose their own adoption +cadence from adopting every beta release, to adopting one per quarter, or one +every 6 months, or one every year. Beyond that, it would likely make more sense +to continue using the stable releases instead. + +For organisations with stricter evaluations or a preference for stability, the +longer release cycle for stable releases will reduce the annual effort required +to update, the longer release candidate period will allow more time to do +internal testing before the X.Y.0 release, and the greater use by others +during the beta period will provide more confidence in the initial releases. +Meanwhile, the organisation can confidently upgrade through maintenance +releases for a longer time without fear of breaking changes. + + +Acknowledgements +================ + +Thanks to Łukasz Langa for creating PEP 602 and prompting this discussion of +possible improvements to the CPython release cadence, and to Kyle Stanley +and h-vetinari for constructive feedback on the initial draft of this PEP. + + +References +========== + +.. [1] Steve Dower's initial "Fast and Stable releases" proposal + (https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/20) + +.. [2] Paul Moore's initial comments on Steve's proposal + (https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/37) + +.. [3] NEP 29 proposes a common policy for dropping support of old Python versions + (https://numpy.org/neps/nep-0029-deprecation_policy.html) + +.. [4] Example implementation for a pre-release SOABI flag + (https://github.com/ncoghlan/cpython/pull/3) + +.. [5] CPython stable ABI documentation + (https://docs.python.org/3/c-api/stable.html) + +.. [6] Travis CI nightly CPython builds + (https://docs.travis-ci.com/user/languages/python/#nightly-build-support) + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 80 + coding: utf-8 + End: