PEP 598: CI implications, provisional APIs, other tweaks (#1129)

* PEP 598: Further discuss testing implications

Also rephrase the section about Python 3.10 vs 4.0
to make it clearer that the PEP isn't *proposing*
releasing 4.0 after 3.9, just discussing the impact
the different choice of version number would have
on the proposal.

* Also propose a tweak to the provisional API policy

pep-0598.rst

@@ -28,6 +28,12 @@ releases would significantly increase the burden of maintaining 3rd party
 Python libraries and applications across all actively supported CPython
 releases.
 
+It's also arguable whether such an approach would actually noticeably reduce
+the typical feature delivery latency, as the adoption cycle for new major
+releases is typically measured in months or years, so more frequent releases
+may just lead to end users updating to every 3rd or 4th major release, rather
+than every 2nd or 3rd major release (as already happens today).
+
 This PEP presents a competing proposal to instead *slow down* the frequency of
 parallel installable major feature releases that change the filesystem layout
 and CPython ABI to a consistent 24 month cycle, but to compensate for this by
@@ -332,6 +338,19 @@ For release series definition purposes though, the feature complete release
 is the first one that sets that flag to "True".
 
 
+Proposed policy adjustment for provisional APIs
+-----------------------------------------------
+
+To help improve consistency in management of provisional APIs, this PEP proposes
+that provisional APIs be subject to regular backwards compatibility requirements
+following the feature complete release for a given release series.
+
+Other aspects of managing provisional APIs would remain as they are today, so as
+long as an API remains in the provisional state, regular backwards compatibility
+requirements would not apply to that API in baseline and incremental feature
+releases.
+
+
 Motivation
 ==========
 
@@ -393,7 +412,7 @@ in a major version update create additional work for large sections of the
 wider Python community.
 
 Decoupling those layout changes from the Python version numbering scheme is also
-something that would in and of itself involves making backwards incompatible
+something that would in and of itself involve making backwards incompatible
 changes, as well as adjusting community expectations around which versions will
 install over the top of each other, and which can be installed in parallel on
 a single system.
@@ -406,7 +425,109 @@ So this PEP takes as its starting assumption that the vast majority of Python
 users simply *shouldn't need to care* that we're changing our release policy,
 and the only folks that should be affected are those that are eagerly waiting
 for standard library improvements, and other backwards compatible interpreter
-enhancements.
+enhancements, and those that need to manage mission critical applications in
+complex deployment environments.
+
+
+Implications for Python library development
+-------------------------------------------
+
+Many Python libraries (both open source and proprietary) currently adopt the
+practice of testing solely against the latest minor release within each major
+release series that the project still supports.
+
+The design assumption in this PEP is that this practice will continue to be
+followed during the feature release phase of a release series, with the
+expectation being that anyone choosing to adopt a new release series before it
+is feature complete will closely track the incremental feature releases.
+
+Libraries that support a previous major release series are unlikely to adopt
+features added in an incremental feature release, and if they do adopt such
+a feature, then any associated fallback compatibility strategies should be
+implemented in such a way that they're also effective on the earlier releases
+in that release series.
+
+
+Implications for simple deployment environments
+-----------------------------------------------
+
+For the purposes of this PEP, a "simple" deployment environment is any use case
+where it is straightforward to ensure that all target environments are updated
+to a new Python minor version at the same time (or at least in advance of the
+rollout of new higher level application versions), such that any pre-release
+testing that occurs need only target a single Python minor version.
+
+The simplest such case would be scripting for personal use, where the testing
+and target environments are the exact same environment.
+
+Similarly simple environments would be containerised web services, where the
+same Python container is used in the CI pipeline as is used on deployment, and
+any application that bundles its own Python runtime, rather than relying on a
+pre-existing Python deployment on the target system.
+
+For these use cases, this PEP shouldn't have any significant implications - only
+a single minor Python version needs to be tested, independently of whether that
+version is feature complete or not.
+
+
+Implications for complex deployment environments
+------------------------------------------------
+
+For the purposes of this PEP, "complex" deployment environments are use cases
+which don't meet the "simple deployment" criterion above: new application
+versions are combined with two or more distinct minor Python versions within
+the same release series as part of the deployment process, rather than always
+targeting exactly one minor version at a time.
+
+If the proposal in this PEP has the desired effect of reducing feature delivery
+latency, then it can be expected that developers using a release series that is
+not yet feature complete will actually make use of the new features as they're
+made available. This then means that testing against a newer incremental feature
+release becomes an even less valid test of compatibility with the baseline
+feature release and older incremental feature releases than testing against a
+newer maintenance release is for older maintenance releases.
+
+One option for handling such cases is to simply prohibit the use of new Python
+versions until the series has reached "feature complete" status. Such a policy
+is effectively already adopted by many organisations when it comes to new major
+feature releases, with acceptance into operational environments occurring months
+or years after the original release. If this policy is adopted, then such
+organisations could potentially still adopt a new Python version every other
+year - it would just be based on the availability of the feature complete
+releases, rather than the baseline feature releases.
+
+A less strict alternative to outright prohibition would be to restrict the use
+of any release series that is still in its feature release phase to applications
+where the occasional outage or failed deployment due to a lack of forwards
+compatibility is considered acceptable.
+
+However, a third variant, which allows selective adoption of new language
+features where appropriate, while also degrading gracefully enough to be
+suitable for mission critical applications, would be to institute a policy that
+applications wishing to target a release series that is not yet feature complete
+must also support the previous major release series for compatibility testing
+purposes.
+
+If this last policy is adopted, then testing against the previous release series
+becomes the new proxy for testing against the baseline feature release and any
+older incremental feature releases of the newer release series, without actually
+needing to install and test against all of them.
+
+Only after the newer release series is feature complete would support for the
+previous release series be dropped entirely.
+
+In a sufficiently complex environment, the second and third policies could also
+be combined, with critical applications maintaining compatibility with the
+previous release series, while newer, more experimental, applications and
+services would be permitted to target the newer release series directly without
+any additional safeguards in their testing process.
+
+Depending on demand and interest, there are also further enhancements that could
+be made to continuous integration pipelines to help ensure compatibility with
+a chosen minimum version within a release series, without needing to run tests
+against multiple minor releases. For example, applications could potentially be
+tested against the latest minor feature release, but typechecked against the
+oldest still deployed minor feature release.
 
 
 Duration of the feature additions period
@@ -527,11 +648,12 @@ the subsequent October 2022 major feature release, as there are already inherent
 compatibility risks associated with the choice of either "Python 4.0" (erroneous
 checks for the major version being exactly 3 rather than 3 or greater), or
 "Python 3.10" (code incorrectly assuming that the minor version will always
-contain exactly one decimal digit).
+contain exactly one decimal digit) [1].
 
-While the test of this PEP assumes that the release published in 2022 would be
-3.10, there are complex pros and cons on both sides of that future choice, and
-this PEP does arguably add a potential pro in favour of choosing the
+While the text of this PEP assumes that the release published in 2022 will be
+3.10 (as the PEP author personally considers that the more reasonable and most
+likely choice), there are complex pros and cons on both sides of that decision,
+and this PEP does arguably add a potential pro in favour of choosing the
 "Python 4.0" option (with the caveat that we would also need to amend the
 affected installation layout and compatibility markers to only consider the
 major version number, rather than both the major and minor version).
@@ -572,7 +694,11 @@ And the 5 year schedule forecast would look like:
 * 2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y maintenance
 * ... etc
 
+References
+==========
 
+[1] Anthony Sottile created a pseudo "Python 3.10" to find and fix such issues
+    (https://github.com/asottile/python3.10)
 
 Copyright
 =========