PEP 598: Add a feature limiting capability (#1156)
The PEP's previous handling of the micro version compatibility testing question really wasn't very good, and didn't address the question of pickle compatibility at all. So if this idea is going to be considered seriously, it needs to provide a way to explicitly opt out of the feature additions, without having to downgrade back to an earlier point release.
This commit is contained in:
Nick Coghlan
GitHub
parent
f97aa37e6a
commit
ff06fa35a3
219
pep-0598.rst
219
pep-0598.rst
|
|
@ -19,20 +19,39 @@ standard library and CPython reference interpreter by increasing the frequency
|
|||
of CPython feature releases from every 18-24 months to instead occur every 9-12
|
||||
months.
|
||||
|
||||
Adopting such an approach has several significant practical downsides, as a
|
||||
CPython feature release carries certain expectations (most notably, a 5 year
|
||||
maintenance lifecycle, support for parallel installation with the previous
|
||||
This PEP proposes to instead *reduce* the frequency of new baseline feature
|
||||
releases (with the associated filesystem layout changes, bytecode format
|
||||
changes, and C ABI compatibility breaks) to occur only every other year (2020,
|
||||
2022, 2024, etc), but to combine that change with a new policy and approach that
|
||||
allows the introduction of backwards compatible features in the initial set of
|
||||
point releases within a given release series.
|
||||
|
||||
In the event that this more complex proposal is *not* accepted (and either a
|
||||
faster release cycle is adopted, or the status quo continues), the PEP aims to
|
||||
provide useful design input for any other "Long Term Support branch"
|
||||
proposals that may be put forward in the future (such as the EL Python draft at
|
||||
[3_]).
|
||||
|
||||
|
||||
Summary
|
||||
=======
|
||||
|
||||
The proposal to keep the current CPython release compatibility management
|
||||
process, but go through it more often has significant practical downsides,
|
||||
as a CPython feature release carries certain expectations (most notably, a 5
|
||||
year maintenance lifecycle, support for parallel installation with the previous
|
||||
feature release, and the possibility of breaking changes to the CPython-specific
|
||||
ABI, requiring recompilation of all extension modules) that mean faster feature
|
||||
releases would significantly increase the burden of maintaining 3rd party
|
||||
Python libraries and applications across all actively supported CPython
|
||||
releases.
|
||||
releases in their current form have the potential to significantly increase the
|
||||
burden of maintaining 3rd party Python libraries and applications across all
|
||||
actively supported CPython releases.
|
||||
|
||||
It's also arguable whether such an approach would actually noticeably reduce
|
||||
the typical feature delivery latency, as the adoption cycle for new feature
|
||||
releases is typically measured in months or years, so more frequent releases
|
||||
may just lead to end users updating to every 3rd or 4th feature release, rather
|
||||
than every 2nd or 3rd feature release (as already happens today).
|
||||
It's also arguable whether such an approach would noticeably reduce the typical
|
||||
feature delivery latency in practice for most end users, as the adoption cycle
|
||||
for new feature releases is typically measured in months or years, so more
|
||||
frequent releases may just lead to end users updating to every 3rd or 4th
|
||||
feature release, rather than every 2nd or 3rd feature release (as often happens
|
||||
today).
|
||||
|
||||
This PEP presents a competing proposal to instead *slow down* the frequency of
|
||||
parallel installable feature releases that change the filesystem layout
|
||||
|
|
@ -48,6 +67,12 @@ open to further incremental feature releases. Alternate implementations of
|
|||
Python would also be free to clear this flag to indicate that their support for
|
||||
their nominal Python version may still be a work in progress.
|
||||
|
||||
For compatibility testing purposes, and to maintain pickle compatibility in
|
||||
mixed version environments, a new ``sys.feature_limit`` attribute (and
|
||||
corresponding CPython CLI parameter, ``--feature-limit X.Y.Z``, and environment
|
||||
variable, ``PYTHONFEATURELIMIT``) will provide a way to limit the runtime
|
||||
availability of features added in incremental feature releases.
|
||||
|
||||
The existing cycle and the new cycle would be synchronised on their feature
|
||||
freeze release dates, so the full Python 3.9.x feature freeze would occur in
|
||||
October 2021, 24 months after the Python 3.8.0 feature release, with the initial
|
||||
|
|
@ -290,6 +315,17 @@ The intent of this change in policy is to allow usability improvements for new
|
|||
rather than requiring users to incur the inherent delay and costs of waiting for
|
||||
and then upgrading to the next feature release series.
|
||||
|
||||
It is also designed such that the approval to add a feature to the next baseline
|
||||
feature release can be considered separately from the question of whether or not
|
||||
to make it available in the next incremental feature release for the current
|
||||
release series, potentially allowing the first task to be completed by volunteer
|
||||
contributors, while the latter activity could be handled by paid contributors
|
||||
(e.g. customers of commercial Python redistributors could potentially request
|
||||
that their vendor backport a feature, or core developers could offer to
|
||||
undertake specific backports on a contract basis). (There would be potential
|
||||
ethical concerns with gating bug fixes this way, but those concerns don't apply
|
||||
for backports of new features)
|
||||
|
||||
Key characteristics of an incremental feature release:
|
||||
|
||||
- in an incremental feature release, ``sys.version_info.feature_complete == False``
|
||||
|
|
@ -303,6 +339,11 @@ Key characteristics of an incremental feature release:
|
|||
``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available``
|
||||
set, and the various ``os.supports_*`` sets that the ``os`` module already
|
||||
offers for platform dependent capability detection
|
||||
- to maintain pickle compatibility in mixed version environments, and to enable
|
||||
easier compatibility testing across multiple API versions within the same
|
||||
release series, all API additions made in an incremental feature release
|
||||
must support the new ``sys.feature_limit`` setting as described in the next
|
||||
section
|
||||
|
||||
Key expectations around incremental feature releases:
|
||||
|
||||
|
|
@ -317,12 +358,103 @@ Key expectations around incremental feature releases:
|
|||
in earlier versions)
|
||||
|
||||
Some standard library modules may also impose their own restrictions on
|
||||
acceptable changes in incremental feature releases (for example, new hash
|
||||
algorithms should only ever be added to ``hashlib.algorithms_guaranteed`` in
|
||||
a baseline feature release - incremental feature releases would only be
|
||||
acceptable changes in incremental feature releases (for example, only a
|
||||
baseline feature release should ever add new hash algorithms to
|
||||
``hashlib.algorithms_guaranteed`` - incremental feature releases would only be
|
||||
permitted to add algorithms to ``hashlib.algorithms_available``)
|
||||
|
||||
|
||||
Maintaining interoperability across incremental feature releases
|
||||
----------------------------------------------------------------
|
||||
|
||||
It is a common practice to use Python's ``pickle`` module to exchange
|
||||
information between Python processes running on different versions of Python.
|
||||
Between release series, this compatibility is expected to only run one way
|
||||
(i.e. excluding deprecated APIs, Python "X.Y+1" processes should be able to
|
||||
read pickle archives produced by Python "X.Y" processes, but the reverse does
|
||||
not hold, as the newer archives may reference attributes and parameters that
|
||||
don't exist in the older version).
|
||||
|
||||
Within a release series, however, it is expected to hold in both directions,
|
||||
as the "No new features" policy means that almost all pickle archives created
|
||||
on Python "X.Y.Z+1" will be readable by Python "X.Y.Z" processes.
|
||||
|
||||
Similarly, Python libraries and applications are often only tested against
|
||||
the latest version in a release series, and this is usually sufficient to keep
|
||||
code working on earlier releases in that same series.
|
||||
|
||||
Allowing feature additions in later "X.Y.Z" releases with no way to turn them
|
||||
off would pose a problem for these common practices, as a library or application
|
||||
that works fine when tested on CPython version "X.Y.Z" would fail on earlier
|
||||
versions if it used a feature newly introduced in "X.Y.Z", and any pickle
|
||||
archives it creates that rely on those new interfaces may also not be readable
|
||||
on the older versions.
|
||||
|
||||
To help address these problems, a new ``sys.feature_limit`` attribute would be
|
||||
added, as a structured sequence corresponding to the first 3 fields in
|
||||
``sys.version_info`` (``major``, ``minor``, ``micro``).
|
||||
|
||||
A new CLI option (``--feature-limit X.Y.Z``) and environment variable
|
||||
(``PYTHONFEATURELIMIT=X.Y.Z``) would be used to set this attribute. The
|
||||
``PyCoreConfig`` struct would also gain a new field::
|
||||
|
||||
wchar_t *feature_limit;
|
||||
|
||||
If the limit is not set explicitly, it would default to the first 3 fields in
|
||||
``sys.version_info``. If the limit is set to a value outside the lower bound of
|
||||
``sys.version_info[:2]`` and the upper bound of ``sys.version_info[:3]``, it
|
||||
will be clamped to those bounds, padding with zeroes if necessary.
|
||||
|
||||
For example, given a current version of "3.9.3", nominal limits would be
|
||||
converted to runtime ``sys.feature_limit`` values as follows::
|
||||
|
||||
3 => (3, 9, 0)
|
||||
3.8.1 => (3, 9, 0)
|
||||
3.9 => (3, 9, 0)
|
||||
3.9.2 => (3, 9, 2)
|
||||
<unset> => (3, 9, 3)
|
||||
3.9.3 => (3, 9, 3)
|
||||
3.9.4 => (3, 9, 3)
|
||||
4 => (3, 9, 3)
|
||||
|
||||
New APIs backported to an incremental feature release would be expected to
|
||||
include a guard that deletes the API from the module if the feature limit is
|
||||
too low::
|
||||
|
||||
def feature_api():
|
||||
...
|
||||
|
||||
_version_feature_api_added = (3, 9, 1)
|
||||
if _version_feature_api_added > sys.feature_limit:
|
||||
del feature_api
|
||||
|
||||
Similarly, new parameters would be expected to include a guard that adjusts the
|
||||
function signature to match the old one::
|
||||
|
||||
|
||||
def feature_api(old_param1, old_param2, new_param=default):
|
||||
"""Updated API docstring"""
|
||||
...
|
||||
|
||||
_version_feature_api_changed = (3, 9, 1)
|
||||
if _version_feature_api_changed > sys.feature_limit:
|
||||
_new_feature_api = feature_api
|
||||
def feature_api(old_param1, old_param2):
|
||||
"""Legacy API docstring"""
|
||||
return _new_feature_api(old_param1, old_param2)
|
||||
|
||||
|
||||
Structuring the guards this way would keep the code structure as similar as
|
||||
possible between the main development branch and the backport branches, so
|
||||
future bug fixes can still be backported automatically.
|
||||
|
||||
It is expected that convenience functions and/or additional automated tests
|
||||
would eventually be added to help ensure these backported APIs are guarded
|
||||
appropriately, but it seems reasonable to wait until specific concrete
|
||||
examples are available to drive the design of those APIs and automated tests,
|
||||
rather than designing them solely on the basis of hypothetical examples.
|
||||
|
||||
|
||||
Feature complete release and subsequent maintenance releases
|
||||
------------------------------------------------------------
|
||||
|
||||
|
|
@ -446,8 +578,8 @@ X.Y+1 is out, but support X.Z until X.Z+2 is out".
|
|||
So this PEP takes as its starting assumption that the vast majority of Python
|
||||
users simply *shouldn't need to care* that we're changing our release policy,
|
||||
and the only folks that should be affected are those that are eagerly waiting
|
||||
for standard library improvements, and other backwards compatible interpreter
|
||||
enhancements, and those that need to manage mission critical applications in
|
||||
for standard library improvements (and other backwards compatible interpreter
|
||||
enhancements), and those that need to manage mission critical applications in
|
||||
complex deployment environments.
|
||||
|
||||
|
||||
|
|
@ -514,8 +646,10 @@ Implications for simple deployment environments
|
|||
For the purposes of this PEP, a "simple" deployment environment is any use case
|
||||
where it is straightforward to ensure that all target environments are updated
|
||||
to a new Python micro version at the same time (or at least in advance of the
|
||||
rollout of new higher level application versions), such that any pre-release
|
||||
testing that occurs need only target a single Python micro version.
|
||||
rollout of new higher level application versions), and there isn't any
|
||||
requirement for older Python versions to be able to reliably read pickle streams
|
||||
generated with the newer Python version, such that any pre-release testing that
|
||||
occurs need only target a single Python micro version.
|
||||
|
||||
The simplest such case would be scripting for personal use, where the testing
|
||||
and target environments are the exact same environment.
|
||||
|
|
@ -556,38 +690,22 @@ organisations could potentially still adopt a new Python version every other
|
|||
year - it would just be based on the availability of the feature complete
|
||||
releases, rather than the baseline feature releases.
|
||||
|
||||
A less strict alternative to outright prohibition would be to restrict the use
|
||||
of any release series that is still in its feature release phase to applications
|
||||
where the occasional outage or failed deployment due to a lack of forwards
|
||||
compatibility is considered acceptable.
|
||||
A less strict alternative to outright prohibition would be to make use of the
|
||||
proposed ``PYTHONFEATURELIMIT`` setting to enable phased migrations to new
|
||||
incremental feature releases:
|
||||
|
||||
However, a third variant, which allows selective adoption of new language
|
||||
features where appropriate, while also degrading gracefully enough to be
|
||||
suitable for mission critical applications, would be to institute a policy that
|
||||
applications wishing to target a release series that is not yet feature complete
|
||||
must also support the previous feature release series for compatibility testing
|
||||
purposes.
|
||||
|
||||
If this last policy is adopted, then testing against the previous release series
|
||||
becomes the new proxy for testing against the baseline feature release and any
|
||||
older incremental feature releases of the newer release series, without actually
|
||||
needing to install and test against all of them.
|
||||
|
||||
Only after the newer release series is feature complete would support for the
|
||||
previous release series be dropped entirely.
|
||||
|
||||
In a sufficiently complex environment, the second and third policies could also
|
||||
be combined, with critical applications maintaining compatibility with the
|
||||
previous release series, while newer, more experimental, applications and
|
||||
services would be permitted to target the newer release series directly without
|
||||
any additional safeguards in their testing process.
|
||||
|
||||
Depending on demand and interest, there are also further enhancements that could
|
||||
be made to continuous integration pipelines to help ensure compatibility with
|
||||
a chosen minimum version within a release series, without needing to run tests
|
||||
against multiple micro releases. For example, applications could potentially be
|
||||
tested against the latest incremental feature release, but typechecked against
|
||||
the oldest still deployed incremental feature release.
|
||||
* initially roll out Python X.Y.0 with ``PYTHONFEATURELIMIT=X.Y.0`` set in CI
|
||||
and on deployment
|
||||
* roll out Python X.Y.1 to CI, keeping the ``PYTHONFEATURELIMIT=X.Y.0`` setting
|
||||
* deploy Python X.Y.1 to production based on successful CI results
|
||||
* update deployment environments to set ``PYTHONFEATURELIMIT=X.Y.1``
|
||||
* set ``PYTHONFEATURELIMIT=X.Y.1`` in CI only after all deployment environments
|
||||
have been updated
|
||||
* repeat this process for each new release up to and including the feature
|
||||
complete release for the release series
|
||||
* once the series is feature complete, either continue with this same process
|
||||
for consistency's sake, or else stop updating ``PYTHONFEATURELIMIT`` and leave
|
||||
it at the feature complete version number
|
||||
|
||||
|
||||
Duration of the feature additions period
|
||||
|
|
@ -763,6 +881,9 @@ References
|
|||
.. [2] NEP proposing a standard policy for dropping support of old Python versions
|
||||
(https://github.com/numpy/numpy/pull/14086)
|
||||
|
||||
.. [3] Draft Extended Lifecycle for Python (ELPython) design concept
|
||||
(https://github.com/elpython/elpython-meta/blob/master/README.md)
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue