298 lines
9.9 KiB
ReStructuredText
298 lines
9.9 KiB
ReStructuredText
PEP: 608
|
||
Title: Coordinated Python release
|
||
Author: Miro Hrončok <miro@hroncok.cz>,
|
||
Victor Stinner <vstinner@python.org>
|
||
Status: Rejected
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 25-Oct-2019
|
||
Python-Version: 3.9
|
||
|
||
Abstract
|
||
========
|
||
|
||
Block a Python release until a compatible version of selected projects
|
||
is available.
|
||
|
||
The Python release manager can decide to release Python even if a
|
||
project is not compatible, if they decide that the project is going to
|
||
be fixed soon enough, or if the issue severity is low enough.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
The PEP involves maintainers of the selected projects in the Python
|
||
release cycle. There are multiple benefit:
|
||
|
||
* Detect more bugs before a Python final release
|
||
* Discuss and maybe revert incompatible changes before a Python final
|
||
release
|
||
* Increase the number of compatible projects when the new Python final
|
||
version is released
|
||
|
||
Too few projects are involved in the Python beta phase
|
||
------------------------------------------------------
|
||
|
||
Currently, Python beta versions are available four months before the
|
||
final 3.x.0 release.
|
||
|
||
Bugs reported during the beta phase can be easily fixed and can block a
|
||
release if they are serious enough.
|
||
|
||
Incompatible changes are discussed during the beta phase: enhance
|
||
documentation explaining how to update code, or consider to revert these
|
||
changes.
|
||
|
||
Even if more and more projects are tested on the master branch of Python
|
||
in their CI, too many projects of the top 50 PyPI projects are only
|
||
compatible with the new Python a few weeks, or even months, after the
|
||
final Python release.
|
||
|
||
DeprecationWarning is being ignored
|
||
-----------------------------------
|
||
|
||
Python has well defined process to deprecate features. A
|
||
DeprecationWarning must be emitted during at least one Python release,
|
||
before a feature can be removed.
|
||
|
||
In practice, DeprecationWarning warnings are ignored for years in major
|
||
Python projects. Usually, maintainers explain that there are too many
|
||
warnings and so they simply ignore warnings. Moreover, DeprecationWarning
|
||
is silent by default (except in the ``__main__`` module: `PEP 565
|
||
<https://www.python.org/dev/peps/pep-0565/>`_).
|
||
|
||
Even if more and more projects are running their test suite with
|
||
warnings treated as errors (``-Werror``), Python core developers still
|
||
have no idea how many projects are broken when a feature is removed.
|
||
|
||
Need to coordinate
|
||
------------------
|
||
|
||
When issues and incompatible changes are discovered and discussed after
|
||
the final Python release, it becomes way more complicated and expensive
|
||
to fix Python. Once an API is part of an official final release, Python
|
||
should provide backward compatibility for the whole 3.x release
|
||
lifetime. Some operating systems can be shipped with the buggy final
|
||
release and can take several months before being updated.
|
||
|
||
Too many projects are only updated to the new Python after the final
|
||
Python release, which makes this new Python version barely usable to run
|
||
large applications when Python is released.
|
||
|
||
It is proposed to block a Python release until a compatible version of
|
||
all selected projects is available.
|
||
|
||
Shorter Python release schedule
|
||
-------------------------------
|
||
|
||
The `PEP 602: Annual Release Cycle for Python
|
||
<https://www.python.org/dev/peps/pep-0602/>`_ and the `PEP 605: A
|
||
rolling feature release stream for CPython
|
||
<https://www.python.org/dev/peps/pep-0605/>`_ would like to release
|
||
Python more often to ship new features more quickly.
|
||
|
||
The problem is that each Python ``3.x`` release breaks many projects.
|
||
|
||
Coordinated Python releases reduces the number of broken projects and
|
||
makes new Python release more usable.
|
||
|
||
|
||
Specification
|
||
=============
|
||
|
||
By default, a Python release is blocked until a compatible version of
|
||
all selected projects is available.
|
||
|
||
Before releasing the final Python version, the Python release manager is
|
||
responsible to send a report of the compatibility status of each project
|
||
of the selected projects. It is recommended to send such report at
|
||
each beta release to see the evolution and detects issues as soon as
|
||
possible.
|
||
|
||
The Python release manager can decide to release Python even if a
|
||
project is not compatible, if they decide that the project is going to
|
||
be fixed soon enough, or if the issue severity is low enough.
|
||
|
||
After each Python release, the project list can be updated to remove
|
||
projects and add new ones. For example, to remove old unused
|
||
dependencies and add new ones. The list can grow if the whole process
|
||
doesn't block Python releases for too long.
|
||
|
||
Limit the delay
|
||
---------------
|
||
|
||
When a build or test issue with the next Python version is reported to a
|
||
project, maintainers have one month to answer. With no answer, the
|
||
project can be excluded from the list of projects blocking the Python
|
||
release.
|
||
|
||
Multiple projects are already tested on the master branch of Python in a
|
||
CI. Problems can be detected very early in a Python release which should
|
||
provide enough time to handle them. More CI can be added for projects
|
||
which are not tested on the next Python yet.
|
||
|
||
Once selected projects issues are known, exceptions can be discussed
|
||
between the Python release manager and involved project maintainers on a
|
||
case-by-case basis. Not all issues deserve to block a Python release.
|
||
|
||
Selected projects
|
||
-----------------
|
||
|
||
List of projects blocking a Python release (total: 27):
|
||
|
||
* Projects (13):
|
||
|
||
* aiohttp
|
||
* cryptography
|
||
* Cython
|
||
* Django
|
||
* numpy
|
||
* pandas
|
||
* pip
|
||
* requests
|
||
* scipy
|
||
* Sphinx (needed to build Python)
|
||
* sqlalchemy
|
||
* pytest
|
||
* tox
|
||
|
||
* Direct and indirect dependencies (14):
|
||
|
||
* certifi (needed by urllib3)
|
||
* cffi (needed by cryptography)
|
||
* chardet (needed by Sphinx)
|
||
* colorama (needed by pip)
|
||
* docutils (needed by Sphinx)
|
||
* idna (needed by Sphinx and requests)
|
||
* jinja2 (needed by Sphinx)
|
||
* MarkupSafe (needed by Sphinx)
|
||
* psycopg2 (needed by Django)
|
||
* pycparser (needed by cffi)
|
||
* setuptools (needed by pip and tons of Python projects)
|
||
* six (needed by tons of Python projects)
|
||
* urllib3 (needed by requests)
|
||
* wheel (needed by pip)
|
||
|
||
How projects are selected
|
||
-------------------------
|
||
|
||
Projects used by to build Python should be in the list, like Sphinx.
|
||
|
||
Most popular projects are picked from the most downloaded PyPI projects.
|
||
|
||
Most of project dependencies are included in the list as well, since a
|
||
single incompatible dependency can block a whole project. Some
|
||
dependencies are excluded to reduce the list length.
|
||
|
||
Test dependencies as pytest and tox should be included as well. If a
|
||
project cannot be tested, a new version cannot be shipped neither.
|
||
|
||
The list should be long enough to have a good idea of the cost of
|
||
porting a project to the next Python, but small enough to not block a
|
||
Python release for too long.
|
||
|
||
Obviously, projects which are not part of the list also are encouraged
|
||
to report issues with the next Python and to have a CI running on the
|
||
next Python version.
|
||
|
||
|
||
Incompatible changes
|
||
====================
|
||
|
||
The definition here is large: any Python change which cause an issue
|
||
when building or testing a project.
|
||
|
||
See also the `PEP 606: Python Compatibility Version
|
||
<https://www.python.org/dev/peps/pep-0606/>`_ for more examples of
|
||
incompatible changes.
|
||
|
||
Examples
|
||
--------
|
||
|
||
There are different kinds of incompatible changes:
|
||
|
||
* Change in the Python build. For example, Python 3.8 removed ``'m'``
|
||
(which stands for pymalloc) from ``sys.abiflags`` which impacts Python
|
||
vendors like Linux distributions.
|
||
* Change in the C extensions build. For example, Python 3.8 no longer
|
||
links C extensions to libpython, and Python 3.7 removed
|
||
``os.errno`` alias to the ``errno`` module.
|
||
* Removed function. For example, collections aliases to ABC classes
|
||
have been removed in Python 3.9.
|
||
* Changed function signature:
|
||
|
||
* Reject a type which was previously accepted (ex: only accept ``int``,
|
||
reject ``float``).
|
||
* Add a new mandatory parameter.
|
||
* Convert a positional-or-keyword parameter to positional-only.
|
||
|
||
* Behavior change. For example, Python 3.8 now serializes XML attributes
|
||
in their insertion order, rather than sorting them by name.
|
||
* New warning. Since more and more projects are tested with all warnings
|
||
treated as errors, any new warning can cause a project test to fail.
|
||
* Function removed from the C API.
|
||
* Structure made opaque in the C API. For example, PyInterpreterState
|
||
became opaque in Python 3.8 which broke projects accessing
|
||
``interp->modules`` (``PyImport_GetModuleDict()`` should be used
|
||
instead).
|
||
|
||
Cleaning up Python and DeprecationWarning
|
||
-----------------------------------------
|
||
|
||
One of the `Zen of Python (PEP 20)
|
||
<https://www.python.org/dev/peps/pep-0020/>`_ motto is:
|
||
|
||
There should be one-- and preferably only one --obvious way to do
|
||
it.
|
||
|
||
When Python evolves, new ways emerge inevitably. ``DeprecationWarning``
|
||
are emitted to suggest to use the new way, but many developers ignore
|
||
these warnings which are silent by default.
|
||
|
||
Sometimes, supporting both ways has a minor maintenance cost, but Python
|
||
core developers prefer to drop the old way to clean up the Python code
|
||
base and standard library. Such kind of change is backward incompatible.
|
||
|
||
More incompatible changes than usual should be expected with the end of
|
||
the Python 2 support which is a good opportunity to cleaning up old
|
||
Python code.
|
||
|
||
|
||
Distributed CI
|
||
==============
|
||
|
||
Checking if selected projects are compatible with the master branch
|
||
of Python can be automated using a distributed CI.
|
||
|
||
Existing CIs can be reused.
|
||
|
||
New CIs can be added for projects which are not tested on the next
|
||
Python yet.
|
||
|
||
It is recommended to treat DeprecationWarning warnings as errors when
|
||
testing on the next Python.
|
||
|
||
A job testing a project on the next Python doesn't have to be
|
||
"mandatory" (block the whole CI). It is fine to have failures during the
|
||
beta phase of a Python release. The job only has to pass for the final
|
||
Python release.
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document is placed in the public domain or under the
|
||
CC0-1.0-Universal license, whichever is more permissive.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|