diff --git a/pep-0606.rst b/pep-0606.rst new file mode 100644 index 000000000..0fb45ea28 --- /dev/null +++ b/pep-0606.rst @@ -0,0 +1,625 @@ +PEP: 606 +Title: Python Compatibility Version +Author: Victor Stinner +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 18-Oct-2019 +Python-Version: 3.9 + + +Abstract +======== + +Add ``sys.set_python_compat_version(version)`` to enable partial +compatibility with requested Python version. Add +``sys.get_python_compat_version()``. + +Modify a few functions of the standard library to implement a partial +compatibility with Python 3.8. + +Add ``sys.set_python_min_compat_version(version)`` to deny backward +compatibility with Python older than *version*. + +Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` +command line options. Add ``PYTHONCOMPATVERSION`` and +``PYTHONCOMPATMINVERSION`` environment variables. + + +Rationale +========= + +The need to evolve frequently +----------------------------- + +To remain relevant and useful, Python has to evolve frequently. Some +enhancements require incompatible changes. Any incompatible change can +break an unknown number of Python projects. Developers can decide to +not implement a feature because of that. + +Users want to get the latest Python version to get new features and +better performance. A few incompatible changes prevent them to use their +applications on the latest Python version. + +This PEP proposes to add a partial compatibility with old Python +versions as a tradeoff to fit both use cases. + +The main issue with the migration from Python 2 to Python 3 is not that +Python 3 is backward incompatible, but how incompatible changes were +introduced. + + +Partial compatibility to minimize the Python maintenance burden +--------------------------------------------------------------- + +While technically it would be possible to provide a full compatibility +with old Python versions, this PEP proposes to minimize the number of +functions handling backward compatibility to reduce the maintenance +burden of the Python project ("CPython"). + +Each change introducing backport compatibility to a function should be +properly discussed to estimate the maintenance cost in the long-term. + +Backward compatibility code will be dropped at each Python release, on a +case by case basis. Each compatibility function can be supported for a +different number of Python releases depending on its maintenance cost +and the estimated risk (number of broken projects) if it's removed. + +The maintenance cost does not only come from the code implementing the +backward compatibility, but come also from additional tests. + + +Cases excluded from backward compatibility +------------------------------------------ + +The performance overhead of a compatibility code must be low when +``sys.set_python_compat_version()`` is not called. + +The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and +the stable ABI are solving this problem differently, see the `PEP 384: +Defining a Stable ABI `_. + +Security fixes which break the backward compatibility on purpose will +not get a compatibility layer. Security matters more than compatibility. +For example, ``http.client.HTTPSConnection`` was modified in Python +3.4.3 to performs all the necessary certificate and hostname checks by +default. It was a deliberate change motivated by the `PEP 476: Enabling +certificate verification by default for stdlib http clients +`_ (`bpo-22417 +`_). + +The Python language does not provide backward compatibility. + +Changes which are not clearly incompatible are not covered by this PEP. +For example, Python 3.9 changed the default protocol in the ``pickle`` +module to Protocol 4 which was first introduced in Python 3.4. This +change is backward compatible up to Python 3.4. There is no need to use +the Protocol 3 by default when compatibility with Python 3.8 is +requested. + +New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings +of Python 3.9 will not be disabled in Python 3.8 compatibility mode. +If a project runs its test suite using ``-Werror`` (treat any warning as +an error), these warnings must be fixed, or specific deprecation +warnings must be ignored on a case by case basis. + + +Upgrade a project to a newer Python +----------------------------------- + +Without backward compatibility, all incompatible changes must be fixed +at once, which can be a blocker issue. It is even worse when a project +is upgraded to a newer Python which is separated by multiple releases +from the old Python. + +Postponing an upgrade only makes things worse: each skipped release adds +more incompatible changes. The technical debt is only steadily +increasing. + +With backward compatibility, it becomes possible to upgrade Python +increamentally in a project, without having to fix all issues at once. + +The "all-or-nothing" is a showstopper to port large Python 2 code bases +to Python 3. The list of incompatible changes between Python 2 and +Python 3 is long, and it's getting longer at each Python 3.x release. + + +Cleaning up Python and DeprecationWarning +----------------------------------------- + +One of the `Zen of Python (PEP 20) +`_ motto is: + + There should be one-- and preferably only one --obvious way to do + it. + +When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` +are emitted to suggest to use the new way, but many developers ignore +these warnings, which are silent by default (except in the ``__main__`` +module: see the `PEP 565 _`). +Some developers simply ignore all warnings since they are too many +warnings, and so only bother with exceptions when deprecated code is +removed. + +Sometimes, supporting both ways has a minor maintenance cost, but +developers prefer to drop the old way to clean up the code. Such kind of +change is backward incompatible. + +Some developers can take the end of the Python 2 support as an +opportunity to push even more incompatible changes than usual. + +Adding backward compatibility as an opt-in prevents to break +applications and allows developers to continue to do such cleanup. + + +Redistribute the maintenance burden +----------------------------------- + +The backward compatibility involves authors of backward incompatible +changes more in the upgrade path. + + +Examples of backward compatibility +================================== + +collections ABC aliases +----------------------- + +``collections.abc`` aliases to ABC classes have been removed from the +``collections`` module in Python 3.9, after being deprecated since +Python 3.3. For example, ``collections.Mapping`` no longer exists. + +In Python 3.6, aliases were created in ``collections/__init__.py`` by +``from _collections_abc import *``. + +In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` +module to emit a DeprecationWarning at the first access to an +attribute:: + + def __getattr__(name): + # For backwards compatibility, continue to make the collections ABCs + # through Python 3.6 available through the collections module. + # Note, no new collections ABCs were added in Python 3.7 + if name in _collections_abc.__all__: + obj = getattr(_collections_abc, name) + import warnings + warnings.warn("Using or importing the ABCs from 'collections' instead " + "of from 'collections.abc' is deprecated since Python 3.3, " + "and in 3.9 it will stop working", + DeprecationWarning, stacklevel=2) + globals()[name] = obj + return obj + raise AttributeError(f'module {__name__!r} has no attribute {name!r}') + +Compatibility with Python 3.8 can be restored in Python 3.9 by adding +back the ``__getattr__()`` function, but only when backward +compatibility is requested:: + + def __getattr__(name): + if (sys.get_python_compat_version() < (3, 9) + and name in _collections_abc.__all__): + ... + raise AttributeError(f'module {__name__!r} has no attribute {name!r}') + + +Deprecated open() "U" mode +-------------------------- + +The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a +``DeprecationWarning``. The `bpo-37330 +`_ proposes to drop this mode: +``open()`` would raise an exception if ``U`` mode is used. + +This change falls into the "cleanup" category: it is not required to +implement a feature. + +A backward compatibility mode would be trivial to implement and would be +welcomed here by users. + + +Specification +============= + +sys functions +------------- + +Add 3 functions to the ``sys`` module: + +* ``sys.set_python_compat_version(version)``: set the Python + compatibility version. If it has been called previously, use the + minimum of requested versions. Raise an exception if + ``sys.set_python_min_compat_version(min_version)`` has been called and + ``version < min_version``. + *version* must be greater than or equal to ``(3, 0)``. + +* ``sys.set_python_min_compat_version(min_version)``: set the + **minimum** compatibility version. Raise an exception if + ``sys.set_python_compat_version(old_version)`` has been called + previously and ``old_version < min_version``. + *min_version* must be greater than or equal to ``(3, 0)``. + +* ``sys.get_python_compat_version()``: get the Python compatibility + version. Return a ``tuple`` of 3 integers. + +A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version +is equivalent to ``(major, minor, 0)``. + +By default, ``sys.get_python_compat_version()`` returns the current +Python version. + +Example to request compatibility with Python 3.8.0:: + + import collections + + sys.set_python_compat_version((3, 8)) + + # collections.Mapping alias, removed from Python 3.9, is available + # again, even if collections has been imported before calling + # set_python_compat_version(). + parent = collections.Mapping + +Obviously, calling ``sys.set_python_compat_version(version)`` has no +effect on code executed before the call. Use ``-X +compat_version=VERSION`` command line option or +``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the +compatibility version at Python startup. + +Command line +------------ + +Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` +command line options: call respectivelly +``sys.set_python_compat_version()`` and +``sys.set_python_min_compat_version()``. ``VERSION`` is a version string +with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For +example, ``-X compat_version=3.8`` calls +``sys.set_python_compat_version((3, 8))``. + +Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and +``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call +respectivelly ``sys.set_python_compat_version()`` and +``sys.set_python_min_compat_version()``. ``VERSION`` is a version +string with the same format that the command line options. + + +Backwards Compatibility +======================= + +Introducing ``sys.set_python_compat_version()`` function means that an +application will behave differently depending on the compatibility +version. Moreover, since the version can be decreased multiple times, +the application can behave differently depending on the import order. + +Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully +compatible with Python 3.8: the compatibility is only partial. + + +Security Implications +===================== + +``sys.set_python_compat_version()`` must not disable security fixes. + + +Alternatives +============ + +Provide a workaround for each incompatible change +------------------------------------------------- + +An application can works around most of the incompatible changes which +impacts it. + +For example, ``collections`` aliases can be added again using:: + + import collections.abc + collections.Mapping = collections.abc.Mapping + collections.Sequence = collections.abc.Sequence + +Handle backward compatibility in the parser +------------------------------------------- + +The parser is modified to support multiple versions of the Python +language (grammar). + +The current Python parser cannot be easily modified for that. AST and +grammar are hardcoded to a single Python version. + +In Python 3.8, ``compile()`` has an undocumented +``_feature_version`` to not consider ``async`` and ``await`` as +keywords. + +The latest major language backward incompatible change was Python 3.7 +which made ``async`` and ``await`` real keywords. It seems like Twisted +was the only affected project, and Twisted had a single affected +function (it used a parameter called ``async``). + +Handling backward compatibility in the parser seems quite complex, not +only to modify the parser, but also for developers who have to check +which version of the Python language is used. + +from __future__ import python38_syntax +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable +backward compatibility with Python X.Y syntax, but only for the current +file. + +With this option, there is no need to change +``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, +since the parser would always produce the same output for the same input +(except of the optimization level). + +Example:: + + from __future__ import python35_syntax + + async = 1 + await = 2 + +Update cache_tag +^^^^^^^^^^^^^^^^ + +Modify the parser to use ``sys.get_python_compat_version()`` to choose +the version of the Python language. + +``sys.set_python_compat_version()`` updates +``sys.implementation.cache_tag`` to include the compatibility version +without the micro version as a suffix. For example, Python 3.9 uses +``'cpython-39'`` by default, but +``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to +``'cpython-39-37'``. Changes of the Python language are now allowed +in micro releases. + +One problem is that ``import asyncio`` is likely to fail if +``sys.set_python_compat_version((3, 6))`` has been called previously. +The code of the ``asyncio`` module requires ``async`` and ``await`` to +be real keywords (change done in Python 3.7). + +Another problem is that regular users cannot write ``.pyc`` files into +system directories, and so cannot create them on demand. It means that +``.pyc`` optimization cannot be used in the backward compatibility mode. + +One solution for that is to modify the Python installer and Python +package installers to precompile ``.pyc`` files not only for the current +Python version, but also for multiple older Python versions (up to +Python 3.0?). + +Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), +where ``n`` is the number of supported Python versions. For example, it +means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python +3.9. + + +Temporary moratorium on incompatible changes +-------------------------------------------- + +In 2009, the PEP 3003 "Python Language Moratorium" proposed to a +temporary moratorium (suspension) of all changes to the Python language +syntax, semantics, and built-ins for Python 3.1 and Python 3.2. + +In May 2018, during PEP 572 discussions, it was also proposed to slow +down Python changes: see the python-dev thread `Slow down... +`_ + +`Barry Warsaw's call on this +`_: + + I don’t believe that the way for Python to remain relevant and + useful for the next 10 years is to cease all language evolution. + Who knows what the computing landscape will look like in 5 years, + let alone 10? Something as arbitrary as a 10 year moratorium is + (again, IMHO) a death sentence for the language. + +PEP 387 +------- + +`PEP 387 -- Backwards Compatibility Policy +`_ proposes a process to make +incompatible changes. The main point is the 4th step of the process: + + See if there's any feedback. Users not involved in the original + discussions may comment now after seeing the warning. Perhaps + reconsider. + +PEP 497 +------- + +`PEP 497 -- A standard mechanism for backward compatibility +`_ proposes different +solutions to provide backward compatibility. + +Except of the ``__past__`` mechanism idea, the PEP 497 does not propose +concrete solutions: + + When an incompatible change to core language syntax or semantics is + being made, Python-dev's policy is to prefer and expect that, + wherever possible, a mechanism for backward compatibility be + considered and provided for future Python versions after the + breaking change is adopted by default, in addition to any mechanisms + proposed for forward compatibility such as new future_statements. + + +Examples of incompatible changes +================================ + +Python 3.8 +---------- + +Examples of Python 3.8 incompatible changes: + +* (During beta phase) ``PyCode_New()`` required a new parameter: it + broke all Cython extensions (all projects distributing precompiled + Cython code). This change has been reverted during the 3.8 beta phase + and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead. + +* ``types.CodeType`` requires an additional mandatory parameter. + The ``CodeType.replace()`` function was added to help projects to no + longer depend on the exact signature of the ``CodeType`` constructor. + +* C extensions are no longer linked to libpython. + +* ``sys.abiflags`` changed from ``'m'`` to an empty string. + For example, ``python3.8m`` program is gone. + +* The C structure ``PyInterpreterState`` was made opaque. + + * Blender: + + * https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 + * https://developer.blender.org/D6038 + +* XML attribute order: `bpo-34160 + `_. Broken projects: + + * `coverage `_ + * `docutils `_ + * `pcs `_ + * `python-glyphsLib + `_ + +Backward compatibility cannot be added for all these changes. For +example, changes in the C API and in the build system are out of the +scope of this PEP. + +See `What’s New In Python 3.8: API and Feature Removals +`_ +for all changes. + +See also the `Porting to Python 3.8 +`_ +section of What’s New In Python 3.8. + + +Python 3.7 +---------- + +Examples of Python 3.7 incompatible changes: + +* ``async`` and ``await`` are now reserved keywords. +* Several undocumented internal imports were removed. One example is + that ``os.errno`` is no longer available; use ``import errno`` + directly instead. Note that such undocumented internal imports may be + removed any time without notice, even in micro version releases. +* Unknown escapes consisting of ``'\'`` and an ASCII letter in + replacement templates for ``re.sub()`` were deprecated in Python 3.5, + and will now cause an error. +* The ``asyncio.windows_utils.socketpair()`` function has been removed: + it was an alias to ``socket.socketpair()``. +* ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` + modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace + ``from asyncio import selectors`` with ``import selectors``. +* PEP 479 is enabled for all code in Python 3.7, meaning that + ``StopIteration`` exceptions raised directly or indirectly in + coroutines and generators are transformed into ``RuntimeError`` + exceptions. +* ``socketserver.ThreadingMixIn.server_close()`` now waits until all + non-daemon threads complete. Set the new ``block_on_close`` class + attribute to ``False`` to get the pre-3.7 behaviour. +* The ``struct.Struct.format`` type is now ``str`` instead of + ``bytes``. +* ``repr`` for ``datetime.timedelta`` has changed to include the keyword + arguments in the output. +* ``tracemalloc.Traceback`` frames are now sorted from oldest to most + recent to be more consistent with ``traceback``. + +Adding backward compatibility for most of these changes would be easy. + +See also the `Porting to Python 3.7 +`_ +section of What’s New In Python 3.7. + + +Micro releases +-------------- + +Sometimes, incompatible changes are introduced in micro releases +(``micro`` in ``major.minor.micro``) to fix bugs or security +vulnerabilities. Examples: + +* Python 3.7.2, ``compileall`` and ``py_compile`` module: the + *invalidation_mode* parameter's default value is updated to ``None``; + the ``SOURCE_DATE_EPOCH`` environment variable no longer + overrides the value of the *invalidation_mode* argument, and + determines its default value instead. + +* Python 3.7.1, ``xml`` modules: the SAX parser no longer processes + general external entities by default to increase security by default. + +* Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` + syscall blocks (the urandom entropy pool is not initialized yet), fall + back on reading ``/dev/urandom``. + +* Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` + exception is now raised if the new limit is too low at the current + recursion depth. + +* Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from + the default cipher string. + +* Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all + the necessary certificate and hostname checks by default. + +* Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is + now a method instead of a property, for consistency with + ``Message.is_multipart()``. + +* Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: + Before Python 3.4.1, if *exist_ok* was ``True`` and the directory + existed, ``makedirs()`` would still raise an error if *mode* did not + match the mode of the existing directory. Since this behavior was + impossible to implement safely, it was removed in Python 3.4.1 + (`bpo-21082 `_). + +Examples of changes made in micro releases which are not backward +incompatible: + +* ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 + for backwards compatibility with OpenSSL 1.0.2. +* ``typing.AsyncContextManager`` was added to Python 3.6.2. +* The ``zipfile`` module accepts a path-like object since Python 3.6.2. +* ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` + module. + +No backward compatibility code is needed for such kind of changes. + + +References +========== + +Accepted PEPs: + +* `PEP 5 -- Guidelines for Language Evolution + `_ +* `PEP 236 -- Back to the __future__ + `_ +* `PEP 411 -- Provisional packages in the Python standard library + `_ +* `PEP 3002 -- Procedure for Backwards-Incompatible Changes + `_ + +Draft PEPs: + +* `PEP 602 -- Annual Release Cycle for Python + `_ +* `PEP 605 -- A rolling feature release stream for CPython + `_ +* See also withdrawn `PEP 598 -- Introducing incremental feature + releases `_ + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: