python-peps/pep-0565.rst

371 lines
16 KiB
ReStructuredText
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

PEP: 565
Title: Show DeprecationWarning in __main__
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Nov-2017
Python-Version: 3.7
Post-History: 12-Nov-2017, 25-Nov-2017
Abstract
========
In Python 2.7 and Python 3.2, the default warning filters were updated to hide
DeprecationWarning by default, such that deprecation warnings in development
tools that were themselves written in Python (e.g. linters, static analysers,
test runners, code generators), as well as any other applications that merely
happened to be written in Python, wouldn't be visible to their users unless
those users explicitly opted in to seeing them.
However, this change has had the unfortunate side effect of making
DeprecationWarning markedly less effective at its primary intended purpose:
providing advance notice of breaking changes in APIs (whether in CPython, the
standard library, or in third party libraries) to users of those APIs.
To improve this situation, this PEP proposes a single adjustment to the
default warnings filter: displaying deprecation warnings attributed to the main
module by default.
This change will mean that code entered at the interactive prompt and code in
single file scripts will revert to reporting these warnings by default, while
they will continue to be silenced by default for packaged code distributed as
part of an importable module.
The PEP also proposes a number of small adjustments to the reference
interpreter and standard library documentation to help make the warnings
subsystem more approachable for new Python developers.
As part of the documentation updates, it will be made clearer that the
``unittest`` test runner displays all warnings by default when executing
test cases, and that other test runners are advised to follow that example.
Specification
=============
New default warnings filter entry
---------------------------------
The current set of default warnings filters consists of::
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning
The default ``unittest`` test runner then uses ``warnings.catch_warnings()``
``warnings.simplefilter('default')`` to override the default filters while
running test cases.
The change proposed in this PEP is to update the default warning filter list
to be::
default::DeprecationWarning:__main__
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning
This means that in cases where the nominal location of the warning (as
determined by the ``stacklevel`` parameter to ``warnings.warn``) is in the
``__main__`` module, the first occurrence of each DeprecationWarning will once
again be reported.
This change will lead to DeprecationWarning being displayed by default for:
* code executed directly at the interactive prompt
* code executed directly as part of a single-file script
While continuing to be hidden by default for:
* code imported from another module in a ``zipapp`` archive's ``__main__.py``
file
* code imported from another module in an executable package's ``__main__``
submodule
* code imported from an executable script wrapper generated at installation time
based on a ``console_scripts`` or ``gui_scripts`` entry point definition
This means that tool developers that create an installable or executable
artifact (such as a ``zipapp`` archive) for distribution to their users
shouldn't see any change from the status quo, while users of more ad hoc
personal or locally distributed scripts are likely to start seeing relevant
deprecation warnings again (as they did in Python 2.6 and earlier).
Additional use case for ``FutureWarning``
-----------------------------------------
The standard library documentation will be updated to explicitly recommend the
use of ``FutureWarning`` (rather than ``DeprecationWarning``) for backwards
compatibility warnings that are intended to be seen by *users* of an
application. (This will be in addition to the existing use of ``FutureWarning``
to warn about constructs that will remain valid code in the future,
but will have different semantics).
This will give the following three distinct categories of backwards
compatibility warning, with three different intended audiences:
* ``PendingDeprecationWarning``: hidden by default for all code.
The intended audience is Python developers that take an active interest in
ensuring the future compatibility of their software (e.g. professional
Python application developers with specific support obligations).
* ``DeprecationWarning``: reported by default for code that runs directly in
the ``__main__`` module (as such code is considered relatively unlikely to
have a dedicated test suite), but hidden by default for code in other modules.
The intended audience is Python developers that are at risk of upgrades to
their dependencies (including upgrades to Python itself) breaking their
software (e.g. developers using Python to script environments where someone
else is in control of the timing of dependency upgrades).
* ``FutureWarning``: reported by default for all code.
The intended audience is users of applications written in Python, rather than
other Python developers (e.g. warning about use of a deprecated setting in a
configuration file format).
For library and framework authors that want to ensure their API compatibility
warnings are more reliably seen by their users, the recommendation is to use a
custom warning class that derives from ``DeprecationWarning`` in Python 3.7+,
and from ``FutureWarning`` in earlier versions.
Recommended filter settings for test runners
--------------------------------------------
Developers of test runners are advised to implement logic equivalent to the
following when determining their default warnings filters::
if not sys.warnoptions:
warnings.simplefilter("default")
This effectively enables all warnings by default, as if the ``-Wd`` command
line option had been passed.
Note that actually enabling ``BytesWarning`` in a test suite still requires
passing the ``-b`` option to the interpreter at the command line. For implicit
bytes conversion and bytes comparison warnings, the warnings filter machinery
is only used to determine whether they should be printed as warnings or raised
as exceptions - when the command line flag isn't set, the interpreter doesn't
even emit the warning in the first place.
Recommended filter settings for interactive shells
--------------------------------------------------
Developers of interactive shells are advised to add a filter that enables
``DeprecationWarning`` in the namespace where user code is entered and executed.
If that namespace is ``__main__`` (as it is for the default CPython REPL), then
no changes are needed beyond those in this PEP.
Interactive shell implementations which use a namespace other than
``__main__`` will need to add their own filter. For example, IPython uses the
following command ([8_]) to set up a suitable filter::
warnings.filterwarnings("default", category=DeprecationWarning,
module=self.user_ns.get("__name__"))
Other documentation updates
---------------------------
The current reference documentation for the warnings system is relatively short
on specific *examples* of possible settings for the ``-W`` command line option
or the ``PYTHONWARNINGS`` environment variably that achieve particular end
results.
The following improvements are proposed as part of the implementation of this
PEP:
* Explicitly list the following entries under the description of the
``PYTHONWARNINGS`` environment variable::
PYTHONWARNINGS=error # Convert to exceptions
PYTHONWARNINGS=always # Warn every time
PYTHONWARNINGS=default # Warn once per call location
PYTHONWARNINGS=module # Warn once per calling module
PYTHONWARNINGS=once # Warn once per Python process
PYTHONWARNINGS=ignore # Never warn
* Explicitly list the corresponding short options
(``-We``, ``-Wa``, ``-Wd``, ``-Wm``, ``-Wo``, ``-Wi``) for each of the
warning actions listed under the ``-W`` command line switch documentation
* Explicitly list the default filter set in the ``warnings`` module
documentation, using the ``action::category`` and ``action::category:module``
notation
* Explicitly list the following snippet in the ``warnings.simplefilter``
documentation as a recommended approach to turning off all warnings by
default in a Python application while still allowing them to be turned
back on via ``PYTHONWARNINGS`` or the ``-W`` command line switch::
if not sys.warnoptions:
warnings.simplefilter("ignore")
None of these are *new* (they already work in all still supported Python
versions), but they're not especially obvious given the current structure
of the related documentation.
Reference Implementation
========================
A reference implementation is available in the PR [4_] linked from the
related tracker issue for this PEP [5_].
As a side-effect of implementing this PEP, the internal warnings filter list
will start allowing the use of plain strings as part of filter definitions (in
addition to the existing use of compiled regular expressions). When present,
the plain strings will be compared for exact matches only. This approach allows
the new default filter to be added during interpreter startup without requiring
early access to the ``re`` module.
Motivation
==========
As discussed in [1_] and mentioned in [2_], Python 2.7 and Python 3.2 changed
the default handling of ``DeprecationWarning`` such that:
* the warning was hidden by default during normal code execution
* the ``unittest`` test runner was updated to re-enable it when running tests
The intent was to avoid cases of tooling output like the following::
$ devtool mycode/
/usr/lib/python3.6/site-packages/devtool/cli.py:1: DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7
async = True
... actual tool output ...
Even when `devtool` is a tool specifically for Python programmers, this is not
a particularly useful warning, as it will be shown on every invocation, even
though the main helpful step an end user can take is to report a bug to the
developers of ``devtool``.
The warning is even less helpful for general purpose developer tools that are
used across more languages than just Python, and almost entirely \*un\*helpful
for applications that simply happen to be written in Python, and aren't
necessarily intended for a developer audience at all.
However, this change proved to have unintended consequences for the following
audiences:
* anyone using a test runner other than the default one built into ``unittest``
(the request for third party test runners to change their default warnings
filters was never made explicitly, so many of them still rely on the
interpreter defaults that are designed to suit deployed applications)
* anyone using the default ``unittest`` test runner to test their Python code
in a subprocess (since even ``unittest`` only adjusts the warnings settings
in the current process)
* anyone writing Python code at the interactive prompt or as part of a directly
executed script that didn't have a Python level test suite at all
In these cases, ``DeprecationWarning`` ended up become almost entirely
equivalent to ``PendingDeprecationWarning``: it was simply never seen at all.
Limitations on PEP Scope
========================
This PEP exists specifically to explain both the proposed addition to the
default warnings filter for 3.7, *and* to more clearly articulate the rationale
for the original change to the handling of DeprecationWarning back in Python 2.7
and 3.2.
This PEP does not solve all known problems with the current approach to handling
deprecation warnings. Most notably:
* The default ``unittest`` test runner does not currently report deprecation
warnings emitted at module import time, as the warnings filter override is only
put in place during test execution, not during test discovery and loading.
* The default ``unittest`` test runner does not currently report deprecation
warnings in subprocesses, as the warnings filter override is applied directly
to the loaded ``warnings`` module, not to the ``PYTHONWARNINGS`` environment
variable.
* The standard library doesn't provide a straightforward way to opt-in to seeing
all warnings emitted *by* a particular dependency prior to upgrading it
(the third-party ``warn`` module [3_] does provide this, but enabling it
involves monkeypatching the standard library's ``warnings`` module).
* When software has been factored out into support modules, but those modules
have little or no automated test coverage, re-enabling deprecation warnings
by default in ``__main__`` isn't likely to help find API compatibility
problems. Near term, the best currently available answer is to run affected
applications with ``PYTHONWARNINGS=default::DeprecationWarning`` or
``python -W default::DeprecationWarning`` and pay attention to their
``stderr`` output. Longer term, this is really a question for researchers
working on static analysis of Python code: how to reliably find usage of
deprecated APIs, and how to infer that an API or parameter is deprecated
based on ``warnings.warn`` calls, without actually running either the code
providing the API or the code accessing it.
While these are real problems with the status quo, they're excluded from
consideration in this PEP because they're going to require more complex
solutions than a single additional entry in the default warnings filter,
and resolving them at least potentially won't require going through the PEP
process.
For anyone interested in pursuing them further, the first two would be
``unittest`` module enhancement requests, the third would be a ``warnings``
module enhancement request, while the last would only require a PEP if
inferring API deprecations from their contents was deemed to be an intractable
code analysis problem, and an explicit function and parameter marker syntax in
annotations was proposed instead.
The CPython reference implementation will also include the following related
changes in 3.7:
* a new ``-X dev`` command line option that combines several developer centric
settings (including ``-Wd``) into one command line flag:
https://bugs.python.org/issue32043
* changing the behaviour in debug builds to show more of the warnings that are
off by default in regular interpeter builds: https://bugs.python.org/issue32088
References
==========
.. [1] stdlib-sig thread proposing the original default filter change
(https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)
.. [2] Python 2.7 notification of the default warnings filter change
(https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)
.. [3] Emitting warnings based on the location of the warning itself
(https://pypi.org/project/warn/)
.. [4] GitHub PR for PEP 565 implementation
(https://github.com/python/cpython/pull/4458)
.. [5] Tracker issue for PEP 565 implementation
(https://bugs.python.org/issue31975)
.. [6] First python-dev discussion thread
(https://mail.python.org/pipermail/python-dev/2017-November/150477.html)
.. [7] Second python-dev discussion thread
(https://mail.python.org/pipermail/python-dev/2017-November/150819.html)
.. [8] IPython's DeprecationWarning auto-configuration
(https://github.com/ipython/ipython/blob/6.2.x/IPython/core/interactiveshell.py#L619)
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: