371 lines
16 KiB
ReStructuredText
371 lines
16 KiB
ReStructuredText
PEP: 565
|
||
Title: Show DeprecationWarning in __main__
|
||
Author: Nick Coghlan <ncoghlan@gmail.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 12-Nov-2017
|
||
Python-Version: 3.7
|
||
Post-History: 12-Nov-2017, 25-Nov-2017
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
In Python 2.7 and Python 3.2, the default warning filters were updated to hide
|
||
DeprecationWarning by default, such that deprecation warnings in development
|
||
tools that were themselves written in Python (e.g. linters, static analysers,
|
||
test runners, code generators), as well as any other applications that merely
|
||
happened to be written in Python, wouldn't be visible to their users unless
|
||
those users explicitly opted in to seeing them.
|
||
|
||
However, this change has had the unfortunate side effect of making
|
||
DeprecationWarning markedly less effective at its primary intended purpose:
|
||
providing advance notice of breaking changes in APIs (whether in CPython, the
|
||
standard library, or in third party libraries) to users of those APIs.
|
||
|
||
To improve this situation, this PEP proposes a single adjustment to the
|
||
default warnings filter: displaying deprecation warnings attributed to the main
|
||
module by default.
|
||
|
||
This change will mean that code entered at the interactive prompt and code in
|
||
single file scripts will revert to reporting these warnings by default, while
|
||
they will continue to be silenced by default for packaged code distributed as
|
||
part of an importable module.
|
||
|
||
The PEP also proposes a number of small adjustments to the reference
|
||
interpreter and standard library documentation to help make the warnings
|
||
subsystem more approachable for new Python developers.
|
||
|
||
As part of the documentation updates, it will be made clearer that the
|
||
``unittest`` test runner displays all warnings by default when executing
|
||
test cases, and that other test runners are advised to follow that example.
|
||
|
||
|
||
Specification
|
||
=============
|
||
|
||
|
||
New default warnings filter entry
|
||
---------------------------------
|
||
|
||
The current set of default warnings filters consists of::
|
||
|
||
ignore::DeprecationWarning
|
||
ignore::PendingDeprecationWarning
|
||
ignore::ImportWarning
|
||
ignore::BytesWarning
|
||
ignore::ResourceWarning
|
||
|
||
The default ``unittest`` test runner then uses ``warnings.catch_warnings()``
|
||
``warnings.simplefilter('default')`` to override the default filters while
|
||
running test cases.
|
||
|
||
The change proposed in this PEP is to update the default warning filter list
|
||
to be::
|
||
|
||
default::DeprecationWarning:__main__
|
||
ignore::DeprecationWarning
|
||
ignore::PendingDeprecationWarning
|
||
ignore::ImportWarning
|
||
ignore::BytesWarning
|
||
ignore::ResourceWarning
|
||
|
||
This means that in cases where the nominal location of the warning (as
|
||
determined by the ``stacklevel`` parameter to ``warnings.warn``) is in the
|
||
``__main__`` module, the first occurrence of each DeprecationWarning will once
|
||
again be reported.
|
||
|
||
This change will lead to DeprecationWarning being displayed by default for:
|
||
|
||
* code executed directly at the interactive prompt
|
||
* code executed directly as part of a single-file script
|
||
|
||
While continuing to be hidden by default for:
|
||
|
||
* code imported from another module in a ``zipapp`` archive's ``__main__.py``
|
||
file
|
||
* code imported from another module in an executable package's ``__main__``
|
||
submodule
|
||
* code imported from an executable script wrapper generated at installation time
|
||
based on a ``console_scripts`` or ``gui_scripts`` entry point definition
|
||
|
||
This means that tool developers that create an installable or executable
|
||
artifact (such as a ``zipapp`` archive) for distribution to their users
|
||
shouldn't see any change from the status quo, while users of more ad hoc
|
||
personal or locally distributed scripts are likely to start seeing relevant
|
||
deprecation warnings again (as they did in Python 2.6 and earlier).
|
||
|
||
|
||
Additional use case for ``FutureWarning``
|
||
-----------------------------------------
|
||
|
||
The standard library documentation will be updated to explicitly recommend the
|
||
use of ``FutureWarning`` (rather than ``DeprecationWarning``) for backwards
|
||
compatibility warnings that are intended to be seen by *users* of an
|
||
application. (This will be in addition to the existing use of ``FutureWarning``
|
||
to warn about constructs that will remain valid code in the future,
|
||
but will have different semantics).
|
||
|
||
This will give the following three distinct categories of backwards
|
||
compatibility warning, with three different intended audiences:
|
||
|
||
* ``PendingDeprecationWarning``: hidden by default for all code.
|
||
The intended audience is Python developers that take an active interest in
|
||
ensuring the future compatibility of their software (e.g. professional
|
||
Python application developers with specific support obligations).
|
||
* ``DeprecationWarning``: reported by default for code that runs directly in
|
||
the ``__main__`` module (as such code is considered relatively unlikely to
|
||
have a dedicated test suite), but hidden by default for code in other modules.
|
||
The intended audience is Python developers that are at risk of upgrades to
|
||
their dependencies (including upgrades to Python itself) breaking their
|
||
software (e.g. developers using Python to script environments where someone
|
||
else is in control of the timing of dependency upgrades).
|
||
* ``FutureWarning``: reported by default for all code.
|
||
The intended audience is users of applications written in Python, rather than
|
||
other Python developers (e.g. warning about use of a deprecated setting in a
|
||
configuration file format).
|
||
|
||
For library and framework authors that want to ensure their API compatibility
|
||
warnings are more reliably seen by their users, the recommendation is to use a
|
||
custom warning class that derives from ``DeprecationWarning`` in Python 3.7+,
|
||
and from ``FutureWarning`` in earlier versions.
|
||
|
||
|
||
Recommended filter settings for test runners
|
||
--------------------------------------------
|
||
|
||
Developers of test runners are advised to implement logic equivalent to the
|
||
following when determining their default warnings filters::
|
||
|
||
if not sys.warnoptions:
|
||
warnings.simplefilter("default")
|
||
|
||
This effectively enables all warnings by default, as if the ``-Wd`` command
|
||
line option had been passed.
|
||
|
||
Note that actually enabling ``BytesWarning`` in a test suite still requires
|
||
passing the ``-b`` option to the interpreter at the command line. For implicit
|
||
bytes conversion and bytes comparison warnings, the warnings filter machinery
|
||
is only used to determine whether they should be printed as warnings or raised
|
||
as exceptions - when the command line flag isn't set, the interpreter doesn't
|
||
even emit the warning in the first place.
|
||
|
||
|
||
Recommended filter settings for interactive shells
|
||
--------------------------------------------------
|
||
|
||
Developers of interactive shells are advised to add a filter that enables
|
||
``DeprecationWarning`` in the namespace where user code is entered and executed.
|
||
|
||
If that namespace is ``__main__`` (as it is for the default CPython REPL), then
|
||
no changes are needed beyond those in this PEP.
|
||
|
||
Interactive shell implementations which use a namespace other than
|
||
``__main__`` will need to add their own filter. For example, IPython uses the
|
||
following command ([8_]) to set up a suitable filter::
|
||
|
||
warnings.filterwarnings("default", category=DeprecationWarning,
|
||
module=self.user_ns.get("__name__"))
|
||
|
||
|
||
Other documentation updates
|
||
---------------------------
|
||
|
||
The current reference documentation for the warnings system is relatively short
|
||
on specific *examples* of possible settings for the ``-W`` command line option
|
||
or the ``PYTHONWARNINGS`` environment variably that achieve particular end
|
||
results.
|
||
|
||
The following improvements are proposed as part of the implementation of this
|
||
PEP:
|
||
|
||
* Explicitly list the following entries under the description of the
|
||
``PYTHONWARNINGS`` environment variable::
|
||
|
||
PYTHONWARNINGS=error # Convert to exceptions
|
||
PYTHONWARNINGS=always # Warn every time
|
||
PYTHONWARNINGS=default # Warn once per call location
|
||
PYTHONWARNINGS=module # Warn once per calling module
|
||
PYTHONWARNINGS=once # Warn once per Python process
|
||
PYTHONWARNINGS=ignore # Never warn
|
||
|
||
* Explicitly list the corresponding short options
|
||
(``-We``, ``-Wa``, ``-Wd``, ``-Wm``, ``-Wo``, ``-Wi``) for each of the
|
||
warning actions listed under the ``-W`` command line switch documentation
|
||
|
||
* Explicitly list the default filter set in the ``warnings`` module
|
||
documentation, using the ``action::category`` and ``action::category:module``
|
||
notation
|
||
|
||
* Explicitly list the following snippet in the ``warnings.simplefilter``
|
||
documentation as a recommended approach to turning off all warnings by
|
||
default in a Python application while still allowing them to be turned
|
||
back on via ``PYTHONWARNINGS`` or the ``-W`` command line switch::
|
||
|
||
if not sys.warnoptions:
|
||
warnings.simplefilter("ignore")
|
||
|
||
None of these are *new* (they already work in all still supported Python
|
||
versions), but they're not especially obvious given the current structure
|
||
of the related documentation.
|
||
|
||
|
||
Reference Implementation
|
||
========================
|
||
|
||
A reference implementation is available in the PR [4_] linked from the
|
||
related tracker issue for this PEP [5_].
|
||
|
||
As a side-effect of implementing this PEP, the internal warnings filter list
|
||
will start allowing the use of plain strings as part of filter definitions (in
|
||
addition to the existing use of compiled regular expressions). When present,
|
||
the plain strings will be compared for exact matches only. This approach allows
|
||
the new default filter to be added during interpreter startup without requiring
|
||
early access to the ``re`` module.
|
||
|
||
|
||
Motivation
|
||
==========
|
||
|
||
As discussed in [1_] and mentioned in [2_], Python 2.7 and Python 3.2 changed
|
||
the default handling of ``DeprecationWarning`` such that:
|
||
|
||
* the warning was hidden by default during normal code execution
|
||
* the ``unittest`` test runner was updated to re-enable it when running tests
|
||
|
||
The intent was to avoid cases of tooling output like the following::
|
||
|
||
$ devtool mycode/
|
||
/usr/lib/python3.6/site-packages/devtool/cli.py:1: DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7
|
||
async = True
|
||
... actual tool output ...
|
||
|
||
Even when `devtool` is a tool specifically for Python programmers, this is not
|
||
a particularly useful warning, as it will be shown on every invocation, even
|
||
though the main helpful step an end user can take is to report a bug to the
|
||
developers of ``devtool``.
|
||
|
||
The warning is even less helpful for general purpose developer tools that are
|
||
used across more languages than just Python, and almost entirely \*un\*helpful
|
||
for applications that simply happen to be written in Python, and aren't
|
||
necessarily intended for a developer audience at all.
|
||
|
||
However, this change proved to have unintended consequences for the following
|
||
audiences:
|
||
|
||
* anyone using a test runner other than the default one built into ``unittest``
|
||
(the request for third party test runners to change their default warnings
|
||
filters was never made explicitly, so many of them still rely on the
|
||
interpreter defaults that are designed to suit deployed applications)
|
||
* anyone using the default ``unittest`` test runner to test their Python code
|
||
in a subprocess (since even ``unittest`` only adjusts the warnings settings
|
||
in the current process)
|
||
* anyone writing Python code at the interactive prompt or as part of a directly
|
||
executed script that didn't have a Python level test suite at all
|
||
|
||
In these cases, ``DeprecationWarning`` ended up become almost entirely
|
||
equivalent to ``PendingDeprecationWarning``: it was simply never seen at all.
|
||
|
||
|
||
Limitations on PEP Scope
|
||
========================
|
||
|
||
This PEP exists specifically to explain both the proposed addition to the
|
||
default warnings filter for 3.7, *and* to more clearly articulate the rationale
|
||
for the original change to the handling of DeprecationWarning back in Python 2.7
|
||
and 3.2.
|
||
|
||
This PEP does not solve all known problems with the current approach to handling
|
||
deprecation warnings. Most notably:
|
||
|
||
* The default ``unittest`` test runner does not currently report deprecation
|
||
warnings emitted at module import time, as the warnings filter override is only
|
||
put in place during test execution, not during test discovery and loading.
|
||
* The default ``unittest`` test runner does not currently report deprecation
|
||
warnings in subprocesses, as the warnings filter override is applied directly
|
||
to the loaded ``warnings`` module, not to the ``PYTHONWARNINGS`` environment
|
||
variable.
|
||
* The standard library doesn't provide a straightforward way to opt-in to seeing
|
||
all warnings emitted *by* a particular dependency prior to upgrading it
|
||
(the third-party ``warn`` module [3_] does provide this, but enabling it
|
||
involves monkeypatching the standard library's ``warnings`` module).
|
||
* When software has been factored out into support modules, but those modules
|
||
have little or no automated test coverage, re-enabling deprecation warnings
|
||
by default in ``__main__`` isn't likely to help find API compatibility
|
||
problems. Near term, the best currently available answer is to run affected
|
||
applications with ``PYTHONWARNINGS=default::DeprecationWarning`` or
|
||
``python -W default::DeprecationWarning`` and pay attention to their
|
||
``stderr`` output. Longer term, this is really a question for researchers
|
||
working on static analysis of Python code: how to reliably find usage of
|
||
deprecated APIs, and how to infer that an API or parameter is deprecated
|
||
based on ``warnings.warn`` calls, without actually running either the code
|
||
providing the API or the code accessing it.
|
||
|
||
While these are real problems with the status quo, they're excluded from
|
||
consideration in this PEP because they're going to require more complex
|
||
solutions than a single additional entry in the default warnings filter,
|
||
and resolving them at least potentially won't require going through the PEP
|
||
process.
|
||
|
||
For anyone interested in pursuing them further, the first two would be
|
||
``unittest`` module enhancement requests, the third would be a ``warnings``
|
||
module enhancement request, while the last would only require a PEP if
|
||
inferring API deprecations from their contents was deemed to be an intractable
|
||
code analysis problem, and an explicit function and parameter marker syntax in
|
||
annotations was proposed instead.
|
||
|
||
The CPython reference implementation will also include the following related
|
||
changes in 3.7:
|
||
|
||
* a new ``-X dev`` command line option that combines several developer centric
|
||
settings (including ``-Wd``) into one command line flag:
|
||
https://bugs.python.org/issue32043
|
||
* changing the behaviour in debug builds to show more of the warnings that are
|
||
off by default in regular interpeter builds: https://bugs.python.org/issue32088
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] stdlib-sig thread proposing the original default filter change
|
||
(https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)
|
||
|
||
.. [2] Python 2.7 notification of the default warnings filter change
|
||
(https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)
|
||
|
||
.. [3] Emitting warnings based on the location of the warning itself
|
||
(https://pypi.org/project/warn/)
|
||
|
||
.. [4] GitHub PR for PEP 565 implementation
|
||
(https://github.com/python/cpython/pull/4458)
|
||
|
||
.. [5] Tracker issue for PEP 565 implementation
|
||
(https://bugs.python.org/issue31975)
|
||
|
||
.. [6] First python-dev discussion thread
|
||
(https://mail.python.org/pipermail/python-dev/2017-November/150477.html)
|
||
|
||
.. [7] Second python-dev discussion thread
|
||
(https://mail.python.org/pipermail/python-dev/2017-November/150819.html)
|
||
|
||
.. [8] IPython's DeprecationWarning auto-configuration
|
||
(https://github.com/ipython/ipython/blob/6.2.x/IPython/core/interactiveshell.py#L619)
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|