python-peps/pep-0702.rst

PEP: 702
Title: Marking deprecations using the type system
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036
Status: Draft
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 30-Dec-2022
Python-Version: 3.12
Post-History: `01-Jan-2023 <https://mail.python.org/archives/list/typing-sig@python.org/thread/AKTFUYW3WDT7R7PGRIJQZMYHMDJNE4QH/>`__,
              `22-Jan-2023 <https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036>`__


Abstract
========

This PEP adds an ``@typing.deprecated()`` decorator that marks a class or function
as deprecated, enabling static checkers to warn when it is used.

Motivation
==========

As software evolves, new functionality is added and old functionality becomes
obsolete. Library developers want to work towards removing obsolete code while
giving their users time to migrate to new APIs. Python provides a mechanism for
achieving these goals: the :exc:`DeprecationWarning` warning class, which is
used to show warnings when deprecated functionality is used. This mechanism is
widely used: as of the writing of this PEP, the CPython main branch contains
about 150 distinct code paths that raise :exc:`!DeprecationWarning`. Many
third-party libraries also use :exc:`!DeprecationWarning` to mark deprecations.
In the `top 5000 PyPI packages <https://dev.to/hugovk/how-to-search-5000-python-projects-31gk>`__,
there are:

- 1911 matches for the regex ``warnings\.warn.*\bDeprecationWarning\b``,
  indicating use of :exc:`!DeprecationWarning` (not including cases where the
  warning is split over multiple lines);
- 1661 matches for the regex ``^\s*@deprecated``, indicating use of some sort of
  deprecation decorator.

However, the current mechanism is often insufficient to ensure that users
of deprecated functionality update their code in time. For example, the
removal of various long-deprecated :mod:`unittest` features had to be
`reverted <https://github.com/python/cpython/commit/b50322d20337ca468f2070eedb051a16ee1eba94>`__
from Python 3.11 to give users more time to update their code.
Users may run their test suite with warnings disabled for practical reasons,
or deprecations may be triggered in code paths that are not covered by tests.

Providing more ways for users to find out about deprecated functionality
can speed up the migration process. This PEP proposes to leverage static type
checkers to communicate deprecations to users. Such checkers have a thorough
semantic understanding of user code, enabling them to detect and report
deprecations that a single ``grep`` invocation could not find. In addition, many type
checkers integrate with IDEs, enabling users to see deprecation warnings
right in their editors.

Rationale
=========

At first glance, deprecations may not seem like a topic that type checkers should
touch. After all, type checkers are concerned with checking whether code will
work as is, not with potential future changes. However, the analysis that type
checkers perform on code to find type errors is very similar to the analysis
that would be needed to detect usage of many deprecations. Therefore, type
checkers are well placed to find and report deprecations.

Other languages already have similar functionality:

* GCC supports a ``deprecated`` `attribute <https://gcc.gnu.org/onlinedocs/gcc-3.1.1/gcc/Type-Attributes.html>`__
  on function declarations. This powers CPython's ``Py_DEPRECATED`` macro.
* GraphQL `supports <https://spec.graphql.org/June2018/#sec-Field-Deprecation>`__
  marking fields as ``@deprecated``.
* Kotlin `supports <https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-deprecated/>`__
  a ``Deprecated`` annotation.
* Scala `supports <https://www.scala-lang.org/api/2.12.5/scala/deprecated.html>`__
  an ``@deprecated`` annotation.
* Swift `supports <https://docs.swift.org/swift-book/ReferenceManual/Attributes.html>`__
  using the ``@available`` attribute to mark APIs as deprecated.
* TypeScript `uses <https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html#deprecated>`__
  the ``@deprecated`` JSDoc tag to issue a hint marking use of
  deprecated functionality.

Several users have requested support for such a feature:

* `typing-sig thread <https://mail.python.org/archives/list/typing-sig@python.org/thread/E24WTMQUTGKPFKEXVCGGEFFMG7LDF3WT/>`__
* `Pyright feature request <https://github.com/microsoft/pyright/discussions/2300>`__
* `mypy feature request <https://github.com/python/mypy/issues/11439>`__

There are similar existing third-party tools:

* `Deprecated <https://pypi.org/project/Deprecated/>`__ provides a decorator to
  mark classes, functions, or methods as deprecated. Access to decorated objects
  raises a runtime warning, but is not detected by type checkers.
* `flake8-deprecated <https://pypi.org/project/flake8-deprecated/>`__ is a linter
  plugin that warns about use of deprecated features. However, it is limited to
  a short, hard-coded list of deprecations.

Specification
=============

A new decorator ``@deprecated()`` is added to the :mod:`typing` module. This
decorator can be used on a class, function or method to mark it as deprecated.
This includes :class:`typing.TypedDict` and :class:`typing.NamedTuple` definitions.
With overloaded functions, the decorator may be applied to individual overloads,
indicating that the particular overload is deprecated. The decorator may also be
applied to the overload implementation function, indicating that the entire function
is deprecated.

The decorator takes a single argument of type ``str``, which is a message that should
be shown by the type checker when it encounters a usage of the decorated object.
The message must be a string literal.
The content of deprecation messages is up to the user, but it may include the version
in which the deprecated object is to be removed, and information about suggested
replacement APIs.

Type checkers should produce a diagnostic whenever they encounter a usage of an
object marked as deprecated. For deprecated overloads, this includes all calls
that resolve to the deprecated overload.
For deprecated classes and functions, this includes:

* References through module, class, or instance attributes (``module.deprecated_object``,
  ``module.SomeClass.deprecated_method``, ``module.SomeClass().deprecated_method``)
* Any usage of deprecated objects in their defining module
  (``x = deprecated_object()`` in ``module.py``)
* If ``import *`` is used, usage of deprecated objects from the
  module (``from module import *; x = deprecated_object()``)
* ``from`` imports (``from module import deprecated_object``)

There are some additional scenarios where deprecations could come into play:

* An object implements a :class:`typing.Protocol`, but one of the methods
  required for protocol compliance is deprecated.
* A class uses the ``@override`` decorator from :pep:`698` to assert that
  its method overrides a base class method, but the base class method is
  deprecated.

As these scenarios appear complex and relatively unlikely to come up in practice,
this PEP does not mandate that type checkers detect them.

Example
-------

As an example, consider this library stub named ``library.pyi``:

.. code-block:: python

   from typing import deprecated

   @deprecated("Use Spam instead")
   class Ham: ...

   @deprecated("It is pining for the fiords")
   def norwegian_blue(x: int) -> int: ...

   @overload
   @deprecated("Only str will be allowed")
   def foo(x: int) -> str: ...
   @overload
   def foo(x: str) -> str: ...

Here is how type checkers should handle usage of this library:

.. code-block:: python

   from library import Ham  # error: Use of deprecated class Ham. Use Spam instead.

   import library

   library.norwegian_blue(1)  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.
   map(library.norwegian_blue, [1, 2, 3])  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.

   library.foo(1)  # error: Use of deprecated overload for foo. Only str will be allowed.
   library.foo("x")  # no error

Runtime behavior
----------------

At runtime, the decorator sets an attribute ``__deprecated__`` on the decorated
object. The value of the attribute is the message passed to the decorator.
The decorator returns the original object. Notably, it does not issue a runtime
:exc:`DeprecationWarning`.

For compatibility with :func:`typing.get_overloads`, the ``@deprecated``
decorator should be placed after the ``@overload`` decorator.

Type checker behavior
---------------------

This PEP does not specify exactly how type checkers should present deprecation
diagnostics to their users. However, some users (e.g., application developers
targeting only a specific version of Python) may not care about deprecations,
while others (e.g., library developers who want their library to remain
compatible with future versions of Python) would want to catch any use of
deprecated functionality in their CI pipeline. Therefore, it is recommended
that type checkers provide configuration options that cover both use cases.
As with any other type checker error, it is also possible to ignore deprecations
using ``# type: ignore`` comments.

Deprecation policy
------------------

We propose that CPython's deprecation policy (:pep:`387`) is updated to require that new deprecations
use the functionality in this PEP to alert users
about the deprecation, if possible. Concretely, this means that new
deprecations should be accompanied by a change to the ``typeshed`` repo to
add the ``@deprecated`` decorator in the appropriate place.
This requirement does not apply to deprecations that cannot be expressed
using this PEP's functionality.

Backwards compatibility
=======================

Creating a new decorator poses no backwards compatibility concerns.
As with all new typing functionality, the ``@deprecated`` decorator
will be added to the ``typing_extensions`` module, enabling its use
in older versions of Python.

How to teach this
=================

For users who encounter deprecation warnings in their IDE or type
checker output, the messages they receive should be clear and self-explanatory.
Usage of the ``@deprecated`` decorator will be an advanced feature
mostly relevant to library authors. The decorator should be mentioned
in relevant documentation (e.g., :pep:`387` and the :exc:`DeprecationWarning`
documentation) as an additional way to mark deprecated functionality.

Reference implementation
========================

A runtime implementation of the ``@deprecated`` decorator is
`available <https://github.com/python/typing_extensions/pull/105>`__.
The ``pyanalyze`` type checker has
`prototype support <https://github.com/quora/pyanalyze/pull/578>`__
for emitting deprecation errors.

Open issues
===========

Runtime warnings
----------------

Users might expect usage of the ``@deprecated`` decorator to issue a
:exc:`DeprecationWarning` at runtime. However, this would raise a number of
thorny issues:

* When the decorator is applied to a class or an overload, the warning
  would not be raised as expected. For classes, the warning could be
  raised on instantiation, but this would not cover usage in type
  annotations or :func:`isinstance` checks.
* Users may want to control the :func:`~warnings.warn` call in more detail
  (e.g., changing the warning class).
* ``typing.py`` generally aims to avoid affecting runtime behavior.
* To raise a wwarning, the ``@deprecated`` decorator would have to wrap
  functions and patch ``__new__`` on classes. This would complicate runtime
  introspection.
* Users may not expect usage of an object from the ``typing`` module to
  affect runtime behavior.

Users who want to use ``@deprecated`` while also issuing a runtime warning
can use the ``if TYPE_CHECKING:`` idiom, for example:

.. code-block:: python

   from typing import TYPE_CHECKING
   import functools
   import warnings

   if TYPE_CHECKING:
       from typing import deprecated
   else:
       def deprecated(msg):
           def decorator(func):
               @functools.wraps(func)
               def wrapper(*args, **kwargs):
                   warnings.warn(msg, DeprecationWarning, stacklevel=2)
                   return func(*args, **kwargs)
               wrapper.__deprecated__ = msg
               return wrapper
           return decorator

While this code block looks complex, it could be encapsulated in a library.

Still, the behavior could be made opt-in, and perhaps the benefits of
incorporating a runtime warning outweigh the costs.

Rejected ideas
==============

Deprecation of modules and attributes
-------------------------------------

This PEP covers deprecations of classes, functions and overloads. This
allows type checkers to detect many but not all possible deprecations.
To evaluate whether additional functionality would be worthwhile, I
`examined <https://gist.github.com/JelleZijlstra/ff459edc5ff0918e22b56740bb28eb8b>`__
all current deprecations in the CPython standard library.

I found:

* 74 deprecations of functions, methods and classes (supported by this PEP)
* 28 deprecations of whole modules (largely due to :pep:`594`)
* 9 deprecations of function parameters (supported by this PEP through
  decorating overloads)
* 1 deprecation of a constant
* 38 deprecations that are not easily detectable in the type system (for
  example, for calling :func:`asyncio.get_event_loop` without an active
  event loop)

Modules could be marked as deprecated by adding a ``__deprecated__``
module-level constant. However, the need for this is limited, and it
is relatively easy to detect usage of deprecated modules simply by
grepping. Therefore, this PEP omits support for whole-module deprecations.
As a workaround, users could mark all module-level classes and functions
with ``@deprecated``.

For deprecating module-level constants, object attributes, and function
parameters, a ``Deprecated[type, message]`` type modifier, similar to
``Annotated`` could be added. However, this would create a new place
in the type system where strings are just strings, not forward references,
complicating the implementation of type checkers. In addition, my data
show that this feature is not commonly needed.

Features for deprecating more kinds of objects could be added in a future
PEP.

Acknowledgments
===============

A call with the typing-sig meetup group led to useful feedback on this
proposal.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.