python-peps/pep-0698.rst

PEP: 698
Title: Override Decorator for Static Typing
Author: Steven Troxler <steven.troxler@gmail.com>,
        Joshua Xu <jxu425@fb.com>,
        Shannon Zhu <szhu@fb.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/TOIYZ3SNPBJZDBRU3ZSBREXV2NNHF4KW/
Status: Draft
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 05-Sep-2022
Python-Version: 3.12
Post-History: `20-May-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/V23I4D6DEOFW4BBPWBMYTHZUOMKR7KQE/>`__,
              `17-Aug-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/7JDW2PKGF6YTERUJGWM3BRP3GDHRFP4O/>`__,
              `11-Oct-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/TOIYZ3SNPBJZDBRU3ZSBREXV2NNHF4KW/>`__,


Abstract
========

This PEP proposes adding an ``@override`` decorator to the Python type system.
This will allow type checkers to prevent a class of bugs that occur when a base
class changes methods that are inherited by derived classes.


Motivation
==========

A primary purpose of type checkers is to flag when refactors or changes break
pre-existing semantic structures in the code, so users can identify and make
fixes across their project without doing a manual audit of their code.


Safe Refactoring
----------------

Python’s type system does not provide a way to identify call sites that need to
be changed to stay consistent when an overridden function API changes. This
makes refactoring and transforming code more dangerous.

Consider this simple inheritance structure:

.. code-block:: python

    class Parent:
        def foo(self, x: int) -> int:
            return x

    class Child(Parent):
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        parent.foo(1)

    def child_callsite(child: Child) -> None:
        child.foo(1)


If the overridden method on the superclass is renamed or deleted, type checkers
will only alert us to update call sites that deal with the base type directly.
But the type checker can only see the new code, not the change we made, so it
has no way of knowing that we probably also needed to rename the same method on
child classes.

A type checker will happily accept this code, even though we are likely
introducing bugs:

.. code-block:: python

    class Parent:
        # Rename this method
        def new_foo(self, x: int) -> int:
            return x

    class Child(Parent):
        # This (unchanged) method used to override `foo` but is unrelated to `new_foo`
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        # If we pass a Child instance we’ll now run Parent.new_foo - likely a bug
        parent.new_foo(1)

    def child_callsite(child: Child) -> None:
        # We probably wanted to invoke new_foo here. Instead, we forked the method
        child.foo(1)


This code will type check, but there are two potential sources of bugs:

- If we pass a ``Child`` instance to the parent_callsite function, it will
  invoke the implementation in ``Parent.new_foo``. rather than ``Child.foo``.
  This is probably a bug - we presumably would not have written ``Child.foo`` in
  the first place if we didn’t need custom behavior.
- Our system was likely relying on ``Child.foo``  behaving in a similar way to
  ``Parent.foo``. But unless we catch this early, we have now forked the
  methods, and future refactors it is likely no one will realize that major
  changes to the behavior of new_foo likely require updating ``Child.foo`` as
  well, which could lead to major bugs later.

The incorrectly-refactored code is type-safe, but is probably not what we
intended and could cause our system to behave incorrectly. The bug can be
difficult to track down because our new code likely does execute without
throwing exceptions. Tests are less likely to catch the problem, and silent
errors can take longer to track down in production.

We are aware of several production outages in multiple typed codebases caused by
such incorrect refactors. This is our primary motivation for adding an ``@override``
decorator to the type system, which lets developers express the relationship
between ``Parent.foo`` and ``Child.foo`` so that type checkers can detect the problem.


Rationale
=========


Subclass Implementations Become More Explicit
---------------------------------------------

We believe that explicit overrides will make unfamiliar code easier to read than
implicit overrides. A developer reading the implementation of a subclass that
uses ``@override``  can immediately see which methods are overriding
functionality in some base class; without this decorator, the only way to
quickly find out is using a static analysis tool.


Precedent in Other Languages and Runtime Libraries
--------------------------------------------------

Static Override Checks in Other Languages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Many popular programming languages support override checks. For example:

- `C++ has <https://en.cppreference.com/w/cpp/language/override/>`_ ``override``.
- `C# has <https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/override/>`_ ``override``.
- `Hack has <https://docs.hhvm.com/hack/attributes/predefined-attributes#__override/>`_ ``<<__Override>>``.
- `Java has <https://docs.oracle.com/javase/tutorial/java/IandI/override.html/>`_ ``@Override``.
- `Kotlin has <https://kotlinlang.org/docs/inheritance.html#overriding-methods/>`_ ``override``.
- `Scala has <https://www.javatpoint.com/scala-method-overriding/>`_ ``override``.
- `Swift has <https://docs.swift.org/swift-book/LanguageGuide/Inheritance.html#ID198/>`_ ``override``.
- `TypeScript has <https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-3.html#override-and-the---noimplicitoverride-flag/>`_ ``override``.

Runtime Override Checks in Python
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Today, there is an `Overrides library <https://pypi.org/project/overrides/>`_
that provides decorators ``@overrides`` (sic) and ``@final`` and will enforce
them at runtime.

:pep:`591` added a ``@final`` decorator with the same semantics as those in the
Overrides library. But the override component of the runtime library is not
supported statically at all, which has added some confusion around the
mix/matched support.

Providing support for ``@override`` in static checks would add value because

- Bugs can be caught earlier, often in-editor.
- Static checks come with no performance overhead, unlike runtime checks.
- Bugs will be caught quickly even in rarely-used modules, whereas with runtime
  checks these might go undetected for a time without automated tests of all
  imports.


Disadvantages
-------------

Using ``@override`` will make code more verbose.


Specification
=============

When type checkers encounter a method decorated with ``@typing.override`` they
should treat it as a type error unless that method is overriding a compatible
method or attribute in some ancestor class.


.. code-block:: python

    from typing import override

    class Parent:
        def foo(self) -> int:
            return 1

        def bar(self, x: str) -> str:
            return x

    class Child(Parent):
        @override
        def foo(self) -> int:
            return 2

        @override
        def baz() -> int:  # Type check error: no matching signature in ancestor
            return 1


The ``@override`` decorator should be permitted anywhere a type checker
considers a method to be a valid override, which typically includes not only
normal methods but also ``@property``, ``@staticmethod``, and ``@classmethod``.


Override Compatibility Rules are Unchanged
------------------------------------------

Type checkers already enforce compatibility rules for overrides; for example, a
subclass method’s type signature should be compatible with that of the
superclass method. These compatibility rules do not change due to the presence
or absence of ``@override``.

Note that when a ``@property`` overrides a regular attribute of the base class,
this should not be considered an error due to the use of ``@override``, but the
type checker may still consider the override to be incompatible. For example a
type checker may consider it illegal to override a non-final attribute with a
getter property and no setter, as this does not respect the substitution
principle.


Strict Enforcement Per-Project
==============================

We believe that ``@override`` is most useful if checkers also allow developers
to opt into a strict mode where methods that override a parent class are
required to use the decorator. Strict enforcement should be opt-in for backward
compatibility.


Motivation
----------

The primary reason for a strict mode that requires ``@override`` is that developers
can only trust that refactors are override-safe if they know that the ``@override``
decorator is used throughout the project.

There is another class of bug related to overrides that we can only catch using a strict mode.

Consider the following code:

.. code-block:: python

    class Parent:
        pass

    class Child(Parent):
        def foo() -> int:
            return 2

Imagine we refactor it as follows:


.. code-block:: python

    class Parent
        def foo() -> int:   # This method is new
            return 1

    class Child(Parent):
        def foo() -> int:  # This is now an override!
            return 2

    def call_foo(parent: Parent) -> int:
        return parent.foo()  # This could invoke Child.foo, which may be surprising.

The semantics of our code changed here, which could cause two problems:

- If the author of the code change did not know that ``Child.foo`` already
  existed (which is very possible in a large codebase), they might be surprised
  to see that ``call_foo`` does not always invoke ``Parent.foo``.
- If the codebase authors tried to manually apply ``@override`` everywhere when
  writing overrides in subclasses, they are likely to miss the fact that
  ``Child.foo`` needs it here.


At first glance this kind of change may seem unlikely, but it can actually
happen often if one or more subclasses have functionality that developers later
realize belongs in the base class.

With a strict mode, we will always alert developers when this occurs.

Precedent
---------

Most of the typed, object-oriented programming languages we looked at have an
easy way to require explicit overrides throughout a project:

- C#, Kotlin, Scala, and Swift always require explicit overrides
- TypeScript has a
  `--no-implicit-override <https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-3.html#override-and-the---noimplicitoverride-flag/>`_
  flag to force explicit overrides
- In Hack and Java the type checker always treats overrides as opt-in, but
  widely-used linters can warn if explicit overrides are missing.


Backward Compatibility
======================

By default, the ``@override`` decorator will be opt-in. Codebases that do not
use it will type-check as before, without the additional type safety.


Runtime Behavior
================

Set ``__override__ = True`` when possible
-----------------------------------------

At runtime, ``@typing.override`` will make a best-effort attempt to add an
attribute ``__override__`` with value ``True`` to its argument. By "best-effort"
we mean that we will try adding the attribute, but if that fails (for example
because the input is a descriptor type with fixed slots) we will silently
return the argument as-is.

This is exactly what the ``@typing.final`` decorator does, and the motivation
is similar - it gives runtime libraries the ability to use ``@override``. As a
concrete example, a runtime library could check ``__override__`` in order
to automatically populate the ``__doc__`` attribute of child class methods
using the parent method docstring.

Limitations of setting ``__override__``
---------------------------------------

As described above, adding ``__override__`` may fail at runtime, in which
case we will simply return the argument as-is.

In addition, even in cases where it does work it may be difficult for users
to correctly work with multiple decorators, because getting the ``__override__``
field to exist on the final output requires understanding the implementation
of each decorator:

- The ``@override`` decorator needs to execute *after* ordinary decorators
  like ``@functools.lru_cache`` that use wrapper functions, since we want to
  set ``__override__`` on the outermost wrapper. This means it needs to
  go *above* all these other decorators.
- But ``@override`` needs to execute *before* many special descriptor-based
  decorators like ``@property``, ``@staticmethod``, and ``@classmethod``.
- As discussed above, in some cases (for example a descriptor with fixed
  slots or a descriptor that also wraps) it may be impossible to get the
  ``__override__`` attribute at all.

As a result, runtime support for setting ``__override__`` is best effort
only, and we do not expect type checkers to validate the ordering of
decorators.


Rejected Alternatives
=====================


Rely on Integrated Development Environments for safety
------------------------------------------------------

Modern Integrated Development Environments (IDEs) often provide the ability to
automatically update subclasses when renaming a method. But we view this as
insufficient for several reasons:

- If a codebase is split into multiple projects, an IDE will not help and the
  bug appears when upgrading dependencies. Type checkers are a fast way to catch
  breaking changes in dependencies.
- Not all developers use such IDEs. And library maintainers, even if they do use
  an IDE, should not need to assume pull request authors use the same IDE. We
  prefer being able to detect problems in continuous integration without
  assuming anything about developers’ choice of editor.



Runtime enforcement
-------------------

We considered having ``@typing.override`` enforce override safety at runtime,
similarly to how ``@overrides.overrides``
`does today <https://pypi.org/project/overrides/>`_.

We rejected this for four reasons:

- For users of static type checking, it is not clear this brings any benefits.
- There would be at least some performance overhead, leading to projects
  importing slower with runtime enforcement. We estimate the
  ``@overrides.overrides`` implementation takes around 100 microseconds, which
  is fast but could still add up to a second or more of extra initialization
  time in million-plus line codebases, which is exactly where we think
  ``@typing.override`` will be most useful.
- An implementation may have edge cases where it doesn’t work well (we heard
  from a maintainer of one such closed-source library that this has been a
  problem). We expect static enforcement to be simple and reliable.
- The implementation approaches we know of are not simple. The decorator
  executes before the class is finished evaluating, so the options we know of
  are either to inspect the bytecode of the caller (as ``@overrides.overrrides``
  does) or to use a metaclass-based approach. Neither approach seems ideal.


Mark a base class to force explicit overrides on subclasses
-----------------------------------------------------------

We considered including a class decorator ``@require_explicit_overrides``, which
would have provided a way for base classes to declare that all subclasses must
use the ``@override`` decorator on method overrides. The overrides library has a
mixin class, ``EnforceExplicitOverrides``, which provides similar behavior in
runtime checks.

We decided against this because we expect owners of large codebases will benefit
most from ``@override``, and for these use cases having a strict mode where
explicit ``@override`` is required (see the Backward Compatibility section)
provides more benefits than a way to mark base classes.

Moreover we believe that authors of projects who do not consider the extra type
safety to be worth the additional boilerplate of using ``@override`` should not
be forced to do so. Having an optional strict mode puts the decision in the
hands of project owners, whereas the use of ``@require_explicit_overrides`` in
libraries would force project owners to use ``@override`` even if they prefer
not to.

Include the name of the ancestor class being overridden
-------------------------------------------------------

We considered allowing the caller of ``@override`` to specify a specific
ancestor class where the overridden method should be defined:

.. code-block:: python

    class Parent0:
        def foo() -> int:
            return 1


    class Parent1:
        def bar() -> int:
            return 1


    class Child(Parent0, Parent1):
        @override(Parent0)  # okay, Parent0 defines foo
        def foo() -> int:
            return 2

        @override(Parent0)  # type error, Parent0 does not define bar
        def bar() -> int:
            return 2


This could be useful for code readability because it makes the override
structure more explicit for deep inheritance trees. It also might catch bugs by
prompting developers to check that the implementation of an override still makes
sense whenever a method being overridden moves from one base class to another.

We decided against it because:

- Supporting this would add complexity to the implementation of both
  ``@override`` and type checker support for it, so there would need to
  be considerable benefits.
- We believe that it would be rarely used and catch relatively few bugs.

  - The author of the ``overrides`` package
    `has noted <https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839/4>`_
    that early versions of his library included this capability but it was
    rarely useful and seemed to have little benefit. After it was removed, the
    ability was never requested by users.



Reference Implementation
========================

Pyre: A proof of concept is implemented in Pyre:

- The decorator
  `@pyre_extensions.override <https://github.com/facebook/pyre-check/blob/f4d3f676d17b2e59c4c55d09dfa3caead8ec2e7c/pyre_extensions/__init__.py#L95/>`_
  can mark overrides
- Pyre can `type-check this decorator <https://github.com/facebook/pyre-check/blob/ae68c44f4e5b263ce0e175f0798272d9318589af/source/analysis/test/integration/methodTest.ml#L2515-L2638/>`_
  as specified in this PEP


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.