749 lines
29 KiB
ReStructuredText
749 lines
29 KiB
ReStructuredText
PEP: 728
|
|
Title: TypedDict with Typed Extra Items
|
|
Author: Zixuan James Li <p359101898@gmail.com>
|
|
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
|
|
Discussions-To: https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Topic: Typing
|
|
Content-Type: text/x-rst
|
|
Created: 12-Sep-2023
|
|
Python-Version: 3.13
|
|
Post-History: `09-Feb-2024 <https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443>`__,
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes a way to limit extra items for :class:`~typing.TypedDict`
|
|
using a ``closed`` argument and to type them with the special ``__extra_items__``
|
|
key. This addresses the need to define closed TypedDict type or to type a subset
|
|
of keys that might appear in a ``dict`` while permitting additional items of a
|
|
specified type.
|
|
|
|
Motivation
|
|
==========
|
|
|
|
A :py:class:`typing.TypedDict` type can annotate the value type of each known
|
|
item in a dictionary. However, due to structural subtyping, a TypedDict can have
|
|
extra items that are not visible through its type. There is currently no way to
|
|
restrict the types of items that might be present in the TypedDict type's
|
|
structural subtypes.
|
|
|
|
Defining a Closed TypedDict Type
|
|
--------------------------------
|
|
|
|
The current behavior of TypedDict prevents users from defining a closed
|
|
TypedDict type when it is expected that the type contains no additional items.
|
|
|
|
Due to the possible presence of extra items, type checkers cannot infer more
|
|
precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can
|
|
also be resolved by
|
|
`defining a closed TypedDict type <https://github.com/python/mypy/issues/7981>`__.
|
|
|
|
Another possible use case for this is a sound way to
|
|
`enable type narrowing <https://github.com/python/mypy/issues/9953>`__ with the
|
|
``in`` check::
|
|
|
|
class Movie(TypedDict):
|
|
name: str
|
|
director: str
|
|
|
|
class Book(TypedDict):
|
|
name: str
|
|
author: str
|
|
|
|
def fun(entry: Movie | Book) -> None:
|
|
if "author" in entry:
|
|
reveal_type(entry) # Revealed type is 'Movie | Book'
|
|
|
|
Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to
|
|
have the ``author`` key, and under the current specification it would be
|
|
incorrect for the type checker to narrow its type.
|
|
|
|
Allowing Extra Items of a Certain Type
|
|
--------------------------------------
|
|
|
|
For supporting API interfaces or legacy codebase where only a subset of possible
|
|
keys are known, it would be useful to explicitly expect additional keys of
|
|
certain value types.
|
|
|
|
However, the typing spec is more restrictive on type checking the construction of a
|
|
TypedDict, `preventing users <https://github.com/python/mypy/issues/4617>`__
|
|
from doing this::
|
|
|
|
class MovieBase(TypedDict):
|
|
name: str
|
|
|
|
def fun(movie: MovieBase) -> None:
|
|
# movie can have extra items that are not visible through MovieBase
|
|
...
|
|
|
|
movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK
|
|
fun({"name": "Blade Runner", "year": 1982}) # Not OK
|
|
|
|
While the restriction is enforced when constructing a TypedDict, due to
|
|
structural subtyping, the TypedDict may have extra items that are not visible
|
|
through its type. For example::
|
|
|
|
class Movie(MovieBase):
|
|
year: int
|
|
|
|
movie: Movie = {"name": "Blade Runner", "year": 1982}
|
|
fun(movie) # OK
|
|
|
|
It is not possible to acknowledge the existence of the extra items through
|
|
``in`` checks and access them without breaking type safety, even though they
|
|
might exist from arbitrary structural subtypes of ``MovieBase``::
|
|
|
|
def g(movie: MovieBase) -> None:
|
|
if "year" in movie:
|
|
reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'year'
|
|
|
|
Some workarounds have already been implemented in response to the need to allow
|
|
extra keys, but none of them is ideal. For mypy,
|
|
``--disable-error-code=typeddict-unknown-key``
|
|
`suppresses type checking error <https://github.com/python/mypy/pull/14225>`__
|
|
specifically for unknown keys on TypedDict. This sacrifices type safety over
|
|
flexibility, and it does not offer a way to specify that the TypedDict type
|
|
expects additional keys compatible with a certain type.
|
|
|
|
Support Additional Keys for ``Unpack``
|
|
--------------------------------------
|
|
|
|
:pep:`692` adds a way to precisely annotate the types of individual keyword
|
|
arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However,
|
|
because TypedDict cannot be defined to accept arbitrary extra items, it is not
|
|
possible to
|
|
`allow additional keyword arguments <https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87>`__
|
|
that are not known at the time the TypedDict is defined.
|
|
|
|
Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing
|
|
codebases, it will be valuable to accept and type extra items on TypedDict so
|
|
that the old typing behavior can be supported in combination with the new
|
|
``Unpack`` construct.
|
|
|
|
Rationale
|
|
=========
|
|
|
|
A type that allows extra items of type ``str`` on a TypedDict can be loosely
|
|
described as the intersection between the TypedDict and ``Mapping[str, str]``.
|
|
|
|
`Index Signatures <https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures>`__
|
|
in TypeScript achieve this:
|
|
|
|
.. code-block:: typescript
|
|
|
|
type Foo = {
|
|
a: string
|
|
[key: string]: string
|
|
}
|
|
|
|
This proposal aims to support a similar feature without introducing general
|
|
intersection of types or syntax changes, offering a natural extension to the
|
|
existing type consistency rules.
|
|
|
|
We propose that we add an argument ``closed`` to TypedDict. Similar to
|
|
``total``, only a literal ``True`` or ``False`` value is allowed. When
|
|
``closed=True`` is used in the TypedDict type definition, we give the dunder
|
|
attribute ``__extra_items__`` a special meaning: extra items are allowed, and
|
|
their types should be compatible with the value type of ``__extra_items__``.
|
|
|
|
If ``closed=True`` is set, but there is no ``__extra_items__`` key, the
|
|
TypedDict is treated as if it contained an item ``__extra_items__: Never``.
|
|
|
|
Note that ``__extra_items__`` on the same TypedDict type definition will remain
|
|
as a regular item if ``closed=True`` is not used.
|
|
|
|
Different from index signatures, the types of the known items do not need to be
|
|
consistent with the value type of ``__extra_items__``.
|
|
|
|
There are some advantages to this approach:
|
|
|
|
- Inheritance works naturally. ``__extra_items__`` defined on a TypedDict will
|
|
also be available to its subclasses.
|
|
|
|
- We can build on top of the `type consistency rules defined in the typing spec
|
|
<https://typing.readthedocs.io/en/latest/spec/typeddict.html#type-consistency>`__.
|
|
``__extra_items__`` can be treated as a pseudo-item in terms of type
|
|
consistency.
|
|
|
|
- There is no need to introduce a grammar change to specify the type of the
|
|
extra items.
|
|
|
|
- We can precisely type the extra items without making ``__extra_items__`` the
|
|
union of known items.
|
|
|
|
- We do not lose backwards compatibility as ``__extra_items__`` still can be
|
|
used as a regular key.
|
|
|
|
Specification
|
|
=============
|
|
|
|
This specification is structured to parallel :pep:`589` to highlight changes to
|
|
the original TypedDict specification.
|
|
|
|
If ``closed=True`` is specified, extra items are treated as non-required items
|
|
having the same type of ``__extra_items__`` whose keys are allowed when
|
|
determining
|
|
`supported and unsupported operations
|
|
<https://typing.readthedocs.io/en/latest/spec/typeddict.html#supported-and-unsupported-operations>`__.
|
|
|
|
Using TypedDict Types
|
|
---------------------
|
|
|
|
Assuming that ``closed=True`` is used in the TypedDict type definition.
|
|
|
|
For a TypedDict type that has the special ``__extra_items__`` key, during
|
|
construction, the value type of each unknown item is expected to be non-required
|
|
and compatible with the value type of ``__extra_items__``. For example::
|
|
|
|
class Movie(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: bool
|
|
|
|
a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # OK
|
|
b: Movie = {
|
|
"name": "Blade Runner",
|
|
"year": 1982, # Not OK. 'int' is incompatible with 'bool'
|
|
}
|
|
|
|
In this example, ``__extra_items__: bool`` does not mean that ``Movie`` has a
|
|
required string key ``"__extra_items__"`` whose value type is ``bool``. Instead,
|
|
it specifies that keys other than "name" have a value type of ``bool`` and are
|
|
non-required.
|
|
|
|
The alternative inline syntax is also supported::
|
|
|
|
Movie = TypedDict("Movie", {"name": str, "__extra_items__": bool}, closed=True)
|
|
|
|
Accessing extra keys is allowed. Type checkers must infer its value type from
|
|
the value type of ``__extra_items__``::
|
|
|
|
def f(movie: Movie) -> None:
|
|
reveal_type(movie["name"]) # Revealed type is 'str'
|
|
reveal_type(movie["novel_adaptation"]) # Revealed type is 'bool'
|
|
|
|
When a TypedDict type defines ``__extra_items__`` without ``closed=True``,
|
|
``closed`` defaults to ``False`` and the key is assumed to be a regular key::
|
|
|
|
class Movie(TypedDict):
|
|
name: str
|
|
__extra_items__: bool
|
|
|
|
a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # Not OK. Unexpected key 'novel_adaptation'
|
|
b: Movie = {
|
|
"name": "Blade Runner",
|
|
"__extra_items__": True, # OK
|
|
}
|
|
|
|
For such non-closed TypedDict types, it is assumed that they allow non-required
|
|
extra items of value type ``ReadOnly[object]`` during inheritance or type
|
|
consistency checks. However, extra keys found during construction should still
|
|
be rejected by the type checker.
|
|
|
|
``closed`` is not inherited through subclassing::
|
|
|
|
class MovieBase(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: ReadOnly[str | None]
|
|
|
|
class Movie(MovieBase):
|
|
__extra_items__: str # A regular key
|
|
|
|
a: Movie = {"name": "Blade Runner", "__extra_items__": None} # Not OK. 'None' is incompatible with 'str'
|
|
b: Movie = {"name": "Blade Runner", "other_extra_key": None} # OK
|
|
|
|
Here, ``"__extra_items__"`` in ``a`` is a regular key defined on ``Movie`` where
|
|
its value type is narrowed from ``ReadOnly[str | None]`` to ``str``,
|
|
``"other_extra_key"`` in ``b`` is an extra key whose value type must be
|
|
consistent with the value type of ``"__extra_items__"`` defined on
|
|
``MovieBase``.
|
|
|
|
Interaction with Totality
|
|
-------------------------
|
|
|
|
It is an error to use ``Required[]`` or ``NotRequired[]`` with the special
|
|
``__extra_items__`` item. ``total=False`` and ``total=True`` have no effect on
|
|
``__extra_items__`` itself.
|
|
|
|
The extra items are non-required, regardless of the totality of the TypedDict.
|
|
Operations that are available to ``NotRequired`` items should also be available
|
|
to the extra items::
|
|
|
|
class Movie(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: int
|
|
|
|
def f(movie: Movie) -> None:
|
|
del movie["name"] # Not OK
|
|
del movie["year"] # OK
|
|
|
|
Interaction with ``Unpack``
|
|
---------------------------
|
|
|
|
For type checking purposes, ``Unpack[TypedDict]`` with extra items should be
|
|
treated as its equivalent in regular parameters, and the existing rules for
|
|
function parameters still apply::
|
|
|
|
class Movie(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: int
|
|
|
|
def f(**kwargs: Unpack[Movie]) -> None: ...
|
|
|
|
# Should be equivalent to
|
|
def f(*, name: str, **kwargs: int) -> None: ...
|
|
|
|
Interaction with PEP 705
|
|
------------------------
|
|
|
|
When the special ``__extra_items__`` item is annotated with ``ReadOnly[]``, the
|
|
extra items on the TypedDict have the properties of read-only items. This
|
|
interacts with inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`.
|
|
|
|
Notably, if the TypedDict type declares ``__extra_items__`` to be read-only, a
|
|
subclass of the TypedDict type may redeclare ``__extra_items__``'s value type or
|
|
additional non-extra items' value type.
|
|
|
|
Because a non-closed TypedDict type implicitly allows non-required extra items
|
|
of value type ``ReadOnly[object]``, its subclass can override the special
|
|
``__extra_items__`` with more specific types.
|
|
|
|
More details are discussed in the later sections.
|
|
|
|
Inheritance
|
|
-----------
|
|
|
|
When the TypedDict type is defined as ``closed=False`` (the default),
|
|
``__extra_items__`` should behave and be inherited the same way a regular key
|
|
would. A regular ``__extra_items__`` key can coexist with the special
|
|
``__extra_items__`` and both should be inherited when subclassing.
|
|
|
|
We assume that ``closed=True`` whenever ``__extra_items__`` is mentioned for the
|
|
rest of this section.
|
|
|
|
``__extra_items__`` is inherited the same way as a regular ``key: value_type``
|
|
item. As with the other keys, the same rules from
|
|
`the typing spec <https://typing.readthedocs.io/en/latest/spec/typeddict.html#inheritance>`__
|
|
and :pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the
|
|
context of ``__extra_items__``.
|
|
|
|
We need to reinterpret the following rule to define how ``__extra_items__``
|
|
interacts with it:
|
|
|
|
* Changing a field type of a parent TypedDict class in a subclass is not allowed.
|
|
|
|
First, it is not allowed to change the value type of ``__extra_items__`` in a subclass
|
|
unless it is declared to be ``ReadOnly`` in the superclass::
|
|
|
|
class Parent(TypedDict, closed=True):
|
|
__extra_items__: int | None
|
|
|
|
class Child(Parent, closed=True):
|
|
__extra_items__: int # Not OK. Like any other TypedDict item, __extra_items__'s type cannot be changed
|
|
|
|
Second, ``__extra_items__: T`` effectively defines the value type of any unnamed
|
|
items accepted to the TypedDict and marks them as non-required. Thus, the above
|
|
restriction applies to any additional items defined in a subclass. For each item
|
|
added in a subclass, all of the following conditions should apply:
|
|
|
|
- If ``__extra_items__`` is read-only
|
|
|
|
- The item can be either required or non-required
|
|
|
|
- The item's value type is consistent with ``T``
|
|
|
|
- If ``__extra_items__`` is not read-only
|
|
|
|
- The item is non-required
|
|
|
|
- The item's value type is consistent with ``T``
|
|
|
|
- ``T`` is consistent with the item's value type
|
|
|
|
- If ``__extra_items__`` is not redeclared, the subclass inherits it as-is.
|
|
|
|
For example::
|
|
|
|
class MovieBase(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: int | None
|
|
|
|
class AdaptedMovie(MovieBase): # Not OK. 'bool' is not consistent with 'int | None'
|
|
adapted_from_novel: bool
|
|
|
|
class MovieRequiredYear(MovieBase): # Not OK. Required key 'year' is not known to 'Parent'
|
|
year: int | None
|
|
|
|
class MovieNotRequiredYear(MovieBase): # Not OK. 'int | None' is not consistent with 'int'
|
|
year: NotRequired[int]
|
|
|
|
class MovieWithYear(MovieBase): # OK
|
|
year: NotRequired[int | None]
|
|
|
|
Due to this nature, an important side effect allows us to define a TypedDict
|
|
type that disallows additional items::
|
|
|
|
class MovieFinal(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: Never
|
|
|
|
Here, annotating ``__extra_items__`` with :class:`typing.Never` specifies that
|
|
there can be no other keys in ``MovieFinal`` other than the known ones.
|
|
Because of its potential common use, this is equivalent to::
|
|
|
|
class MovieFinal(TypedDict, closed=True):
|
|
name: str
|
|
|
|
where we implicitly assume the ``__extra_items__: Never`` field by default
|
|
if only ``closed=True`` is specified.
|
|
|
|
Type Consistency
|
|
----------------
|
|
|
|
In addition to the set ``S`` of keys of the explicitly defined items, a
|
|
TypedDict type that has the item ``__extra_items__: T`` is considered to have an
|
|
infinite set of items that all satisfy the following conditions:
|
|
|
|
- If ``__extra_items__`` is read-only
|
|
|
|
- The key's value type is consistent with ``T``
|
|
|
|
- The key is not in ``S``.
|
|
|
|
- If ``__extra_items__`` is not read-only
|
|
|
|
- The key is non-required
|
|
|
|
- The key's value type is consistent with ``T``
|
|
|
|
- ``T`` is consistent with the key's value type
|
|
|
|
- The key is not in ``S``.
|
|
|
|
For type checking purposes, let ``__extra_items__`` be a non-required pseudo-item to
|
|
be included whenever "for each ... item/key" is stated in
|
|
:pep:`the existing type consistency rules from PEP 705 <705#type-consistency>`,
|
|
and we modify it as follows:
|
|
|
|
A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is
|
|
structurally compatible with ``B``. This is true if and only if all of the
|
|
following are satisfied:
|
|
|
|
* For each item in ``B``, ``A`` has the corresponding key, unless the item
|
|
in ``B`` is read-only, not required, and of top value type
|
|
(``ReadOnly[NotRequired[object]]``). **[Edit: Otherwise, if the
|
|
corresponding key with the same name cannot be found in ``A``,
|
|
"__extra_items__" is considered the corresponding key.]**
|
|
|
|
* For each item in ``B``, if ``A`` has the corresponding key **[Edit: or
|
|
"__extra_items__"]**, the corresponding value type in ``A`` is consistent
|
|
with the value type in ``B``.
|
|
|
|
* For each non-read-only item in ``B``, its value type is consistent with
|
|
the corresponding value type in ``A``. **[Edit: if the corresponding key
|
|
with the same name cannot be found in ``A``, "__extra_items__" is
|
|
considered the corresponding key.]**
|
|
|
|
* For each required key in ``B``, the corresponding key is required in ``A``.
|
|
For each non-required key in ``B``, if the item is not read-only in ``B``,
|
|
the corresponding key is not required in ``A``.
|
|
**[Edit: if the corresponding key with the same name cannot be found in
|
|
``A``, "__extra_items__" is considered to be non-required as the
|
|
corresponding key.]**
|
|
|
|
The following examples illustrate these checks in action.
|
|
|
|
``__extra_items__`` puts various restrictions on additional items for type
|
|
consistency checks::
|
|
|
|
class Movie(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: int | None
|
|
|
|
class MovieDetails(TypedDict, closed=True):
|
|
name: str
|
|
year: NotRequired[int]
|
|
__extra_items__: int | None
|
|
|
|
details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003}
|
|
movie: Movie = details # Not OK. While 'int' is consistent with 'int | None',
|
|
# 'int | None' is not consistent with 'int'
|
|
|
|
class MovieWithYear(TypedDict, closed=True):
|
|
name: str
|
|
year: int | None
|
|
__extra_items__: int | None
|
|
|
|
details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003}
|
|
movie: Movie = details # Not OK. 'year' is not required in 'Movie',
|
|
# so it shouldn't be required in 'MovieWithYear' either
|
|
|
|
Because "year" is absent in ``Movie``, ``__extra_items__`` is considered the
|
|
corresponding key. ``"year"`` being required violates the rule "For each
|
|
required key in ``B``, the corresponding key is required in ``A``".
|
|
|
|
When ``__extra_items__`` is defined to be read-only in a TypedDict type, it is possible
|
|
for an item to have a narrower type than ``__extra_items__``'s value type::
|
|
|
|
class Movie(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: ReadOnly[str | int]
|
|
|
|
class MovieDetails(TypedDict, closed=True):
|
|
name: str
|
|
year: NotRequired[int]
|
|
__extra_items__: int
|
|
|
|
details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004}
|
|
movie: Movie = details # OK. 'int' is consistent with 'str | int'.
|
|
|
|
This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]``
|
|
is an item defined in ``Movie``.
|
|
|
|
``__extra_items__`` as a pseudo-item follows the same rules that other items have, so
|
|
when both TypedDicts contain ``__extra_items__``, this check is naturally enforced::
|
|
|
|
class MovieExtraInt(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: int
|
|
|
|
class MovieExtraStr(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: str
|
|
|
|
extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
|
|
extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""}
|
|
extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for item '__extra_items__'
|
|
extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for item '__extra_items__'
|
|
|
|
A non-closed TypedDict type implicitly allows non-required extra keys of value
|
|
type ``ReadOnly[object]``. This allows to apply the type consistency rules
|
|
between this type and a closed TypedDict type::
|
|
|
|
class MovieNotClosed(TypedDict):
|
|
name: str
|
|
|
|
extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
|
|
not_closed: MovieNotClosed = {"name": "No Country for Old Men"}
|
|
extra_int = not_closed # Not OK. 'ReadOnly[object]' implicitly on 'MovieNotClosed' is not consistent with 'int' for item '__extra_items__'
|
|
not_closed = extra_int # OK
|
|
|
|
Interaction with Mapping[KT, VT]
|
|
--------------------------------
|
|
|
|
A TypedDict type can be consistent with ``Mapping[KT, VT]`` types other than
|
|
``Mapping[str, object]`` as long as the union of value types on the TypedDict
|
|
type is consistent with ``VT``. It is an extension of this rule from the typing
|
|
spec:
|
|
|
|
* A TypedDict with all ``int`` values is not consistent with
|
|
``Mapping[str, int]``, since there may be additional non-``int``
|
|
values not visible through the type, due to structural subtyping.
|
|
These can be accessed using the ``values()`` and ``items()``
|
|
methods in ``Mapping``
|
|
|
|
For example::
|
|
|
|
class MovieExtraStr(TypedDict, closed=True):
|
|
name: str
|
|
__extra_items__: str
|
|
|
|
extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""}
|
|
str_mapping: Mapping[str, str] = extra_str # OK
|
|
|
|
int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int'
|
|
int_str_mapping: Mapping[str, int | str] = extra_int # OK
|
|
|
|
Furthermore, type checkers should be able to infer the precise return types of
|
|
``values()`` and ``items()`` on such TypedDict types::
|
|
|
|
def fun(movie: MovieExtraStr) -> None:
|
|
reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]'
|
|
reveal_type(movie.values()) # Revealed type is 'dict_values[str, str]'
|
|
|
|
Interaction with dict[KT, VT]
|
|
-----------------------------
|
|
|
|
Note that because the presence of ``__extra_items__`` on a closed TypedDict type
|
|
prohibits additional required keys in its structural subtypes, we can determine
|
|
if the TypedDict type and its structural subtypes will ever have any required
|
|
key during static analysis.
|
|
|
|
If there is no required key, the TypedDict type is consistent with ``dict[KT,
|
|
VT]`` and vice versa if all items on the TypedDict type satisfy the following
|
|
conditions:
|
|
|
|
- ``VT`` is consistent with the value type of the item
|
|
|
|
- The value type of the item is consistent with ``VT``
|
|
|
|
For example::
|
|
|
|
class IntDict(TypedDict, closed=True):
|
|
__extra_items__: int
|
|
|
|
class IntDictWithNum(IntDict):
|
|
num: NotRequired[int]
|
|
|
|
def f(x: IntDict) -> None:
|
|
v: dict[str, int] = x # OK
|
|
v.clear() # OK
|
|
|
|
not_required_num: IntDictWithNum = {"num": 1, "bar": 2}
|
|
regular_dict: dict[str, int] = not_required_num # OK
|
|
f(not_required_num) # OK
|
|
|
|
In this case, methods that are previously unavailable on a TypedDict are allowed::
|
|
|
|
not_required_num.clear() # OK
|
|
|
|
reveal_type(not_required_num.popitem()) # OK. Revealed type is tuple[str, int]
|
|
|
|
How to Teach this
|
|
=================
|
|
|
|
The choice of ``"__extra_items__"`` and the requirement of ``closed=True``
|
|
whenever it is used as a special key intended to make it more understandable to
|
|
new users even without former knowledge of this feature.
|
|
|
|
Details of this should be documented in both the typing spec and the
|
|
:mod:`typing` documentation.
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
Because ``__extra_items__`` remains as a regular key if ``closed=True`` is not
|
|
specified, no existing codebase will break due to this change.
|
|
|
|
If the proposal is accepted, none of ``__required_keys__``,
|
|
``__optional_keys__``, ``__readonly_keys__`` and ``__mutable_keys__`` should
|
|
include ``"__extra_items__"`` defined on the same TypedDict type when
|
|
``closed=True`` is specified.
|
|
|
|
Note that ``closed`` as a keyword argument does not collide with the keyword
|
|
arguments alternative to define keys with the functional syntax that allows
|
|
things like ``TD = TypedDict("TD", foo=str, bar=int)``, because it is scheduled
|
|
to be removed in Python 3.13.
|
|
|
|
Because this is a type-checking feature, it can be made available to older
|
|
versions as long as the type checker supports it.
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
Allowing Extra Items without Specifying the Type
|
|
------------------------------------------------
|
|
|
|
``extra=True`` was originally proposed for defining a TypedDict that accepts extra
|
|
items regardless of the type, like how ``total=True`` works::
|
|
|
|
class TypedDict(extra=True):
|
|
pass
|
|
|
|
Because it did not offer a way to specify the type of the extra items, the type
|
|
checkers will need to assume that the type of the extra items is ``Any``, which
|
|
compromises type safety. Furthermore, the current behavior of TypedDict already
|
|
allows untyped extra items to be present in runtime, due to structural
|
|
subtyping. ``closed=True`` plays a similar role in the current proposal.
|
|
|
|
Supporting ``TypedDict(extra=type)``
|
|
------------------------------------
|
|
|
|
While this design is potentially viable, there are several partially addressable
|
|
concerns to consider. Adding everything up, it is slightly less favorable than
|
|
the current proposal.
|
|
|
|
- Usability of forward reference
|
|
As in the functional syntax, using a quoted type or a type alias will be
|
|
required when SomeType is a forward reference. This is already a requirement
|
|
for the functional syntax, so implementations can potentially reuse that piece
|
|
of logic, but this is still extra work that the ``closed=True`` proposal doesn't
|
|
have.
|
|
|
|
- Concerns about using type as a value
|
|
Whatever is not allowed as the value type in the functional syntax should not
|
|
be allowed as the argument for extra either. While type checkers might be able
|
|
to reuse this check, it still needs to be somehow special-cased for the
|
|
class-based syntax.
|
|
|
|
- How to teach
|
|
Notably, the ``extra=type`` often gets brought up due to it being an intuitive
|
|
solution for the use case, so it is potentially simpler to learn than the less
|
|
obvious solution. However, the more common used case only requires
|
|
``closed=True``, and the other drawbacks mentioned earlier outweigh what is
|
|
need to teach the usage of the special key.
|
|
|
|
Support Extra Items with Intersection
|
|
-------------------------------------
|
|
|
|
Supporting intersections in Python's type system requires a lot of careful
|
|
consideration, and it can take a long time for the community to reach a
|
|
consensus on a reasonable design.
|
|
|
|
Ideally, extra items in TypedDict should not be blocked by work on
|
|
intersections, nor does it necessarily need to be supported through
|
|
intersections.
|
|
|
|
Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not
|
|
equivalent to a TypedDict type with the proposed ``__extra_items__`` special
|
|
item, as the value type of all known items in ``TypedDict`` needs to satisfy the
|
|
is-subtype-of relation with the value type of ``Mapping[...]``.
|
|
|
|
Requiring Type Compatibility of the Known Items with ``__extra_items__``
|
|
------------------------------------------------------------------------
|
|
|
|
``__extra_items__`` restricts the value type for keys that are *unknown* to the
|
|
TypedDict type. So the value type of any *known* item is not necessarily
|
|
consistent with ``__extra_items__``'s type, and ``__extra_items__``'s type is
|
|
not necessarily consistent with the value types of all known items.
|
|
|
|
This differs from TypeScript's `Index Signatures
|
|
<https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures>`__
|
|
syntax, which requires all properties' types to match the string index's type.
|
|
For example:
|
|
|
|
.. code-block:: typescript
|
|
|
|
interface MovieWithExtraNumber {
|
|
name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'.
|
|
[index: string]: number
|
|
}
|
|
|
|
interface MovieWithExtraNumberOrString {
|
|
name: string // OK
|
|
[index: string]: number | string
|
|
}
|
|
|
|
This is a known limitation discussed in `TypeScript's issue tracker
|
|
<https://github.com/microsoft/TypeScript/issues/17867>`__,
|
|
where it is suggested that there should be a way to exclude the defined keys
|
|
from the index signature so that it is possible to define a type like
|
|
``MovieWithExtraNumber``.
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
pyanalyze has `experimental support
|
|
<https://github.com/quora/pyanalyze/blob/9bfc2c58467c87774a9950838402d2657b1486a0/pyanalyze/extensions.py#L590>`__
|
|
for a similar feature.
|
|
|
|
Reference implementation for this specific proposal, however, is not currently
|
|
available.
|
|
|
|
Acknowledgments
|
|
===============
|
|
|
|
Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback,
|
|
Eric Traut who `proposed the original design
|
|
<https://mail.python.org/archives/list/typing-sig@python.org/message/3Z72OQWVTOVS6UYUUCCII2UZN56PV5II/>`__
|
|
this PEP iterates on, and Alice Purcell for offering their perspective as the
|
|
author of :pep:`705`.
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|