PEP: 728 Title: TypedDict with Typed Extra Items Author: Zixuan James Li Sponsor: Jelle Zijlstra Discussions-To: https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443 Status: Draft Type: Standards Track Topic: Typing Content-Type: text/x-rst Created: 12-Sep-2023 Python-Version: 3.13 Post-History: `09-Feb-2024 `__, Abstract ======== This PEP proposes a way to limit extra items for :class:`~typing.TypedDict` using a ``closed`` argument and to type them with the special ``__extra_items__`` key. This addresses the need to define closed TypedDict type or to type a subset of keys that might appear in a ``dict`` while permitting additional items of a specified type. Motivation ========== A :py:class:`typing.TypedDict` type can annotate the value type of each known item in a dictionary. However, due to structural subtyping, a TypedDict can have extra items that are not visible through its type. There is currently no way to restrict the types of items that might be present in the TypedDict type's structural subtypes. Defining a Closed TypedDict Type -------------------------------- The current behavior of TypedDict prevents users from defining a closed TypedDict type when it is expected that the type contains no additional items. Due to the possible presence of extra items, type checkers cannot infer more precise return types for ``.items()`` and ``.values()`` on a TypedDict. This can also be resolved by `defining a closed TypedDict type `__. Another possible use case for this is a sound way to `enable type narrowing `__ with the ``in`` check:: class Movie(TypedDict): name: str director: str class Book(TypedDict): name: str author: str def fun(entry: Movie | Book) -> None: if "author" in entry: reveal_type(entry) # Revealed type is 'Movie | Book' Nothing prevents a ``dict`` that is structurally compatible with ``Movie`` to have the ``author`` key, and under the current specification it would be incorrect for the type checker to narrow its type. Allowing Extra Items of a Certain Type -------------------------------------- For supporting API interfaces or legacy codebase where only a subset of possible keys are known, it would be useful to explicitly expect additional keys of certain value types. However, the typing spec is more restrictive on type checking the construction of a TypedDict, `preventing users `__ from doing this:: class MovieBase(TypedDict): name: str def fun(movie: MovieBase) -> None: # movie can have extra items that are not visible through MovieBase ... movie: MovieBase = {"name": "Blade Runner", "year": 1982} # Not OK fun({"name": "Blade Runner", "year": 1982}) # Not OK While the restriction is enforced when constructing a TypedDict, due to structural subtyping, the TypedDict may have extra items that are not visible through its type. For example:: class Movie(MovieBase): year: int movie: Movie = {"name": "Blade Runner", "year": 1982} fun(movie) # OK It is not possible to acknowledge the existence of the extra items through ``in`` checks and access them without breaking type safety, even though they might exist from arbitrary structural subtypes of ``MovieBase``:: def g(movie: MovieBase) -> None: if "year" in movie: reveal_type(movie["year"]) # Error: TypedDict 'MovieBase' has no key 'year' Some workarounds have already been implemented in response to the need to allow extra keys, but none of them is ideal. For mypy, ``--disable-error-code=typeddict-unknown-key`` `suppresses type checking error `__ specifically for unknown keys on TypedDict. This sacrifices type safety over flexibility, and it does not offer a way to specify that the TypedDict type expects additional keys compatible with a certain type. Support Additional Keys for ``Unpack`` -------------------------------------- :pep:`692` adds a way to precisely annotate the types of individual keyword arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However, because TypedDict cannot be defined to accept arbitrary extra items, it is not possible to `allow additional keyword arguments `__ that are not known at the time the TypedDict is defined. Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing codebases, it will be valuable to accept and type extra items on TypedDict so that the old typing behavior can be supported in combination with the new ``Unpack`` construct. Rationale ========= A type that allows extra items of type ``str`` on a TypedDict can be loosely described as the intersection between the TypedDict and ``Mapping[str, str]``. `Index Signatures `__ in TypeScript achieve this: .. code-block:: typescript type Foo = { a: string [key: string]: string } This proposal aims to support a similar feature without introducing general intersection of types or syntax changes, offering a natural extension to the existing type consistency rules. We propose that we add an argument ``closed`` to TypedDict. Similar to ``total``, only a literal ``True`` or ``False`` value is allowed. When ``closed=True`` is used in the TypedDict type definition, we give the dunder attribute ``__extra_items__`` a special meaning: extra items are allowed, and their types should be compatible with the value type of ``__extra_items__``. If ``closed=True`` is set, but there is no ``__extra_items__`` key, the TypedDict is treated as if it contained an item ``__extra_items__: Never``. Note that ``__extra_items__`` on the same TypedDict type definition will remain as a regular item if ``closed=True`` is not used. Different from index signatures, the types of the known items do not need to be consistent with the value type of ``__extra_items__``. There are some advantages to this approach: - Inheritance works naturally. ``__extra_items__`` defined on a TypedDict will also be available to its subclasses. - We can build on top of the `type consistency rules defined in the typing spec `__. ``__extra_items__`` can be treated as a pseudo-item in terms of type consistency. - There is no need to introduce a grammar change to specify the type of the extra items. - We can precisely type the extra items without making ``__extra_items__`` the union of known items. - We do not lose backwards compatibility as ``__extra_items__`` still can be used as a regular key. Specification ============= This specification is structured to parallel :pep:`589` to highlight changes to the original TypedDict specification. If ``closed=True`` is specified, extra items are treated as non-required items having the same type of ``__extra_items__`` whose keys are allowed when determining `supported and unsupported operations `__. Using TypedDict Types --------------------- Assuming that ``closed=True`` is used in the TypedDict type definition. For a TypedDict type that has the special ``__extra_items__`` key, during construction, the value type of each unknown item is expected to be non-required and compatible with the value type of ``__extra_items__``. For example:: class Movie(TypedDict, closed=True): name: str __extra_items__: bool a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # OK b: Movie = { "name": "Blade Runner", "year": 1982, # Not OK. 'int' is incompatible with 'bool' } In this example, ``__extra_items__: bool`` does not mean that ``Movie`` has a required string key ``"__extra_items__"`` whose value type is ``bool``. Instead, it specifies that keys other than "name" have a value type of ``bool`` and are non-required. The alternative inline syntax is also supported:: Movie = TypedDict("Movie", {"name": str, "__extra_items__": bool}, closed=True) Accessing extra keys is allowed. Type checkers must infer its value type from the value type of ``__extra_items__``:: def f(movie: Movie) -> None: reveal_type(movie["name"]) # Revealed type is 'str' reveal_type(movie["novel_adaptation"]) # Revealed type is 'bool' When a TypedDict type defines ``__extra_items__`` without ``closed=True``, ``closed`` defaults to ``False`` and the key is assumed to be a regular key:: class Movie(TypedDict): name: str __extra_items__: bool a: Movie = {"name": "Blade Runner", "novel_adaptation": True} # Not OK. Unexpected key 'novel_adaptation' b: Movie = { "name": "Blade Runner", "__extra_items__": True, # OK } For such non-closed TypedDict types, it is assumed that they allow non-required extra items of value type ``ReadOnly[object]`` during inheritance or type consistency checks. However, extra keys found during construction should still be rejected by the type checker. ``closed`` is not inherited through subclassing:: class MovieBase(TypedDict, closed=True): name: str __extra_items__: ReadOnly[str | None] class Movie(MovieBase): __extra_items__: str # A regular key a: Movie = {"name": "Blade Runner", "__extra_items__": None} # Not OK. 'None' is incompatible with 'str' b: Movie = { "name": "Blade Runner", "__extra_items__": "A required regular key", "other_extra_key": None, } # OK Here, ``"__extra_items__"`` in ``a`` is a regular key defined on ``Movie`` where its value type is narrowed from ``ReadOnly[str | None]`` to ``str``, ``"other_extra_key"`` in ``b`` is an extra key whose value type must be consistent with the value type of ``"__extra_items__"`` defined on ``MovieBase``. Interaction with Totality ------------------------- It is an error to use ``Required[]`` or ``NotRequired[]`` with the special ``__extra_items__`` item. ``total=False`` and ``total=True`` have no effect on ``__extra_items__`` itself. The extra items are non-required, regardless of the totality of the TypedDict. Operations that are available to ``NotRequired`` items should also be available to the extra items:: class Movie(TypedDict, closed=True): name: str __extra_items__: int def f(movie: Movie) -> None: del movie["name"] # Not OK del movie["year"] # OK Interaction with ``Unpack`` --------------------------- For type checking purposes, ``Unpack[TypedDict]`` with extra items should be treated as its equivalent in regular parameters, and the existing rules for function parameters still apply:: class Movie(TypedDict, closed=True): name: str __extra_items__: int def f(**kwargs: Unpack[Movie]) -> None: ... # Should be equivalent to def f(*, name: str, **kwargs: int) -> None: ... Interaction with PEP 705 ------------------------ When the special ``__extra_items__`` item is annotated with ``ReadOnly[]``, the extra items on the TypedDict have the properties of read-only items. This interacts with inheritance rules specified in :pep:`PEP 705 <705#Inheritance>`. Notably, if the TypedDict type declares ``__extra_items__`` to be read-only, a subclass of the TypedDict type may redeclare ``__extra_items__``'s value type or additional non-extra items' value type. Because a non-closed TypedDict type implicitly allows non-required extra items of value type ``ReadOnly[object]``, its subclass can override the special ``__extra_items__`` with more specific types. More details are discussed in the later sections. Inheritance ----------- When the TypedDict type is defined as ``closed=False`` (the default), ``__extra_items__`` should behave and be inherited the same way a regular key would. A regular ``__extra_items__`` key can coexist with the special ``__extra_items__`` and both should be inherited when subclassing. We assume that ``closed=True`` whenever ``__extra_items__`` is mentioned for the rest of this section. ``__extra_items__`` is inherited the same way as a regular ``key: value_type`` item. As with the other keys, the same rules from `the typing spec `__ and :pep:`PEP 705 <705#inheritance>` apply. We interpret the existing rules in the context of ``__extra_items__``. We need to reinterpret the following rule to define how ``__extra_items__`` interacts with it: * Changing a field type of a parent TypedDict class in a subclass is not allowed. First, it is not allowed to change the value type of ``__extra_items__`` in a subclass unless it is declared to be ``ReadOnly`` in the superclass:: class Parent(TypedDict, closed=True): __extra_items__: int | None class Child(Parent, closed=True): __extra_items__: int # Not OK. Like any other TypedDict item, __extra_items__'s type cannot be changed Second, ``__extra_items__: T`` effectively defines the value type of any unnamed items accepted to the TypedDict and marks them as non-required. Thus, the above restriction applies to any additional items defined in a subclass. For each item added in a subclass, all of the following conditions should apply: - If ``__extra_items__`` is read-only - The item can be either required or non-required - The item's value type is consistent with ``T`` - If ``__extra_items__`` is not read-only - The item is non-required - The item's value type is consistent with ``T`` - ``T`` is consistent with the item's value type - If ``__extra_items__`` is not redeclared, the subclass inherits it as-is. For example:: class MovieBase(TypedDict, closed=True): name: str __extra_items__: int | None class AdaptedMovie(MovieBase): # Not OK. 'bool' is not consistent with 'int | None' adapted_from_novel: bool class MovieRequiredYear(MovieBase): # Not OK. Required key 'year' is not known to 'Parent' year: int | None class MovieNotRequiredYear(MovieBase): # Not OK. 'int | None' is not consistent with 'int' year: NotRequired[int] class MovieWithYear(MovieBase): # OK year: NotRequired[int | None] Due to this nature, an important side effect allows us to define a TypedDict type that disallows additional items:: class MovieFinal(TypedDict, closed=True): name: str __extra_items__: Never Here, annotating ``__extra_items__`` with :class:`typing.Never` specifies that there can be no other keys in ``MovieFinal`` other than the known ones. Because of its potential common use, this is equivalent to:: class MovieFinal(TypedDict, closed=True): name: str where we implicitly assume the ``__extra_items__: Never`` field by default if only ``closed=True`` is specified. Type Consistency ---------------- In addition to the set ``S`` of keys of the explicitly defined items, a TypedDict type that has the item ``__extra_items__: T`` is considered to have an infinite set of items that all satisfy the following conditions: - If ``__extra_items__`` is read-only - The key's value type is consistent with ``T`` - The key is not in ``S``. - If ``__extra_items__`` is not read-only - The key is non-required - The key's value type is consistent with ``T`` - ``T`` is consistent with the key's value type - The key is not in ``S``. For type checking purposes, let ``__extra_items__`` be a non-required pseudo-item to be included whenever "for each ... item/key" is stated in :pep:`the existing type consistency rules from PEP 705 <705#type-consistency>`, and we modify it as follows: A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if all of the following are satisfied: * For each item in ``B``, ``A`` has the corresponding key, unless the item in ``B`` is read-only, not required, and of top value type (``ReadOnly[NotRequired[object]]``). **[Edit: Otherwise, if the corresponding key with the same name cannot be found in ``A``, "__extra_items__" is considered the corresponding key.]** * For each item in ``B``, if ``A`` has the corresponding key **[Edit: or "__extra_items__"]**, the corresponding value type in ``A`` is consistent with the value type in ``B``. * For each non-read-only item in ``B``, its value type is consistent with the corresponding value type in ``A``. **[Edit: if the corresponding key with the same name cannot be found in ``A``, "__extra_items__" is considered the corresponding key.]** * For each required key in ``B``, the corresponding key is required in ``A``. For each non-required key in ``B``, if the item is not read-only in ``B``, the corresponding key is not required in ``A``. **[Edit: if the corresponding key with the same name cannot be found in ``A``, "__extra_items__" is considered to be non-required as the corresponding key.]** The following examples illustrate these checks in action. ``__extra_items__`` puts various restrictions on additional items for type consistency checks:: class Movie(TypedDict, closed=True): name: str __extra_items__: int | None class MovieDetails(TypedDict, closed=True): name: str year: NotRequired[int] __extra_items__: int | None details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003} movie: Movie = details # Not OK. While 'int' is consistent with 'int | None', # 'int | None' is not consistent with 'int' class MovieWithYear(TypedDict, closed=True): name: str year: int | None __extra_items__: int | None details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003} movie: Movie = details # Not OK. 'year' is not required in 'Movie', # so it shouldn't be required in 'MovieWithYear' either Because "year" is absent in ``Movie``, ``__extra_items__`` is considered the corresponding key. ``"year"`` being required violates the rule "For each required key in ``B``, the corresponding key is required in ``A``". When ``__extra_items__`` is defined to be read-only in a TypedDict type, it is possible for an item to have a narrower type than ``__extra_items__``'s value type:: class Movie(TypedDict, closed=True): name: str __extra_items__: ReadOnly[str | int] class MovieDetails(TypedDict, closed=True): name: str year: NotRequired[int] __extra_items__: int details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004} movie: Movie = details # OK. 'int' is consistent with 'str | int'. This behaves the same way as :pep:`705` specified if ``year: ReadOnly[str | int]`` is an item defined in ``Movie``. ``__extra_items__`` as a pseudo-item follows the same rules that other items have, so when both TypedDicts contain ``__extra_items__``, this check is naturally enforced:: class MovieExtraInt(TypedDict, closed=True): name: str __extra_items__: int class MovieExtraStr(TypedDict, closed=True): name: str __extra_items__: str extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007} extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""} extra_int = extra_str # Not OK. 'str' is inconsistent with 'int' for item '__extra_items__' extra_str = extra_int # Not OK. 'int' is inconsistent with 'str' for item '__extra_items__' A non-closed TypedDict type implicitly allows non-required extra keys of value type ``ReadOnly[object]``. This allows to apply the type consistency rules between this type and a closed TypedDict type:: class MovieNotClosed(TypedDict): name: str extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007} not_closed: MovieNotClosed = {"name": "No Country for Old Men"} extra_int = not_closed # Not OK. 'ReadOnly[object]' implicitly on 'MovieNotClosed' is not consistent with 'int' for item '__extra_items__' not_closed = extra_int # OK Interaction with Constructors ----------------------------- TypedDicts that allow extra items of type ``T`` also allow arbitrary keyword arguments of this type when constructed by calling the class object:: class OpenMovie(TypedDict): name: str OpenMovie(name="No Country for Old Men") # OK OpenMovie(name="No Country for Old Men", year=2007) # Not OK. Unrecognized key class ExtraMovie(TypedDict, closed=True): name: str __extra_items__: int ExtraMovie(name="No Country for Old Men") # OK ExtraMovie(name="No Country for Old Men", year=2007) # OK ExtraMovie( name="No Country for Old Men", language="English", ) # Not OK. Wrong type for extra key # This implies '__extra_items__: Never', # so extra keyword arguments produce an error class ClosedMovie(TypedDict, closed=True): name: str ClosedMovie(name="No Country for Old Men") # OK ClosedMovie( name="No Country for Old Men", year=2007, ) # Not OK. Extra items not allowed Interaction with Mapping[KT, VT] -------------------------------- A TypedDict type can be consistent with ``Mapping[KT, VT]`` types other than ``Mapping[str, object]`` as long as the union of value types on the TypedDict type is consistent with ``VT``. It is an extension of this rule from the typing spec: * A TypedDict with all ``int`` values is not consistent with ``Mapping[str, int]``, since there may be additional non-``int`` values not visible through the type, due to structural subtyping. These can be accessed using the ``values()`` and ``items()`` methods in ``Mapping`` For example:: class MovieExtraStr(TypedDict, closed=True): name: str __extra_items__: str extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""} str_mapping: Mapping[str, str] = extra_str # OK int_mapping: Mapping[str, int] = extra_int # Not OK. 'int | str' is not consistent with 'int' int_str_mapping: Mapping[str, int | str] = extra_int # OK Furthermore, type checkers should be able to infer the precise return types of ``values()`` and ``items()`` on such TypedDict types:: def fun(movie: MovieExtraStr) -> None: reveal_type(movie.items()) # Revealed type is 'dict_items[str, str]' reveal_type(movie.values()) # Revealed type is 'dict_values[str, str]' Interaction with dict[KT, VT] ----------------------------- Note that because the presence of ``__extra_items__`` on a closed TypedDict type prohibits additional required keys in its structural subtypes, we can determine if the TypedDict type and its structural subtypes will ever have any required key during static analysis. The TypedDict type is consistent with ``dict[str, VT]`` if all items on the TypedDict type satisfy the following conditions: - ``VT`` is consistent with the value type of the item - The value type of the item is consistent with ``VT`` - The item is not read-only. - The item is not required. For example:: class IntDict(TypedDict, closed=True): __extra_items__: int class IntDictWithNum(IntDict): num: NotRequired[int] def f(x: IntDict) -> None: v: dict[str, int] = x # OK v.clear() # OK not_required_num: IntDictWithNum = {"num": 1, "bar": 2} regular_dict: dict[str, int] = not_required_num # OK f(not_required_num) # OK In this case, methods that are previously unavailable on a TypedDict are allowed:: not_required_num.clear() # OK reveal_type(not_required_num.popitem()) # OK. Revealed type is tuple[str, int] However, ``dict[str, VT]`` is not necessarily consistent with a TypedDict type, because such dict can be a subtype of dict:: class CustomDict(dict[str, int]): ... not_a_regular_dict: CustomDict = {"num": 1} int_dict: IntDict = not_a_regular_dict # Not OK How to Teach This ================= The choice of the spelling ``"__extra_items__"`` is intended to make this feature more understandable to new users compared to shorter alternatives like ``"__extra__"``. Details of this should be documented in both the typing spec and the :mod:`typing` documentation. Backwards Compatibility ======================= Because ``__extra_items__`` remains as a regular key if ``closed=True`` is not specified, no existing codebase will break due to this change. If the proposal is accepted, none of ``__required_keys__``, ``__optional_keys__``, ``__readonly_keys__`` and ``__mutable_keys__`` should include ``"__extra_items__"`` defined on the same TypedDict type when ``closed=True`` is specified. Note that ``closed`` as a keyword argument does not collide with the keyword arguments alternative to define keys with the functional syntax that allows things like ``TD = TypedDict("TD", foo=str, bar=int)``, because it is scheduled to be removed in Python 3.13. Because this is a type-checking feature, it can be made available to older versions as long as the type checker supports it. Rejected Ideas ============== Allowing Extra Items without Specifying the Type ------------------------------------------------ ``extra=True`` was originally proposed for defining a TypedDict that accepts extra items regardless of the type, like how ``total=True`` works:: class TypedDict(extra=True): pass Because it did not offer a way to specify the type of the extra items, the type checkers will need to assume that the type of the extra items is ``Any``, which compromises type safety. Furthermore, the current behavior of TypedDict already allows untyped extra items to be present in runtime, due to structural subtyping. ``closed=True`` plays a similar role in the current proposal. Supporting ``TypedDict(extra=type)`` ------------------------------------ During the discussion of the PEP, there were strong objections against adding another place where types are passed as values instead of annotations from some authors of type checkers. While this design is potentially viable, there are also several partially addressable concerns to consider. - Usability of forward reference As in the functional syntax, using a quoted type or a type alias will be required when SomeType is a forward reference. This is already a requirement for the functional syntax, so implementations can potentially reuse that piece of logic, but this is still extra work that the ``closed=True`` proposal doesn't have. - Concerns about using type as a value Whatever is not allowed as the value type in the functional syntax should not be allowed as the argument for extra either. While type checkers might be able to reuse this check, it still needs to be somehow special-cased for the class-based syntax. - How to teach Notably, the ``extra=type`` often gets brought up due to it being an intuitive solution for the use case, so it is potentially simpler to learn than the less obvious solution. However, the more common used case only requires ``closed=True``, and the other drawbacks mentioned earlier outweigh what is need to teach the usage of the special key. Support Extra Items with Intersection ------------------------------------- Supporting intersections in Python's type system requires a lot of careful consideration, and it can take a long time for the community to reach a consensus on a reasonable design. Ideally, extra items in TypedDict should not be blocked by work on intersections, nor does it necessarily need to be supported through intersections. Moreover, the intersection between ``Mapping[...]`` and ``TypedDict`` is not equivalent to a TypedDict type with the proposed ``__extra_items__`` special item, as the value type of all known items in ``TypedDict`` needs to satisfy the is-subtype-of relation with the value type of ``Mapping[...]``. Requiring Type Compatibility of the Known Items with ``__extra_items__`` ------------------------------------------------------------------------ ``__extra_items__`` restricts the value type for keys that are *unknown* to the TypedDict type. So the value type of any *known* item is not necessarily consistent with ``__extra_items__``'s type, and ``__extra_items__``'s type is not necessarily consistent with the value types of all known items. This differs from TypeScript's `Index Signatures `__ syntax, which requires all properties' types to match the string index's type. For example: .. code-block:: typescript interface MovieWithExtraNumber { name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'. [index: string]: number } interface MovieWithExtraNumberOrString { name: string // OK [index: string]: number | string } This is a known limitation discussed in `TypeScript's issue tracker `__, where it is suggested that there should be a way to exclude the defined keys from the index signature so that it is possible to define a type like ``MovieWithExtraNumber``. Reference Implementation ======================== This proposal is supported in `pyright 1.1.352 `_, and `pyanalyze 0.12.0 `_. Acknowledgments =============== Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback, Eric Traut who `proposed the original design `__ this PEP iterates on, and Alice Purcell for offering their perspective as the author of :pep:`705`. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.