iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/peps/pep-0705.rst

585 lines
25 KiB
ReStructuredText
Raw Blame History

PEP: 705
Title: TypedDict: Read-only items
Author: Alice Purcell <alicederyn@gmail.com>
Sponsor: Pablo Galindo <pablogsal@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867
Status: Final
Type: Standards Track
Topic: Typing
Created: 07-Nov-2022
Python-Version: 3.13
Post-History: `30-Sep-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/6FR6RKNUZU4UY6B6RXC2H4IAHKBU3UKV/>`__,
`02-Nov-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/2P26R4VH2ZCNNNOQCBZWEM4RNF35OXOW/>`__,
`14-Mar-2023 <https://discuss.python.org/t/pep-705-typedmapping/24827>`__,
`17-Oct-2023 <https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457>`__,
`04-Nov-2023 <https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867>`__,
Resolution: `29-Feb-2024 <https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867/39>`__
.. canonical-typing-spec:: :ref:`typing:readonly` and
:external+py3.13:data:`typing.ReadOnly`
Abstract
========
:pep:`589` defines the structural type :class:`~typing.TypedDict` for dictionaries with a fixed set of keys.
As ``TypedDict`` is a mutable type, it is difficult to correctly annotate methods which accept read-only parameters in a way that doesn't prevent valid inputs.
This PEP proposes a new type qualifier, ``typing.ReadOnly``, to support these usages. It makes no Python grammar changes. Correct usage of read-only keys of TypedDicts is intended to be enforced only by static type checkers, and will not be enforced by Python itself at runtime.
Motivation
==========
Representing structured data using (potentially nested) dictionaries with string keys is a common pattern in Python programs. :pep:`589` allows these values to be type checked when the exact type is known up-front, but it is hard to write read-only code that accepts more specific variants: for instance, where values may be subtypes or restrict a union of possible types. This is an especially common issue when writing APIs for services, which may support a wide range of input structures, and typically do not need to modify their input.
Pure functions
--------------
Consider trying to add type hints to a function ``movie_string``::
def movie_string(movie: Movie) -> str:
if movie.get("year") is None:
return movie["name"]
else:
return f'{movie["name"]} ({movie["year"]})'
We could define this ``Movie`` type using a ``TypedDict``::
from typing import NotRequired, TypedDict
class Movie(TypedDict):
name: str
year: NotRequired[int | None]
But suppose we have another type where year is required::
class MovieRecord(TypedDict):
name: str
year: int
Attempting to pass a ``MovieRecord`` into ``movie_string`` results in the error (using mypy):
.. code-block:: text
Argument 1 to "movie_string" has incompatible type "MovieRecord"; expected "Movie"
This particular use case should be type-safe, but the type checker correctly stops the
user from passing a ``MovieRecord`` into a ``Movie`` parameter in the general case, because
the ``Movie`` class has mutator methods that could potentially allow the function to break
the type constraints in ``MovieRecord`` (e.g. with ``movie["year"] = None`` or ``del movie["year"]``).
The problem disappears if we don't have mutator methods in ``Movie``. This could be achieved by defining an immutable interface using a :pep:`544` :class:`~typing.Protocol`::
from typing import Literal, Protocol, overload
class Movie(Protocol):
@overload
def get(self, key: Literal["name"]) -> str: ...
@overload
def get(self, key: Literal["year"]) -> int | None: ...
@overload
def __getitem__(self, key: Literal["name"]) -> str: ...
@overload
def __getitem__(self, key: Literal["year"]) -> int | None: ...
This is very repetitive, easy to get wrong, and is still missing important method definitions like ``__contains__()`` and ``keys()``.
Updating nested dicts
---------------------
The structural typing of ``TypedDict`` is supposed to permit writing update functions that only constrain the types of items they modify::
class HasTimestamp(TypedDict):
timestamp: float
class Logs(TypedDict):
timestamp: float
loglines: list[str]
def update_timestamp(d: HasTimestamp) -> None:
d["timestamp"] = now()
def add_logline(logs: Logs, logline: str) -> None:
logs["loglines"].append(logline)
update_timestamp(logs) # Accepted by type checker
However, this no longer works once you start nesting dictionaries::
class HasTimestampedMetadata(TypedDict):
metadata: HasTimestamp
class UserAudit(TypedDict):
name: str
metadata: Logs
def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
d["metadata"]["timestamp"] = now()
def rename_user(d: UserAudit, name: str) -> None:
d["name"] = name
update_metadata_timestamp(d) # Type check error: "metadata" is not of type HasTimestamp
This looks like an error, but is simply due to the (unwanted) ability to overwrite the ``metadata`` item held by the ``HasTimestampedMetadata`` instance with a different ``HasTimestamp`` instance, that may no longer be a ``Logs`` instance.
It is possible to work around this issue with generics (as of Python 3.11), but it is very complicated, requiring a type parameter for every nested dict.
Rationale
=========
These problems can be resolved by removing the ability to update one or more of the items in a ``TypedDict``. This does not mean the items are immutable: a reference to the underlying dictionary could still exist with a different but compatible type in which those items have mutator operations. These items are "read-only", and we introduce a new ``typing.ReadOnly`` type qualifier for this purpose.
The ``movie_string`` function in the first motivating example can then be typed as follows::
from typing import NotRequired, ReadOnly, TypedDict
class Movie(TypedDict):
name: ReadOnly[str]
year: ReadOnly[NotRequired[int | None]]
def movie_string(movie: Movie) -> str:
if movie.get("year") is None:
return movie["name"]
else:
return f'{movie["name"]} ({movie["year"]})'
A mixture of read-only and non-read-only items is permitted, allowing the second motivating example to be correctly annotated::
class HasTimestamp(TypedDict):
timestamp: float
class HasTimestampedMetadata(TypedDict):
metadata: ReadOnly[HasTimestamp]
def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
d["metadata"]["timestamp"] = now()
class Logs(HasTimestamp):
loglines: list[str]
class UserAudit(TypedDict):
name: str
metadata: Logs
def rename_user(d: UserAudit, name: str) -> None:
d["name"] = name
update_metadata_timestamp(d) # Now OK
In addition to these benefits, by flagging arguments of a function as read-only (by using a ``TypedDict`` like ``Movie`` with read-only items), it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desirable property of a function interface.
This PEP proposes making ``ReadOnly`` valid only in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols.
Specification
=============
A new ``typing.ReadOnly`` type qualifier is added.
``typing.ReadOnly`` type qualifier
----------------------------------
The ``typing.ReadOnly`` type qualifier is used to indicate that an item declared in a ``TypedDict`` definition may not be mutated (added, modified, or removed)::
from typing import ReadOnly
class Band(TypedDict):
name: str
members: ReadOnly[list[str]]
blur: Band = {"name": "blur", "members": []}
blur["name"] = "Blur" # OK: "name" is not read-only
blur["members"] = ["Damon Albarn"] # Type check error: "members" is read-only
blur["members"].append("Damon Albarn") # OK: list is mutable
Alternative functional syntax
-----------------------------
The :pep:`alternative functional syntax <589#alternative-syntax>` for TypedDict also supports the new type qualifier::
Band = TypedDict("Band", {"name": str, "members": ReadOnly[list[str]]})
Interaction with other special types
------------------------------------
``ReadOnly[]`` can be used with ``Required[]``, ``NotRequired[]`` and ``Annotated[]``, in any nesting order:
::
class Movie(TypedDict):
title: ReadOnly[Required[str]] # OK
year: ReadOnly[NotRequired[Annotated[int, ValueRange(-9999, 9999)]]] # OK
::
class Movie(TypedDict):
title: Required[ReadOnly[str]] # OK
year: Annotated[NotRequired[ReadOnly[int]], ValueRange(-9999, 9999)] # OK
This is consistent with the behavior introduced in :pep:`655`.
Inheritance
-----------
Subclasses can redeclare read-only items as non-read-only, allowing them to be mutated::
class NamedDict(TypedDict):
name: ReadOnly[str]
class Album(NamedDict):
name: str
year: int
album: Album = { "name": "Flood", "year": 1990 }
album["year"] = 1973
album["name"] = "Dark Side Of The Moon" # OK: "name" is not read-only in Album
If a read-only item is not redeclared, it remains read-only::
class Album(NamedDict):
year: int
album: Album = { "name": "Flood", "year": 1990 }
album["name"] = "Dark Side Of The Moon" # Type check error: "name" is read-only in Album
Subclasses can narrow value types of read-only items::
class AlbumCollection(TypedDict):
albums: ReadOnly[Collection[Album]]
class RecordShop(AlbumCollection):
name: str
albums: ReadOnly[list[Album]] # OK: "albums" is read-only in AlbumCollection
Subclasses can require items that are read-only but not required in the superclass::
class OptionalName(TypedDict):
name: ReadOnly[NotRequired[str]]
class RequiredName(OptionalName):
name: ReadOnly[Required[str]]
d: RequiredName = {} # Type check error: "name" required
Subclasses can combine these rules::
class OptionalIdent(TypedDict):
ident: ReadOnly[NotRequired[str | int]]
class User(OptionalIdent):
ident: str # Required, mutable, and not an int
Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`.
Type consistency
----------------
*This section updates the type consistency rules introduced in* :pep:`589` *to cover the new feature in this PEP. In particular, any pair of types that do not use the new feature will be consistent under these new rules if (and only if) they were already consistent.*
A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if all of the following are satisfied:
* For each item in ``B``, ``A`` has the corresponding key, unless the item in ``B`` is read-only, not required, and of top value type (``ReadOnly[NotRequired[object]]``).
* For each item in ``B``, if ``A`` has the corresponding key, the corresponding value type in ``A`` is consistent with the value type in ``B``.
* For each non-read-only item in ``B``, its value type is consistent with the corresponding value type in ``A``.
* For each required key in ``B``, the corresponding key is required in ``A``.
* For each non-required key in ``B``, if the item is not read-only in ``B``, the corresponding key is not required in ``A``.
Discussion:
* All non-specified items in a TypedDict implicitly have value type ``ReadOnly[NotRequired[object]]``.
* Read-only items behave covariantly, as they cannot be mutated. This is similar to container types such as ``Sequence``, and different from non-read-only items, which behave invariantly. Example::
class A(TypedDict):
x: ReadOnly[int | None]
class B(TypedDict):
x: int
def f(a: A) -> None:
print(a["x"] or 0)
b: B = {"x": 1}
f(b) # Accepted by type checker
* A TypedDict type ``A`` with no explicit key ``'x'`` is not consistent with a TypedDict type ``B`` with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if the item in ``B`` is read-only, and the value type is of top type (``object``). For example::
class A(TypedDict):
x: int
class B(TypedDict):
x: int
y: ReadOnly[NotRequired[object]]
a: A = { "x": 1 }
b: B = a # Accepted by type checker
Update method
-------------
In addition to existing type checking rules, type checkers should error if a TypedDict with a read-only item is updated with another TypedDict that declares that key::
class A(TypedDict):
x: ReadOnly[int]
y: int
a1: A = { "x": 1, "y": 2 }
a2: A = { "x": 3, "y": 4 }
a1.update(a2) # Type check error: "x" is read-only in A
Unless the declared value is of bottom type (:data:`~typing.Never`)::
class B(TypedDict):
x: NotRequired[typing.Never]
y: ReadOnly[int]
def update_a(a: A, b: B) -> None:
a.update(b) # Accepted by type checker: "x" cannot be set on b
Note: Nothing will ever match the ``Never`` type, so an item annotated with it must be absent.
Keyword argument typing
-----------------------
:pep:`692` introduced ``Unpack`` to annotate ``**kwargs`` with a ``TypedDict``. Marking one or more of the items of a ``TypedDict`` used in this way as read-only will have no effect on the type signature of the method. However, it *will* prevent the item from being modified in the body of the function::
class Args(TypedDict):
key1: int
key2: str
class ReadOnlyArgs(TypedDict):
key1: ReadOnly[int]
key2: ReadOnly[str]
class Function(Protocol):
def __call__(self, **kwargs: Unpack[Args]) -> None: ...
def impl(**kwargs: Unpack[ReadOnlyArgs]) -> None:
kwargs["key1"] = 3 # Type check error: key1 is readonly
fn: Function = impl # Accepted by type checker: function signatures are identical
Runtime behavior
----------------
``TypedDict`` types will gain two new attributes, ``__readonly_keys__`` and ``__mutable_keys__``, which will be frozensets containing all read-only and non-read-only keys, respectively::
class Example(TypedDict):
a: int
b: ReadOnly[int]
c: int
d: ReadOnly[int]
assert Example.__readonly_keys__ == frozenset({'b', 'd'})
assert Example.__mutable_keys__ == frozenset({'a', 'c'})
``typing.get_type_hints`` will strip out any ``ReadOnly`` type qualifiers, unless ``include_extras`` is ``True``::
assert get_type_hints(Example)['b'] == int
assert get_type_hints(Example, include_extras=True)['b'] == ReadOnly[int]
``typing.get_origin`` and ``typing.get_args`` will be updated to recognize ``ReadOnly``::
assert get_origin(ReadOnly[int]) is ReadOnly
assert get_args(ReadOnly[int]) == (int,)
Backwards compatibility
=======================
This PEP adds a new feature to ``TypedDict``, so code that inspects ``TypedDict`` types will have to change to support types using it. This is expected to mainly affect type-checkers.
Security implications
=====================
There are no known security consequences arising from this PEP.
How to teach this
=================
Suggested changes to the :mod:`typing` module documentation, in line with current practice:
* Add this PEP to the others listed.
* Add ``typing.ReadOnly``, linked to TypedDict and this PEP.
* Add the following text to the TypedDict entry:
The ``ReadOnly`` type qualifier indicates that an item declared in a ``TypedDict`` definition may be read but not mutated (added, modified or removed). This is useful when the exact type of the value is not known yet, and so modifying it would break structural subtypes. *insert example*
Reference implementation
========================
`pyright 1.1.333 fully implements this proposal <https://github.com/microsoft/pyright/releases/tag/1.1.333>`_.
Rejected alternatives
=====================
A TypedMapping protocol type
----------------------------
An earlier version of this PEP proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in the current version of this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity.
A higher-order ReadOnly type
----------------------------
A generalized higher-order type could be added that removes mutator methods from its parameter, e.g. ``ReadOnly[MovieRecord]``. For a TypedDict, this would be like adding ``ReadOnly`` to every item, including those declared in superclasses. This would naturally want to be defined for a wider set of types than just TypedDict subclasses, and also raises questions about whether and how it applies to nested types. We decided to keep the scope of this PEP narrower.
Calling the type ``Readonly``
-----------------------------
``Read-only`` is generally hyphenated, and it appears to be common convention to put initial caps onto words separated by a dash when converting to CamelCase. This appears consistent with the definition of CamelCase on Wikipedia: CamelCase uppercases the first letter of each word. That said, Python examples or counter-examples, ideally from the core Python libraries, or better explicit guidance on the convention, would be greatly appreciated.
Reusing the ``Final`` annotation
--------------------------------
The :class:`~typing.Final` annotation prevents an attribute from being modified, like the proposed ``ReadOnly`` qualifier does for ``TypedDict`` items. However, it is also documented as preventing redefinition in subclasses too; from :pep:`591`:
The ``typing.Final`` type qualifier is used to indicate that a variable or attribute should not be reassigned, redefined, or overridden.
This does not fit with the intended use of ``ReadOnly``. Rather than introduce confusion by having ``Final`` behave differently in different contexts, we chose to introduce a new qualifier.
A readonly flag
---------------
Earlier versions of this PEP introduced a boolean flag that would ensure all items in a TypedDict were read-only::
class Movie(TypedDict, readonly=True):
name: str
year: NotRequired[int | None]
movie: Movie = { "name": "A Clockwork Orange" }
movie["year"] = 1971 # Type check error: "year" is read-only
However, this led to confusion when inheritance was introduced::
class A(TypedDict):
key1: int
class B(A, TypedDict, readonly=True):
key2: int
b: B = { "key1": 1, "key2": 2 }
b["key1"] = 4 # Accepted by type checker: "key1" is not read-only
It would be reasonable for someone familiar with ``frozen`` (from :mod:`dataclasses`), on seeing just the definition of B, to assume that the whole type was read-only. On the other hand, it would be reasonable for someone familiar with ``total`` to assume that read-only only applies to the current type.
The original proposal attempted to eliminate this ambiguity by making it both a type check and a runtime error to define ``B`` in this way. This was still a source of surprise to people expecting it to work like ``total``.
Given that no extra types could be expressed with the ``readonly`` flag, it has been removed from the proposal to avoid ambiguity and surprise.
Supporting type-checked removal of read-only qualifier via copy and other methods
---------------------------------------------------------------------------------
An earlier version of this PEP mandated that code like the following be supported by type-checkers::
class A(TypedDict):
x: ReadOnly[int]
class B(TypedDict):
x: ReadOnly[str]
class C(TypedDict):
x: int | str
def copy_and_modify(a: A) -> C:
c: C = copy.copy(a)
if not c['x']:
c['x'] = "N/A"
return c
def merge_and_modify(a: A, b: B) -> C:
c: C = a | b
if not c['x']:
c['x'] = "N/A"
return c
However, there is currently no way to express this in the typeshed, meaning type-checkers would be forced to special-case these functions. There is already a way to code these operations that mypy and pyright do support, though arguably this is less readable::
copied: C = { **a }
merged: C = { **a, **b }
While not as flexible as would be ideal, the current typeshed stubs are sound, and remain so if this PEP is accepted. Updating the typeshed would require new typing features, like a type constructor to express the type resulting from merging two or more dicts, and a type qualifier to indicate a returned value is not shared (so may have type constraints like read-only and invariance of generics loosened in specific ways), plus details of how type-checkers would be expected to interpret these features. These could be valuable additions to the language, but are outside the scope of this PEP.
Given this, we have deferred any update of the typeshed stubs.
Preventing unspecified keys in TypedDicts
-----------------------------------------
Consider the following "type discrimination" code::
class A(TypedDict):
foo: int
class B(TypedDict):
bar: int
def get_field(d: A | B) -> int:
if "foo" in d:
return d["foo"] # !!!
else:
return d["bar"]
This is a common idiom, and other languages like Typescript allow it. Technically, however, this code is unsound: ``B`` does not declare ``foo``, but instances of ``B`` may still have the key present, and the associated value may be of any type::
class C(TypedDict):
foo: str
bar: int
c: C = { "foo": "hi", "bar" 3 }
b: B = c # OK: C is structurally compatible with B
v = get_field(b) # Returns a string at runtime, not an int!
mypy rejects the definition of ``get_field`` on the marked line with the error ``TypedDict "B" has no key "foo"``, which is a rather confusing error message, but is caused by this unsoundness.
One option for correcting this would be to explicitly prevent ``B`` from holding a ``foo``::
class B(TypedDict):
foo: NotRequired[Never]
bar: int
b: B = c # Type check error: key "foo" not allowed in B
However, this requires every possible key that might be used to discriminate on to be explicitly declared in every type, which is not generally feasible. A better option would be to have a way of preventing all unspecified keys from being included in ``B``. mypy supports this using the ``@final`` decorator from :pep:`591`::
@final
class B(TypedDict):
bar: int
The reasoning here is that this prevents ``C`` or any other type from being considered a "subclass" of ``B``, so instances of ``B`` can now be relied on to never hold the key ``foo``, even though it is not explicitly declared to be of bottom type.
With the introduction of read-only items, however, this reasoning would imply type-checkers should ban the following::
@final
class D(TypedDict):
field: ReadOnly[Collection[str]]
@final
class E(TypedDict):
field: list[str]
e: E = { "field": ["value1", "value2"] }
d: D = e # Error?
The conceptual problem here is that TypedDicts are structural types: they cannot really be subclassed. As such, using ``@final`` on them is not well-defined; it is certainly not mentioned in :pep:`591`.
An earlier version of this PEP proposed resolving this by adding a new flag to ``TypedDict`` that would explicitly prevent other keys from being used, but not other kinds of structural compatibility::
class B(TypedDict, other_keys=Never):
bar: int
b: B = c # Type check error: key "foo" not allowed in B
However, during the process of drafting, the situation changed:
* pyright, which previously worked similarly to mypy in this type discrimination case, `changed to allow the original example without error <https://github.com/microsoft/pyright/commit/6a25a7bf0b5cb3721a06d0e0d6245b2ebfbf053b>`_, despite the unsoundness, due to it being a common idiom
* mypy has `an open issue <https://github.com/python/mypy/issues/15697>`_ to follow the lead of pyright and Typescript and permit the idiom as well
* a `draft of PEP-728 <https://github.com/python/peps/pull/3441>`_ was created that is a superset of the ``other_keys`` functionality
As such, there is less urgency to address this issue in this PEP, and it has been deferred to PEP-728.
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
Reference in New Issue View Git Blame Copy Permalink