python-peps/pep-0604.rst

PEP: 604
Title: Complementary syntax for ``Union[]`` and ``Optional[]``
Author: Philippe PRADOS <python@prados.fr>
Sponsor: Chris Angelico <rosuav@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Aug-2019
Python-Version: 3.9


Introduction
============

This PEP describes an extension to Python language, which aims to add a complementary
syntax to write ``Union[X,Y]`` and ``Optional[X]`` easier.


Motivation
==========

The PEP484 [1]_ and PEP526 [2]_ propose a generic syntax to add typing to variables,
parameters and function returns.
The PEP585 [3]_ proposes to `expose parameters to generics at runtime <https://www.python.org/dev/peps/pep-0585/#id7>`_.
MyPy [4]_ accepts a syntax which looks like something like this:

::

    annotation: name_type
    name_type: NAME (args)?
    args: '[' paramslist ']'
    paramslist: annotation (',' annotation)* [',']

- To describe a disjunction, the user must use ``Union[X,Y]``.

- To describe an optional value, the user must use ``Optional[X]``.

The verbosity of this syntax does not help the adoption.

Proposal
========

Inspired by Scala language [5]_, this proposal adds two operators in the root ``type``:

Strong proposition
------------------
Add operator ``__or__()`` in the root ``type``.

With this new operator, it is possible to write ``int | str`` in place of ``Union[int,str]``.

This proposition uses the standard meaning of the ``|`` operator.

Optional proposition 1
----------------------
Add operator ``__invert__()`` in the root ``type``.

With this new operator, it is possible to write ``~int`` in place of ``Optional[int]``.

This proposition uses this operator because it is present in the language and it's conform to the
`usage of tilde <https://www.thecut.com/article/why-the-internet-tilde-is-our-most-perfect-tool-for-snark.html>`_

So, the new syntax for annotations will be:

::

    annotation: ( name_type | or_type | invert_type )
    or_type: name_type '|' annotation
    invert_type: '~' annotation
    name_type: NAME (args)?
    args: '[' paramslist ']'
    paramslist: annotation (',' annotation)* [',']

Optional proposition 2
----------------------
Then, it is possible to extend ``isinstance()`` and ``issubclass()``
to accept this new syntax:

::

  isinstance(int, int | str)

Examples
========

Here are some examples of what we can do with this feature.

::

    # in place of
    # def f(list: List[Union[int, str]], param: Optional[int]) -> Union[float, str]
    def f(list: List[int | str], param: ~int) -> float | str:
        pass

    f([1,"abc"],None)

    assert str | int == Union[str,int]
    assert str | int | float == Union[str, int, float]
    assert ~str == Optional[str]

    assert isinstance("", int | str)
    assert issubclass(int, int | str)

Once the Python language is extended, MyPy [3]_ must be updated to accept this new syntax.

Incompatible changes
====================
In some situations, some exceptions will not be raised as expected.


Dissenting Opinion
==================

- `Discussion in python-ideas <https://mail.python.org/archives/list/python-ideas@python.org/thread/FCTXGDT2NNKRJQ6CDEPWUXHVG2AAQZZY/>`_

1. Add a new operator for ``Union[type1|type2]``?
--------------------------------------------------

- CONS: This is not a new proposal. If I recall correctly, it was proposed way back at the very beginning of the
  type-hinting discussion, and there has been at least one closed feature request for it:
  `Issue 387 <https://github.com/python/typing/issues/387>`_

  - It is maybe too late to change this, many people are already get used to current notation.

    - *This PEP propose to add a new notation, not to replace the notation*

  - This syntax is difficult to google, if someone encounters it in code
  - It is still not possible to use ``|`` for unions because of built-in types. (This would require a corresponding
    slot in type which is a non-starter)

    - *The proposed implementation do it*

  - There are currently no volunteer to implement this in mypy

    - *An implementation is proposed now (One patch for CPython and one for MyPy).*

  - "but as @ilevkivskyi pointed out, that is not an option (at least until Python 4)."

    - *Is it time now ?*

- PRO: It's similar of Scala
- PRO: Seems like ``foo | None`` is just as readable
- PRO: Which means you couldn't use this feature in Python 3.7, much less 2.7. I'm not sure it maintaining backward
  compatibility in typing and in mypy is still as important today as it was 5 years ago, but I'm pretty sure it hasn't
  been abandoned entirely.
- CONS: add operator introducing a dependency to typing in builtins
- CONS:  supporting this would likely break compatibility with existing code that overloads ``|`` for class objects
  using a metaclass. We could perhaps work around this by making ``|`` inside an annotation context different from
  the regular ``|`` operator.

  - *A work around is to use* ``Union[type1,type2]`` *in this case*

- CONS: You need ``__ror__`` as well as ``__or__``
  - *No, in this situation, Python auto invoke ``__or__`` in case of ``__ror__``.*
- CONS: as breaking the backport (in that ``typing.py`` can easily be backported but core ``types`` can't)

  - There are several things in the typing syntax that require a certain minimum version. E.g. type annotations require
    Python 3 (whereas type comments work in Python 2 too), type annotations on variables (PEP 526) require 3.6+,
    ``from __future__ import annotations`` (PEP 563) requires 3.7+.

- PRO: I mean that at run-time, ``int|str`` might return a very simple object in 3.9, rather than everything that
  you'd need to grab from importing ``typing``. Wondering if doing so would close off the possibility of, in 3.12 or
  something, making it a more directly usable "type union" that has other value.
- CONS: if Python itself doesn't have to be changed, we'd still need to implement it in mypy, Pyre, PyCharm,
  Pytype, and who knows what else.

  - *A proposed patch of mypy is just 20 lines of codes*

- If yes,

Change only the PEP484 (Type hints) to accept the syntax ``type1 | type2`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- PRO: The PEP563 [6]_ (Postponed Evaluation of Annotations) is enough to accept this proposition
- CONS: The Resolving type hints at runtime says: “For code which uses annotations for other purposes, a
  regular ``eval(ann, globals, locals)`` call is enough to resolve the annotation.". Without add a new
  operator ``__or__`` in type ``type``, it's not possible to resolve type hints at runtime.

::

    >>> from __future__ import annotations
    >>> def foo() -> int | str: pass
    ...
    >>> eval(foo.__annotations__['return'])
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
    TypeError: unsupported operand type(s) for |: 'type' and 'type'

  - CONS: Without operator, it's not possible to write

::

    >>> u = int | str
    >>> u
    typing.Union[int, str]

Use ``(int, str)`` in place of ``Union[int,str]`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- PRO: This doesn't have compatibility issues and it's similar to ``isinstance(foo, (int, str))``
- PRO: Either better than breaking backward compatibility by adding new operator methods to the type ``type``.
- CONS: In most languages with similar-ish type syntax, ``(int, str)`` means ``Tuple[int, str]``,
  not ``Union[int, str]``.

Use ``{int, str}`` in place of ``Union[int,str]`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- PRO: big advantage of ``{int, str}`` over ``int|str``. It doesn't require adding anything to ``type``,
  and we don't need to introduce a new lightweight builtin union type.

2. Add a new operator for ``Optional[type]`` ?
----------------------------------------------

- CONS: ``foo | None`` is short and readable
- CONS: ``foo | None`` it's 3 fewer characters than ``Optional[foo]``, or 30 fewer if you include the full
  removal of ``from typing import Optional``. the additional gain of ``~foo`` is only 6 characters.
- PRO: help the readability, with a lot of parameters:

::

    def f(source: str | None, destination: str | None, param: int | None):...
    def f(source: ~str, destination: ~str, param: ~int):...

- PRO: I'm currently working on annotating a very large codebase, and ``Optional[T]`` is so frequent that I
  think ``T | None`` would not be enough of an improvement.
- PRO: Adding a default ``__or__`` overload to ``type`` seems a reasonable price to pay in 3.9, and
  ditto for ``__invert__``. Type checkers can support this in older Python versions using PEP 563 or in type
  comments or in "forward references" (types hidden in string literals).
- CONS: The ``~`` is easy to be missed (at least by human readers) and the meaning not obvious.
- PRO: Also, Python's typing system is a lot easier to grasp if you're familiar with an established modern-typed
  language (Swift, Scala, Haskell, F#, etc.), and they also use ``Optional[T]`` (or ``optional<T>`` or ``Maybe t``
  or some other spelling of the same idea) all over be place—so often that many of them have added shortcuts
  like ``T?`` to make it easier to write and less intrusive to read.

- if yes,

Add operator ``__revert__`` in type type to use syntax like ``~int`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- CONS: ``~`` is not automatically readable

  - *like ``:`` to separate variable and typing.*

- CONS: ``~`` means complement, which is a completely different thing from ``|None``. ``~int`` seems like it
  would actually harm comprehension instead of helping.
- PRO: the slight abuse of ``~int`` meaning "maybe int" is pretty plausible (consider how "approximately equal"
  is written mathematically).
- PRO: `Possibly relevant for tilde <https://www.thecut.com/article/why-the-internet-tilde-is-our-most-perfect-tool-for-snark.html>`_
- CONS: With ``~`` there probably won't be a confusion in that sense, but someone reading it for the first time will
  definitely need to look it up (which is fine i.m.o.).

  - *Like the first time someone reading the annotation*

::

    def f(a=int):...
    def f(a:int):...

Add operator ``__add__`` in type type to use syntax like ``+int`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- PRO: ``+foo`` definitely seems to say "foo, plus something else" to me much more than ``~foo``.
- CONS: ``+foo`` is less intuitive than ``~foo`` for ``Optional``

Like Kotlin, add a new ``?`` operator to use syntax like ``int?`` or ``?int`` ?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- CONS: It's not compatible with IPython and Jupyter Lab ``?smth`` displays help for symbol ``smth``
- CONS: With default arguments, ``?=`` looks... not great

::

    def f(source: str?=def_src, destination: str?=MISSING, param: int?=1): ...

3. Extend ``isinstance()`` and ``issubclass()`` to accept ``Union`` ?
---------------------------------------------------------------------

::

    isinstance(x, str | int) ==> "is x an instance of str or int"

- PRO: if they were permitted, then instance checks could use an extremely clean-looking notation for "any of these":
- PRO: The implementation can use the tuple present in ``Union`` parameter, without create a new instance.
- CONS: Why not accept this syntax in ``except`` ?

Reference Implementation
========================

A proposed implementation for `cpython is here
<https://github.com/pprados/cpython/tree/updage_isinstance>`_.
A proposed implementation for `mypy is here
<https://github.com/pprados/mypy/tree/add_INVERT_to_types>`_.


References
==========

.. [1] PEP484,
   https://www.python.org/dev/peps/pep-0484/
.. [2] PEP526,
   https://www.python.org/dev/peps/pep-0526/
.. [3] PEP585,
   https://www.python.org/dev/peps/pep-0585/
.. [4] MyPy
   http://mypy-lang.org/
.. [5] Scala Union Types
   https://dotty.epfl.ch/docs/reference/new-types/union-types.html
.. [6] PEP563,
   https://www.python.org/dev/peps/pep-0563/

Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: