iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 505: Improve examples and refer to other relevant PEPs (#736)

This commit is contained in:
Steve Dower 2018-07-11 14:11:15 -07:00 committed by GitHub
parent c39a17cf11
commit 8ace3c5e1f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 287 additions and 148 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

435
pep-0505.rst
View File

@ -31,7 +31,8 @@ definitions and other language's implementations of those above. Specifically:
* The "``None`` coalescing`` binary operator ``??`` returns the left hand side
if it evaluates to a value that is not ``None``, or else it evaluates and
returns the right hand side.
returns the right hand side. A coalescing ``??=`` augmented assignment
operator is included.
* The "``None``-aware attribute access" operator ``?.`` evaluates the complete
expression if the left hand side evaluates to a value that is not ``None``
* The "``None``-aware indexing" operator ``?[]`` evaluates the complete
@ -55,6 +56,8 @@ Some argue that this makes ``None`` special. We contend that ``None`` is
already special, and that using it as both the test and the result of these
operators does not change the existing semantics in any way.
See the `Rejected Ideas`_ section for discussion on the rejected approaches.
Grammar changes
---------------
@ -95,12 +98,15 @@ evaluated. For example::
a = None
b = ''
c = 0
a ??= 'value'
b ??= undefined_name
c ??= shutil.rmtree('/') # don't try this at home, kids
assert a == 'value'
assert b == ''
assert c == '0' and any(os.scandir('/'))
Adding new trailers for the other ``None``-aware operators ensures that they
may be used in all valid locations for the existing equivalent operators,
@ -172,15 +178,109 @@ though unlikely to be useful unless combined with a coalescing operation::
Examples
========
This section presents some examples of common ``None`` patterns and explains
the drawbacks.
This section presents some examples of common ``None`` patterns and shows what
conversion to use ``None``-aware operators may look like.
Standard Library
----------------
Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 standard
library discovered up to 678 code snippets that could be replaced with use of
one of the ``None``-aware operators::
$ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py
<snip>
Total None-coalescing `if` blocks: 449
Total [possible] None-coalescing `or`: 120
Total None-coalescing ternaries: 27
Total Safe navigation `and`: 13
Total Safe navigation `if` blocks: 61
Total Safe navigation ternaries: 8
Some of these are shown below as examples before and after converting to use the
new operators.
From ``bisect.py``::
def insort_right(a, x, lo=0, hi=None):
# ...
if hi is None:
hi = len(a)
# ...
After updating to use the ``??=`` augmented assignment statement::
def insort_right(a, x, lo=0, hi=None):
# ...
hi ??= len(a)
# ...
From ``calendar.py``::
encoding = options.encoding
if encoding is None:
encoding = sys.getdefaultencoding()
optdict = dict(encoding=encoding, css=options.css)
After updating to use the ``??`` operator::
optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
css=options.css)
From ``dis.py``::
def _get_const_info(const_index, const_list):
argval = const_index
if const_list is not None:
argval = const_list[const_index]
return argval, repr(argval)
After updating to use the ``?[]`` and ``??`` operators::
def _get_const_info(const_index, const_list):
argval = const_list?[const_index] ?? const_index
return argval, repr(argval)
From ``inspect.py``::
for base in object.__bases__:
for name in getattr(base, "__abstractmethods__", ()):
value = getattr(object, name, None)
if getattr(value, "__isabstractmethod__", False):
return True
After updating to use the ``?.`` operator (and deliberately not converting to
use ``any()``)::
for base in object.__bases__:
for name in base?.__abstractmethods__ ?? ():
if object?.name?.__isabstractmethod__:
return True
From ``os.py``::
if entry.is_dir():
dirs.append(name)
if entries is not None:
entries.append(entry)
else:
nondirs.append(name)
After updating to use the ``?.`` operator::
if entry.is_dir():
dirs.append(name)
entries?.append(entry)
else:
nondirs.append(name)
jsonify
-------
This first example is from a Python web crawler that uses the popular Flask
framework as a front-end. This function retrieves information about a web site
from a SQL database and formats it as JSON to send to an HTTP client::
This example is from a Python web crawler that uses the Flask framework as its
front-end. This function retrieves information about a web site from a SQL
database and formats it as JSON to send to an HTTP client::
class SiteView(FlaskView):
@route('/site/<id_>', methods=['GET'])
@ -200,12 +300,12 @@ database, and they are also allowed to be ``null`` in the JSON response. JSON
does not have a native way to represent a ``datetime``, so the server's contract
states that any non-``null`` date is represented as an ISO-8601 string.
However, without knowing the exact semantics of the ``first_seen`` and
``last_seen`` attributes, it is impossible to know whether the attribute can
be safely or performantly accessed multiple times.
Without knowing the exact semantics of the ``first_seen`` and ``last_seen``
attributes, it is impossible to know whether the attribute can be safely or
performantly accessed multiple times.
One way to fix this code is to replace each ternary with explicit value
assignment and a full ``if/else`` block::
One way to fix this code is to replace each conditional expression with an
explicit value assignment and a full ``if``/``else`` block::
class SiteView(FlaskView):
@route('/site/<id_>', methods=['GET'])
@ -233,9 +333,9 @@ assignment and a full ``if/else`` block::
)
This adds ten lines of code and four new code paths to the function,
dramatically increasing the apparently complexity. Rewriting using
``None``-aware attribute operator on the other hand, results in shorter
code::
dramatically increasing the apparent complexity. Rewriting using the
``None``-aware attribute operator results in shorter code with more clear
intent::
class SiteView(FlaskView):
@route('/site/<id_>', methods=['GET'])
@ -290,7 +390,7 @@ ab/upload.py>`_::
self.content_type = content_type
This example contains several good examples of needing to provide default
values. Rewriting to use a conditional expression reduces the overall lines of
values. Rewriting to use conditional expressions reduces the overall lines of
code, but does not necessarily improve readability::
class BaseUploadObject(object):
@ -339,117 +439,148 @@ Rewriting using the ``None`` coalescing operator::
self.filename = filename ?? os.path.split(path)[1]
self.content_type = content_type ?? self.find_content_type(self.filename)
This syntax has an intuitive ordering of the operands. In
``find_content_type``, for example, the preferred value ``ctype`` appears before
the fallback value. The terseness of the syntax also makes for fewer lines of
code and less code to visually parse.
This syntax has an intuitive ordering of the operands. In ``find_content_type``,
for example, the preferred value ``ctype`` appears before the fallback value.
The terseness of the syntax also makes for fewer lines of code and less code to
visually parse, and reading from left-to-right and top-to-bottom more accurately
follows the execution flow.
Alternatives
============
Python does not have any existing ``None``-aware operators, but it does have
operators that can be used for a similar purpose. This section describes why
these alternatives are undesirable for some common ``None`` patterns.
``or`` Operator
---------------
Similar behavior can be achieved with the ``or`` operator, but ``or`` checks
whether its left operand is false-y, not specifically ``None``. This can lead
to unexpected behavior when the value is zero, an empty string, or an empty
collection::
>>> def f(s=None):
... s = s or []
... s.append(123)
...
>>> my_list = []
>>> f(my_list)
>>> my_list
[]
# Expected: [123]
Rewritten using the ``None`` coalescing operator, the function could read::
def f(s=None):
s = s ?? []
s.append(123)
Or using the ``None``-aware attribute operator::
def f(s=None):
s?.append(123)
(Rewriting using a conditional expression is covered in a later section.)
``getattr`` Builtin
-------------------
Using the ``getattr`` builtin with a default value is often a suitable approach
for getting an attribute from a target that may be ``None``::
client = maybe_get_client()
name = getattr(client, 'name', None)
However, the semantics of this call are different from the proposed operators.
Using ``getattr`` will suppress ``AttributeError`` for misspelled or missing
attributes, even when ``client`` has a value.
Spelled correctly for these semantics, this example would read::
client = maybe_get_client()
if client is not None:
name = client.name
else:
name = None
Written using the ``None``-aware attribute operator::
client = maybe_get_client()
name = client?.name
Ternary Operator
----------------
Another common way to initialize default values is to use the ternary operator.
Here is an excerpt from the popular `Requests package
<https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
a8149f5/requests/models.py#L212>`_::
data = [] if data is None else data
files = [] if files is None else files
headers = {} if headers is None else headers
params = {} if params is None else params
hooks = {} if hooks is None else hooks
This particular formulation has the undesirable effect of putting the operands
in an unintuitive order: the brain thinks, "use ``data`` if possible and use
``[]`` as a fallback," but the code puts the fallback *before* the preferred
value.
The author of this package could have written it like this instead::
data = data if data is not None else []
files = files if files is not None else []
headers = headers if headers is not None else {}
params = params if params is not None else {}
hooks = hooks if hooks is not None else {}
This ordering of the operands is more intuitive, but it requires 4 extra
characters (for "not "). It also highlights the repetition of identifiers:
``data if data``, ``files if files``, etc.
When written using the ``None`` coalescing operator, the sample reads::
data = data ?? []
files = files ?? []
headers = headers ?? {}
params = params ?? {}
hooks = hooks ?? {}
Rejected Ideas
==============
The first three ideas in this section are oft-proposed alternatives to treating
``None`` as special. For further background on why these are rejected, see their
treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and
`PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated
discussions.
No-Value Protocol
-----------------
The operators could be generalised to user-defined types by defining a protocol
to indicate when a value represents "no value". Such a protocol may be a dunder
method ``__has_value__(self)` that returns ``True`` if the value should be
treated as having a value, and ``False`` if the value should be treated as no
value.
With this generalization, ``object`` would implement a dunder method equivalent
to this::
def __has_value__(self):
return True
``NoneType`` would implement a dunder method equivalent to this::
def __has_value__(self):
return False
In the specification section, all uses of ``x is None`` would be replaced with
``not x.__has_value__()``.
This generalization would allow for domain-specific "no-value" objects to be
coalesced just like ``None``. For example the ``pyasn1`` package has a type
called ``Null`` that represents an ASN.1 ``null``::
>>> from pyasn1.type import univ
>>> univ.Null() ?? univ.Integer(123)
Integer(123)
Similarly, values such as ``math.nan`` and ``NotImplemented`` could be treated
as representing no value.
However, the "no-value" nature of these values is domain-specific, which means
they *should* be treated as a value by the language. For example,
``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting
``math.nan?.imag`` to return ``math.nan`` would be incorrect.
As ``None`` is already defined by the language as being the value that
represents "no value", and the current specification would not preclude
switching to a protocol in the future (though changes to built-in objects would
not be compatible), this idea is rejected for now.
Boolean-aware operators
-----------------------
This suggestion is fundamentally the same as adding a no-value protocol, and so
the discussion above also applies.
Similar behavior to the ``??`` operator can be achieved with an ``or``
expression, however ``or`` checks whether its left operand is false-y and not
specifically ``None``. This approach is attractive, as it requires fewer changes
to the language, but ultimately does not solve the underlying problem correctly.
Assuming the check is for truthiness rather than ``None``, there is no longer a
need for the ``??`` operator. However, applying this check to the ``?.`` and
``?[]`` operators prevents perfectly valid operations applying
Consider the following example, where ``get_log_list()`` may return either a
list containing current log messages (potentially empty), or ``None`` if logging
is not enabled::
lst = get_log_list()
lst?.append('A log message')
If ``?.`` is checking for true values rather than specifically ``None`` and the
log has not been initialized with any items, no item will ever be appended. This
violates the obvious intent of the code, which is to append an item. The
``append`` method is available on an empty list, as are all other list methods,
and there is no reason to assume that these members should not be used because
the list is presently empty.
Further, there is no sensible result to use in place of the expression. A
normal ``lst.append`` returns ``None``, but under this idea ``lst?.append`` may
result in either ``[]`` or ``None``, depending on the value of ``lst``. As with
the examples in the previous section, this makes no sense.
As checking for truthiness rather than ``None`` results in apparently valid
expressions no longer executing as intended, this idea is rejected.
Exception-aware operators
-------------------------
Arguably, the reason to short-circuit an expression when ``None`` is encountered
is to avoid the ``AttributeError`` or ``TypeError`` that would be raised under
normal circumstances. As an alternative to testing for ``None``, the ``?.`` and
``?[]`` operators could instead handle ``AttributeError`` and ``TypeError``
raised by the operation and skip the remainder of the expression.
This produces a transformation for ``a?.b.c?.d.e`` similar to this::
_v = a
try:
_v = _v.b
except AttributeError:
pass
else:
_v = _v.c
try:
_v = _v.d
except AttributeError:
pass
else:
_v = _v.e
One open question is which value should be returned as the expression when an
exception is handled. The above example simply leaves the partial result, but
this is not helpful for replacing with a default value. An alternative would be
to force the result to ``None``, which then raises the question as to why
``None`` is special enough to be the result but not special enough to be the
test.
Secondly, this approach masks errors within code executed implicitly as part of
the expression. For ``?.``, any ``AttributeError`` within a property or
``__getattr__`` implementation would be hidden, and similarly for ``?[]`` and
``__getitem__`` implementations.
Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed.
Existing conventions for handling these kinds of errors in the form of the
``getattr`` builtin and the ``.get(key, default)`` method pattern established by
``dict`` show that it is already possible to explicitly use this behaviour.
As this approach would hide errors in code, it is rejected.
``None``-aware Function Call
----------------------------
@ -457,7 +588,7 @@ The ``None``-aware syntax applies to attribute and index access, so it seems
natural to ask if it should also apply to function invocation syntax. It might
be written as ``foo?()``, where ``foo`` is only called if it is not None.
This has been rejected on the basis of the proposed operators being intended
This has been deferred on the basis of the proposed operators being intended
to aid traversal of partially populated hierarchical data structures, *not*
for traversal of arbitrary class hierarchies. This is reflected in the fact
that none of the other mainstream languages that already offer this syntax
@ -518,8 +649,8 @@ This degree of generalization is not useful. The operators actually proposed
herein are intentionally limited to a few operators that are expected to make it
easier to write common code patterns.
Haskell-style ``Maybe``
-----------------------
Built-in ``maybe``
------------------
Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that
encapsulates the idea of an optional value without relying on any special
@ -548,39 +679,45 @@ not nearly as powerful as support built into the language.
The idea of adding a builtin ``maybe`` type to enable this scenario is rejected.
No-Value Protocol
-----------------
Just use a conditional expression
---------------------------------
These operators could be generalised to user-defined types by using a protocol
to indicate when a value represents no value. Such a protocol may be a dunder
method ``__has_value__(self)` that returns ``True`` if the value should be
treated as having a value, and ``False`` if the ``None``-aware operators should
Another common way to initialize default values is to use the ternary operator.
Here is an excerpt from the popular `Requests package
<https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
a8149f5/requests/models.py#L212>`_::
With this generalization, ``object`` would implement a dunder method equivalent
to this::
data = [] if data is None else data
files = [] if files is None else files
headers = {} if headers is None else headers
params = {} if params is None else params
hooks = {} if hooks is None else hooks
def __has_value__(self):
return True
This particular formulation has the undesirable effect of putting the operands
in an unintuitive order: the brain thinks, "use ``data`` if possible and use
``[]`` as a fallback," but the code puts the fallback *before* the preferred
value.
``NoneType`` would implement a dunder method equivalent to this::
The author of this package could have written it like this instead::
def __has_value__(self):
return False
data = data if data is not None else []
files = files if files is not None else []
headers = headers if headers is not None else {}
params = params if params is not None else {}
hooks = hooks if hooks is not None else {}
In the specification section, all uses of ``x is None`` would be replaces with
``not x.__has_value__()``.
This ordering of the operands is more intuitive, but it requires 4 extra
characters (for "not "). It also highlights the repetition of identifiers:
``data if data``, ``files if files``, etc.
This generalization would allow for domain-specific ``null`` objects to be
coalesced just like ``None``. For example the ``pyasn1`` package has a type
called ``Null`` that represents an ASN.1 ``null``::
When written using the ``None`` coalescing operator, the sample reads::
>>> from pyasn1.type import univ
>>> univ.Null() ?? univ.Integer(123)
Integer(123)
data = data ?? []
files = files ?? []
headers = headers ?? {}
params = params ?? {}
hooks = hooks ?? {}
However, as ``None`` is already defined as being the value that represents
"no value", and the current specification would not preclude switching to a
protocol in the future, this idea is rejected for now.
References
==========
@ -591,6 +728,8 @@ References
.. [2] A Tour of the Dart Language: Operators
(https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)
.. [3] Associated scripts
(https://github.com/python/peps/tree/master/pep-0505/)
Copyright
=========