1107 lines
42 KiB
Plaintext
1107 lines
42 KiB
Plaintext
PEP: 505
|
||
Title: None-aware operators
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Mark E. Haase <mehaase@gmail.com>
|
||
Status: Deferred
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 18-Sep-2015
|
||
Python-Version: 3.8
|
||
|
||
PEP Deferral
|
||
============
|
||
|
||
Further consideration of this PEP has been deferred until Python 3.8 at the
|
||
earliest.
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
Several modern programming languages have so-called "``null``-coalescing" or
|
||
"``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, and PHP
|
||
(starting in version 7). These operators provide syntactic sugar for common
|
||
patterns involving null references.
|
||
|
||
* The "``null``-coalescing" operator is a binary operator that returns its left
|
||
operand if it is not ``null``. Otherwise it returns its right operand.
|
||
* The "``null``-aware member access" operator accesses an instance member only
|
||
if that instance is non-``null``. Otherwise it returns ``null``. (This is also
|
||
called a "safe navigation" operator.)
|
||
* The "``null``-aware index access" operator accesses an element of a collection
|
||
only if that collection is non-``null``. Otherwise it returns ``null``. (This
|
||
is another type of "safe navigation" operator.)
|
||
|
||
The purpose of this PEP is to explore the possibility of implementing similar
|
||
operators in Python. It provides some background material and then offers
|
||
several competing alternatives for implementation.
|
||
|
||
The initial reaction to this idea is majority negative. Even if ultimately
|
||
rejected, this PEP still serves a purpose: to fully document the reasons why
|
||
Python should not add this behavior, so that it can be pointed to in the future
|
||
when the question inevitably arises again. (This is the null alternative, so to
|
||
speak!)
|
||
|
||
This proposal advances multiple alternatives, and it should be considered
|
||
severable. It may be accepted in whole or in part. For example, the safe
|
||
navigation operators might be rejected even if the ``null``-coalescing operator
|
||
is approved, or vice-versa.
|
||
|
||
Of course, Python does not have ``null``; it has ``None``, which is conceptually
|
||
distinct. Although this PEP is inspired by "``null``-aware" operators in other
|
||
languages, it uses the term "``None``-aware" operators to describe some
|
||
hypothetical Python implementations.
|
||
|
||
|
||
Background
|
||
==========
|
||
|
||
Specialness of ``None``
|
||
-----------------------
|
||
|
||
The Python language does not currently define any special behavior for ``None``.
|
||
This PEP suggests making ``None`` a special case. This loss of generality is a
|
||
noticeable drawback of the proposal. A generalization of ``None``-aware
|
||
operators is set forth later in this document in order to avoid this
|
||
specialization.
|
||
|
||
|
||
Utility of ``None``
|
||
-------------------
|
||
|
||
One common criticism of adding special syntax for ``None`` is that ``None``
|
||
shouldn't be used in the first place: it's a code smell. A related criticism is
|
||
that ``None``-aware operators are used to silence errors (such as the novice
|
||
misunderstanding of an implicit ``return None``) akin to `PHP's @ operator
|
||
<http://php.net/manual/en/language.operators.errorcontrol.php>`_. Therefore,
|
||
the utility of ``None`` must be debated before discussing whether to add new
|
||
behavior around it.
|
||
|
||
Python does not have any concept of ``null``. Every Python identifier must
|
||
refer to an instance, so there cannot be any ``null`` references. Python does
|
||
have a special instance called ``None`` that can be used to represent missing
|
||
values, but ``None`` is conceptually distinct from ``null``.
|
||
|
||
The most frequent use of ``None`` in Python is to provide a default value for
|
||
optional arguments when some other default object is unwieldy. For example:
|
||
``def get(url, proxy=None):``. In this case, ``proxy`` is an optional
|
||
argument. If ``proxy`` is ``None``, then the request should be sent directly to
|
||
the server; otherwise, the request should be routed through the specified proxy
|
||
server. This use of ``None`` is preferred here to some other sentinel value or
|
||
the Null Object Pattern. [3]_
|
||
|
||
Examples of this form abound. Consider ``types.py`` in the standard library::
|
||
|
||
def prepare_class(name, bases=(), kwds=None):
|
||
if kwds is None:
|
||
kwds = {}
|
||
else:
|
||
kwds = dict(kwds)
|
||
...
|
||
|
||
Another frequent use of ``None`` is interfacing with external systems. Many of
|
||
those other systems have a concept of ``null``. Therefore, Python code must have
|
||
a way of representing ``null``, and typically it is represented by ``None``. For
|
||
example, databases can have ``null`` values, and most Python database drivers
|
||
will convert ``null`` to ``None`` when retrieving data from a database, and will
|
||
convert from ``None`` back to ``null`` when sending data to a database.
|
||
|
||
This convention of interchanging ``null`` and ``None`` is widespread in Python.
|
||
It is canonized in the Python DBAPI (PEP-249). [4]_ The ``json`` module in the
|
||
standard library and the third party PyYAML package both use ``None`` to
|
||
represent their respective languages' ``null``.
|
||
|
||
The C language ``null`` often bleeds into Python, too, particularly for thin
|
||
wrappers around C libraries. For example, in ``pyopenssl``, the ``X509`` class
|
||
has a ``get_notBefore()`` `method <https://github.com/pyca/pyopenssl/blob/325787
|
||
7f8846e4357b495fa6c9344d01b11cf16d/OpenSSL/crypto.py#L1219>`_ that returns
|
||
either a timestamp or ``None``. This function is a thin wrapper around an
|
||
OpenSSL function with the return type ``ASN1_TIME *``. Because this C pointer
|
||
may be ``null``, the Python wrapper must be able to represent ``null``, and
|
||
``None`` is the chosen representation.
|
||
|
||
The representation of ``null`` is particularly noticeable when Python code is
|
||
marshalling data between two systems. For example, consider a Python server that
|
||
fetches data from a database and converts it to JSON for consumption by another
|
||
process. In this case, it's often desirable that ``null`` in the database can be
|
||
easily translated to ``null`` in JSON. If ``None`` is not used for this purpose,
|
||
then each package will have to define its own representation of ``null``, and
|
||
converting between these representations adds unnecessary complexity to the
|
||
Python glue code.
|
||
|
||
Therefore, the preference for avoiding ``None`` is nothing more than a
|
||
preference. ``None`` has legitimate uses, particularly in specific types of
|
||
software. Any hypothetical ``None``-aware operators should be construed as
|
||
syntactic sugar for simplifying common patterns involving ``None``, and *should
|
||
not be construed* as error handling behavior.
|
||
|
||
|
||
Behavior In Other Languages
|
||
---------------------------
|
||
|
||
Given that ``null``-aware operators exist in other modern languages, it may be
|
||
helpful to quickly understand how they work in those languages::
|
||
|
||
/* Null-coalescing. */
|
||
|
||
String s1 = null;
|
||
String s2 = "hello";
|
||
String s3 = s1 ?? s2;
|
||
Console.WriteLine("s3 is: " + s3);
|
||
// s3 is: hello
|
||
|
||
/* Null-aware member access, a.k.a. safe navigation. */
|
||
|
||
Console.WriteLine("s1.Length is: " + s1?.Length);
|
||
Console.WriteLine("s2.Length is: " + s2?.Length);
|
||
// s1.Length is:
|
||
// s2.Length is: 5
|
||
|
||
/* Null-aware index access, a.k.a. safe navigation. */
|
||
|
||
Dictionary<string,string> d1 = null;
|
||
Dictionary<string,string> d2 = new Dictionary<string, string>
|
||
{
|
||
{ "foo", "bar" },
|
||
{ "baz", "bat" }
|
||
};
|
||
|
||
Console.WriteLine("d1[\"foo\"] is: " + d1?["foo"]);
|
||
Console.WriteLine("d2[\"foo\"] is: " + d2?["foo"]);
|
||
// d1["foo"] is:
|
||
// d2["foo"] is: bar
|
||
|
||
/* Short Circuiting */
|
||
|
||
Console.WriteLine("s1 trim/upper is: " + s1?.Trim().Length);
|
||
Console.WriteLine("s2 trim/upper is: " + s2?.Trim().Length);
|
||
// s1 trimmed length is:
|
||
// s2 trimmed length is: 5
|
||
|
||
String s4 = s1 ?? s2 ?? DoError();
|
||
Console.WriteLine("s4 is: " + s4)
|
||
// s4 is: hello
|
||
|
||
A `working example <https://dotnetfiddle.net/SxQNG8>`_ can be viewed online.
|
||
|
||
Of utmost importance, notice the short circuiting behavior. The short circuiting
|
||
of ``??`` is similar to short circuiting of other boolean operators such as
|
||
``||`` or ``&&`` and should not be surprising. Helpfully, `?.` is *also* short
|
||
circuiting: ``s1?.Trim()`` evaluates to null, but ``s1?.Trim().Length`` does not
|
||
attempt to dereference the ``null`` pointer.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Existing Alternatives
|
||
---------------------
|
||
|
||
Python does not have any specific ``None``-aware operators, but it does have
|
||
operators that can be used for a similar purpose. This section describes why
|
||
these alternatives may be undesirable for some common ``None`` patterns.
|
||
|
||
|
||
``or`` Operator
|
||
~~~~~~~~~~~~~~~
|
||
|
||
Similar behavior can be achieved with the ``or`` operator, but ``or`` checks
|
||
whether its left operand is false-y, not specifically ``None``. This can lead
|
||
to surprising behavior. Consider the scenario of computing the price of some
|
||
products a customer has in his/her shopping cart::
|
||
|
||
>>> price = 100
|
||
>>> default_quantity = 1
|
||
# If user didn't specify a quantity, then assume the default.
|
||
>>> requested_quantity = None
|
||
>>> (requested_quantity or default_quantity) * price
|
||
100
|
||
# The user added 5 items to the cart.
|
||
>>> requested_quantity = 5
|
||
>>> (requested_quantity or default_quantity) * price
|
||
500
|
||
# User removed 5 items from cart.
|
||
>>> requested_quantity = 0
|
||
>>> (requested_quantity or default_quantity) * price # oops!
|
||
100
|
||
|
||
An experienced Python developer should know how ``or`` works and be capable of
|
||
avoiding bugs like this. However, getting in the habit of using ``or`` for this
|
||
purpose still might cause an experienced developer to occasionally make this
|
||
mistake, especially when refactoring existing code and not carefully paying
|
||
attention to the possible values of the left-hand operand.
|
||
|
||
For inexperienced developers, the problem is worse. The top Google hit for
|
||
"python null coalesce" is a `StackOverflow page
|
||
<http://stackoverflow.com/questions/4978738/is-there-a-python-equivalent-of-
|
||
the-c-sharp-null-coalescing-operator>`_, and the top answer says to use ``or``.
|
||
The top answer goes on to explain the caveats of using ``or`` like this, but how
|
||
many beginning developers go on to read all those caveats? The accepted answer
|
||
on `a more recent question <http://stackoverflow.com/questions/13710631/is-
|
||
there-shorthand-for-returning-a -default-value-if-none-in-python>`_ says to use
|
||
``or`` without any caveats at all. These two questions have a combined 26,000
|
||
views!
|
||
|
||
The common usage of ``or`` for the purpose of providing default values is
|
||
undeniable, and yet it is also booby-trapped for unsuspecting newcomers. This
|
||
suggests that a safe operator for providing default values would have positive
|
||
utility. While some critics claim that ``None``-aware operators will be abused
|
||
for error handling, they are no more prone to abuse than ``or`` is.
|
||
|
||
|
||
Ternary Operator
|
||
~~~~~~~~~~~~~~~~
|
||
|
||
Another common way to initialize default values is to use the ternary operator.
|
||
Here is an excerpt from the popular `Requests package
|
||
<https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
|
||
a8149f5/requests/models.py#L212>`_::
|
||
|
||
data = [] if data is None else data
|
||
files = [] if files is None else files
|
||
headers = {} if headers is None else headers
|
||
params = {} if params is None else params
|
||
hooks = {} if hooks is None else hooks
|
||
|
||
This particular formulation has the undesirable effect of putting the operands
|
||
in an unintuitive order: the brain thinks, "use ``data`` if possible and use
|
||
``[]`` as a fallback," but the code puts the fallback *before* the preferred
|
||
value.
|
||
|
||
The author of this package could have written it like this instead::
|
||
|
||
data = data if data is not None else []
|
||
files = files if files is not None else []
|
||
headers = headers if headers is not None else {}
|
||
params = params if params is not None else {}
|
||
hooks = hooks if hooks is not None else {}
|
||
|
||
This ordering of the operands is more intuitive, but it requires 4 extra
|
||
characters (for "not "). It also highlights the repetition of identifiers:
|
||
``data if data``, ``files if files``, etc. This example benefits from short
|
||
identifiers, but what if the tested expression is longer and/or has side
|
||
effects? This is addressed in the next section.
|
||
|
||
|
||
Motivating Examples
|
||
-------------------
|
||
|
||
The purpose of this PEP is to simplify some common patterns involving ``None``.
|
||
This section presents some examples of common ``None`` patterns and explains
|
||
the drawbacks.
|
||
|
||
This first example is from a Python web crawler that uses the popular Flask
|
||
framework as a front-end. This function retrieves information about a web site
|
||
from a SQL database and formats it as JSON to send to an HTTP client::
|
||
|
||
class SiteView(FlaskView):
|
||
@route('/site/<id_>', methods=['GET'])
|
||
def get_site(self, id_):
|
||
site = db.query('site_table').find(id_)
|
||
|
||
return jsonify(
|
||
first_seen=site.first_seen.isoformat() if site.first_seen is not None else None,
|
||
id=site.id,
|
||
is_active=site.is_active,
|
||
last_seen=site.last_seen.isoformat() if site.last_seen is not None else None,
|
||
url=site.url.rstrip('/')
|
||
)
|
||
|
||
Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the
|
||
database, and they are also allowed to be ``null`` in the JSON response. JSON
|
||
does not have a native way to represent a ``datetime``, so the server's contract
|
||
states that any non-``null`` date is represented as an ISO-8601 string.
|
||
|
||
Note that this code is invalid by PEP-8 standards: several lines are over the
|
||
line length limit. In fact, *including it in this document* violates the PEP
|
||
formatting standard! But it's not unreasonably indented, nor are any of the
|
||
identifiers excessively long. The excessive line length is due to the
|
||
repetition of identifiers on both sides of the ternary ``if`` and the verbosity
|
||
of the ternary itself (10 characters out of a 78 character line length).
|
||
|
||
One way to fix this code is to replace each ternary with a full ``if/else``
|
||
block::
|
||
|
||
class SiteView(FlaskView):
|
||
@route('/site/<id_>', methods=['GET'])
|
||
def get_site(self, id_):
|
||
site = db.query('site_table').find(id_)
|
||
|
||
if site.first_seen is None:
|
||
first_seen = None
|
||
else:
|
||
first_seen = site.first_seen.isoformat()
|
||
|
||
if site.last_seen is None:
|
||
last_seen = None
|
||
else:
|
||
last_seen = site.last_seen.isoformat()
|
||
|
||
return jsonify(
|
||
first_seen=first_seen,
|
||
id=site.id,
|
||
is_active=site.is_active,
|
||
last_seen=last_seen,
|
||
url=site.url.rstrip('/')
|
||
)
|
||
|
||
This version definitely isn't *bad*. It is easy to read and understand. On the
|
||
other hand, adding 8 lines of code to express this common behavior feels a bit
|
||
heavy, especially for a deliberately simplified example. If a larger, more
|
||
complicated data model was being used, then it would get tedious to continually
|
||
write in this long form. The readability would start to suffer as the number of
|
||
lines in the function grows, and a refactoring would be forced.
|
||
|
||
Another alternative is to rename some of the identifiers::
|
||
|
||
class SiteView(FlaskView):
|
||
@route('/site/<id_>', methods=['GET'])
|
||
def get_site(self, id_):
|
||
site = db.query('site_table').find(id_)
|
||
|
||
fs = site.first_seen
|
||
ls = site.last_seen
|
||
|
||
return jsonify(
|
||
first_seen=fs.isodate() if fs is not None else None,
|
||
id=site.id,
|
||
is_active=site.is_active,
|
||
last_seen=ls.isodate() if ls is not None else None,,
|
||
url=site.url.rstrip('/')
|
||
)
|
||
|
||
This adds fewer lines of code than the previous example, but it comes at the
|
||
expense of introducing extraneous identifiers that amount to nothing more than
|
||
aliases. These new identifiers are short enough to fit a ternary expression onto
|
||
one line, but the identifiers are also less intuitive, e.g. ``fs`` versus
|
||
``first_seen``.
|
||
|
||
As a quick preview, consider an alternative rewrite using a new operator::
|
||
|
||
class SiteView(FlaskView):
|
||
@route('/site/<id_>', methods=['GET'])
|
||
def get_site(self, id_):
|
||
site = db.query('site_table').find(id_)
|
||
|
||
return jsonify(
|
||
first_seen=site.first_seen?.isoformat(),
|
||
id=site.id,
|
||
is_active=site.is_active,
|
||
last_seen=site.last_seen?.isoformat(),
|
||
url=site.url.rstrip('/')
|
||
)
|
||
|
||
The ``?.`` operator behaves as a "safe navigation" operator, allowing a more
|
||
concise syntax where the expression ``site.first_seen`` is not duplicated.
|
||
|
||
The next example is from a trending project on GitHub called `Grab
|
||
<https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
|
||
ab/upload.py>`_, which is a Python scraping library::
|
||
|
||
class BaseUploadObject(object):
|
||
def find_content_type(self, filename):
|
||
ctype, encoding = mimetypes.guess_type(filename)
|
||
if ctype is None:
|
||
return 'application/octet-stream'
|
||
else:
|
||
return ctype
|
||
|
||
class UploadContent(BaseUploadObject):
|
||
def __init__(self, content, filename=None, content_type=None):
|
||
self.content = content
|
||
if filename is None:
|
||
self.filename = self.get_random_filename()
|
||
else:
|
||
self.filename = filename
|
||
if content_type is None:
|
||
self.content_type = self.find_content_type(self.filename)
|
||
else:
|
||
self.content_type = content_type
|
||
|
||
class UploadFile(BaseUploadObject):
|
||
def __init__(self, path, filename=None, content_type=None):
|
||
self.path = path
|
||
if filename is None:
|
||
self.filename = os.path.split(path)[1]
|
||
else:
|
||
self.filename = filename
|
||
if content_type is None:
|
||
self.content_type = self.find_content_type(self.filename)
|
||
else:
|
||
self.content_type = content_type
|
||
|
||
.. note::
|
||
|
||
I don't know the author of the Grab project. I used it as an example
|
||
because it is a trending repo on GitHub and it has good examples of common
|
||
``None`` patterns.
|
||
|
||
This example contains several good examples of needing to provide default
|
||
values. It is a bit verbose as it is, and it is certainly not improved by the
|
||
ternary operator::
|
||
|
||
class BaseUploadObject(object):
|
||
def find_content_type(self, filename):
|
||
ctype, encoding = mimetypes.guess_type(filename)
|
||
return 'application/octet-stream' if ctype is None else ctype
|
||
|
||
class UploadContent(BaseUploadObject):
|
||
def __init__(self, content, filename=None, content_type=None):
|
||
self.content = content
|
||
self.filename = self.get_random_filename() if filename \
|
||
is None else filename
|
||
self.content_type = self.find_content_type(self.filename) \
|
||
if content_type is None else content_type
|
||
|
||
class UploadFile(BaseUploadObject):
|
||
def __init__(self, path, filename=None, content_type=None):
|
||
self.path = path
|
||
self.filename = os.path.split(path)[1] if filename is \
|
||
None else filename
|
||
self.content_type = self.find_content_type(self.filename) \
|
||
if content_type is None else content_type
|
||
|
||
The first ternary expression is tidy, but it reverses the intuitive order of
|
||
the operands: it should return ``ctype`` if it has a value and use the string
|
||
literal as fallback. The other ternary expressions are unintuitive and so
|
||
long that they must be wrapped. The overall readability is worsened, not
|
||
improved.
|
||
|
||
This code *might* be improved, though, if there was a syntactic shortcut for
|
||
this common need to supply a default value::
|
||
|
||
class BaseUploadObject(object):
|
||
def find_ctype(self, filename):
|
||
ctype, encoding = mimetypes.guess_type(filename)
|
||
return ctype ?? 'application/octet-stream'
|
||
|
||
class UploadContent(BaseUploadObject):
|
||
def __init__(self, content, filename=None, content_type=None):
|
||
self.content = content
|
||
self.filename = filename ?? self.get_random_filename()
|
||
self.content_type = content_type ?? self.find_ctype(self.filename)
|
||
|
||
class UploadFile(BaseUploadObject):
|
||
def __init__(self, path, filename=None, content_type=None):
|
||
self.path = path
|
||
self.filename = filename ?? os.path.split(path)[1]
|
||
self.content_type = content_type ?? self.find_ctype(self.filename)
|
||
|
||
This syntax has an intuitive ordering of the operands, e.g. ``ctype`` -- the
|
||
preferred value -- comes before the fallback value. The terseness of the syntax
|
||
also makes for fewer lines of code and less code to visually parse.
|
||
|
||
.. note::
|
||
|
||
I cheated on the last example: I renamed ``find_content_type`` to
|
||
``find_ctype`` in order to fit two of the lines under 80 characters. If you
|
||
find this underhanded, you can go back and apply the same renaming to the
|
||
previous 2 examples. You'll find that it doesn't change the
|
||
conclusions.
|
||
|
||
|
||
Usage Of ``None`` In The Standard Library
|
||
-----------------------------------------
|
||
|
||
The previous sections show some code patterns that are claimed to be "common",
|
||
but how common are they? The attached script `find-pep505.py
|
||
<https://github.com/python/peps/blob/master/pep-0505/find-pep505.py>`_ is meant
|
||
to answer this question. It uses the ``ast`` module to search for variations of
|
||
the following patterns in any ``*.py`` file::
|
||
|
||
>>> # None-coalescing if block
|
||
...
|
||
>>> if a is None:
|
||
... a = b
|
||
|
||
>>> # [Possible] None-coalescing "or" operator
|
||
...
|
||
>>> a or 'foo'
|
||
>>> a or []
|
||
>>> a or {}
|
||
|
||
>>> # None-coalescing ternary
|
||
...
|
||
>>> a if a is not None else b
|
||
>>> b if a is None else a
|
||
|
||
>>> # Safe navigation "and" operator
|
||
...
|
||
>>> a and a.foo
|
||
>>> a and a['foo']
|
||
>>> a and a.foo()
|
||
|
||
>>> # Safe navigation if block
|
||
...
|
||
>>> if a is not None:
|
||
... a.foo()
|
||
|
||
>>> # Safe navigation ternary
|
||
...
|
||
>>> a.foo if a is not None else b
|
||
>>> b if a is None else a.foo
|
||
|
||
This script takes one or more names of Python source files to analyze::
|
||
|
||
$ python3 find-pep505.py test.py
|
||
$ find /usr/lib/python3.4 -name '*.py' | xargs python3 find-pep505.py
|
||
|
||
The script prints out any matches it finds. Sample::
|
||
|
||
None-coalescing if block: /usr/lib/python3.4/inspect.py:594
|
||
if _filename is None:
|
||
_filename = getsourcefile(object) or getfile(object)
|
||
|
||
[Possible] None-coalescing `or`: /usr/lib/python3.4/lib2to3/refactor.py:191
|
||
self.explicit = explicit or []
|
||
|
||
None-coalescing ternary: /usr/lib/python3.4/decimal.py:3909
|
||
self.clamp = clamp if clamp is not None else dc.clamp
|
||
|
||
Safe navigation `and`: /usr/lib/python3.4/weakref.py:512
|
||
obj = info and info.weakref()
|
||
|
||
Safe navigation `if` block: /usr/lib/python3.4/http/cookiejar.py:1895
|
||
if k is not None:
|
||
lc = k.lower()
|
||
else:
|
||
lc = None
|
||
|
||
Safe navigation ternary: /usr/lib/python3.4/sre_parse.py:856
|
||
literals = [None if s is None else s.encode('latin-1') for s in literals]
|
||
|
||
.. note::
|
||
|
||
Coalescing with ``or`` is marked as a "possible" match, because it's not
|
||
trivial to infer whether ``or`` is meant to coalesce False-y values
|
||
(correct) or if it meant to coalesce ``None`` (incorrect). On the other
|
||
hand, we assume that ``and`` is always incorrect for safe navigation.
|
||
|
||
The script has been tested against `test.py
|
||
<https://github.com/python/peps/blob/master/pep-0505/test.py>`_ and the Python
|
||
3.4 standard library, but it should work on any arbitrary Python 3 source code.
|
||
The complete output from running it against the standard library is attached to
|
||
this proposal as `find-pep505.out
|
||
<https://github.com/python/peps/blob/master/pep-0505/find-pep505.out>`_.
|
||
|
||
The script counts how many matches it finds and prints the totals at the
|
||
end::
|
||
|
||
Total None-coalescing `if` blocks: 426
|
||
Total [possible] None-coalescing `or`: 119
|
||
Total None-coalescing ternaries: 21
|
||
Total Safe navigation `and`: 9
|
||
Total Safe navigation `if` blocks: 55
|
||
Total Safe navigation ternaries: 7
|
||
|
||
This is a total of 637 possible matches for these common code patterns in the
|
||
standard library. Allowing for some false positives and false negatives, it is
|
||
fair to say that these code patterns are definitely common in the standard
|
||
library.
|
||
|
||
|
||
Rejected Ideas
|
||
--------------
|
||
|
||
Several related ideas were discussed on python-ideas, and some of these were
|
||
roundly rejected by BDFL, the community, or both. For posterity's sake, some of
|
||
those ideas are recorded here.
|
||
|
||
``None``-aware Function Call
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
The ``None``-aware syntax applies to attribute and index access, so it seems
|
||
natural to ask if it should also apply to function invocation syntax. It might
|
||
be written as ``foo?()``, where ``foo`` is only called if it is not None. This
|
||
idea was quickly rejected, for several reasons.
|
||
|
||
First, no other mainstream language has such syntax. Second, Python evaluates
|
||
arguments to a function before it looks up the function itself, so
|
||
``foo?(bar())`` would still call ``bar()`` even if ``foo`` is ``None``. This
|
||
behaviour is unexpected for a so-called "short-circuiting" operator.
|
||
|
||
``?`` Unary Postfix Operator
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
To generalize the ``None``-aware behavior and limit the number of new operators
|
||
introduced, a unary, postfix operator spelled ``?`` was suggested. The idea is
|
||
that ``?`` might return a special object that could would override dunder
|
||
methods that return ``self``. For example, ``foo?`` would evaluate to ``foo`` if
|
||
it is not ``None``, otherwise it would evaluate to an instance of
|
||
``NoneQuestion``::
|
||
|
||
class NoneQuestion():
|
||
def __call__(self, *args, **kwargs):
|
||
return self
|
||
|
||
def __getattr__(self, name):
|
||
return self
|
||
|
||
def __getitem__(self, key):
|
||
return self
|
||
|
||
|
||
With this new operator and new type, an expression like ``foo?.bar[baz]``
|
||
evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty
|
||
generalization, but it's difficult to use in practice since most existing code
|
||
won't know what ``NoneQuestion`` is.
|
||
|
||
Going back to one of the motivating examples above, consider the following::
|
||
|
||
>>> import json
|
||
>>> created = None
|
||
>>> json.dumps({'created': created?.isoformat()})``
|
||
|
||
The JSON serializer does not know how to serialize ``NoneQuestion``, nor will
|
||
any other API. This proposal actually requires *lots of specialized logic*
|
||
throughout the standard library and any third party library.
|
||
|
||
At the same time, the ``?`` operator may also be **too general**, in the sense
|
||
that it can be combined with any other operator. What should the following
|
||
expressions mean?::
|
||
|
||
>>> x? + 1
|
||
>>> x? -= 1
|
||
>>> x? == 1
|
||
>>> ~x?
|
||
|
||
This degree of generalization is not useful. The operators actually proposed
|
||
herein are intentionally limited to a few operators that are expected to make it
|
||
easier to write common code patterns.
|
||
|
||
Haskell-style ``Maybe``
|
||
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that
|
||
encapsulates the idea of an optional value without relying on any special
|
||
keyword (e.g. ``null``) or any special instance (e.g. ``None``). In Haskell, the
|
||
purpose of ``Maybe`` is to avoid separate handling of "something" and nothing".
|
||
The concept is so heavily intertwined with Haskell's lazy evaluation that it
|
||
doesn't translate cleanly into Python.
|
||
|
||
There is a Python package called `pymaybe
|
||
<https://pypi.python.org/pypi/pymaybe/0.1.1>`_ that provides a rough
|
||
approximation. The documentation shows the following example that appears
|
||
relevant to the discussion at hand::
|
||
|
||
>>> maybe('VALUE').lower()
|
||
'value'
|
||
|
||
>>> maybe(None).invalid().method().or_else('unknown')
|
||
'unknown'
|
||
|
||
The function ``maybe()`` returns either a ``Something`` instance or a
|
||
``Nothing`` instance. Similar to the unary postfix operator described in the
|
||
previous section, ``Nothing`` overrides dunder methods in order to allow
|
||
chaining on a missing value.
|
||
|
||
Note that ``or_else()`` is eventually required to retrieve the underlying value
|
||
from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short circuit any
|
||
evaluation. Although ``pymaybe`` has some strengths and may be useful in its own
|
||
right, it also demonstrates why a pure Python implementation of coalescing is
|
||
not nearly as powerful as support built into the language.
|
||
|
||
|
||
Specification
|
||
=============
|
||
|
||
This PEP suggests 3 new operators be added to Python:
|
||
|
||
1. ``None``-coalescing operator
|
||
2. ``None``-aware attribute access
|
||
3. ``None``-aware index access/slicing
|
||
|
||
We will continue to assume the same spellings as in
|
||
the previous sections in order to focus on behavior before diving into the much
|
||
more contentious issue of how to spell these operators.
|
||
|
||
A generalization of these operators is also proposed below under the heading
|
||
"Generalized Coalescing".
|
||
|
||
|
||
Operator Spelling
|
||
-----------------
|
||
|
||
Despite significant support for the proposed operators, the majority of
|
||
discussion on python-ideas fixated on the spelling. Many alternative spellings
|
||
were proposed, both punctuation and keywords, but each alternative drew some
|
||
criticism. Spelling the operator as a keyword is problematic, because adding new
|
||
keywords to the language is not backwards compatible.
|
||
|
||
It is not impossible to add a new keyword, however, and we can look at several
|
||
other PEPs for inspiration. For example, `PEP-492
|
||
<https://www.python.org/dev/peps/pep-0492/>`_ introduced the new keywords
|
||
``async`` and ``await`` into Python 3.5. These new keywords are fully backwards
|
||
compatible, because that PEP also introduces a new lexical context such that
|
||
``async`` and ``await`` are only treated as keywords when used inside of an
|
||
``async def`` function. In other locations, ``async`` and ``await`` may be used
|
||
as identifiers.
|
||
|
||
It is also possible to craft a new operator out of existing keywords, as was
|
||
the case with `PEP-308 <https://www.python.org/dev/peps/pep-0308/>`_, which
|
||
created a ternary operator by cobbling together the `if` and `else` keywords
|
||
into a new operator.
|
||
|
||
In addition to the lexical acrobatics required to create a new keyword, keyword
|
||
operators are also undesirable for creating an assignment shortcut syntax. In
|
||
Dart, for example, ``x ??= y`` is an assignment shortcut that approximately
|
||
means ``x = x ?? y`` except that ``x`` is only evaluated once. If Python's
|
||
coalesce operator is a keyword, e.g. ``foo``, then the assignment shortcut would
|
||
be very ugly: ``x foo= y``.
|
||
|
||
Spelling new logical operators with punctuation is unlikely, for several
|
||
reasons. First, Python eschews punctuation for logical operators. For example,
|
||
it uses ``not`` instead of ``!``, ``or`` instead of ``||``, and ``… if … else …``
|
||
instead of ``… ? … : …``.
|
||
|
||
Second, nearly every single punctuation character on a standard keyboard already
|
||
has special meaning in Python. The only exceptions are ``$``, ``!``, ``?``, and
|
||
backtick (as of Python 3). This leaves few options for a new, single-character
|
||
operator.
|
||
|
||
Third, other projects in the Python universe assign special meaning to
|
||
punctuation. For example, `IPython
|
||
<https://ipython.org/ipython-doc/2/interactive/reference.html>`_ assigns
|
||
special meaning to ``%``, ``%%``, ``?``, ``??``, ``$``, and ``$$``, among
|
||
others. Out of deference to those projects and the large communities using them,
|
||
introducing conflicting syntax into Python is undesirable.
|
||
|
||
The spellings ``??`` and ``?.`` will be familiar to programmers who have seen
|
||
them in other popular programming languages. Any alternative punctuation will be
|
||
just as ugly but without the benefit of familiarity from other languages.
|
||
Therefore, this proposal spells the new operators using the same punctuation
|
||
that already exists in other languages.
|
||
|
||
|
||
``None``-Coalescing Operator
|
||
----------------------------
|
||
|
||
The ``None``-coalescing operator is a short-circuiting, binary operator that
|
||
behaves in the following way.
|
||
|
||
1. Evaluate the left operand first.
|
||
2. If the left operand is not ``None``, then return it immediately.
|
||
3. Else, evaluate the right operand and return the result.
|
||
|
||
Consider the following examples. We will continue to use the spelling ``??``
|
||
here, but keep in mind that alternative spellings will be discussed below::
|
||
|
||
>>> 1 ?? 2
|
||
1
|
||
>>> None ?? 2
|
||
2
|
||
|
||
Importantly, note that the right operand is not evaluated unless the left
|
||
operand is None::
|
||
|
||
>>> def err(): raise Exception('foo')
|
||
>>> 1 ?? err()
|
||
1
|
||
>>> None ?? err()
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
File "<stdin>", line 1, in err
|
||
Exception: foo
|
||
|
||
The operator is left associative. Combined with its short circuiting behavior,
|
||
this makes the operator easy to chain::
|
||
|
||
>>> timeout = None
|
||
>>> local_timeout = 60
|
||
>>> global_timeout = 300
|
||
>>> timeout ?? local_timeout ?? global_timeout
|
||
60
|
||
|
||
>>> local_timeout = None
|
||
>>> timeout ?? local_timeout ?? global_timeout
|
||
300
|
||
|
||
The operator has higher precedence than the comparison operators ``==``, ``>``,
|
||
``is``, etc., but lower precedence than any bitwise or arithmetic operators.
|
||
This precedence is chosen for making "default value" expressions intuitive to
|
||
read and write::
|
||
|
||
>>> not None ?? True
|
||
>>> not (None ?? True) # Same precedence
|
||
|
||
>>> 1 == None ?? 1
|
||
>>> 1 == (None ?? 1) # Same precedence
|
||
|
||
>>> 'foo' in None ?? ['foo', 'bar']
|
||
>>> 'foo' in (None ?? ['foo', 'bar']) # Same precedence
|
||
|
||
>>> 1 + None ?? 2
|
||
>>> 1 + (None ?? 2) # Same precedence
|
||
|
||
Recall the example above of calculating the cost of items in a shopping cart,
|
||
and the easy-to-miss bug. This type of bug is not possible with the ``None``-
|
||
coalescing operator, because there is no implicit type coersion to ``bool``::
|
||
|
||
>>> requested_quantity = 0
|
||
>>> default_quantity = 1
|
||
>>> price = 100
|
||
>>> requested_quantity ?? default_quantity * price
|
||
0
|
||
|
||
The ``None``-coalescing operator also has a corresponding assignment shortcut.
|
||
The following assignments are semantically similar, except that ``foo`` is only
|
||
looked up once when using the assignment shortcut::
|
||
|
||
>>> foo ??= []
|
||
>>> foo = foo ?? []
|
||
|
||
The ``None`` coalescing operator improves readability, especially when handling
|
||
default function arguments. Consider again the example from the Requests
|
||
library, rewritten to use ``None``-coalescing::
|
||
|
||
def __init__(self, data=None, files=None, headers=None, params=None, hooks=None):
|
||
self.data = data ?? []
|
||
self.files = files ?? []
|
||
self.headers = headers ?? {}
|
||
self.params = params ?? {}
|
||
self.hooks = hooks ?? {}
|
||
|
||
The operator makes the intent easier to follow (by putting operands in an
|
||
intuitive order) and is more concise than the ternary operator, while still
|
||
preserving the short circuit semantics of the code that it replaces.
|
||
|
||
|
||
``None``-Aware Attribute Access Operator
|
||
----------------------------------------
|
||
|
||
The ``None``-aware attribute access operator (also called "safe navigation")
|
||
checks its left operand. If the left operand is ``None``, then the operator
|
||
evaluates to ``None``. If the the left operand is not ``None``, then the
|
||
operator accesses the attribute named by the right operand::
|
||
|
||
>>> from datetime import date
|
||
>>> d = date.today()
|
||
>>> d.year
|
||
2015
|
||
|
||
>>> d = None
|
||
>>> d.year
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
AttributeError: 'NoneType' object has no attribute 'year'
|
||
|
||
>>> d?.year
|
||
None
|
||
|
||
The operator has the same precedence and associativity as the plain attribute
|
||
access operator ``.``, but this operator is also short-circuiting in a unique
|
||
way: if the left operand is ``None``, then any series of attribute access, index
|
||
access, slicing, or function call operators immediately to the right of it *are
|
||
not evaluated*::
|
||
|
||
>>> name = ' The Black Knight '
|
||
>>> name.strip()[4:].upper()
|
||
'BLACK KNIGHT'
|
||
|
||
>>> name = None
|
||
>>> name?.strip()[4:].upper()
|
||
None
|
||
|
||
If this operator did not short circuit in this way, then the second example
|
||
would partially evaluate ``name?.strip()`` to ``None()`` and then fail with
|
||
``TypeError: 'NoneType' object is not callable``.
|
||
|
||
To put it another way, the following expressions are semantically similar,
|
||
except that ``name`` is only looked up once on the first line::
|
||
|
||
>>> name?.strip()[4:].upper()
|
||
>>> name.strip()[4:].upper() if name is not None else None
|
||
|
||
.. note::
|
||
|
||
C# implements its safe navigation operators with the same short-circuiting
|
||
semantics, but Dart does not. In Dart, the second example (suitably
|
||
translated) would fail. The C# semantics are obviously superior, given the
|
||
original goal of writing common cases more concisely. The Dart semantics are
|
||
nearly useless.
|
||
|
||
This operator short circuits one or more attribute access, index access,
|
||
slicing, or function call operators that are adjacent to its right, but it
|
||
does not short circuit any other operators (logical, bitwise, arithmetic, etc.),
|
||
nor does it escape parentheses::
|
||
|
||
>>> d = date.today()
|
||
>>> d?.year.numerator + 1
|
||
2016
|
||
|
||
>>> d = None
|
||
>>> d?.year.numerator + 1
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
|
||
|
||
>>> (d?.year).numerator + 1
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
AttributeError: 'NoneType' object has no attribute 'numerator'
|
||
|
||
Note that the error in the second example is not on the attribute access
|
||
``numerator``. In fact, that attribute access is never performed. The error
|
||
occurs when adding ``None + 1``, because the ``None``-aware attribute access
|
||
does not short circuit ``+``.
|
||
|
||
The third example fails because the operator does not escape parentheses. In
|
||
that example, the attribute access ``numerator`` is evaluated and fails because
|
||
``None`` does not have that attribute.
|
||
|
||
Finally, observe that short circuiting adjacent operators is not at all the same
|
||
thing as propagating ``None`` throughout an expression::
|
||
|
||
>>> user?.first_name.upper()
|
||
|
||
If ``user`` is not ``None``, then ``user.first_name`` is evaluated. If
|
||
``user.first_name`` evaluates to ``None``, then ``user.first_name.upper()`` is
|
||
an error! In English, this expression says, "``user`` is optional but if it has
|
||
a value, then it must have a ``first_name``, too."
|
||
|
||
If ``first_name`` is supposed to be optional attribute, then the expression must
|
||
make that explicit::
|
||
|
||
>>> user?.first_name?.upper()
|
||
|
||
The operator is not intended as an error silencing mechanism, and it would be
|
||
undesirable if its presence infected nearby operators.
|
||
|
||
|
||
``None``-Aware Index Access/Slicing Operator
|
||
--------------------------------------------
|
||
|
||
The ``None``-aware index access/slicing operator (also called "safe navigation")
|
||
is nearly identical to the ``None``-aware attribute access operator. It combines
|
||
the familiar square bracket syntax ``[]`` with new punctuation or a new keyword,
|
||
the spelling of which is discussed later::
|
||
|
||
>>> person = {'name': 'Mark', 'age': 32}
|
||
>>> person['name']
|
||
'Mark'
|
||
|
||
>>> person = None
|
||
>>> person['name']
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
TypeError: 'NoneType' object is not subscriptable
|
||
|
||
>>> person?.['name']
|
||
None
|
||
|
||
The ``None``-aware slicing operator behaves similarly::
|
||
|
||
>>> name = 'The Black Knight'
|
||
>>> name[4:]
|
||
'Black Knight'
|
||
|
||
>>> name = None
|
||
>>> name[4:]
|
||
Traceback (most recent call last):
|
||
File "<stdin>", line 1, in <module>
|
||
TypeError: 'NoneType' object is not subscriptable
|
||
|
||
>>> name?.[4:]
|
||
None
|
||
|
||
These operators have the same precedence as the plain index access and slicing
|
||
operators. They also have the same short-circuiting behavior as the
|
||
``None``-aware attribute access.
|
||
|
||
|
||
Generalized Coalescing
|
||
----------------------
|
||
|
||
Making ``None`` a special case is too specialized and magical. The behavior can
|
||
be generalized by making the ``None``-aware operators invoke a dunder method,
|
||
e.g. ``__coalesce__(self)`` that returns ``True`` if an object should be
|
||
coalesced and ``False`` otherwise.
|
||
|
||
With this generalization, ``object`` would implement a dunder method equivalent
|
||
to this::
|
||
|
||
def __coalesce__(self):
|
||
return False
|
||
|
||
``NoneType`` would implement a dunder method equivalent to this::
|
||
|
||
def __coalesce__(self):
|
||
return True
|
||
|
||
If this generalization is accepted, then the operators will need to be renamed
|
||
such that the term ``None`` is not used, e.g. "Coalescing Operator", "Coalesced
|
||
Member Access Operator", etc.
|
||
|
||
The coalesce operator would invoke this dunder method. The following two
|
||
expressions are semantically similar, except `foo` is only looked up once when
|
||
using the coalesce operator::
|
||
|
||
>>> foo ?? bar
|
||
>>> bar if foo.__coalesce__() else foo
|
||
|
||
The coalesced attribute and index access operators would invoke the same dunder
|
||
method::
|
||
|
||
>>> user?.first_name.upper()
|
||
>>> None if user.__coalesce__() else user.first_name.upper()
|
||
|
||
This generalization allows for domain-specific ``null`` objects to be coalesced
|
||
just like ``None``. For example the ``pyasn1`` package has a type called
|
||
``Null`` that represents an ASN.1 ``null``::
|
||
|
||
>>> from pyasn1.type import univ
|
||
>>> univ.Null() ?? univ.Integer(123)
|
||
Integer(123)
|
||
|
||
In addition to making the proposed operators less specialized, this
|
||
generalization also makes it easier to work with the Null Object Pattern, [3]_
|
||
for those developers who prefer to avoid using ``None``.
|
||
|
||
|
||
Implementation
|
||
--------------
|
||
|
||
The author of this PEP is not competent with grammars or lexers, and given the
|
||
contentiousness of this proposal, the implementation details for CPython will be
|
||
deferred until we have a clearer idea that one or more of the proposed
|
||
enhancements will be approved.
|
||
|
||
...TBD...
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] C# Reference: Operators
|
||
(https://msdn.microsoft.com/en-us/library/6a71f45d.aspx)
|
||
|
||
.. [2] A Tour of the Dart Language: Operators
|
||
(https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)
|
||
|
||
.. [3] Wikipedia: Null Object Pattern
|
||
(https://en.wikipedia.org/wiki/Null_Object_pattern)
|
||
|
||
.. [4] PEP-249:
|
||
(https://www.python.org/dev/peps/pep-0249/)
|
||
|
||
.. [5] PEP-308
|
||
(https://www.python.org/dev/peps/pep-0308/)
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|