From 33877f7ac33b64b1b3ee89ca2b91b8a7b3645816 Mon Sep 17 00:00:00 2001 From: Steve Dower Date: Fri, 6 Jul 2018 10:48:47 -0700 Subject: [PATCH] PEP 505: prepare for 3.8 (#693) * Add myself as PEP 505 coauthor, and prepare for the rewrite * First pass rewrite of PEP 505 --- pep-0505.txt | 1002 ++++++++++++-------------------------------------- 1 file changed, 239 insertions(+), 763 deletions(-) diff --git a/pep-0505.txt b/pep-0505.txt index fffbd3266..a6e675fc8 100644 --- a/pep-0505.txt +++ b/pep-0505.txt @@ -2,20 +2,13 @@ PEP: 505 Title: None-aware operators Version: $Revision$ Last-Modified: $Date$ -Author: Mark E. Haase +Author: Mark E. Haase , Steve Dower Status: Deferred Type: Standards Track Content-Type: text/x-rst Created: 18-Sep-2015 Python-Version: 3.8 -PEP Deferral -============ - -Further consideration of this PEP has been deferred until Python 3.8 at the -earliest. - - Abstract ======== @@ -33,264 +26,132 @@ patterns involving null references. only if that collection is non-``null``. Otherwise it returns ``null``. (This is another type of "safe navigation" operator.) -The purpose of this PEP is to explore the possibility of implementing similar -operators in Python. It provides some background material and then offers -several competing alternatives for implementation. +This PEP proposes three ``None``-aware operators for Python, based on the +definitions and other language's implementations of those above. Specifically: -The initial reaction to this idea is majority negative. Even if ultimately -rejected, this PEP still serves a purpose: to fully document the reasons why -Python should not add this behavior, so that it can be pointed to in the future -when the question inevitably arises again. (This is the null alternative, so to -speak!) +* The "``None`` coalescing`` binary operator ``??`` returns the left hand side + if it evaluates to a value that is not ``None``, or else it evaluates and + returns the right hand side. +* The "``None``-aware attribute access" operator ``?.`` evaluates the complete + expression if the left hand side evaluates to a value that is not ``None`` +* The "``None``-aware indexing" operator ``?[]`` evaluates the complete + expression if the left hand site evaluates to a value that is not ``None`` -This proposal advances multiple alternatives, and it should be considered -severable. It may be accepted in whole or in part. For example, the safe -navigation operators might be rejected even if the ``null``-coalescing operator -is approved, or vice-versa. - -Of course, Python does not have ``null``; it has ``None``, which is conceptually -distinct. Although this PEP is inspired by "``null``-aware" operators in other -languages, it uses the term "``None``-aware" operators to describe some -hypothetical Python implementations. - - -Background -========== +Syntax and Semantics +==================== Specialness of ``None`` ----------------------- -The Python language does not currently define any special behavior for ``None``. -This PEP suggests making ``None`` a special case. This loss of generality is a -noticeable drawback of the proposal. A generalization of ``None``-aware -operators is set forth later in this document in order to avoid this -specialization. +The ``None`` object denotes the lack of a value. For the purposes of these +operators, the lack of a value indicates that the remainder of the expression +also lacks a value and should not be evaluated. +A rejected proposal was to treat any value that evaluates to false in a +Boolean context as not having a value. However, the purpose of these operators +is to propagate the "lack of value" state, rather that the "false" state. -Utility of ``None`` -------------------- +Some argue that this makes ``None`` special. We contend that ``None`` is +already special, and that using it as both the test and the result of these +operators does not change the existing semantics in any way. -One common criticism of adding special syntax for ``None`` is that ``None`` -shouldn't be used in the first place: it's a code smell. A related criticism is -that ``None``-aware operators are used to silence errors (such as the novice -misunderstanding of an implicit ``return None``) akin to `PHP's @ operator -`_. Therefore, -the utility of ``None`` must be debated before discussing whether to add new -behavior around it. +Grammar changes +--------------- -Python does not have any concept of ``null``. Every Python identifier must -refer to an instance, so there cannot be any ``null`` references. Python does -have a special instance called ``None`` that can be used to represent missing -values, but ``None`` is conceptually distinct from ``null``. +The following rules of the Python grammar are updated to read:: -The most frequent use of ``None`` in Python is to provide a default value for -optional arguments when some other default object is unwieldy. For example: -``def get(url, proxy=None):``. In this case, ``proxy`` is an optional -argument. If ``proxy`` is ``None``, then the request should be sent directly to -the server; otherwise, the request should be routed through the specified proxy -server. This use of ``None`` is preferred here to some other sentinel value or -the Null Object Pattern. [3]_ + power: coalesce_expr ['**' factor] + coalesce_expr: atom_expr ['??' atom_expr] + atom_expr: ['await'] atom trailer* + trailer: '(' [arglist] ')' | + '[' subscriptlist ']' | + '?[' subscriptlist ']' | + '.' NAME | + '?.' NAME -Examples of this form abound. Consider ``types.py`` in the standard library:: +Inserting the ``coalesce_expr`` rule in this location ensures that +expressions resulting in ``None`` are natuarlly coalesced before they are used +in operations that would typically raise ``TypeError``. Like ``and`` and ``or`` +the right-hand expression is not evaluated until the left-hand side is +determined to be ``None``. For example:: - def prepare_class(name, bases=(), kwds=None): - if kwds is None: - kwds = {} - else: - kwds = dict(kwds) - ... + a, b = None, None + def c(): return None + def ex(): raise Exception() -Another frequent use of ``None`` is interfacing with external systems. Many of -those other systems have a concept of ``null``. Therefore, Python code must have -a way of representing ``null``, and typically it is represented by ``None``. For -example, databases can have ``null`` values, and most Python database drivers -will convert ``null`` to ``None`` when retrieving data from a database, and will -convert from ``None`` back to ``null`` when sending data to a database. + (a ?? 2 ** b ?? 3) == (a ?? 2) ** (b ?? 3) + (a ?? True and b ?? False) == (a ?? True) and (b ?? False) + (c() ?? c() ?? True) == True + (True ?? ex()) == True + (c ?? ex)() == c() -This convention of interchanging ``null`` and ``None`` is widespread in Python. -It is canonized in the Python DBAPI (PEP-249). [4]_ The ``json`` module in the -standard library and the third party PyYAML package both use ``None`` to -represent their respective languages' ``null``. +Adding new trailers for the other ``None``-aware operators ensures that they +may be used in all valid locations for the existing equivalent operators. +However, as the existing evaluation rules are not directly embedded in the +grammar, we specify the required changes here. -The C language ``null`` often bleeds into Python, too, particularly for thin -wrappers around C libraries. For example, in ``pyopenssl``, the ``X509`` class -has a ``get_notBefore()`` `method `_ that returns -either a timestamp or ``None``. This function is a thin wrapper around an -OpenSSL function with the return type ``ASN1_TIME *``. Because this C pointer -may be ``null``, the Python wrapper must be able to represent ``null``, and -``None`` is the chosen representation. +Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is +then evaluated from left to right, applying its own parameter (either its +arguments, subscripts or attribute name) to produce the value for the next +``trailer``. Finally, if present, ``await`` is applied. -The representation of ``null`` is particularly noticeable when Python code is -marshalling data between two systems. For example, consider a Python server that -fetches data from a database and converts it to JSON for consumption by another -process. In this case, it's often desirable that ``null`` in the database can be -easily translated to ``null`` in JSON. If ``None`` is not used for this purpose, -then each package will have to define its own representation of ``null``, and -converting between these representations adds unnecessary complexity to the -Python glue code. +For example, ``await a.b(c).d[e]`` is currently parsed as +``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated:: -Therefore, the preference for avoiding ``None`` is nothing more than a -preference. ``None`` has legitimate uses, particularly in specific types of -software. Any hypothetical ``None``-aware operators should be construed as -syntactic sugar for simplifying common patterns involving ``None``, and *should -not be construed* as error handling behavior. + _v = a + _v = _v.b + _v = _v(c) + _v = _v.d + _v = _v[e] + await _v +When a ``None``-aware operator is present, the left-to-right evaluation may be +short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated:: -Behavior In Other Languages ---------------------------- + _v = a + if _v is not None: + _v = _v.b + _v = _v(c) + _v = _v.d + if _v is not None: + _v = _v[e] + await _v -Given that ``null``-aware operators exist in other modern languages, it may be -helpful to quickly understand how they work in those languages:: +.. note:: + ``await`` will almost certainly fail in this context, as it would in + the case where code attempts ``await None``. We are not proposing to add a + ``None``-aware ``await`` keyword here, and merely include it in this + example for completeness of the specification, since the ``atom_expr`` + grammar rule includes the keyword. If it were in its own rule, we would have + never mentioned it. - /* Null-coalescing. */ +Parenthesised expressions are handled by the ``atom`` rule (not shown above), +which will implicitly terminate the short-circuiting behaviour of the above +transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as:: - String s1 = null; - String s2 = "hello"; - String s3 = s1 ?? s2; - Console.WriteLine("s3 is: " + s3); - // s3 is: hello + # a?.b + _v = a + if _v is not None: + _v = _v.b - /* Null-aware member access, a.k.a. safe navigation. */ + # ... ?? c + if _v is None: + _v = c - Console.WriteLine("s1.Length is: " + s1?.Length); - Console.WriteLine("s2.Length is: " + s2?.Length); - // s1.Length is: - // s2.Length is: 5 + # (...).d?.e + _v = _v.d + if _v is not None: + _v = _v.e - /* Null-aware index access, a.k.a. safe navigation. */ +Examples +======== - Dictionary d1 = null; - Dictionary d2 = new Dictionary - { - { "foo", "bar" }, - { "baz", "bat" } - }; - - Console.WriteLine("d1[\"foo\"] is: " + d1?["foo"]); - Console.WriteLine("d2[\"foo\"] is: " + d2?["foo"]); - // d1["foo"] is: - // d2["foo"] is: bar - - /* Short Circuiting */ - - Console.WriteLine("s1 trim/upper is: " + s1?.Trim().Length); - Console.WriteLine("s2 trim/upper is: " + s2?.Trim().Length); - // s1 trimmed length is: - // s2 trimmed length is: 5 - - String s4 = s1 ?? s2 ?? DoError(); - Console.WriteLine("s4 is: " + s4) - // s4 is: hello - -A `working example `_ can be viewed online. - -Of utmost importance, notice the short circuiting behavior. The short circuiting -of ``??`` is similar to short circuiting of other boolean operators such as -``||`` or ``&&`` and should not be surprising. Helpfully, `?.` is *also* short -circuiting: ``s1?.Trim()`` evaluates to null, but ``s1?.Trim().Length`` does not -attempt to dereference the ``null`` pointer. - - -Rationale -========= - -Existing Alternatives ---------------------- - -Python does not have any specific ``None``-aware operators, but it does have -operators that can be used for a similar purpose. This section describes why -these alternatives may be undesirable for some common ``None`` patterns. - - -``or`` Operator -~~~~~~~~~~~~~~~ - -Similar behavior can be achieved with the ``or`` operator, but ``or`` checks -whether its left operand is false-y, not specifically ``None``. This can lead -to surprising behavior. Consider the scenario of computing the price of some -products a customer has in his/her shopping cart:: - - >>> price = 100 - >>> default_quantity = 1 - # If user didn't specify a quantity, then assume the default. - >>> requested_quantity = None - >>> (requested_quantity or default_quantity) * price - 100 - # The user added 5 items to the cart. - >>> requested_quantity = 5 - >>> (requested_quantity or default_quantity) * price - 500 - # User removed 5 items from cart. - >>> requested_quantity = 0 - >>> (requested_quantity or default_quantity) * price # oops! - 100 - -An experienced Python developer should know how ``or`` works and be capable of -avoiding bugs like this. However, getting in the habit of using ``or`` for this -purpose still might cause an experienced developer to occasionally make this -mistake, especially when refactoring existing code and not carefully paying -attention to the possible values of the left-hand operand. - -For inexperienced developers, the problem is worse. The top Google hit for -"python null coalesce" is a `StackOverflow page -`_, and the top answer says to use ``or``. -The top answer goes on to explain the caveats of using ``or`` like this, but how -many beginning developers go on to read all those caveats? The accepted answer -on `a more recent question `_ says to use -``or`` without any caveats at all. These two questions have a combined 26,000 -views! - -The common usage of ``or`` for the purpose of providing default values is -undeniable, and yet it is also booby-trapped for unsuspecting newcomers. This -suggests that a safe operator for providing default values would have positive -utility. While some critics claim that ``None``-aware operators will be abused -for error handling, they are no more prone to abuse than ``or`` is. - - -Ternary Operator -~~~~~~~~~~~~~~~~ - -Another common way to initialize default values is to use the ternary operator. -Here is an excerpt from the popular `Requests package -`_:: - - data = [] if data is None else data - files = [] if files is None else files - headers = {} if headers is None else headers - params = {} if params is None else params - hooks = {} if hooks is None else hooks - -This particular formulation has the undesirable effect of putting the operands -in an unintuitive order: the brain thinks, "use ``data`` if possible and use -``[]`` as a fallback," but the code puts the fallback *before* the preferred -value. - -The author of this package could have written it like this instead:: - - data = data if data is not None else [] - files = files if files is not None else [] - headers = headers if headers is not None else {} - params = params if params is not None else {} - hooks = hooks if hooks is not None else {} - -This ordering of the operands is more intuitive, but it requires 4 extra -characters (for "not "). It also highlights the repetition of identifiers: -``data if data``, ``files if files``, etc. This example benefits from short -identifiers, but what if the tested expression is longer and/or has side -effects? This is addressed in the next section. - - -Motivating Examples -------------------- - -The purpose of this PEP is to simplify some common patterns involving ``None``. This section presents some examples of common ``None`` patterns and explains the drawbacks. +jsonify +------- + This first example is from a Python web crawler that uses the popular Flask framework as a front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client:: @@ -313,30 +174,29 @@ database, and they are also allowed to be ``null`` in the JSON response. JSON does not have a native way to represent a ``datetime``, so the server's contract states that any non-``null`` date is represented as an ISO-8601 string. -Note that this code is invalid by PEP-8 standards: several lines are over the -line length limit. In fact, *including it in this document* violates the PEP -formatting standard! But it's not unreasonably indented, nor are any of the -identifiers excessively long. The excessive line length is due to the -repetition of identifiers on both sides of the ternary ``if`` and the verbosity -of the ternary itself (10 characters out of a 78 character line length). +However, without knowing the exact semantics of the ``first_seen`` and +``last_seen`` attributes, it is impossible to know whether the attribute can +be safely or performantly accessed multiple times. -One way to fix this code is to replace each ternary with a full ``if/else`` -block:: +One way to fix this code is to replace each ternary with explicit value +assignment and a full ``if/else`` block:: class SiteView(FlaskView): @route('/site/', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) - if site.first_seen is None: + first_seen_dt = site.first_seen + if first_seen_dt is None: first_seen = None else: - first_seen = site.first_seen.isoformat() + first_seen = first_seen_dt.isoformat() - if site.last_seen is None: + last_seen_dt = site.last_seen + if last_seen_dt is None: last_seen = None else: - last_seen = site.last_seen.isoformat() + last_seen = last_seen_dt.isoformat() return jsonify( first_seen=first_seen, @@ -346,38 +206,10 @@ block:: url=site.url.rstrip('/') ) -This version definitely isn't *bad*. It is easy to read and understand. On the -other hand, adding 8 lines of code to express this common behavior feels a bit -heavy, especially for a deliberately simplified example. If a larger, more -complicated data model was being used, then it would get tedious to continually -write in this long form. The readability would start to suffer as the number of -lines in the function grows, and a refactoring would be forced. - -Another alternative is to rename some of the identifiers:: - - class SiteView(FlaskView): - @route('/site/', methods=['GET']) - def get_site(self, id_): - site = db.query('site_table').find(id_) - - fs = site.first_seen - ls = site.last_seen - - return jsonify( - first_seen=fs.isodate() if fs is not None else None, - id=site.id, - is_active=site.is_active, - last_seen=ls.isodate() if ls is not None else None,, - url=site.url.rstrip('/') - ) - -This adds fewer lines of code than the previous example, but it comes at the -expense of introducing extraneous identifiers that amount to nothing more than -aliases. These new identifiers are short enough to fit a ternary expression onto -one line, but the identifiers are also less intuitive, e.g. ``fs`` versus -``first_seen``. - -As a quick preview, consider an alternative rewrite using a new operator:: +This adds ten lines of code and four new code paths to the function, +dramatically increasing the apparently complexity. Rewriting using +``None``-aware attribute operator on the other hand, results in shorter +code:: class SiteView(FlaskView): @route('/site/', methods=['GET']) @@ -392,12 +224,12 @@ As a quick preview, consider an alternative rewrite using a new operator:: url=site.url.rstrip('/') ) -The ``?.`` operator behaves as a "safe navigation" operator, allowing a more -concise syntax where the expression ``site.first_seen`` is not duplicated. +Grab +---- -The next example is from a trending project on GitHub called `Grab +The next example is from a Python scraping library called `Grab `_, which is a Python scraping library:: +ab/upload.py>`_:: class BaseUploadObject(object): def find_content_type(self, filename): @@ -431,15 +263,9 @@ ab/upload.py>`_, which is a Python scraping library:: else: self.content_type = content_type -.. note:: - - I don't know the author of the Grab project. I used it as an example - because it is a trending repo on GitHub and it has good examples of common - ``None`` patterns. - This example contains several good examples of needing to provide default -values. It is a bit verbose as it is, and it is certainly not improved by the -ternary operator:: +values. Rewriting to use a conditional expression reduces the overall lines of +code, but does not necessarily improve readability:: class BaseUploadObject(object): def find_content_type(self, filename): @@ -449,18 +275,18 @@ ternary operator:: class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content - self.filename = self.get_random_filename() if filename \ - is None else filename - self.content_type = self.find_content_type(self.filename) \ - if content_type is None else content_type + self.filename = (self.get_random_filename() if filename + is None else filename) + self.content_type = (self.find_content_type(self.filename) + if content_type is None else content_type) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path - self.filename = os.path.split(path)[1] if filename is \ - None else filename - self.content_type = self.find_content_type(self.filename) \ - if content_type is None else content_type + self.filename = (os.path.split(path)[1] if filename is + None else filename) + self.content_type = (self.find_content_type(self.filename) + if content_type is None else content_type) The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ``ctype`` if it has a value and use the string @@ -468,11 +294,10 @@ literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved. -This code *might* be improved, though, if there was a syntactic shortcut for -this common need to supply a default value:: +Rewriting using the ``None`` coalescing operator:: class BaseUploadObject(object): - def find_ctype(self, filename): + def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) return ctype ?? 'application/octet-stream' @@ -480,136 +305,127 @@ this common need to supply a default value:: def __init__(self, content, filename=None, content_type=None): self.content = content self.filename = filename ?? self.get_random_filename() - self.content_type = content_type ?? self.find_ctype(self.filename) + self.content_type = content_type ?? self.find_content_type(self.filename) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path self.filename = filename ?? os.path.split(path)[1] - self.content_type = content_type ?? self.find_ctype(self.filename) + self.content_type = content_type ?? self.find_content_type(self.filename) -This syntax has an intuitive ordering of the operands, e.g. ``ctype`` -- the -preferred value -- comes before the fallback value. The terseness of the syntax -also makes for fewer lines of code and less code to visually parse. +This syntax has an intuitive ordering of the operands. In +``find_content_type``, for example, the preferred value ``ctype`` appears before +the fallback value. The terseness of the syntax also makes for fewer lines of +code and less code to visually parse. -.. note:: +Alternatives +============ - I cheated on the last example: I renamed ``find_content_type`` to - ``find_ctype`` in order to fit two of the lines under 80 characters. If you - find this underhanded, you can go back and apply the same renaming to the - previous 2 examples. You'll find that it doesn't change the - conclusions. +Python does not have any existing ``None``-aware operators, but it does have +operators that can be used for a similar purpose. This section describes why +these alternatives are undesirable for some common ``None`` patterns. +``or`` Operator +--------------- -Usage Of ``None`` In The Standard Library ------------------------------------------ +Similar behavior can be achieved with the ``or`` operator, but ``or`` checks +whether its left operand is false-y, not specifically ``None``. This can lead +to unexpected behavior when the value is zero, an empty string, or an empty +collection:: -The previous sections show some code patterns that are claimed to be "common", -but how common are they? The attached script `find-pep505.py -`_ is meant -to answer this question. It uses the ``ast`` module to search for variations of -the following patterns in any ``*.py`` file:: - - >>> # None-coalescing if block + >>> def f(s=None): + ... s = s or [] + ... s.append(123) ... - >>> if a is None: - ... a = b + >>> my_list = [] + >>> f(my_list) + >>> my_list + [] + # Expected: [123] - >>> # [Possible] None-coalescing "or" operator - ... - >>> a or 'foo' - >>> a or [] - >>> a or {} +Rewritten using the ``None`` coalescing operator, the function could read:: - >>> # None-coalescing ternary - ... - >>> a if a is not None else b - >>> b if a is None else a + def f(s=None): + s = s ?? [] + s.append(123) - >>> # Safe navigation "and" operator - ... - >>> a and a.foo - >>> a and a['foo'] - >>> a and a.foo() +Or using the ``None``-aware attribute operator:: - >>> # Safe navigation if block - ... - >>> if a is not None: - ... a.foo() + def f(s=None): + s?.append(123) - >>> # Safe navigation ternary - ... - >>> a.foo if a is not None else b - >>> b if a is None else a.foo +(Rewriting using a conditional expression is covered in a later section.) -This script takes one or more names of Python source files to analyze:: +``getattr`` Builtin +------------------- - $ python3 find-pep505.py test.py - $ find /usr/lib/python3.4 -name '*.py' | xargs python3 find-pep505.py +Using the ``getattr`` builtin with a default value is often a suitable approach +for getting an attribute from a target that may be ``None``:: -The script prints out any matches it finds. Sample:: + client = maybe_get_client() + name = getattr(client, 'name', None) - None-coalescing if block: /usr/lib/python3.4/inspect.py:594 - if _filename is None: - _filename = getsourcefile(object) or getfile(object) +However, the semantics of this call are different from the proposed operators. +Using ``getattr`` will suppress ``AttributeError`` for misspelled or missing +attributes, even when ``client`` has a value. - [Possible] None-coalescing `or`: /usr/lib/python3.4/lib2to3/refactor.py:191 - self.explicit = explicit or [] +Spelled correctly for these semantics, this example would read:: - None-coalescing ternary: /usr/lib/python3.4/decimal.py:3909 - self.clamp = clamp if clamp is not None else dc.clamp + client = maybe_get_client() + if client is not None: + name = client.name + else: + name = None - Safe navigation `and`: /usr/lib/python3.4/weakref.py:512 - obj = info and info.weakref() +Written using the ``None``-aware attribute operator:: - Safe navigation `if` block: /usr/lib/python3.4/http/cookiejar.py:1895 - if k is not None: - lc = k.lower() - else: - lc = None + client = maybe_get_client() + name = client?.name - Safe navigation ternary: /usr/lib/python3.4/sre_parse.py:856 - literals = [None if s is None else s.encode('latin-1') for s in literals] +Ternary Operator +---------------- -.. note:: +Another common way to initialize default values is to use the ternary operator. +Here is an excerpt from the popular `Requests package +`_:: - Coalescing with ``or`` is marked as a "possible" match, because it's not - trivial to infer whether ``or`` is meant to coalesce False-y values - (correct) or if it meant to coalesce ``None`` (incorrect). On the other - hand, we assume that ``and`` is always incorrect for safe navigation. + data = [] if data is None else data + files = [] if files is None else files + headers = {} if headers is None else headers + params = {} if params is None else params + hooks = {} if hooks is None else hooks -The script has been tested against `test.py -`_ and the Python -3.4 standard library, but it should work on any arbitrary Python 3 source code. -The complete output from running it against the standard library is attached to -this proposal as `find-pep505.out -`_. +This particular formulation has the undesirable effect of putting the operands +in an unintuitive order: the brain thinks, "use ``data`` if possible and use +``[]`` as a fallback," but the code puts the fallback *before* the preferred +value. -The script counts how many matches it finds and prints the totals at the -end:: +The author of this package could have written it like this instead:: - Total None-coalescing `if` blocks: 426 - Total [possible] None-coalescing `or`: 119 - Total None-coalescing ternaries: 21 - Total Safe navigation `and`: 9 - Total Safe navigation `if` blocks: 55 - Total Safe navigation ternaries: 7 + data = data if data is not None else [] + files = files if files is not None else [] + headers = headers if headers is not None else {} + params = params if params is not None else {} + hooks = hooks if hooks is not None else {} -This is a total of 637 possible matches for these common code patterns in the -standard library. Allowing for some false positives and false negatives, it is -fair to say that these code patterns are definitely common in the standard -library. +This ordering of the operands is more intuitive, but it requires 4 extra +characters (for "not "). It also highlights the repetition of identifiers: +``data if data``, ``files if files``, etc. +When written using the ``None`` coalescing operator, the sample reads:: + + data = data ?? [] + files = files ?? [] + headers = headers ?? {} + params = params ?? {} + hooks = hooks ?? {} Rejected Ideas --------------- - -Several related ideas were discussed on python-ideas, and some of these were -roundly rejected by BDFL, the community, or both. For posterity's sake, some of -those ideas are recorded here. +============== ``None``-aware Function Call -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------- The ``None``-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might @@ -622,8 +438,13 @@ that none of the other mainstream languages that already offer this syntax have found it worthwhile to support a similar syntax for optional function invocations. +A workaround similar to that used by C# would be to write +``maybe_none?.__call__(arguments)``. If the callable is ``None``, the +expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` on its +callable type.) + ``?`` Unary Postfix Operator -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------- To generalize the ``None``-aware behavior and limit the number of new operators introduced, a unary, postfix operator spelled ``?`` was suggested. The idea is @@ -672,19 +493,15 @@ herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns. Haskell-style ``Maybe`` -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- Haskell has a concept called `Maybe `_ that encapsulates the idea of an optional value without relying on any special keyword (e.g. ``null``) or any special instance (e.g. ``None``). In Haskell, the purpose of ``Maybe`` is to avoid separate handling of "something" and nothing". -The concept is so heavily intertwined with Haskell's lazy evaluation that it -doesn't translate cleanly into Python. -There is a Python package called `pymaybe -`_ that provides a rough -approximation. The documentation shows the following example that appears -relevant to the discussion at hand:: +A Python package called `pymaybe `_ provides a +rough approximation. The documentation shows the following example:: >>> maybe('VALUE').lower() 'value' @@ -703,373 +520,41 @@ evaluation. Although ``pymaybe`` has some strengths and may be useful in its own right, it also demonstrates why a pure Python implementation of coalescing is not nearly as powerful as support built into the language. +The idea of adding a builtin ``maybe`` type to enable this scenario is rejected. -Specification -============= - -This PEP suggests 3 new operators be added to Python: - -1. ``None``-coalescing operator -2. ``None``-aware attribute access -3. ``None``-aware index access/slicing - -We will continue to assume the same spellings as in -the previous sections in order to focus on behavior before diving into the much -more contentious issue of how to spell these operators. - -A generalization of these operators is also proposed below under the heading -"Generalized Coalescing". - - -Operator Spelling +No-Value Protocol ----------------- -Despite significant support for the proposed operators, the majority of -discussion on python-ideas fixated on the spelling. Many alternative spellings -were proposed, both punctuation and keywords, but each alternative drew some -criticism. Spelling the operator as a keyword is problematic, because adding new -keywords to the language is not backwards compatible. - -It is not impossible to add a new keyword, however, and we can look at several -other PEPs for inspiration. For example, `PEP-492 -`_ introduced the new keywords -``async`` and ``await`` into Python 3.5. These new keywords are fully backwards -compatible, because that PEP also introduces a new lexical context such that -``async`` and ``await`` are only treated as keywords when used inside of an -``async def`` function. In other locations, ``async`` and ``await`` may be used -as identifiers. - -It is also possible to craft a new operator out of existing keywords, as was -the case with `PEP-308 `_, which -created a ternary operator by cobbling together the `if` and `else` keywords -into a new operator. - -In addition to the lexical acrobatics required to create a new keyword, keyword -operators are also undesirable for creating an assignment shortcut syntax. In -Dart, for example, ``x ??= y`` is an assignment shortcut that approximately -means ``x = x ?? y`` except that ``x`` is only evaluated once. If Python's -coalesce operator is a keyword, e.g. ``foo``, then the assignment shortcut would -be very ugly: ``x foo= y``. - -Spelling new logical operators with punctuation is unlikely, for several -reasons. First, Python eschews punctuation for logical operators. For example, -it uses ``not`` instead of ``!``, ``or`` instead of ``||``, and ``… if … else …`` -instead of ``… ? … : …``. - -Second, nearly every single punctuation character on a standard keyboard already -has special meaning in Python. The only exceptions are ``$``, ``!``, ``?``, and -backtick (as of Python 3). This leaves few options for a new, single-character -operator. - -Third, other projects in the Python universe assign special meaning to -punctuation. For example, `IPython -`_ assigns -special meaning to ``%``, ``%%``, ``?``, ``??``, ``$``, and ``$$``, among -others. Out of deference to those projects and the large communities using them, -introducing conflicting syntax into Python is undesirable. - -The spellings ``??`` and ``?.`` will be familiar to programmers who have seen -them in other popular programming languages. Any alternative punctuation will be -just as ugly but without the benefit of familiarity from other languages. -Therefore, this proposal spells the new operators using the same punctuation -that already exists in other languages. - - -``None``-Coalescing Operator ----------------------------- - -The ``None``-coalescing operator is a short-circuiting, binary operator that -behaves in the following way. - -1. Evaluate the left operand first. -2. If the left operand is not ``None``, then return it immediately. -3. Else, evaluate the right operand and return the result. - -Consider the following examples. We will continue to use the spelling ``??`` -here, but keep in mind that alternative spellings will be discussed below:: - - >>> 1 ?? 2 - 1 - >>> None ?? 2 - 2 - -Importantly, note that the right operand is not evaluated unless the left -operand is None:: - - >>> def err(): raise Exception('foo') - >>> 1 ?? err() - 1 - >>> None ?? err() - Traceback (most recent call last): - File "", line 1, in - File "", line 1, in err - Exception: foo - -The operator is left associative. Combined with its short circuiting behavior, -this makes the operator easy to chain:: - - >>> timeout = None - >>> local_timeout = 60 - >>> global_timeout = 300 - >>> timeout ?? local_timeout ?? global_timeout - 60 - - >>> local_timeout = None - >>> timeout ?? local_timeout ?? global_timeout - 300 - -The operator has higher precedence than the comparison operators ``==``, ``>``, -``is``, etc., but lower precedence than any bitwise or arithmetic operators. -This precedence is chosen for making "default value" expressions intuitive to -read and write:: - - >>> not None ?? True - >>> not (None ?? True) # Same precedence - - >>> 1 == None ?? 1 - >>> 1 == (None ?? 1) # Same precedence - - >>> 'foo' in None ?? ['foo', 'bar'] - >>> 'foo' in (None ?? ['foo', 'bar']) # Same precedence - - >>> 1 + None ?? 2 - >>> 1 + (None ?? 2) # Same precedence - -Recall the example above of calculating the cost of items in a shopping cart, -and the easy-to-miss bug. This type of bug is not possible with the ``None``- -coalescing operator, because there is no implicit type coersion to ``bool``:: - - >>> requested_quantity = 0 - >>> default_quantity = 1 - >>> price = 100 - >>> requested_quantity ?? default_quantity * price - 0 - -The ``None``-coalescing operator also has a corresponding assignment shortcut. -The following assignments are semantically similar, except that ``foo`` is only -looked up once when using the assignment shortcut:: - - >>> foo ??= [] - >>> foo = foo ?? [] - -The ``None`` coalescing operator improves readability, especially when handling -default function arguments. Consider again the example from the Requests -library, rewritten to use ``None``-coalescing:: - - def __init__(self, data=None, files=None, headers=None, params=None, hooks=None): - self.data = data ?? [] - self.files = files ?? [] - self.headers = headers ?? {} - self.params = params ?? {} - self.hooks = hooks ?? {} - -The operator makes the intent easier to follow (by putting operands in an -intuitive order) and is more concise than the ternary operator, while still -preserving the short circuit semantics of the code that it replaces. - - -``None``-Aware Attribute Access Operator ----------------------------------------- - -The ``None``-aware attribute access operator (also called "safe navigation") -checks its left operand. If the left operand is ``None``, then the operator -evaluates to ``None``. If the the left operand is not ``None``, then the -operator accesses the attribute named by the right operand:: - - >>> from datetime import date - >>> d = date.today() - >>> d.year - 2015 - - >>> d = None - >>> d.year - Traceback (most recent call last): - File "", line 1, in - AttributeError: 'NoneType' object has no attribute 'year' - - >>> d?.year - None - -The operator has the same precedence and associativity as the plain attribute -access operator ``.``, but this operator is also short-circuiting in a unique -way: if the left operand is ``None``, then any series of attribute access, index -access, slicing, or function call operators immediately to the right of it *are -not evaluated*:: - - >>> name = ' The Black Knight ' - >>> name.strip()[4:].upper() - 'BLACK KNIGHT' - - >>> name = None - >>> name?.strip()[4:].upper() - None - -If this operator did not short circuit in this way, then the second example -would partially evaluate ``name?.strip()`` to ``None()`` and then fail with -``TypeError: 'NoneType' object is not callable``. - -To put it another way, the following expressions are semantically similar, -except that ``name`` is only looked up once on the first line:: - - >>> name?.strip()[4:].upper() - >>> name.strip()[4:].upper() if name is not None else None - -.. note:: - - C# implements its safe navigation operators with the same short-circuiting - semantics, but Dart does not. In Dart, the second example (suitably - translated) would fail. The C# semantics are obviously superior, given the - original goal of writing common cases more concisely. The Dart semantics are - nearly useless. - -This operator short circuits one or more attribute access, index access, -slicing, or function call operators that are adjacent to its right, but it -does not short circuit any other operators (logical, bitwise, arithmetic, etc.), -nor does it escape parentheses:: - - >>> d = date.today() - >>> d?.year.numerator + 1 - 2016 - - >>> d = None - >>> d?.year.numerator + 1 - Traceback (most recent call last): - File "", line 1, in - TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' - - >>> (d?.year).numerator + 1 - Traceback (most recent call last): - File "", line 1, in - AttributeError: 'NoneType' object has no attribute 'numerator' - -Note that the error in the second example is not on the attribute access -``numerator``. In fact, that attribute access is never performed. The error -occurs when adding ``None + 1``, because the ``None``-aware attribute access -does not short circuit ``+``. - -The third example fails because the operator does not escape parentheses. In -that example, the attribute access ``numerator`` is evaluated and fails because -``None`` does not have that attribute. - -Finally, observe that short circuiting adjacent operators is not at all the same -thing as propagating ``None`` throughout an expression:: - - >>> user?.first_name.upper() - -If ``user`` is not ``None``, then ``user.first_name`` is evaluated. If -``user.first_name`` evaluates to ``None``, then ``user.first_name.upper()`` is -an error! In English, this expression says, "``user`` is optional but if it has -a value, then it must have a ``first_name``, too." - -If ``first_name`` is supposed to be optional attribute, then the expression must -make that explicit:: - - >>> user?.first_name?.upper() - -The operator is not intended as an error silencing mechanism, and it would be -undesirable if its presence infected nearby operators. - - -``None``-Aware Index Access/Slicing Operator --------------------------------------------- - -The ``None``-aware index access/slicing operator (also called "safe navigation") -is nearly identical to the ``None``-aware attribute access operator. It combines -the familiar square bracket syntax ``[]`` with new punctuation or a new keyword, -the spelling of which is discussed later:: - - >>> person = {'name': 'Mark', 'age': 32} - >>> person['name'] - 'Mark' - - >>> person = None - >>> person['name'] - Traceback (most recent call last): - File "", line 1, in - TypeError: 'NoneType' object is not subscriptable - - >>> person?.['name'] - None - -The ``None``-aware slicing operator behaves similarly:: - - >>> name = 'The Black Knight' - >>> name[4:] - 'Black Knight' - - >>> name = None - >>> name[4:] - Traceback (most recent call last): - File "", line 1, in - TypeError: 'NoneType' object is not subscriptable - - >>> name?.[4:] - None - -These operators have the same precedence as the plain index access and slicing -operators. They also have the same short-circuiting behavior as the -``None``-aware attribute access. - - -Generalized Coalescing ----------------------- - -Making ``None`` a special case is too specialized and magical. The behavior can -be generalized by making the ``None``-aware operators invoke a dunder method, -e.g. ``__coalesce__(self)`` that returns ``True`` if an object should be -coalesced and ``False`` otherwise. +These operators could be generalised to user-defined types by using a protocol +to indicate when a value represents no value. Such a protocol may be a dunder +method ``__has_value__(self)` that returns ``True`` if the value should be +treated as having a value, and ``False`` if the ``None``-aware operators should With this generalization, ``object`` would implement a dunder method equivalent to this:: - def __coalesce__(self): - return False + def __has_value__(self): + return True ``NoneType`` would implement a dunder method equivalent to this:: - def __coalesce__(self): - return True + def __has_value__(self): + return False -If this generalization is accepted, then the operators will need to be renamed -such that the term ``None`` is not used, e.g. "Coalescing Operator", "Coalesced -Member Access Operator", etc. +In the specification section, all uses of ``x is None`` would be replaces with +``not x.__has_value__()``. -The coalesce operator would invoke this dunder method. The following two -expressions are semantically similar, except `foo` is only looked up once when -using the coalesce operator:: - - >>> foo ?? bar - >>> bar if foo.__coalesce__() else foo - -The coalesced attribute and index access operators would invoke the same dunder -method:: - - >>> user?.first_name.upper() - >>> None if user.__coalesce__() else user.first_name.upper() - -This generalization allows for domain-specific ``null`` objects to be coalesced -just like ``None``. For example the ``pyasn1`` package has a type called -``Null`` that represents an ASN.1 ``null``:: +This generalization would allow for domain-specific ``null`` objects to be +coalesced just like ``None``. For example the ``pyasn1`` package has a type +called ``Null`` that represents an ASN.1 ``null``:: >>> from pyasn1.type import univ >>> univ.Null() ?? univ.Integer(123) Integer(123) -In addition to making the proposed operators less specialized, this -generalization also makes it easier to work with the Null Object Pattern, [3]_ -for those developers who prefer to avoid using ``None``. - - -Implementation --------------- - -The author of this PEP is not competent with grammars or lexers, and given the -contentiousness of this proposal, the implementation details for CPython will be -deferred until we have a clearer idea that one or more of the proposed -enhancements will be approved. - -...TBD... - +However, as ``None`` is already defined as being the value that represents +"no value", and the current specification would not preclude switching to a +protocol in the future, this idea is rejected for now. References ========== @@ -1080,15 +565,6 @@ References .. [2] A Tour of the Dart Language: Operators (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators) -.. [3] Wikipedia: Null Object Pattern - (https://en.wikipedia.org/wiki/Null_Object_pattern) - -.. [4] PEP-249: - (https://www.python.org/dev/peps/pep-0249/) - -.. [5] PEP-308 - (https://www.python.org/dev/peps/pep-0308/) - Copyright =========