PEP 505 updates (#747)

Improve readability of PEP 505

pep-0505.rst

@@ -33,10 +33,18 @@ definitions and other language's implementations of those above. Specifically:
   if it evaluates to a value that is not ``None``, or else it evaluates and
   returns the right hand side. A coalescing ``??=`` augmented assignment
   operator is included.
-* The "``None``-aware attribute access" operator ``?.`` evaluates the complete
+* The "``None``-aware attribute access" operator ``?.`` ("maybe dot") evaluates
-  expression if the left hand side evaluates to a value that is not ``None``
+  the complete expression if the left hand side evaluates to a value that is
-* The "``None``-aware indexing" operator ``?[]`` evaluates the complete
+  not ``None``
-  expression if the left hand site evaluates to a value that is not ``None``
+* The "``None``-aware indexing" operator ``?[]`` ("maybe subscript") evaluates
+  the complete expression if the left hand site evaluates to a value that is
+  not ``None``
+
+See the `Grammar changes`_ section for specifics and examples of the required
+grammar changes.
+
+See the `Examples`_ section for more realistic examples of code that could be
+updated to use the new operators.
 
 Syntax and Semantics
 ====================
@@ -48,7 +56,7 @@ The ``None`` object denotes the lack of a value. For the purposes of these
 operators, the lack of a value indicates that the remainder of the expression
 also lacks a value and should not be evaluated.
 
-A rejected proposal was to treat any value that evaluates to false in a
+A rejected proposal was to treat any value that evaluates as "false" in a
 Boolean context as not having a value. However, the purpose of these operators
 is to propagate the "lack of value" state, rather than the "false" state.
 
@@ -56,7 +64,7 @@ Some argue that this makes ``None`` special. We contend that ``None`` is
 already special, and that using it as both the test and the result of these
 operators does not change the existing semantics in any way.
 
-See the `Rejected Ideas`_ section for discussion on the rejected approaches.
+See the `Rejected Ideas`_ section for discussions on alternate approaches.
 
 Grammar changes
 ---------------
@@ -75,11 +83,21 @@ The following rules of the Python grammar are updated to read::
               '.' NAME |
               '?.' NAME)
 
-Inserting the ``coalesce`` rule in this location ensures that expressions
+The coalesce rule
-resulting in ``None`` are natuarlly coalesced before they are used in
+*****************
-operations that would typically raise ``TypeError``. Like ``and`` and ``or``
+
-the right-hand expression is not evaluated until the left-hand side is
+The ``coalesce`` rule provides the ``??`` binary operator. Unlike most binary
-determined to be ``None``. For example::
+operators, the right-hand side is not evaulated until the left-hand side is
+determined to be ``None``.
+
+The ``??`` operator binds more tightly than other binary operators as most
+existing implementations of these do not propagate ``None``s (they will
+typically raise ``TypeError``). Expressions that are known to potentially
+result in ``None`` can be substituted for a default value without needing
+additional parentheses.
+
+Some examples of how implicit parentheses are placed when evaluating operator
+precedence in the presence of the ``??`` operator.
 
     a, b = None, None
     def c(): return None
@@ -92,9 +110,17 @@ determined to be ``None``. For example::
     (True ?? ex()) == True
     (c ?? ex)() == c()
 
-Augmented coalescing assignment only rebinds the name if its current value is
+Particularly for cases such as ``a ?? 2 ** b ?? 3``, parenthesizing the
-``None``. If the target name already has a value, the right-hand side is not
+sub-expressions any other way would result in ``TypeError``, as ``int.__pow__``
-evaluated. For example::
+cannot be called with ``None`` (and the fact that the ``??`` operator is used
+at all implies that ``a`` or ``b`` may be ``None``). However, as usual,
+while parentheses are not required they should be added if it helps improve
+readability.
+
+An augmented assignment for the ``??`` operator is also added. Augmented
+coalescing assignment only rebinds the name if its current value is ``None``.
+If the target name already has a value, the right-hand side is not evaluated.
+For example::
 
     a = None
     b = ''
@@ -106,13 +132,16 @@ evaluated. For example::
 
     assert a == 'value'
     assert b == ''
-    assert c == '0' and any(os.scandir('/'))
+    assert c == 0 and any(os.scandir('/'))
 
-Adding new trailers for the other ``None``-aware operators ensures that they
+The maybe-dot and maybe-subscript operators
-may be used in all valid locations for the existing equivalent operators,
+*******************************************
-including as part of an assignment target (more details below). As the existing
+
-evaluation rules are not directly embedded in the grammar, we specify the
+The maybe-dot and maybe-subscript operators are added as trailers for atoms,
-required changes here.
+so that they may be used in all the same locations as the regular operators,
+including as part of an assignment target (more details below). As the
+existing evaluation rules are not directly embedded in the grammar, we specify
+the required changes below.
 
 Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is
 then evaluated from left to right, applying its own parameter (either its
@@ -174,6 +203,28 @@ though unlikely to be useful unless combined with a coalescing operation::
 
     (a?.b ?? d).c = 1
 
+Reading expressions
+-------------------
+
+For the maybe-dot and maybe-subscript operators, the intention is that
+expressions including these operators should be read and interpreted as for the
+regular versions of these operators. In "normal" cases, the end results are
+going to be identical between an expression such as ``a?.b?[c]`` and
+``a.b[c]``, and just as we do not currently read "a.b" as "read attribute b
+from a *if it has an attribute a or else it raises AttributeError*", there is
+no need to read "a?.b" as "read attribute b from a *if a is not None*"
+(unless in a context where the listener needs to be aware of the specific
+behaviour).
+
+For coalescing expressions using the ``??`` operator, expressions should either
+be read as "or ... if None" or "coalesced with". For example, the expression
+``a.get_value() ?? 100`` would be read "call a dot get_value or 100 if None",
+or "call a dot get_value coalesced with 100".
+
+.. note::
+   Reading code in spoken text is always lossy, and so we make no attempt to
+   define an unambiguous way of speaking these operators. These suggestions
+   are intended to add context to the implications of adding the new syntax.
 
 Examples
 ========
@@ -227,35 +278,67 @@ After updating to use the ``??`` operator::
     optdict = dict(encoding=encoding ?? sys.getdefaultencoding(),
                    css=options.css)
 
-From ``dis.py``::
+From ``email/generator.py`` (and importantly note that there is no way to
+substitute ``or`` for ``??`` in this situation)::
 
-    def _get_const_info(const_index, const_list):
+    mangle_from_ = True if policy is None else policy.mangle_from_
-        argval = const_index
-        if const_list is not None:
-            argval = const_list[const_index]
-        return argval, repr(argval)
 
-After updating to use the ``?[]`` and ``??`` operators::
+After updating::
 
-    def _get_const_info(const_index, const_list):
+    mangle_from_ = policy?.mangle_from_ ?? True
-        argval = const_list?[const_index] ?? const_index
-        return argval, repr(argval)
 
-From ``inspect.py``::
 
-    for base in object.__bases__:
+From ``asyncio/subprocess.py``::
-        for name in getattr(base, "__abstractmethods__", ()):
-            value = getattr(object, name, None)
-            if getattr(value, "__isabstractmethod__", False):
-                return True
 
-After updating to use the ``?.`` operator (and deliberately not converting to
+    def pipe_data_received(self, fd, data):
-use ``any()``)::
+        if fd == 1:
+            reader = self.stdout
+        elif fd == 2:
+            reader = self.stderr
+        else:
+            reader = None
+        if reader is not None:
+            reader.feed_data(data)
+
+After updating to use the ``?.`` operator::
+
+    def pipe_data_received(self, fd, data):
+        if fd == 1:
+            reader = self.stdout
+        elif fd == 2:
+            reader = self.stderr
+        else:
+            reader = None
+        reader?.feed_data(data)
+
+
+From ``asyncio/tasks.py``::
+
+    try:
+        await waiter
+    finally:
+        if timeout_handle is not None:
+            timeout_handle.cancel()
+
+After updating to use the ``?.`` operator::
+
+    try:
+        await waiter
+    finally:
+        timeout_handle?.cancel()
+
+
+From ``ctypes/_aix.py``::
+
+    if libpaths is None:
+        libpaths = []
+    else:
+        libpaths = libpaths.split(":")
+
+After updating::
+
+    libpaths = libpaths?.split(":") ?? []
 
-    for base in object.__bases__:
-        for name in base?.__abstractmethods__ ?? ():
-            if object?.name?.__isabstractmethod__:
-                return True
 
 From ``os.py``::
 
@@ -275,6 +358,43 @@ After updating to use the ``?.`` operator::
         nondirs.append(name)
 
 
+From ``importlib/abc.py``::
+
+    def find_module(self, fullname, path):
+        if not hasattr(self, 'find_spec'):
+            return None
+        found = self.find_spec(fullname, path)
+        return found.loader if found is not None else None
+
+After partially updating::
+
+    def find_module(self, fullname, path):
+        if not hasattr(self, 'find_spec'):
+            return None
+        return self.find_spec(fullname, path)?.loader
+
+After extensive updating (arguably excessive, though that's for the style
+guides to determine)::
+
+    def find_module(self, fullname, path):
+        return getattr(self, 'find_spec', None)?.__call__(fullname, path)?.loader
+
+
+From ``dis.py``::
+
+    def _get_const_info(const_index, const_list):
+        argval = const_index
+        if const_list is not None:
+            argval = const_list[const_index]
+        return argval, repr(argval)
+
+After updating to use the ``?[]`` and ``??`` operators::
+
+    def _get_const_info(const_index, const_list):
+        argval = const_list?[const_index] ?? const_index
+        return argval, repr(argval)
+
+
 jsonify
 -------
 

pep-0505/find-pep505.py

@@ -8,7 +8,10 @@ Example usage:
 '''
 
 import ast
+import glob
+import os
 import sys
+import tokenize
 
 
 class NoneCoalesceIfBlockVisitor(ast.NodeVisitor):
@@ -379,15 +382,16 @@ def log(text, file_, start_line, stop_line=None):
     if stop_line is None:
         stop_line = start_line
 
-    with open(file_) as source:
+    try:
+        source = tokenize.open(file_)
+    except (SyntaxError, UnicodeDecodeError):
+        return
+
+    with source:
         print(''.join(source.readlines()[start_line-1:stop_line]))
 
 
 def main():
-    if len(sys.argv) < 2:
-        sys.stderr.write('Usage: python3 parse.py <files>\n')
-        sys.exit(1)
-
     def make_callback(text):
         return count_calls_decorator(
             lambda file_, start, stop: log(text, file_, start, stop)
@@ -400,8 +404,24 @@ def main():
     sni_callback = make_callback('Safe navigation `if` block')
     snt_callback = make_callback('Safe navigation ternary')
 
-    for file_ in sys.argv[1:]:
+    files = sys.argv[1:]
-        with open(file_) as source:
+    if files:
+        expanded_files = []
+        for file_ in files:
+            if '*' in file_:
+                expanded_files.extend(glob.glob(file_))
+            else:
+                expanded_files.append(file_)
+    else:
+        files = glob.glob(os.path.join(sys.prefix, 'Lib', '**', '*.py'))
+
+    for file_ in files:
+        try:
+            source = tokenize.open(file_)
+        except (SyntaxError, UnicodeDecodeError):
+            continue
+
+        with source:
             try:
                 tree = ast.parse(source.read(), filename=file_)
             except SyntaxError: