diff --git a/pep-0505.rst b/pep-0505.rst index 41cb7c042..3c0fd188c 100644 --- a/pep-0505.rst +++ b/pep-0505.rst @@ -31,7 +31,8 @@ definitions and other language's implementations of those above. Specifically: * The "``None`` coalescing`` binary operator ``??`` returns the left hand side if it evaluates to a value that is not ``None``, or else it evaluates and - returns the right hand side. + returns the right hand side. A coalescing ``??=`` augmented assignment + operator is included. * The "``None``-aware attribute access" operator ``?.`` evaluates the complete expression if the left hand side evaluates to a value that is not ``None`` * The "``None``-aware indexing" operator ``?[]`` evaluates the complete @@ -55,6 +56,8 @@ Some argue that this makes ``None`` special. We contend that ``None`` is already special, and that using it as both the test and the result of these operators does not change the existing semantics in any way. +See the `Rejected Ideas`_ section for discussion on the rejected approaches. + Grammar changes --------------- @@ -95,12 +98,15 @@ evaluated. For example:: a = None b = '' + c = 0 a ??= 'value' b ??= undefined_name + c ??= shutil.rmtree('/') # don't try this at home, kids assert a == 'value' assert b == '' + assert c == '0' and any(os.scandir('/')) Adding new trailers for the other ``None``-aware operators ensures that they may be used in all valid locations for the existing equivalent operators, @@ -172,15 +178,109 @@ though unlikely to be useful unless combined with a coalescing operation:: Examples ======== -This section presents some examples of common ``None`` patterns and explains -the drawbacks. +This section presents some examples of common ``None`` patterns and shows what +conversion to use ``None``-aware operators may look like. + +Standard Library +---------------- + +Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 standard +library discovered up to 678 code snippets that could be replaced with use of +one of the ``None``-aware operators:: + + $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py + + Total None-coalescing `if` blocks: 449 + Total [possible] None-coalescing `or`: 120 + Total None-coalescing ternaries: 27 + Total Safe navigation `and`: 13 + Total Safe navigation `if` blocks: 61 + Total Safe navigation ternaries: 8 + +Some of these are shown below as examples before and after converting to use the +new operators. + +From ``bisect.py``:: + + def insort_right(a, x, lo=0, hi=None): + # ... + if hi is None: + hi = len(a) + # ... + +After updating to use the ``??=`` augmented assignment statement:: + + def insort_right(a, x, lo=0, hi=None): + # ... + hi ??= len(a) + # ... + +From ``calendar.py``:: + + encoding = options.encoding + if encoding is None: + encoding = sys.getdefaultencoding() + optdict = dict(encoding=encoding, css=options.css) + +After updating to use the ``??`` operator:: + + optdict = dict(encoding=encoding ?? sys.getdefaultencoding(), + css=options.css) + +From ``dis.py``:: + + def _get_const_info(const_index, const_list): + argval = const_index + if const_list is not None: + argval = const_list[const_index] + return argval, repr(argval) + +After updating to use the ``?[]`` and ``??`` operators:: + + def _get_const_info(const_index, const_list): + argval = const_list?[const_index] ?? const_index + return argval, repr(argval) + +From ``inspect.py``:: + + for base in object.__bases__: + for name in getattr(base, "__abstractmethods__", ()): + value = getattr(object, name, None) + if getattr(value, "__isabstractmethod__", False): + return True + +After updating to use the ``?.`` operator (and deliberately not converting to +use ``any()``):: + + for base in object.__bases__: + for name in base?.__abstractmethods__ ?? (): + if object?.name?.__isabstractmethod__: + return True + +From ``os.py``:: + + if entry.is_dir(): + dirs.append(name) + if entries is not None: + entries.append(entry) + else: + nondirs.append(name) + +After updating to use the ``?.`` operator:: + + if entry.is_dir(): + dirs.append(name) + entries?.append(entry) + else: + nondirs.append(name) + jsonify ------- -This first example is from a Python web crawler that uses the popular Flask -framework as a front-end. This function retrieves information about a web site -from a SQL database and formats it as JSON to send to an HTTP client:: +This example is from a Python web crawler that uses the Flask framework as its +front-end. This function retrieves information about a web site from a SQL +database and formats it as JSON to send to an HTTP client:: class SiteView(FlaskView): @route('/site/', methods=['GET']) @@ -200,12 +300,12 @@ database, and they are also allowed to be ``null`` in the JSON response. JSON does not have a native way to represent a ``datetime``, so the server's contract states that any non-``null`` date is represented as an ISO-8601 string. -However, without knowing the exact semantics of the ``first_seen`` and -``last_seen`` attributes, it is impossible to know whether the attribute can -be safely or performantly accessed multiple times. +Without knowing the exact semantics of the ``first_seen`` and ``last_seen`` +attributes, it is impossible to know whether the attribute can be safely or +performantly accessed multiple times. -One way to fix this code is to replace each ternary with explicit value -assignment and a full ``if/else`` block:: +One way to fix this code is to replace each conditional expression with an +explicit value assignment and a full ``if``/``else`` block:: class SiteView(FlaskView): @route('/site/', methods=['GET']) @@ -233,9 +333,9 @@ assignment and a full ``if/else`` block:: ) This adds ten lines of code and four new code paths to the function, -dramatically increasing the apparently complexity. Rewriting using -``None``-aware attribute operator on the other hand, results in shorter -code:: +dramatically increasing the apparent complexity. Rewriting using the +``None``-aware attribute operator results in shorter code with more clear +intent:: class SiteView(FlaskView): @route('/site/', methods=['GET']) @@ -290,7 +390,7 @@ ab/upload.py>`_:: self.content_type = content_type This example contains several good examples of needing to provide default -values. Rewriting to use a conditional expression reduces the overall lines of +values. Rewriting to use conditional expressions reduces the overall lines of code, but does not necessarily improve readability:: class BaseUploadObject(object): @@ -339,117 +439,148 @@ Rewriting using the ``None`` coalescing operator:: self.filename = filename ?? os.path.split(path)[1] self.content_type = content_type ?? self.find_content_type(self.filename) -This syntax has an intuitive ordering of the operands. In -``find_content_type``, for example, the preferred value ``ctype`` appears before -the fallback value. The terseness of the syntax also makes for fewer lines of -code and less code to visually parse. +This syntax has an intuitive ordering of the operands. In ``find_content_type``, +for example, the preferred value ``ctype`` appears before the fallback value. +The terseness of the syntax also makes for fewer lines of code and less code to +visually parse, and reading from left-to-right and top-to-bottom more accurately +follows the execution flow. -Alternatives -============ - -Python does not have any existing ``None``-aware operators, but it does have -operators that can be used for a similar purpose. This section describes why -these alternatives are undesirable for some common ``None`` patterns. - -``or`` Operator ---------------- - -Similar behavior can be achieved with the ``or`` operator, but ``or`` checks -whether its left operand is false-y, not specifically ``None``. This can lead -to unexpected behavior when the value is zero, an empty string, or an empty -collection:: - - >>> def f(s=None): - ... s = s or [] - ... s.append(123) - ... - >>> my_list = [] - >>> f(my_list) - >>> my_list - [] - # Expected: [123] - -Rewritten using the ``None`` coalescing operator, the function could read:: - - def f(s=None): - s = s ?? [] - s.append(123) - -Or using the ``None``-aware attribute operator:: - - def f(s=None): - s?.append(123) - -(Rewriting using a conditional expression is covered in a later section.) - -``getattr`` Builtin -------------------- - -Using the ``getattr`` builtin with a default value is often a suitable approach -for getting an attribute from a target that may be ``None``:: - - client = maybe_get_client() - name = getattr(client, 'name', None) - -However, the semantics of this call are different from the proposed operators. -Using ``getattr`` will suppress ``AttributeError`` for misspelled or missing -attributes, even when ``client`` has a value. - -Spelled correctly for these semantics, this example would read:: - - client = maybe_get_client() - if client is not None: - name = client.name - else: - name = None - -Written using the ``None``-aware attribute operator:: - - client = maybe_get_client() - name = client?.name - -Ternary Operator ----------------- - -Another common way to initialize default values is to use the ternary operator. -Here is an excerpt from the popular `Requests package -`_:: - - data = [] if data is None else data - files = [] if files is None else files - headers = {} if headers is None else headers - params = {} if params is None else params - hooks = {} if hooks is None else hooks - -This particular formulation has the undesirable effect of putting the operands -in an unintuitive order: the brain thinks, "use ``data`` if possible and use -``[]`` as a fallback," but the code puts the fallback *before* the preferred -value. - -The author of this package could have written it like this instead:: - - data = data if data is not None else [] - files = files if files is not None else [] - headers = headers if headers is not None else {} - params = params if params is not None else {} - hooks = hooks if hooks is not None else {} - -This ordering of the operands is more intuitive, but it requires 4 extra -characters (for "not "). It also highlights the repetition of identifiers: -``data if data``, ``files if files``, etc. - -When written using the ``None`` coalescing operator, the sample reads:: - - data = data ?? [] - files = files ?? [] - headers = headers ?? {} - params = params ?? {} - hooks = hooks ?? {} Rejected Ideas ============== +The first three ideas in this section are oft-proposed alternatives to treating +``None`` as special. For further background on why these are rejected, see their +treatment in `PEP 531 `_ and +`PEP 532 `_ and the associated +discussions. + +No-Value Protocol +----------------- + +The operators could be generalised to user-defined types by defining a protocol +to indicate when a value represents "no value". Such a protocol may be a dunder +method ``__has_value__(self)` that returns ``True`` if the value should be +treated as having a value, and ``False`` if the value should be treated as no +value. + +With this generalization, ``object`` would implement a dunder method equivalent +to this:: + + def __has_value__(self): + return True + +``NoneType`` would implement a dunder method equivalent to this:: + + def __has_value__(self): + return False + +In the specification section, all uses of ``x is None`` would be replaced with +``not x.__has_value__()``. + +This generalization would allow for domain-specific "no-value" objects to be +coalesced just like ``None``. For example the ``pyasn1`` package has a type +called ``Null`` that represents an ASN.1 ``null``:: + + >>> from pyasn1.type import univ + >>> univ.Null() ?? univ.Integer(123) + Integer(123) + +Similarly, values such as ``math.nan`` and ``NotImplemented`` could be treated +as representing no value. + +However, the "no-value" nature of these values is domain-specific, which means +they *should* be treated as a value by the language. For example, +``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting +``math.nan?.imag`` to return ``math.nan`` would be incorrect. + +As ``None`` is already defined by the language as being the value that +represents "no value", and the current specification would not preclude +switching to a protocol in the future (though changes to built-in objects would +not be compatible), this idea is rejected for now. + +Boolean-aware operators +----------------------- + +This suggestion is fundamentally the same as adding a no-value protocol, and so +the discussion above also applies. + +Similar behavior to the ``??`` operator can be achieved with an ``or`` +expression, however ``or`` checks whether its left operand is false-y and not +specifically ``None``. This approach is attractive, as it requires fewer changes +to the language, but ultimately does not solve the underlying problem correctly. + +Assuming the check is for truthiness rather than ``None``, there is no longer a +need for the ``??`` operator. However, applying this check to the ``?.`` and +``?[]`` operators prevents perfectly valid operations applying + +Consider the following example, where ``get_log_list()`` may return either a +list containing current log messages (potentially empty), or ``None`` if logging +is not enabled:: + + lst = get_log_list() + lst?.append('A log message') + +If ``?.`` is checking for true values rather than specifically ``None`` and the +log has not been initialized with any items, no item will ever be appended. This +violates the obvious intent of the code, which is to append an item. The +``append`` method is available on an empty list, as are all other list methods, +and there is no reason to assume that these members should not be used because +the list is presently empty. + +Further, there is no sensible result to use in place of the expression. A +normal ``lst.append`` returns ``None``, but under this idea ``lst?.append`` may +result in either ``[]`` or ``None``, depending on the value of ``lst``. As with +the examples in the previous section, this makes no sense. + +As checking for truthiness rather than ``None`` results in apparently valid +expressions no longer executing as intended, this idea is rejected. + +Exception-aware operators +------------------------- + +Arguably, the reason to short-circuit an expression when ``None`` is encountered +is to avoid the ``AttributeError`` or ``TypeError`` that would be raised under +normal circumstances. As an alternative to testing for ``None``, the ``?.`` and +``?[]`` operators could instead handle ``AttributeError`` and ``TypeError`` +raised by the operation and skip the remainder of the expression. + +This produces a transformation for ``a?.b.c?.d.e`` similar to this:: + + _v = a + try: + _v = _v.b + except AttributeError: + pass + else: + _v = _v.c + try: + _v = _v.d + except AttributeError: + pass + else: + _v = _v.e + +One open question is which value should be returned as the expression when an +exception is handled. The above example simply leaves the partial result, but +this is not helpful for replacing with a default value. An alternative would be +to force the result to ``None``, which then raises the question as to why +``None`` is special enough to be the result but not special enough to be the +test. + +Secondly, this approach masks errors within code executed implicitly as part of +the expression. For ``?.``, any ``AttributeError`` within a property or +``__getattr__`` implementation would be hidden, and similarly for ``?[]`` and +``__getitem__`` implementations. + +Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed. + +Existing conventions for handling these kinds of errors in the form of the +``getattr`` builtin and the ``.get(key, default)`` method pattern established by +``dict`` show that it is already possible to explicitly use this behaviour. + +As this approach would hide errors in code, it is rejected. + ``None``-aware Function Call ---------------------------- @@ -457,7 +588,7 @@ The ``None``-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might be written as ``foo?()``, where ``foo`` is only called if it is not None. -This has been rejected on the basis of the proposed operators being intended +This has been deferred on the basis of the proposed operators being intended to aid traversal of partially populated hierarchical data structures, *not* for traversal of arbitrary class hierarchies. This is reflected in the fact that none of the other mainstream languages that already offer this syntax @@ -518,8 +649,8 @@ This degree of generalization is not useful. The operators actually proposed herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns. -Haskell-style ``Maybe`` ------------------------ +Built-in ``maybe`` +------------------ Haskell has a concept called `Maybe `_ that encapsulates the idea of an optional value without relying on any special @@ -548,39 +679,45 @@ not nearly as powerful as support built into the language. The idea of adding a builtin ``maybe`` type to enable this scenario is rejected. -No-Value Protocol ------------------ +Just use a conditional expression +--------------------------------- -These operators could be generalised to user-defined types by using a protocol -to indicate when a value represents no value. Such a protocol may be a dunder -method ``__has_value__(self)` that returns ``True`` if the value should be -treated as having a value, and ``False`` if the ``None``-aware operators should +Another common way to initialize default values is to use the ternary operator. +Here is an excerpt from the popular `Requests package +`_:: -With this generalization, ``object`` would implement a dunder method equivalent -to this:: + data = [] if data is None else data + files = [] if files is None else files + headers = {} if headers is None else headers + params = {} if params is None else params + hooks = {} if hooks is None else hooks - def __has_value__(self): - return True +This particular formulation has the undesirable effect of putting the operands +in an unintuitive order: the brain thinks, "use ``data`` if possible and use +``[]`` as a fallback," but the code puts the fallback *before* the preferred +value. -``NoneType`` would implement a dunder method equivalent to this:: +The author of this package could have written it like this instead:: - def __has_value__(self): - return False + data = data if data is not None else [] + files = files if files is not None else [] + headers = headers if headers is not None else {} + params = params if params is not None else {} + hooks = hooks if hooks is not None else {} -In the specification section, all uses of ``x is None`` would be replaces with -``not x.__has_value__()``. +This ordering of the operands is more intuitive, but it requires 4 extra +characters (for "not "). It also highlights the repetition of identifiers: +``data if data``, ``files if files``, etc. -This generalization would allow for domain-specific ``null`` objects to be -coalesced just like ``None``. For example the ``pyasn1`` package has a type -called ``Null`` that represents an ASN.1 ``null``:: +When written using the ``None`` coalescing operator, the sample reads:: - >>> from pyasn1.type import univ - >>> univ.Null() ?? univ.Integer(123) - Integer(123) + data = data ?? [] + files = files ?? [] + headers = headers ?? {} + params = params ?? {} + hooks = hooks ?? {} -However, as ``None`` is already defined as being the value that represents -"no value", and the current specification would not preclude switching to a -protocol in the future, this idea is rejected for now. References ========== @@ -591,6 +728,8 @@ References .. [2] A Tour of the Dart Language: Operators (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators) +.. [3] Associated scripts + (https://github.com/python/peps/tree/master/pep-0505/) Copyright =========