194 lines
7.5 KiB
ReStructuredText
194 lines
7.5 KiB
ReStructuredText
PEP: 536
|
||
Title: Final Grammar for Literal String Interpolation
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Philipp Angerer <phil.angerer@gmail.com>
|
||
Status: Withdrawn
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 11-Dec-2016
|
||
Python-Version: 3.7
|
||
Post-History: `18-Aug-2016 <https://mail.python.org/archives/list/python-ideas@python.org/thread/FOYKXOFWEINPVQSK2XGEHKXSTEVO5WWA/>`__,
|
||
`23-Dec-2016 <https://mail.python.org/archives/list/python-ideas@python.org/thread/YKKEA5NIMMKHZTMRE5UFHST4WQ4NN3XJ/>`__,
|
||
`15-Mar-2019 <https://mail.python.org/archives/list/python-dev@python.org/thread/N43O4KNLZW4U7YZC4NVPCETZIVRDUVU2/>`__
|
||
Resolution: https://discuss.python.org/t/pep-536-should-be-marked-as-rejected/35226/4
|
||
|
||
Abstract
|
||
========
|
||
|
||
:pep:`498` introduced Literal String Interpolation (or “f-strings”).
|
||
The expression portions of those literals however are subject to
|
||
certain restrictions. This PEP proposes a formal grammar lifting
|
||
those restrictions, promoting “f-strings” to “f expressions” or f-literals.
|
||
|
||
This PEP expands upon the f-strings introduced by :pep:`498`,
|
||
so this text requires familiarity with :pep:`498`.
|
||
|
||
PEP Withdrawal
|
||
==============
|
||
|
||
This PEP has been withdrawn in favour of :pep:`701`.
|
||
:pep:`701` addresses all important points of this PEP.
|
||
|
||
|
||
Terminology
|
||
===========
|
||
|
||
This text will refer to the existing grammar as “f-strings”,
|
||
and the proposed one as “f-literals”.
|
||
|
||
Furthermore, it will refer to the ``{}``-delimited expressions in
|
||
f-literals/f-strings as “expression portions” and the static string content
|
||
around them as “string portions”.
|
||
|
||
Motivation
|
||
==========
|
||
|
||
The current implementation of f-strings in CPython relies on the existing
|
||
string parsing machinery and a post processing of its tokens. This results in
|
||
several restrictions to the possible expressions usable within f-strings:
|
||
|
||
#. It is impossible to use the quote character delimiting the f-string
|
||
within the expression portion::
|
||
|
||
>>> f'Magic wand: { bag['wand'] }'
|
||
^
|
||
SyntaxError: invalid syntax
|
||
|
||
#. A previously considered way around it would lead to escape sequences
|
||
in executed code and is prohibited in f-strings::
|
||
|
||
>>> f'Magic wand { bag[\'wand\'] } string'
|
||
SyntaxError: f-string expression portion cannot include a backslash
|
||
|
||
#. Comments are forbidden even in multi-line f-strings::
|
||
|
||
>>> f'''A complex trick: {
|
||
... bag['bag'] # recursive bags!
|
||
... }'''
|
||
SyntaxError: f-string expression part cannot include '#'
|
||
|
||
#. Expression portions need to wrap ``':'`` and ``'!'`` in braces::
|
||
|
||
>>> f'Useless use of lambdas: { lambda x: x*2 }'
|
||
SyntaxError: unexpected EOF while parsing
|
||
|
||
These limitations serve no purpose from a language user perspective and
|
||
can be lifted by giving f-literals a regular grammar without exceptions
|
||
and implementing it using dedicated parse code.
|
||
|
||
Rationale
|
||
=========
|
||
|
||
.. https://mail.python.org/pipermail/python-ideas/2016-August/041727.html
|
||
|
||
The restrictions mentioned in Motivation_ are non-obvious and counter-intuitive
|
||
unless the user is familiar with the f-literals’ implementation details.
|
||
|
||
As mentioned, a previous version of :pep:`498` allowed escape sequences
|
||
anywhere in f-strings, including as ways to encode the braces delimiting
|
||
the expression portions and in their code. They would be expanded before
|
||
the code is parsed, which would have had several important ramifications:
|
||
|
||
#. It would not be clear to human readers which portions are Expressions
|
||
and which are strings. Great material for an “obfuscated/underhanded
|
||
Python challenge”
|
||
#. Syntax highlighters are good in parsing nested grammar, but not
|
||
in recognizing escape sequences. ECMAScript 2016 (JavaScript) allows
|
||
escape sequences in its identifiers [1]_ and the author knows of no
|
||
syntax highlighter able to correctly highlight code making use of this.
|
||
|
||
As a consequence, the expression portions would be harder to recognize
|
||
with and without the aid of syntax highlighting. With the new grammar,
|
||
it is easy to extend syntax highlighters to correctly parse
|
||
and display f-literals:
|
||
|
||
.. raw:: html
|
||
|
||
<pre><span style=color:#ff5500>f'Magic wand: </span><span style=color:#3daee9>{</span>bag[<span style=color:#bf0303>'wand'</span>]<span style=color:#3daee9>:^10}</span><span style=color:#ff5500>'</span></pre>
|
||
|
||
.. This is the output of kate-syntax-highlighter when given that code
|
||
(with some quotes stripped)
|
||
|
||
Highlighting expression portions with possible escape sequences would
|
||
mean to create a modified copy of all rules of the complete expression
|
||
grammar, accounting for the possibility of escape sequences in key words,
|
||
delimiters, and all other language syntax. One such duplication would
|
||
yield one level of escaping depth and have to be repeated for a deeper
|
||
escaping in a recursive f-literal. This is the case since no highlighting
|
||
engine known to the author supports expanding escape sequences before
|
||
applying rules to a certain context. Nesting contexts however is a
|
||
standard feature of all highlighting engines.
|
||
|
||
Familiarity also plays a role: Arbitrary nesting of expressions
|
||
without expansion of escape sequences is available in every single
|
||
other language employing a string interpolation method that uses
|
||
expressions instead of just variable names. [2]_
|
||
|
||
Specification
|
||
=============
|
||
|
||
:pep:`498` specified f-strings as the following, but places restrictions on it::
|
||
|
||
f ' <text> { <expression> <optional !s, !r, or !a> <optional : format specifier> } <text> ... '
|
||
|
||
All restrictions mentioned in the PEP are lifted from f-literals,
|
||
as explained below:
|
||
|
||
#. Expression portions may now contain strings delimited with the same
|
||
kind of quote that is used to delimit the f-literal.
|
||
#. Backslashes may now appear within expressions just like anywhere else
|
||
in Python code. In case of strings nested within f-literals,
|
||
escape sequences are expanded when the innermost string is evaluated.
|
||
#. Comments, using the ``'#'`` character, are possible only in multi-line
|
||
f-literals, since comments are terminated by the end of the line
|
||
(which makes closing a single-line f-literal impossible).
|
||
#. Expression portions may contain ``':'`` or ``'!'`` wherever
|
||
syntactically valid. The first ``':'`` or ``'!'`` that is not part
|
||
of an expression has to be followed a valid coercion or format specifier.
|
||
|
||
A remaining restriction not explicitly mentioned by :pep:`498` is line breaks
|
||
in expression portions. Since strings delimited by single ``'`` or ``"``
|
||
characters are expected to be single line, line breaks remain illegal
|
||
in expression portions of single line strings.
|
||
|
||
.. note:: Is lifting of the restrictions sufficient,
|
||
or should we specify a more complete grammar?
|
||
|
||
Backwards Compatibility
|
||
=======================
|
||
|
||
f-literals are fully backwards compatible to f-strings,
|
||
and expands the syntax considered legal.
|
||
|
||
Reference Implementation
|
||
========================
|
||
|
||
TBD
|
||
|
||
References
|
||
==========
|
||
|
||
.. [1] ECMAScript ``IdentifierName`` specification
|
||
( http://ecma-international.org/ecma-262/6.0/#sec-names-and-keywords )
|
||
|
||
Yes, ``const cthulhu = { H̹̙̦̮͉̩̗̗ͧ̇̏̊̾Eͨ͆͒̆ͮ̃͏̷̮̣̫̤̣Cͯ̂͐͏̨̛͔̦̟͈̻O̜͎͍͙͚̬̝̣̽ͮ͐͗̀ͤ̍̀͢M̴̡̲̭͍͇̼̟̯̦̉̒͠Ḛ̛̙̞̪̗ͥͤͩ̾͑̔͐ͅṮ̴̷̷̗̼͍̿̿̓̽͐H̙̙̔̄͜\u0042: 42 }`` is valid ECMAScript 2016
|
||
|
||
.. [2] Wikipedia article on string interpolation
|
||
( https://en.wikipedia.org/wiki/String_interpolation )
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|