iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Kill PEP 403 in light of python-ideas discussion. Resurrect PEP 3150 with first cut updates based on PEP 403 feedback

This commit is contained in:
Nick Coghlan 2011-10-16 13:10:10 +10:00
parent 48655ce245
commit 3442cf7d15
2 changed files with 340 additions and 155 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
pep-0403.txt
View File

@ -3,7 +3,7 @@ Title: Prefix syntax for post function definition operations
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Deferred
Status: Withdrawn
Type: Standards Track
Content-Type: text/x-rst
Created: 2011-10-13
@ -32,20 +32,14 @@ Namespaces) so some elements of the rationale will be familiar to readers of
that PEP. That PEP has now been withdrawn in favour of this one.
PEP Deferral
============
PEP Withdrawal
==============
Like PEP 3150, this PEP currently exists in a deferred state. Unlike PEP 3150,
this isn't because I suspect it might be a terrible idea or see nasty problems
lurking in the implementation (aside from one potential parsing issue).
Instead, it's because I think fleshing out the concept, exploring syntax
variants, creating a reference implementation and generally championing
the idea is going to require more time than I can give it in the 3.3 time
frame.
So, it's deferred. If anyone wants to step forward to drive the PEP for 3.3,
let me know and I can add you as co-author and move it to Draft status.
The python-ideas thread discussing this PEP [1]_ persuaded me that it was
essentially am unnecessarily cryptic, wholly inferior version of PEP 3150's
statement local namespaces. The discussion also resolved some of my concerns
with PEP 3150, so I am withdrawing this more limited version of the idea in
favour of resurrecting the original concept.
Basic Examples
@ -259,7 +253,7 @@ Calculating attributes without polluting the local namespace (from os.py)::
del _createenviron
# Becomes:
:environ = @()
postdef environ = def()
def _createenviron():
... # 27 line function
@ -269,14 +263,14 @@ Loop early binding::
funcs = [(lambda x, i=i: x + i) for i in range(10)]
# Becomes:
:funcs = [@(i) for i in range(10)]
postdef funcs = [def(i) for i in range(10)]
def make_incrementor(i):
return lambda x: x + i
# Or even:
:funcs = [@(i) for i in range(10)]
postdef funcs = [def(i) for i in range(10)]
def make_incrementor(i):
:return @
postdef return def
def incrementor(x):
return x + i
@ -287,15 +281,6 @@ Reference Implementation
None as yet.
TO DO
=====
Sort out links and references to everything :)
Start of python-ideas thread:
http://mail.python.org/pipermail/python-ideas/2011-October/012276.html
Acknowledgements
================
@ -303,11 +288,15 @@ Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no
idea what I was talking about in criticising Ruby's blocks, kicking off a
rather enlightening process of investigation.
Even though this PEP has been withdrawn, the process of writing and arguing
in its favour has been quite influential on the future direction of PEP 3150.
References
==========
TBD
[1] Start of python-ideas thread:
http://mail.python.org/pipermail/python-ideas/2011-October/012276.html
Copyright

432
pep-3150.txt
View File

@ -3,11 +3,11 @@ Title: Statement local namespaces (aka "given" clause)
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 2010-07-09
Python-Version: 3.3
Python-Version: 3.4
Post-History: 2010-07-14, 2011-04-21, 2011-06-13
Resolution: TBD
@ -21,46 +21,70 @@ clause will create a statement local namespace for additional names that are
accessible in the associated statement, but do not become part of the
containing namespace.
The primary motivation is to elevate ordinary assignment statements to be
on par with ``class`` and ``def`` statements where the name of the item to
be defined is presented to the reader in advance of the details of how the
value of that item is calculated.
The primary motivation is to enable a more declarative style of programming,
where the operation to be performed is presented to the reader first, and the
details of the necessary subcalculations are presented in the following
indented suite. As a key example, this would elevate ordinary assignment
statements to be on par with ``class`` and ``def`` statements where the name
of the item to be defined is presented to the reader in advance of the
details of how the value of that item is calculated. It also allows named
functions to be used in a "multi-line lambda" fashion, where the name is used
solely as a placeholder in the current expression and then defined in the
following suite.
A secondary motivation is to simplify interim calculations in module and
class level code without polluting the resulting namespaces.
There are additional emergent properties of the proposed solution which may
be of interest to some users. Most notably, it is proposed that this clause
use a new kind of scope that performs early binding of variables, potentially
replacing other techniques that achieve the same effect (such as the "default
argument hack").
The intent is that the relationship between a given clause and a separate
function definition that performs the specified operation will be similar to
the existing relationship between an explicit while loop and a generator that
produces the same sequence of operations as that while loop.
The specific proposal in this PEP has been informed by various explorations
of this and related concepts over the years (e.g. [1], [2], [3], [6]), and is
inspired to some degree by the ``where`` and ``let`` clauses in Haskell. It
avoids some problems that have been identified in past proposals, but has not
yet itself been subject to the test of implementation.
of this and related concepts over the years (e.g. [1]_, [2]_, [3]_, [6]_,
[8]_), and is inspired to some degree by the ``where`` and ``let`` clauses in
Haskell. It avoids some problems that have been identified in past proposals,
but has not yet itself been subject to the test of implementation.
PEP Deferral
============
Despite the lifting of the language moratorium (PEP 3003) for Python 3.3,
this PEP currently remains in a Deferred state. That means the PEP has to
pass at least *two* hurdles to become part of 3.3.
this PEP currently remains in a Deferred state. This idea, if implemented,
will potentially have a deep and pervasive effect on the way people write
Python code.
Firstly, I personally have to be sufficiently convinced of the PEP's value and
feasibility to return it to Draft status. While I do see merit in the concept
of statement local namespaces (otherwise I wouldn't have spent so much time
pondering the idea over the years), I also have grave doubts as to the wisdom
of actually adding it to the language (see "Key Concern" below).
When this PEP was first put forward, even I, as the PEP author, was not
convinced it was a good idea. Instead, I was simply writing it as a way to
avoid endlessly rehashing similar topics on python-ideas. When someone
broached the subject, they could be pointed at this PEP and told "Come back
when you've read and understood the arguments presented there". Subsequent
discussions (most notably, those surrounding PEP 403's attempt at a more
restricted version of the idea) have convinced me that the idea is valuable
and will help address a number of situations where developers feel that
Python "gets in the way" instead of "matching the way they think". For me,
it is this aspect of "let people express what they're thinking, rather than
forcing them to think differently due to Python's limitations" that finally
allowed the idea to clear the "status quo wins a stalemate" bar ([5]_).
Secondly, Guido van Rossum (or his delegate) will need to accept the PEP. At
the very least, that will not occur until a fully functional draft
implementation for CPython is available, and the other three major Python
implementations (PyPy, Jython, IronPython) have indicated that they consider
However, while I now think the idea is worthwhile, I don't think there is
sufficient time left in the 3.3 release cycle for the idea to mature. A
reference implementation is needed, and people need time to experiment with
that implementation and offer feedback on whether or not it helps with
programming paradigms that are currently somewhat clumsy in Python (like
callback programming). Even if a PEP co-author volunteered immediately to
work on the implementation and incorporate feedback into the PEP text, I feel
targetting 3.3 would be unnecessarily rushing things. So, I've marked this
PEP as a candidate for 3.4 rather than 3.3.
Once that process is complete, Guido van Rossum (or his delegate) will need
to be sufficiently convinced of the idea's merit and accept the PEP. Such
acceptance will require not only a fully functional reference implementation
for CPython (as already mentioned), but also indications from the other three
major Python implementations (PyPy, Jython, IronPython) that they consider
it feasible to implement the proposed semantics once they reach the point of
targetting 3.3 compatibility. Input from related projects with a vested
targetting 3.4 compatibility. Input from related projects with a vested
interest in Python's syntax (e.g. Cython) will also be valuable.
@ -69,7 +93,7 @@ Proposal
This PEP proposes the addition of an optional ``given`` clause to the
syntax for simple statements which may contain an expression, or may
substitute for such an expression for purely syntactic purposes. The
substitute for such a statement for purely syntactic purposes. The
current list of simple statements that would be affected by this
addition is as follows:
@ -87,8 +111,9 @@ The ``given`` clause would allow subexpressions to be referenced by
name in the header line, with the actual definitions following in
the indented clause. As a simple example::
c = sqrt(a*a + b*b) given:
a, b = 3, 4
sorted_data = sorted(data, key=sort_key) given:
def sort_key(item):
return item.attr1, item.attr2
The ``pass`` statement is included to provide a consistent way to skip
inclusion of a meaningful expression in the header line. While this is not
@ -150,7 +175,8 @@ interim working variables from polluting the resulting namespace.
One potentially useful way to think of the proposed clause is as a middle
ground between conventional in-line code and separation of an
operation out into a dedicated function.
operation out into a dedicated function, just as an inline while loop may
eventually be factored out into a dedicator generator.
Keyword Choice
@ -158,7 +184,7 @@ Keyword Choice
This proposal initially used ``where`` based on the name of a similar
construct in Haskell. However, it has been pointed out that there
are existing Python libraries (such as Numpy [4]) that already use
are existing Python libraries (such as Numpy [4]_) that already use
``where`` in the SQL query condition sense, making that keyword choice
potentially confusing.
@ -175,6 +201,176 @@ statement would look similar but do completely different things.
That way lies C++ and Perl :)
Anticipated Objections
======================
Two Ways To Do It
-----------------
A lot of code may now be written with values defined either before the
expression where they are used or afterwards in a ``given`` clause, creating
two ways to do it, perhaps without an obvious way of choosing between them.
On reflection, I feel this is a misapplication of the "one obvious way"
aphorism. Python already offers *lots* of ways to write code. We can use
a for loop or a while loop, a functional style or an imperative style or an
object oriented style. The language, in general, is designed to let people
write code that matches the way they think. Since different people think
differently, the way they write their code will change accordingly.
Such stylistic questions in a code base are rightly left to the development
group responsible for that code. When does an expression get so complicated
that the subexpressions should be taken out and assigned to variables, even
though those variables are only going to be used once? When should an inline
while loop be replaced with a generator that implements the same logic?
Opinions differ, and that's OK.
However, explicit PEP 8 guidance will be needed for CPython and the standard
library, and that is discussed below.
Out of Order Execution
----------------------
The ``given`` clause makes execution jump around a little strangely, as the
body of the ``given`` clause is executed before the simple statement in the
clause header. The closest any other part of Python comes to this is the out
of order evaluation in list comprehensions, generator expressions and
conditional expressions and the delayed application of decorator functions to
the function they decorate (the decorator expressions themselves are executed
in the order they are written).
While this is true, the syntax is intended for cases where people are
themselves *thinking* about a problem out of sequence (at least as far as
the language is concerned). As an example of this, consider the following
thought in the mind of a Python user:
I want to sort the items in this sequence according to the values of
attr1 and attr2 on each item.
If they're comfortable with Python's ``lambda`` expressions, then they might
choose to write it like this::
sorted_list = sorted(original, key=(lambda v: v.attr1, v.attr2))
That gets the job done, but it hardly reaches the standard of ``executable
pseudocode`` that Python aspires to, does it?
If they don't like ``lambda`` specifically, the ``operator`` module offers an
alternative that still allows the key function to be defined inline::
sorted_list = sorted(original,
key=operator.attrgetter(v. 'attr1', 'attr2'))
Again, it gets the job done, but executable pseudocode it ain't.
If they think both of the above options are ugly and confusing, or they need
logic in their key function that can't be expressed as an expression (such
as catching an exception), then Python currently forces them to reverse the
order of their original thought and define the sorting criteria first::
def sort_key(item):
return item.attr1, item.attr2
sorted_list = sorted(original, key=sort_key)
"Just define a function" has been the rote response to requests for multi-line
lambda support for years. As with the above options, it gets the job done,
but it really does represent a break between what the user is thinking and
what the language allows them to express.
I believe the proposal in this PEP will finally let Python get close to the
"executable pseudocode" bar for the kind of thought expressed above::
sorted_list = sorted(original, key=sort_key) given:
def sort_key(item):
return item.attr1, item.attr2
Everything is in the same order as it was in the user's original thought, the
only addition they have to make is to give the sorting criteria a name so that
the usage can be linked up to the subsequent definition.
One other useful note on this front, is that this PEP allows existing out of
order execution constructs to be described as special cases of the more
general out of order execution syntax (just as comprehensions are now special
cases of the more general generator expression syntax, even though list
comprehensions existed first)::
@classmethod
def classname(cls):
return cls.__name__
Would be roughly equivalent to::
classname = f1(classname) given:
f1 = classmethod
def classname(cls):
return cls.__name__
A list comprehension like ``squares = [x*x for x in range(10)]``
would be equivalent to::
# Note: this example uses an explicit early binding variant that
# isn't yet reflected in the rest of the PEP. It will get there, though.
squares = seq given outermost=range(10):
seq = []
for x in outermost:
seq.append(x*x)
Harmful to Introspection
------------------------
Poking around in module and class internals is an invaluable tool for
white-box testing and interactive debugging. The ``given`` clause will be
quite effective at preventing access to temporary state used during
calculations (although no more so than current usage of ``del`` statements
in that regard).
While this is a valid concern, design for testability is an issue that
cuts across many aspects of programming. If a component needs to be tested
independently, then a ``given`` statement should be refactored in to separate
statements so that information is exposed to the test suite. This isn't
significantly different from refactoring an operation hidden inside a
function or generator out into its own function purely to allow it to be
tested in isolation.
Lack of Real World Impact Assessment
------------------------------------
The examples in the current PEP are almost all relatively small "toy"
examples. The proposal in this PEP needs to be subjected to the test of
application to a large code base (such as the standard library or a large
Twisted application) in a search for examples where the readability of real
world code is genuinely enhanced.
This is more of a deficiency in the PEP rather than the idea, though.
New PEP 8 Guidelines
====================
As discussed on python-ideas ([7]_, [9]_) new PEP 8 guidelines would also
need to be developed to provide appropriate direction on when to use the
``given`` clause over ordinary variable assignments.
Based on the similar guidelines already present for ``try`` statements, this
PEP proposes the following additions for ``given`` statements to the
"Programming Conventions" section of PEP 8:
- for code that could reasonably be factored out into a separate function,
but is not currently reused anywhere, consider using a ``given`` clause.
This clearly indicates which variables are being used only to define
subcomponents of another statement rather than to hold algorithm or
application state.
- keep ``given`` clauses concise. If they become unwieldy, either break
them up into multiple steps or else move the details into a separate
function.
Syntax Change
=============
@ -202,8 +398,9 @@ New::
assert_stmt: 'assert' test [',' test] [given_clause]
given_clause: "given" ":" suite
(Note that expr_stmt in the grammar covers assignment and augmented
assignment in addition to simple expression statements)
(Note that ``expr_stmt`` in the grammar is a slight misnomer, as it covers
assignment and augmented assignment in addition to simple expression
statements)
The new clause is added as an optional element of the existing statements
rather than as a new kind of compound statement in order to avoid creating
@ -213,6 +410,9 @@ listed so that nonsense like the following is disallowed::
break given:
a = b = 1
import sys given:
a = b = 1
However, the precise Grammar change described above is inadequate, as it
creates problems for the definition of simple_stmt (which allows chaining of
multiple single line statements with ";" rather than "\\n").
@ -348,6 +548,7 @@ would then translate to something like the following::
However, as noted in the abstract, an actual implementation of
this idea has never been tried.
Detailed Semantics #1: Early Binding of Variable References
-----------------------------------------------------------
@ -389,17 +590,19 @@ clause. Name in outer scopes will be referenced as normal.
This intention is subject to revision based on feedback and practicalities
of implementation.
Detailed Semantics #2: Handling of ``nonlocal`` and ``global``
--------------------------------------------------------------
``nonlocal`` and ``global`` will largely operate as if the anonymous
functions were defined as in the expansion above. However, they will also
override the default early-binding semantics from names from the containing
override the default early-binding semantics for names from the containing
scope.
This intention is subject to revision based on feedback and practicalities
of implementation.
Detailed Semantics #3: Handling of ``break`` and ``continue``
-------------------------------------------------------------
@ -408,6 +611,7 @@ defined as in the expansion above. They will be syntax errors if they occur
in the ``given`` clause suite but will work normally if they appear within
a ``for`` or ``while`` loop as part of that suite.
Detailed Semantics #4: Handling of ``return`` and ``yield``
-------------------------------------------------------------
@ -416,6 +620,45 @@ suite and will be syntax errors if they occur. They will work normally if
they appear within a ``def`` statement within that suite.
Alternative Semantics for Name Binding
--------------------------------------
The "early binding" semantics proposed for the ``given`` clause are driven
by the desire to have ``given`` clauses work "normally" in class scopes (that
is, allowing them to see the local variables in the class, even though classes
do not participate in normal lexical scoping).
There is an alternative, which is to simply declare that the ``given`` clause
creates an ordinary nested scope, just like comprehensions and generator
expressions. Thus, the given clause would share the same quirks as those
constructs: they exhibit surprising behaviour at class scope, since they
can't see the local variables in the class definition. While this behaviour
is considered desirable for method definitions (where class variables are
accessed via the class or instance argument passed to the method), it can be
surprising and inconvenient for implicit scopes that are designed to hide
their own name bindings from the containing scope rather than vice-versa.
A third alternative, more analogous to the comprehension case (where the
outermost iterator expression is evaluated in the current scope and hence can
see class locals normally), would be to allow *explicit* early binding in the
``given`` clause, by passing an optional tuple of assignments after the
``given`` keyword::
# Explicit early binding via given clause
seq = []
for i in range(10):
seq.append(f) given i=i:
def f():
return i
assert [f() for f in seq] == list(range(10))
(Note: I actually like the explicit early binding idea significantly more
than I do the implicit early binding - expect a future version of the PEP
to be updated accordingly. I've already used it above when describing how
an existing construct like a list comprehension could be expressed as a
special case of the new syntax)
Examples
========
@ -426,6 +669,17 @@ Defining "one-off" classes which typically only have a single instance::
... # However many lines
public_name = public_name(*params)
# Current Python (custom decorator)
def singleton(*args, **kwds):
def decorator(cls):
return cls(*args, **kwds)
return decorator
@singleton(*params)
class public_name():
... # However many lines
public_name = public_name(*params)
# Becomes:
public_name = MeaningfulClassName(*params) given:
class MeaningfulClassName():
@ -465,57 +719,21 @@ Replacing default argument hack (from functools.lru_cache)::
# nested functions, that isn't entirely clear in the current code.
Anticipated Objections
======================
* Two Ways To Do It: a lot of code may now be written with values
defined either before the expression where they are used or
afterwards in a ``given`` clause, creating two ways to do it,
without an obvious way of choosing between them.
* Out of Order Execution: the ``given`` clause makes execution
jump around a little strangely, as the body of the ``given``
clause is executed before the simple statement in the clause
header. The closest any other part of Python comes to this
is the out of order evaluation in list comprehensions,
generator expressions and conditional expressions.
* Harmful to Introspection: poking around in module and class internals
is an invaluable tool for white-box testing and interactive debugging.
The ``given`` clause will be quite effective at preventing access to
temporary state used during calculations (although no more so than
current usage of ``del`` statements in that regard)
These objections should not be dismissed lightly - the proposal
in this PEP needs to be subjected to the test of application to
a large code base (such as the standard library) in a search
for examples where the readability of real world code is genuinely
enhanced.
New PEP 8 guidelines would also need to be developed to provide
appropriate direction on when to use the ``given`` clause over
ordinary variable assignments. Some thoughts on possible guidelines are
provided at [7]
Possible Additions
==================
* The current proposal allows the addition of a ``given`` clause only
for simple statements. Extending the idea to allow the use of
compound statements would be quite possible, but doing so raises
compound statements would be quite possible (by appending the given
clause as an independent suite at the end), but doing so raises
serious readability concerns (as values defined in the ``given``
clause may be used well before they are defined, exactly the kind
of readability trap that other features like decorators and ``with``
statements are designed to eliminate)
* Currently only the outermost clause of comprehensions and generator
expressions can reference the surrounding namespace when executed
at class level. If this proposal is implemented successfully, the
associated namespace semantics could allow that restriction to be
lifted. There would be backwards compatibility implications in doing
so as existing code may be relying on the behaviour of ignoring
class level variables, but the idea is worth considering.
* The "explicit early binding" variant may be applicable to the discussions
on python-ideas on how to eliminate the default argument hack. A ``given``
clause in the header line for functions may be the answer to that question.
Reference Implementation
@ -525,41 +743,6 @@ None as yet. If you want a crash course in Python namespace
semantics and code compilation, feel free to try ;)
Key Concern
===========
If a decision on the acceptance or rejection of this PEP had to be made
immediately, rejection would be far more likely. Unlike the previous
major syntax addition to Python (PEP 343's ``with`` statement), this
PEP as yet has no "killer application" of common code that is clearly and
obviously improved through the use of the new syntax. The ``with`` statement
(in conjunction with the generator enhancements in PEP 342) allowed
exception handling to be factored out into context managers in a way
that had never before been possible. Code using the new statement was
not only easier to read, but much easier to write correctly in the
first place.
In the case of this PEP. however, the "Two Ways to Do It" objection is a
strong one. While the ability to break out subexpresions of a statement
without having to worry about name clashes with the rest of a
function or script and without distracting from the operation that is
the ultimate aim of the statement is potentially nice to have as a
language feature, it doesn't really provide significant expressive power
over and above what is already possible by assigning subexpressions to
ordinary local variables before the statement of interest. In particular,
explaining to new Python programmers when it is best to use a ``given``
clause and when to use normal local variables is likely to be challenging
and an unnecessary distraction.
"It might be kinda, sorta, nice to have, sometimes" really isn't a strong
argument for a new syntactic construct (particularly one this complicated).
"Status quo wins a stalemate" [5] is a very useful design principle, and I'm
not yet convinced that this PEP clears that hurdle.
The case for it has definitely strengthened over time though, which is why
this PEP remains Deferred rather than Rejected.
TO-DO
=====
@ -571,19 +754,32 @@ TO-DO
References
==========
.. [1] http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
.. [1] Explicitation lines in Python:
http://mail.python.org/pipermail/python-ideas/2010-June/007476.html
.. [2] http://mail.python.org/pipermail/python-ideas/2010-July/007584.html
.. [2] 'where' statement in Python:
http://mail.python.org/pipermail/python-ideas/2010-July/007584.html
.. [3] http://mail.python.org/pipermail/python-ideas/2009-July/005132.html
.. [3] Where-statement (Proposal for function expressions):
http://mail.python.org/pipermail/python-ideas/2009-July/005132.html
.. [4] http://mail.python.org/pipermail/python-ideas/2010-July/007596.html
.. [4] Name conflict with NumPy for 'where' keyword choice:
http://mail.python.org/pipermail/python-ideas/2010-July/007596.html
.. [5] http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html
.. [5] The "Status quo wins a stalemate" design principle:
http://www.boredomandlaziness.org/2011/02/status-quo-wins-stalemate.html
.. [6] http://mail.python.org/pipermail/python-ideas/2011-April/009863.html
.. [6] Assignments in list/generator expressions:
http://mail.python.org/pipermail/python-ideas/2011-April/009863.html
.. [7] http://mail.python.org/pipermail/python-ideas/2011-April/009869.html
.. [7] Possible PEP 3150 style guidelines (#1):
http://mail.python.org/pipermail/python-ideas/2011-April/009869.html
.. [8] Discussion of PEP 403 (statement local function definition):
http://mail.python.org/pipermail/python-ideas/2011-October/012276.html
.. [9] Possible PEP 3150 style guidelines (#2):
http://mail.python.org/pipermail/python-ideas/2011-October/012341.html
Copyright
=========