243 lines
10 KiB
Plaintext
243 lines
10 KiB
Plaintext
|
PEP: 289
|
|||
|
|
Title: Generator Comprehensions
|
|||
|
|
Version: $Revision$
|
|||
|
|
Last-Modified: $Date$
|
|||
|
|
Author: python@rcn.com (Raymond D. Hettinger)
|
|||
|
|
Status: Rejected
|
|||
|
|
Type: Standards Track
|
|||
|
|
Created: 30-Jan-2002
|
|||
|
|
Python-Version: 2.3
|
|||
|
|
Post-History:
|
|||
|
|
|
|||
|
|
|
|||
|
|
Abstract
|
|||
|
|
|
|||
|
|
This PEP introduces generator comprehensions as an idea for
|
|||
|
|
enhancing the generators introduced in Python version 2.2 [1].
|
|||
|
|
The goal is to increase the convenience, utility, and power of
|
|||
|
|
generators by making it easy to convert a list comprehension into
|
|||
|
|
a generator.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Rationale
|
|||
|
|
|
|||
|
|
Python 2.2 introduced the concept of an iterable interface as
|
|||
|
|
proposed in PEP 234 [4]. The iter() factory function was provided
|
|||
|
|
as common calling convention and deep changes were made to use
|
|||
|
|
iterators as a unifying theme throughout Python. The unification
|
|||
|
|
came in the form of establishing a common iterable interface for
|
|||
|
|
mappings, sequences, and file objects.
|
|||
|
|
|
|||
|
|
Generators, as proposed in PEP 255 [1], were introduced as a means
|
|||
|
|
for making it easier to create iterators, especially ones with
|
|||
|
|
complex internal execution or variable states. When I created new
|
|||
|
|
programs, generators were often the tool of choice for creating an
|
|||
|
|
iterator.
|
|||
|
|
|
|||
|
|
However, when updating existing programs, I found that the tool
|
|||
|
|
had another use, one that improved program function as well as
|
|||
|
|
structure. Some programs exhibited a pattern of creating large
|
|||
|
|
lists and then looping over them. As data sizes increased, the
|
|||
|
|
programs encountered scalability limitations owing to excessive
|
|||
|
|
memory consumption (and malloc time) for the intermediate lists.
|
|||
|
|
Generators were found to be directly substitutable for the lists
|
|||
|
|
while eliminating the memory issues through lazy evaluation
|
|||
|
|
a.k.a. just in time manufacturing.
|
|||
|
|
|
|||
|
|
Python itself encountered similar issues. As a result, xrange()
|
|||
|
|
and xreadlines() were introduced. And, in the case of file
|
|||
|
|
objects and mappings, just-in-time evaluation became the norm.
|
|||
|
|
Generators provide a tool to program memory conserving for-loops
|
|||
|
|
whenever complete evaluation is not desired because of memory
|
|||
|
|
restrictions or availability of data.
|
|||
|
|
|
|||
|
|
The next step in the evolution of generators is to establish a
|
|||
|
|
generator alternative to list comprehensions [3]. This
|
|||
|
|
alternative provides a simple way to convert a list comprehension
|
|||
|
|
into a generator whenever memory issues arise.
|
|||
|
|
|
|||
|
|
This suggestion is designed to take advantage of the existing
|
|||
|
|
implementation and require little additional effort to
|
|||
|
|
incorporate. It is backward compatible and requires no new
|
|||
|
|
keywords.
|
|||
|
|
|
|||
|
|
|
|||
|
|
BDFL Pronouncements
|
|||
|
|
|
|||
|
|
Generator comprehensions are REJECTED. The rationale is that the
|
|||
|
|
benefits are marginal since generators can already be coded
|
|||
|
|
directly and the costs are high because implementation and
|
|||
|
|
maintenance require major efforts with the parser.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Reference Implementation
|
|||
|
|
|
|||
|
|
There is not currently a CPython implementation; however, a
|
|||
|
|
simulation module written in pure Python is available on
|
|||
|
|
SourceForge [5]. The simulation is meant to allow direct
|
|||
|
|
experimentation with the proposal.
|
|||
|
|
|
|||
|
|
There is also a module [6] with working source code for all of the
|
|||
|
|
examples used in this PEP. It serves as a test suite for the
|
|||
|
|
simulator and it documents how each the new feature works in
|
|||
|
|
practice.
|
|||
|
|
|
|||
|
|
The authors and implementers of PEP 255 [1] were contacted to
|
|||
|
|
provide their assessment of whether these enhancements were going
|
|||
|
|
to be straight-forward to implement and require only minor
|
|||
|
|
modification of the existing generator code. Neil felt the
|
|||
|
|
assertion was correct. Ka-Ping thought so also. GvR said he
|
|||
|
|
could believe that it was true. Later GvR re-assessed and thought
|
|||
|
|
that it would be difficult to tweak the code generator to produce
|
|||
|
|
a separate object. Tim did not have an opportunity to give an
|
|||
|
|
assessment.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Specification for Generator Comprehensions :
|
|||
|
|
|
|||
|
|
If a list comprehension starts with a 'yield' keyword, then
|
|||
|
|
express the comprehension with a generator. For example:
|
|||
|
|
|
|||
|
|
g = [yield (len(line),line) for line in file if len(line)>5]
|
|||
|
|
|
|||
|
|
This would be implemented as if it had been written:
|
|||
|
|
|
|||
|
|
def __temp(self):
|
|||
|
|
for line in file:
|
|||
|
|
if len(line) > 5:
|
|||
|
|
yield (len(line), line)
|
|||
|
|
g = __temp()
|
|||
|
|
|
|||
|
|
Note A: There is some discussion about whether the enclosing
|
|||
|
|
brackets should be part of the syntax for generator
|
|||
|
|
comprehensions. On the plus side, it neatly parallels list
|
|||
|
|
comprehensions and would be immediately recognizable as a similar
|
|||
|
|
form with similar internal syntax (taking maximum advantage of
|
|||
|
|
what people already know). More importantly, it sets off the
|
|||
|
|
generator comprehension from the rest of the function so as to not
|
|||
|
|
suggest that the enclosing function is a generator (currently the
|
|||
|
|
only cue that a function is really a generator is the presence of
|
|||
|
|
the yield keyword). On the minus side, the brackets may falsely
|
|||
|
|
suggest that the whole expression returns a list. Most of the
|
|||
|
|
feedback received to date indicates that brackets are helpful and
|
|||
|
|
not misleading. Unfortunately, the one dissent is from GvR.
|
|||
|
|
|
|||
|
|
A key advantage of the generator comprehension syntax is that it
|
|||
|
|
makes it trivially easy to transform existing list comprehension
|
|||
|
|
code to a generator by adding yield. Likewise, it can be
|
|||
|
|
converted back to a list by deleting yield. This makes it easy to
|
|||
|
|
scale-up programs from small datasets to ones large enough to
|
|||
|
|
warrant just in time evaluation.
|
|||
|
|
|
|||
|
|
Note B: List comprehensions expose their looping variable and
|
|||
|
|
leave that variable in the enclosing scope. The code, [str(i) for
|
|||
|
|
i in range(8)] leaves 'i' set to 7 in the scope where the
|
|||
|
|
comprehension appears. This behavior is by design and reflects an
|
|||
|
|
intent to duplicate the result of coding a for-loop instead of a
|
|||
|
|
list comprehension. Further, the variable 'i' is in a defined and
|
|||
|
|
potentially useful state on the line immediately following the
|
|||
|
|
list comprehension.
|
|||
|
|
|
|||
|
|
In contrast, generator comprehensions do not expose the looping
|
|||
|
|
variable to the enclosing scope. The code, [yield str(i) for i in
|
|||
|
|
range(8)] leaves 'i' untouched in the scope where the
|
|||
|
|
comprehension appears. This is also by design and reflects an
|
|||
|
|
intent to duplicate the result of coding a generator directly
|
|||
|
|
instead of a generator comprehension. Further, the variable 'i'
|
|||
|
|
is not in a defined state on the line immediately following the
|
|||
|
|
list comprehension. It does not come into existence until
|
|||
|
|
iteration starts (possibly never).
|
|||
|
|
|
|||
|
|
Comments from GvR: Cute hack, but I think the use of the [] syntax
|
|||
|
|
strongly suggests that it would return a list, not an
|
|||
|
|
iterator. I also think that this is trying to turn Python into
|
|||
|
|
a functional language, where most algorithms use lazy infinite
|
|||
|
|
sequences, and I just don't think that's where its future
|
|||
|
|
lies.
|
|||
|
|
|
|||
|
|
I don't think it's worth the trouble. I expect it will take a
|
|||
|
|
lot of work to hack it into the code generator: it has to
|
|||
|
|
create a separate code object in order to be a generator.
|
|||
|
|
List comprehensions are inlined, so I expect that the
|
|||
|
|
generator comprehension code generator can't share much with
|
|||
|
|
the list comprehension code generator. And this for something
|
|||
|
|
that's not that common and easily done by writing a 2-line
|
|||
|
|
helper function. IOW the ROI isn't high enough.
|
|||
|
|
|
|||
|
|
Comments from Ka-Ping Yee: I am very happy with the things you have
|
|||
|
|
proposed in this PEP. I feel quite positive about generator
|
|||
|
|
comprehensions and have no reservations. So a +1 on that.
|
|||
|
|
|
|||
|
|
Comments from Neil Schemenauer: I'm -0 on the generator list
|
|||
|
|
comprehensions. They don't seem to add much. You could
|
|||
|
|
easily use a nested generator to do the same thing. They
|
|||
|
|
smell like lambda.
|
|||
|
|
|
|||
|
|
Comments from Magnus Lie Hetland: Generator comprehensions seem mildly
|
|||
|
|
useful, but I vote +0. Defining a separate, named generator
|
|||
|
|
would probably be my preference. On the other hand, I do see
|
|||
|
|
the advantage of "scaling up" from list comprehensions.
|
|||
|
|
|
|||
|
|
Comments from the Community: The response to the generator comprehension
|
|||
|
|
proposal has been mostly favorable. There were some 0 votes
|
|||
|
|
from people who didn't see a real need or who were not
|
|||
|
|
energized by the idea. Some of the 0 votes were tempered by
|
|||
|
|
comments that the reviewer did not even like list
|
|||
|
|
comprehensions or did not have any use for generators in any
|
|||
|
|
form. The +1 votes outnumbered the 0 votes by about two to
|
|||
|
|
one.
|
|||
|
|
|
|||
|
|
Author response: I've studied several syntactical variations and
|
|||
|
|
concluded that the brackets are essential for:
|
|||
|
|
- teachability (it's like a list comprehension)
|
|||
|
|
- set-off (yield applies to the comprehension not the enclosing
|
|||
|
|
function)
|
|||
|
|
- substitutability (list comprehensions can be made lazy just by
|
|||
|
|
adding yield)
|
|||
|
|
|
|||
|
|
What I like best about generator comprehensions is that I can
|
|||
|
|
design using list comprehensions and then easily switch to a
|
|||
|
|
generator (by adding yield) in response to scalability
|
|||
|
|
requirements (when the list comprehension produces too large
|
|||
|
|
of an intermediate result). Had generators already been
|
|||
|
|
in-place when list comprehensions were accepted, the yield
|
|||
|
|
option might have been incorporated from the start. For
|
|||
|
|
certain, the mathematical style notation is explicit and
|
|||
|
|
readable as compared to a separate function definition with an
|
|||
|
|
embedded yield.
|
|||
|
|
|
|||
|
|
|
|||
|
|
References
|
|||
|
|
|
|||
|
|
[1] PEP 255 Simple Generators
|
|||
|
|
http://python.sourceforge.net/peps/pep-0255.html
|
|||
|
|
|
|||
|
|
[2] PEP 212 Loop Counter Iteration
|
|||
|
|
http://python.sourceforge.net/peps/pep-0212.html
|
|||
|
|
|
|||
|
|
[3] PEP 202 List Comprehensions
|
|||
|
|
http://python.sourceforge.net/peps/pep-0202.html
|
|||
|
|
|
|||
|
|
[4] PEP 234 Iterators
|
|||
|
|
http://python.sourceforge.net/peps/pep-0234.html
|
|||
|
|
|
|||
|
|
[5] A pure Python simulation of every feature in this PEP is at:
|
|||
|
|
http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17348&aid=513752
|
|||
|
|
|
|||
|
|
[6] The full, working source code for each of the examples in this PEP
|
|||
|
|
along with other examples and tests is at:
|
|||
|
|
http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17412&aid=513756
|
|||
|
|
|
|||
|
|
|
|||
|
|
Copyright
|
|||
|
|
|
|||
|
|
This document has been placed in the public domain.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Local Variables:
|
|||
|
|
mode: indented-text
|
|||
|
|
indent-tabs-mode: nil
|
|||
|
|
fill-column: 70
|
|||
|
|
End:
|