287 lines
10 KiB
Plaintext
287 lines
10 KiB
Plaintext
PEP: 325
|
||
Title: Resource-Release Support for Generators
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Samuele Pedroni <pedronis@python.org>
|
||
Status: Rejected
|
||
Type: Standards Track
|
||
Content-Type: text/plain
|
||
Created: 25-Aug-2003
|
||
Python-Version: 2.4
|
||
Post-History:
|
||
|
||
|
||
Abstract
|
||
|
||
Generators allow for natural coding and abstraction of traversal
|
||
over data. Currently if external resources needing proper timely
|
||
release are involved, generators are unfortunately not adequate.
|
||
The typical idiom for timely release is not supported, a yield
|
||
statement is not allowed in the try clause of a try-finally
|
||
statement inside a generator. The finally clause execution can be
|
||
neither guaranteed nor enforced.
|
||
|
||
This PEP proposes that the built-in generator type implement a
|
||
close method and destruction semantics, such that the restriction
|
||
on yield placement can be lifted, expanding the applicability of
|
||
generators.
|
||
|
||
Pronouncement
|
||
|
||
Rejected in favor of PEP 342 which includes substantially all of
|
||
the requested behavior in a more refined form.
|
||
|
||
Rationale
|
||
|
||
Python generators allow for natural coding of many data traversal
|
||
scenarios. Their instantiation produces iterators,
|
||
i.e. first-class objects abstracting traversal (with all the
|
||
advantages of first- classness). In this respect they match in
|
||
power and offer some advantages over the approach using iterator
|
||
methods taking a (smalltalkish) block. On the other hand, given
|
||
current limitations (no yield allowed in a try clause of a
|
||
try-finally inside a generator) the latter approach seems better
|
||
suited to encapsulating not only traversal but also exception
|
||
handling and proper resource acquisition and release.
|
||
|
||
Let's consider an example (for simplicity, files in read-mode are
|
||
used):
|
||
|
||
def all_lines(index_path):
|
||
for path in file(index_path, "r"):
|
||
for line in file(path.strip(), "r"):
|
||
yield line
|
||
|
||
this is short and to the point, but the try-finally for timely
|
||
closing of the files cannot be added. (While instead of a path, a
|
||
file, whose closing then would be responsibility of the caller,
|
||
could be passed in as argument, the same is not applicable for the
|
||
files opened depending on the contents of the index).
|
||
|
||
If we want timely release, we have to sacrifice the simplicity and
|
||
directness of the generator-only approach: (e.g.)
|
||
|
||
class AllLines:
|
||
|
||
def __init__(self,index_path):
|
||
self.index_path = index_path
|
||
self.index = None
|
||
self.document = None
|
||
|
||
def __iter__(self):
|
||
self.index = file(self.index_path,"r")
|
||
for path in self.index:
|
||
self.document = file(path.strip(),"r")
|
||
for line in self.document:
|
||
yield line
|
||
self.document.close()
|
||
self.document = None
|
||
|
||
def close(self):
|
||
if self.index:
|
||
self.index.close()
|
||
if self.document:
|
||
self.document.close()
|
||
|
||
to be used as:
|
||
|
||
all_lines = AllLines("index.txt")
|
||
try:
|
||
for line in all_lines:
|
||
...
|
||
finally:
|
||
all_lines.close()
|
||
|
||
The more convoluted solution implementing timely release, seems
|
||
to offer a precious hint. What we have done is encapsulate our
|
||
traversal in an object (iterator) with a close method.
|
||
|
||
This PEP proposes that generators should grow such a close method
|
||
with such semantics that the example could be rewritten as:
|
||
|
||
# Today this is not valid Python: yield is not allowed between
|
||
# try and finally, and generator type instances support no
|
||
# close method.
|
||
|
||
def all_lines(index_path):
|
||
index = file(index_path,"r")
|
||
try:
|
||
for path in index:
|
||
document = file(path.strip(),"r")
|
||
try:
|
||
for line in document:
|
||
yield line
|
||
finally:
|
||
document.close()
|
||
finally:
|
||
index.close()
|
||
|
||
all = all_lines("index.txt")
|
||
try:
|
||
for line in all:
|
||
...
|
||
finally:
|
||
all.close() # close on generator
|
||
|
||
Currently PEP 255 [1] disallows yield inside a try clause of a
|
||
try-finally statement, because the execution of the finally clause
|
||
cannot be guaranteed as required by try-finally semantics.
|
||
|
||
The semantics of the proposed close method should be such that
|
||
while the finally clause execution still cannot be guaranteed, it
|
||
can be enforced when required. Specifically, the close method
|
||
behavior should trigger the execution of the finally clauses
|
||
inside the generator, either by forcing a return in the generator
|
||
frame or by throwing an exception in it. In situations requiring
|
||
timely resource release, close could then be explicitly invoked.
|
||
|
||
The semantics of generator destruction on the other hand should be
|
||
extended in order to implement a best-effort policy for the
|
||
general case. Specifically, destruction should invoke close().
|
||
The best-effort limitation comes from the fact that the
|
||
destructor's execution is not guaranteed in the first place.
|
||
|
||
This seems to be a reasonable compromise, the resulting global
|
||
behavior being similar to that of files and closing.
|
||
|
||
|
||
Possible Semantics
|
||
|
||
The built-in generator type should have a close method
|
||
implemented, which can then be invoked as:
|
||
|
||
gen.close()
|
||
|
||
where gen is an instance of the built-in generator type.
|
||
Generator destruction should also invoke close method behavior.
|
||
|
||
If a generator is already terminated, close should be a no-op.
|
||
|
||
Otherwise, there are two alternative solutions, Return or
|
||
Exception Semantics:
|
||
|
||
A - Return Semantics: The generator should be resumed, generator
|
||
execution should continue as if the instruction at the re-entry
|
||
point is a return. Consequently finally clauses surrounding the
|
||
re-entry point would be executed, in the case of a then allowed
|
||
try-yield-finally pattern.
|
||
|
||
Issues: is it important to be able to distinguish forced
|
||
termination by close, normal termination, exception propagation
|
||
from generator or generator-called code? In the normal case it
|
||
seems not, finally clauses should be there to work the same in all
|
||
these cases, still this semantics could make such a distinction
|
||
hard.
|
||
|
||
Except-clauses, like by a normal return, are not executed, such
|
||
clauses in legacy generators expect to be executed for exceptions
|
||
raised by the generator or by code called from it. Not executing
|
||
them in the close case seems correct.
|
||
|
||
B - Exception Semantics: The generator should be resumed and
|
||
execution should continue as if a special-purpose exception
|
||
(e.g. CloseGenerator) has been raised at re-entry point. Close
|
||
implementation should consume and not propagate further this
|
||
exception.
|
||
|
||
Issues: should StopIteration be reused for this purpose? Probably
|
||
not. We would like close to be a harmless operation for legacy
|
||
generators, which could contain code catching StopIteration to
|
||
deal with other generators/iterators.
|
||
|
||
In general, with exception semantics, it is unclear what to do if
|
||
the generator does not terminate or we do not receive the special
|
||
exception propagated back. Other different exceptions should
|
||
probably be propagated, but consider this possible legacy
|
||
generator code:
|
||
|
||
try:
|
||
...
|
||
yield ...
|
||
...
|
||
except: # or except Exception:, etc
|
||
raise Exception("boom")
|
||
|
||
If close is invoked with the generator suspended after the yield,
|
||
the except clause would catch our special purpose exception, so we
|
||
would get a different exception propagated back, which in this
|
||
case ought to be reasonably consumed and ignored but in general
|
||
should be propagated, but separating these scenarios seems hard.
|
||
|
||
The exception approach has the advantage to let the generator
|
||
distinguish between termination cases and have more control. On
|
||
the other hand clear-cut semantics seem harder to define.
|
||
|
||
|
||
Remarks
|
||
|
||
If this proposal is accepted, it should become common practice to
|
||
document whether a generator acquires resources, so that its close
|
||
method ought to be called. If a generator is no longer used,
|
||
calling close should be harmless.
|
||
|
||
On the other hand, in the typical scenario the code that
|
||
instantiated the generator should call close if required by it.
|
||
Generic code dealing with iterators/generators instantiated
|
||
elsewhere should typically not be littered with close calls.
|
||
|
||
The rare case of code that has acquired ownership of and need to
|
||
properly deal with all of iterators, generators and generators
|
||
acquiring resources that need timely release, is easily solved:
|
||
|
||
if hasattr(iterator, 'close'):
|
||
iterator.close()
|
||
|
||
|
||
Open Issues
|
||
|
||
Definitive semantics ought to be chosen. Currently Guido favors
|
||
Exception Semantics. If the generator yields a value instead of
|
||
terminating, or propagating back the special exception, a special
|
||
exception should be raised again on the generator side.
|
||
|
||
It is still unclear whether spuriously converted special
|
||
exceptions (as discussed in Possible Semantics) are a problem and
|
||
what to do about them.
|
||
|
||
Implementation issues should be explored.
|
||
|
||
|
||
Alternative Ideas
|
||
|
||
The idea that the yield placement limitation should be removed and
|
||
that generator destruction should trigger execution of finally
|
||
clauses has been proposed more than once. Alone it cannot
|
||
guarantee that timely release of resources acquired by a generator
|
||
can be enforced.
|
||
|
||
PEP 288 [2] proposes a more general solution, allowing custom
|
||
exception passing to generators. The proposal in this PEP
|
||
addresses more directly the problem of resource release. Were PEP
|
||
288 implemented, Exceptions Semantics for close could be layered
|
||
on top of it, on the other hand PEP 288 should make a separate
|
||
case for the more general functionality.
|
||
|
||
|
||
References
|
||
|
||
[1] PEP 255 Simple Generators
|
||
http://www.python.org/peps/pep-0255.html
|
||
|
||
[2] PEP 288 Generators Attributes and Exceptions
|
||
http://www.python.org/peps/pep-0288.html
|
||
|
||
|
||
Copyright
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
End:
|