python-peps/pep-0325.txt

PEP: 325
Title: Resource-Release Support for Generators
Version: $Revision$
Last-Modified: $Date$
Author: Samuele Pedroni <pedronis@python.org>
Status: Rejected
Type: Standards Track
Content-Type: text/plain
Created: 25-Aug-2003
Python-Version: 2.4
Post-History:


Abstract

    Generators allow for natural coding and abstraction of traversal
    over data.  Currently if external resources needing proper timely
    release are involved, generators are unfortunately not adequate.
    The typical idiom for timely release is not supported, a yield
    statement is not allowed in the try clause of a try-finally
    statement inside a generator.  The finally clause execution can be
    neither guaranteed nor enforced.

    This PEP proposes that the built-in generator type implement a
    close method and destruction semantics, such that the restriction
    on yield placement can be lifted, expanding the applicability of
    generators.

Pronouncement

    Rejected in favor of PEP 342 which includes substantially all of
    the requested behavior in a more refined form.

Rationale

    Python generators allow for natural coding of many data traversal
    scenarios.  Their instantiation produces iterators,
    i.e. first-class objects abstracting traversal (with all the
    advantages of first- classness).  In this respect they match in
    power and offer some advantages over the approach using iterator
    methods taking a (smalltalkish) block.  On the other hand, given
    current limitations (no yield allowed in a try clause of a
    try-finally inside a generator) the latter approach seems better
    suited to encapsulating not only traversal but also exception
    handling and proper resource acquisition and release.

    Let's consider an example (for simplicity, files in read-mode are
    used):

        def all_lines(index_path):
            for path in file(index_path, "r"):
                for line in file(path.strip(), "r"):
                    yield line

    this is short and to the point, but the try-finally for timely
    closing of the files cannot be added.  (While instead of a path, a
    file, whose closing then would be responsibility of the caller,
    could be passed in as argument, the same is not applicable for the
    files opened depending on the contents of the index).

    If we want timely release, we have to sacrifice the simplicity and
    directness of the generator-only approach: (e.g.)

        class AllLines:

            def __init__(self,index_path):
                self.index_path = index_path
                self.index = None
                self.document = None

            def __iter__(self):
                self.index = file(self.index_path,"r")
                for path in self.index:
                    self.document = file(path.strip(),"r")
                    for line in self.document:
                        yield line
                    self.document.close()
                    self.document = None

            def close(self):
                if self.index:
                   self.index.close()
                if self.document:
                   self.document.close()

    to be used as:

        all_lines = AllLines("index.txt")
        try:
            for line in all_lines:
                ...
        finally:
            all_lines.close()

    The more convoluted solution implementing timely release, seems
    to offer a precious hint.  What we have done is encapsulate our
    traversal in an object (iterator) with a close method.

    This PEP proposes that generators should grow such a close method
    with such semantics that the example could be rewritten as:

        # Today this is not valid Python: yield is not allowed between
        # try and finally, and generator type instances support no
        # close method.

        def all_lines(index_path):
            index = file(index_path,"r")
            try:
                for path in index:
                    document = file(path.strip(),"r")
                    try:
                        for line in document:
                            yield line
                    finally:
                       document.close()
            finally:
                index.close()

        all = all_lines("index.txt")
        try:
            for line in all:
                ...
        finally:
            all.close() # close on generator

    Currently PEP 255 [1] disallows yield inside a try clause of a
    try-finally statement, because the execution of the finally clause
    cannot be guaranteed as required by try-finally semantics.

    The semantics of the proposed close method should be such that
    while the finally clause execution still cannot be guaranteed, it
    can be enforced when required.  Specifically, the close method
    behavior should trigger the execution of the finally clauses
    inside the generator, either by forcing a return in the generator
    frame or by throwing an exception in it.  In situations requiring
    timely resource release, close could then be explicitly invoked.

    The semantics of generator destruction on the other hand should be
    extended in order to implement a best-effort policy for the
    general case.  Specifically, destruction should invoke close().
    The best-effort limitation comes from the fact that the
    destructor's execution is not guaranteed in the first place.

    This seems to be a reasonable compromise, the resulting global
    behavior being similar to that of files and closing.


Possible Semantics

    The built-in generator type should have a close method
    implemented, which can then be invoked as:

       gen.close()

    where gen is an instance of the built-in generator type.
    Generator destruction should also invoke close method behavior.

    If a generator is already terminated, close should be a no-op.

    Otherwise, there are two alternative solutions, Return or
    Exception Semantics:

    A - Return Semantics: The generator should be resumed, generator
    execution should continue as if the instruction at the re-entry
    point is a return.  Consequently, finally clauses surrounding the
    re-entry point would be executed, in the case of a then allowed
    try-yield-finally pattern.

    Issues: is it important to be able to distinguish forced
    termination by close, normal termination, exception propagation
    from generator or generator-called code?  In the normal case it
    seems not, finally clauses should be there to work the same in all
    these cases, still this semantics could make such a distinction
    hard.

    Except-clauses, like by a normal return, are not executed, such
    clauses in legacy generators expect to be executed for exceptions
    raised by the generator or by code called from it.  Not executing
    them in the close case seems correct.

    B - Exception Semantics: The generator should be resumed and
    execution should continue as if a special-purpose exception
    (e.g. CloseGenerator) has been raised at re-entry point.  Close
    implementation should consume and not propagate further this
    exception.

    Issues: should StopIteration be reused for this purpose?  Probably
    not.  We would like close to be a harmless operation for legacy
    generators, which could contain code catching StopIteration to
    deal with other generators/iterators.

    In general, with exception semantics, it is unclear what to do if
    the generator does not terminate or we do not receive the special
    exception propagated back.  Other different exceptions should
    probably be propagated, but consider this possible legacy
    generator code:

        try:
            ...
            yield ...
            ...
        except: # or except Exception:, etc
            raise Exception("boom")

    If close is invoked with the generator suspended after the yield,
    the except clause would catch our special purpose exception, so we
    would get a different exception propagated back, which in this
    case ought to be reasonably consumed and ignored but in general
    should be propagated, but separating these scenarios seems hard.

    The exception approach has the advantage to let the generator
    distinguish between termination cases and have more control.  On
    the other hand, clear-cut semantics seem harder to define.


Remarks

    If this proposal is accepted, it should become common practice to
    document whether a generator acquires resources, so that its close
    method ought to be called.  If a generator is no longer used,
    calling close should be harmless.

    On the other hand, in the typical scenario the code that
    instantiated the generator should call close if required by it.
    Generic code dealing with iterators/generators instantiated
    elsewhere should typically not be littered with close calls.

    The rare case of code that has acquired ownership of and need to
    properly deal with all of iterators, generators and generators
    acquiring resources that need timely release, is easily solved:

        if hasattr(iterator, 'close'):
            iterator.close()


Open Issues

    Definitive semantics ought to be chosen.  Currently Guido favors
    Exception Semantics.  If the generator yields a value instead of
    terminating, or propagating back the special exception, a special
    exception should be raised again on the generator side.

    It is still unclear whether spuriously converted special
    exceptions (as discussed in Possible Semantics) are a problem and
    what to do about them.

    Implementation issues should be explored.


Alternative Ideas

    The idea that the yield placement limitation should be removed and
    that generator destruction should trigger execution of finally
    clauses has been proposed more than once.  Alone it cannot
    guarantee that timely release of resources acquired by a generator
    can be enforced.

    PEP 288 [2] proposes a more general solution, allowing custom
    exception passing to generators.  The proposal in this PEP
    addresses more directly the problem of resource release.  Were PEP
    288 implemented, Exceptions Semantics for close could be layered
    on top of it, on the other hand PEP 288 should make a separate
    case for the more general functionality.


References

    [1] PEP 255 Simple Generators
        http://www.python.org/dev/peps/pep-0255/

    [2] PEP 288 Generators Attributes and Exceptions
        http://www.python.org/dev/peps/pep-0288/


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End: