python-peps/pep-0377.txt

PEP: 377
Title: Allow __enter__() methods to skip the statement body
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 8-Mar-2009
Python-Version: 2.7, 3.1
Post-History: 8-Mar-2009


Abstract
========

This PEP proposes a backwards compatible mechanism that allows ``__enter__()``
methods to skip the body of the associated ``with`` statment. The lack of
this ability currently means the ``contextlib.nested`` context manager
is unable to fulfil its specification of being equivalent to writing out
multiple nested ``with`` statements [1].

The proposed change is to introduce a new flow control exception
``SkipStatement``, and skip the execution of the ``with``
statement body if ``__enter__()`` raises this exception.


Proposed Change
===============

The semantics of the ``with`` statement will be changed to include a
new ``try``/``except``/``else`` block around the call to ``__enter__()``.
If ``SkipStatement`` is raised by the ``__enter__()`` method, then
the main section of the ``with`` statement (now located in the ``else``
clause) will not be executed. To avoid leaving the names in any ``as``
clause unbound in this case, a new ``StatementSkipped`` singleton
(similar to the existing ``NotImplemented`` singleton) will be
assigned to all names that appear in the ``as`` clause.

The components of the ``with`` statement remain as described in PEP 343 [2]::

    with EXPR as VAR:
        BLOCK

After the modification, the ``with`` statement semantics would
be as follows::

    mgr = (EXPR)
    exit = mgr.__exit__  # Not calling it yet
    try:
        value = mgr.__enter__()
    except SkipStatement:
        VAR = StatementSkipped
        # Only if "as VAR" is present and
        # VAR is a single name
        # If VAR is a tuple of names, then StatementSkipped
        # will be assigned to each name in the tuple
    else:
        exc = True
        try:
            try:
                VAR = value  # Only if "as VAR" is present
                BLOCK
            except:
                # The exceptional case is handled here
                exc = False
                if not exit(*sys.exc_info()):
                    raise
                # The exception is swallowed if exit() returns true
        finally:
            # The normal and non-local-goto cases are handled here
            if exc:
                exit(None, None, None)

With the above change in place for the ``with`` statement semantics,
``contextlib.contextmanager()`` will then be modified to raise
``SkipStatement`` instead of ``RuntimeError`` when the underlying
generator doesn't yield.


Rationale for Change
====================

Currently, some apparently innocuous context managers may raise
``RuntimeError`` when executed. This occurs when the context
manager's ``__enter__()`` method encounters a situation where
the written out version of the code corresponding to the
context manager would skip the code that is now the body
of the ``with`` statement. Since the ``__enter__()`` method
has no mechanism available to signal this to the interpreter,
it is instead forced to raise an exception that not only
skips the body of the ``with`` statement, but also jumps over
all code until the nearest exception handler. This goes against
one of the design goals of the ``with`` statement, which was to
be able to factor out arbitrary common exception handling code
into a single context manager by putting into a generator
function and replacing the variant part of the code with a
``yield`` statement.

Specifically, the following examples behave differently if
``cmB().__enter__()`` raises an exception which ``cmA().__exit__()``
then handles and suppresses::

  with cmA():
    with cmB():
      do_stuff()
  # This will resume here without executing "do_stuff()"

  @contextlib.contextmanager
  def combined():
    with cmA():
      with cmB():
        yield

  with combined():
    do_stuff()
  # This will raise a RuntimeError complaining that the context
  # manager's underlying generator didn't yield

  with contextlib.nested(cmA(), cmB()):
    do_stuff()
  # This will raise the same RuntimeError as the contextmanager()
  # example (unsurprising, given that the nested() implementation
  # uses contextmanager())

  # The following class based version shows that the issue isn't
  # specific to contextlib.contextmanager() (it also shows how
  # much simpler it is to write context managers as generators
  # instead of as classes!)
  class CM(object):
    def __init__(self):
      self.cmA = None
      self.cmB = None

    def __enter__(self):
      if self.cmA is not None:
        raise RuntimeError("Can't re-use this CM")
      self.cmA = cmA()
      self.cmA.__enter__()
      try:
        self.cmB = cmB()
        self.cmB.__enter__()
      except:
        self.cmA.__exit__(*sys.exc_info())
        # Can't suppress in __enter__(), so must raise
        raise

    def __exit__(self, *args):
      suppress = False
      try:
        if self.cmB is not None:
          suppress = self.cmB.__exit__(*args)
      except:
        suppress = self.cmA.__exit__(*sys.exc_info()):
        if not suppress:
          # Exception has changed, so reraise explicitly
          raise
      else:
        if suppress:
          # cmB already suppressed the exception,
          # so don't pass it to cmA
          suppress = self.cmA.__exit__(None, None, None):
        else:
          suppress = self.cmA.__exit__(*args):
      return suppress

With the proposed semantic change in place, the contextlib based examples
above would then "just work", but the class based version would need
a small adjustment to take advantage of the new semantics::

  class CM(object):
    def __init__(self):
      self.cmA = None
      self.cmB = None

    def __enter__(self):
      if self.cmA is not None:
        raise RuntimeError("Can't re-use this CM")
      self.cmA = cmA()
      self.cmA.__enter__()
      try:
        self.cmB = cmB()
        self.cmB.__enter__()
      except:
        if self.cmA.__exit__(*sys.exc_info()):
          # Suppress the exception, but don't run
          # the body of the with statement either
          raise SkipStatement
        raise

    def __exit__(self, *args):
      suppress = False
      try:
        if self.cmB is not None:
          suppress = self.cmB.__exit__(*args)
      except:
        suppress = self.cmA.__exit__(*sys.exc_info()):
        if not suppress:
          # Exception has changed, so reraise explicitly
          raise
      else:
        if suppress:
          # cmB already suppressed the exception,
          # so don't pass it to cmA
          suppress = self.cmA.__exit__(None, None, None):
        else:
          suppress = self.cmA.__exit__(*args):
      return suppress

There is currently a tentative suggestion [3] to add import-style syntax to
the ``with`` statement to allow multiple context managers to be included in
a single ``with`` statement without needing to use ``contextlib.nested``. In
that case the compiler has the option of simply emitting multiple ``with``
statements at the AST level, thus allowing the semantics of actual nested
``with`` statements to be reproduced accurately. However, such a change
would highlight rather than alleviate the problem the current PEP aims to
address: it would not be possible to use ``contextlib.contextmanager`` to
reliably factor out such ``with`` statements, as they would exhibit exactly
the same semantic differences as are seen with the ``combined()`` context
manager in the above example.


Performance Impact
==================

Implementing the new semantics makes it necessary to store the references
to the ``__enter__`` and ``__exit__`` methods in temporary variables instead
of on the stack. This results in a slight regression in ``with`` statement
speed relative to Python 2.6/3.1. However, implementing a custom
``SETUP_WITH`` opcode would negate any differences between the two
approaches (as well as dramatically improving speed by eliminating more
than a dozen unnecessary trips around the eval loop).


Reference Implementation
========================

Patch attached to Issue 5251 [1]. That patch uses only existing opcodes
(i.e. no ``SETUP_WITH``).


Acknowledgements
================

James William Pye both raised the issue and suggested the basic outline of
the solution described in this PEP.


References
==========

.. [1] Issue 5251: contextlib.nested inconsistent with nested with statements
   (http://bugs.python.org/issue5251)

.. [2] PEP 343: The "with" Statement
   (http://www.python.org/dev/peps/pep-0343/)

.. [3] Import-style syntax to reduce indentation of nested with statements
   (http://mail.python.org/pipermail/python-ideas/2009-March/003188.html)


Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: