python-peps/pep-0322.txt

PEP: 322
Title: Reverse Iteration Methods
Version: $Revision$
Last-Modified: $Date$
Author: Raymond Hettinger <python@rcn.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Sep-2003
Python-Version: 2.4
Post-History: 24-Sep-2003


Abstract
========

This proposal is to extend the API of several sequence types
to include methods for iterating over the sequence in reverse.


Motivation
==========

For indexable objects, current approaches for reverse iteration are
error prone, unnatural, and not especially readable::

    for i in xrange(n-1, -1, -1):
        print seqn[i]

One other current approach involves reversing a list before iterating
over it.  That technique wastes computer cycles, memory, and lines of
code::

    rseqn = list(seqn)
    rseqn.reverse()
    for value in rseqn:
        print value

Extending slicing is a third approach that minimizes the code overhead
but does nothing for memory efficiency, beauty, or clarity.

Reverse iteration is much less common than forward iteration, but it
does arise regularly in practice.  See `Real World Use Cases`_ below.


Proposal
========

Add a method called *iter_backwards()* to sequence objects that can
benefit from it.  The above examples then simplify to::

    for i in xrange(n).iter_backwards():
        print seqn[i]

::

    for elem in seqn.iter_backwards():
        print elem

The new protocol would be applied to lists, strings, xrange objects,
and possibly other sequence objects as well (depending on use cases
and implementation issues).  It would not apply to unordered
collections like dicts and sets.

No language syntax changes are needed.


Alternative Method Names
========================

* *iterbackwards* -- like iteritems() but somewhat long
* *backwards* -- more pithy, less explicit
* *ireverse* -- reminiscent of imap(), izip(), and ifilter()


Open Issues
===========

* Should *tuple* objects be included?  In the past, they have been
  denied some list like behaviors such as count() and index().

* Should *file* objects be included?  Implementing reverse iteration
  may not be easy though it would be useful on occasion.

* Should *enumerate* objects be included?  They can provide reverse
  iteration only when the underlying sequences support *__len__*
  and reverse iteration.


Real World Use Cases
====================

Here are some instances of reverse iteration taken from the standard
library and comments on why reverse iteration was necessary:

* atexit.exit_handlers() uses::

    while _exithandlers:
        func, targs, kargs = _exithandlers.pop()
            . . .

  The application dictates the need to run exit handlers in the
  reverse order they were built.  The ``while alist: alist.pop()``
  form is readable and clean; however, it would be slightly faster
  and clearer with::

    for func, target, kargs in _exithandlers.iter_backwards():
        . . .
    del _exithandlers

  Note, if the order of deletion is important, then the first form
  is still needed.

* difflib.get_close_matches() uses::

    result.sort()           # Retain only the best n.
    result = result[-n:]    # Move best-scorer to head of list.
    result.reverse()        # Strip scores.
    return [x for score, x in result]

  The need for reverse iteration arises from a requirement to return
  a portion of a sort in an order opposite of the sort criterion.  The
  list comprehension is incidental (the third step of a Schwartzian
  transform).  This particular use case can met with extended slicing,
  but the code is somewhat unattractive, hard to visually verify,
  and difficult for beginners to construct::

      result.sort()
      return [x for score, x in result[:-n-1:-1]]

  The proposed form is much easier to construct and verify::

      result.sort()
      return [x for score, x in result[-n:].iter_backwards()]

* heapq.heapify() uses ``for i in xrange(n//2 - 1, -1, -1)`` because
  higher-level orderings are more easily formed from pairs of
  lower-level orderings.  A forward version of this algorithm is
  possible; however, that would complicate the rest of the heap code
  which iterates over the underlying list in the opposite direction.

* mhlib.test() uses::

    testfolders.reverse();
    for t in testfolders:
        do('mh.deletefolder(%s)' % `t`)

  The need for reverse iteration arises because the tail of the
  underlying list is altered during iteration.

* platform._dist_try_harder() uses
  ``for n in range(len(verfiles)-1,-1,-1)`` because the loop deletes
  selected elements from *verfiles* but needs to leave the rest of
  the list intact for further iteration.

* random.shuffle() uses ``for i in xrange(len(x)-1, 0, -1)`` because
  the algorithm is most easily understood as randomly selecting
  elements from an ever diminishing pool.  In fact, the algorithm can
  be run in a forward direction but is less intuitive and rarely
  presented that way in literature.  The replacement code
  ``for i in xrange(1, len(x)).iter_backwards()`` is much easier
  to mentally verify.

* rfc822.Message.__delitem__() uses::

    list.reverse()
    for i in list:
        del self.headers[i]

  The need for reverse iteration arises because the tail of the
  underlying list is altered during iteration.


Alternative Ideas
=================

* Add a builtin function, *riter()* which calls a magic method,
  *__riter__*.  I see this as more overhead for no additional benefit.

* Several variants were submitted that provided fallback behavior
  when *__riter__* is not defined:

  - fallback to:  ``for i in xrange(len(obj)-1,-1,-1): yield obj[i]``
  - fallback to:  ``for i in itertools.count(): yield[obj[-i]]``
  - fallback to:  ``tmp=list(obj); tmp.reverse();  return iter(tmp)``

  All of these attempt to save implementing some object methods at the
  expense of adding a new builtin function and of creating a new magic
  method name.

  The approaches using *__getitem__()* are slower than using a custom
  method for each object.  Also, the *__getitem__()* variants produce
  bizarre results when applied to mappings.

  All of the variants crash when applied to an infinite iterator.

  The last variant can invisibly slip into a low performance mode
  (in terms of time and memory) which could be made more visible with
  an explicit ``list(obj).reverse()``.


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: