PEP: 322 Title: Reverse Iteration Methods Version: $Revision$ Last-Modified: $Date$ Author: Raymond Hettinger Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2003 Python-Version: 2.4 Post-History: 24-Sep-2003 Abstract ======== This proposal is to extend the API of several sequence types to include a method for iterating over the sequence in reverse. Motivation ========== For indexable objects, current approaches for reverse iteration are error prone, unnatural, and not especially readable:: for i in xrange(n-1, -1, -1): print seqn[i] One other current approach involves reversing a list before iterating over it. That technique wastes computer cycles, memory, and lines of code:: rseqn = list(seqn) rseqn.reverse() for value in rseqn: print value Extended slicing is a third approach that minimizes the code overhead but does nothing for memory efficiency, beauty, or clarity. Reverse iteration is much less common than forward iteration, but it does arise regularly in practice. See `Real World Use Cases`_ below. Proposal ======== Add a method called *iterreverse()* to sequence objects that can benefit from it. The above examples then simplify to:: for i in xrange(n).iterreverse(): print seqn[i] :: for elem in seqn.iterreverse(): print elem The new protocol would be applied to lists, tuples, strings, and xrange objects. It would not apply to unordered collections like dicts and sets. No language syntax changes are needed. Alternative Method Names ======================== * *iterbackwards* -- like iteritems() but somewhat long * *backwards* -- more pithy, less explicit * *ireverse* -- reminiscent of imap(), izip(), and ifilter() Other Issues ============ * Should *tuple* objects be included? In the past, they have been denied some list like behaviors such as count() and index(). I prefer that it be included. * Should *file* objects be included? Implementing reverse iteration may not be easy though it would be useful on occasion. I think this one should be skipped. * Should *enumerate* objects be included? They can provide reverse iteration only when the underlying sequences support *__len__* and reverse iteration. I think this can be saved for another day if the need arises. Real World Use Cases ==================== Here are some instances of reverse iteration taken from the standard library and comments on why reverse iteration was necessary: * atexit.exit_handlers() uses:: while _exithandlers: func, targs, kargs = _exithandlers.pop() . . . The application dictates the need to run exit handlers in the reverse order they were built. The ``while alist: alist.pop()`` form is readable and clean; however, it would be slightly faster and clearer with:: for func, target, kargs in _exithandlers.iterreverse(): . . . del _exithandlers Note, if the order of deletion is important, then the first form is still needed. * difflib.get_close_matches() uses:: result.sort() # Retain only the best n. result = result[-n:] # Move best-scorer to head of list. result.reverse() # Strip scores. return [x for score, x in result] The need for reverse iteration arises from a requirement to return a portion of a sort in an order opposite of the sort criterion. The list comprehension is incidental (the third step of a Schwartzian transform). This particular use case can met with extended slicing, but the code is somewhat unattractive, hard to visually verify, and difficult for beginners to construct:: result.sort() return [x for score, x in result[:-n-1:-1]] The proposed form is much easier to construct and verify:: result.sort() return [x for score, x in result[-n:].iterreverse()] * heapq.heapify() uses ``for i in xrange(n//2 - 1, -1, -1)`` because higher-level orderings are more easily formed from pairs of lower-level orderings. A forward version of this algorithm is possible; however, that would complicate the rest of the heap code which iterates over the underlying list in the opposite direction. * mhlib.test() uses:: testfolders.reverse(); for t in testfolders: do('mh.deletefolder(%s)' % `t`) The need for reverse iteration arises because the tail of the underlying list is altered during iteration. * platform._dist_try_harder() uses ``for n in range(len(verfiles)-1,-1,-1)`` because the loop deletes selected elements from *verfiles* but needs to leave the rest of the list intact for further iteration. * random.shuffle() uses ``for i in xrange(len(x)-1, 0, -1)`` because the algorithm is most easily understood as randomly selecting elements from an ever diminishing pool. In fact, the algorithm can be run in a forward direction but is less intuitive and rarely presented that way in literature. The replacement code ``for i in xrange(1, len(x)).iterreverse()`` is much easier to mentally verify. * rfc822.Message.__delitem__() uses:: list.reverse() for i in list: del self.headers[i] The need for reverse iteration arises because the tail of the underlying list is altered during iteration. Alternative Ideas ================= * Add a builtin function, *riter()* which calls a magic method, *__riter__*. I see this as more overhead for no additional benefit. * Several variants were submitted that provided fallback behavior when *__riter__* is not defined: - fallback to: ``for i in xrange(len(obj)-1,-1,-1): yield obj[i]`` - fallback to: ``for i in itertools.count(): yield obj[-i]`` - fallback to: ``tmp=list(obj); tmp.reverse(); return iter(tmp)`` All of these attempt to save implementing some object methods at the expense of adding a new builtin function and of creating a new magic method name. The approaches using *__getitem__()* are slower than using a custom method for each object. Also, the *__getitem__()* variants produce bizarre results when applied to mappings. All of the variants crash when applied to an infinite iterator. The last variant can invisibly slip into a low performance mode (in terms of time and memory) which could be made more visible with an explicit ``ro=list(obj); ro.reverse()``. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: