python-peps/pep-0485.txt

PEP: 485
Title: A Function for testing approximate equality
Version: $Revision$
Last-Modified: $Date$
Author: Christopher Barker <Chris.Barker@noaa.gov>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Jan-2015
Python-Version: 3.5
Post-History:


Abstract
========

This PEP proposes the addition of a function to the standard library
that determines whether one value is approximately equal or "close"
to another value.


Rationale
=========

Floating point values contain limited precision, which results in
their being unable to exactly represent some values, and for error to
accumulate with repeated computation.  As a result, it is common
advice to only use an equality comparison in very specific situations.
Often a inequality comparison fits the bill, but there are times
(often in testing) where the programmer wants to determine whether a
computed value is "close" to an expected value, without requiring them
to be exactly equal. This is common enough, particularly in testing,
and not always obvious how to do it, so it would be useful addition to
the standard library.


Existing Implementations
------------------------

The standard library includes the
``unittest.TestCase.assertAlmostEqual`` method, but it:

* Is buried in the unittest.TestCase class

* Is an assertion, so you can't use it as a general test (easily)

* Uses number of decimal digits or an absolute delta, which are
  particular use cases that don't provide a general relative error.

The numpy package has the ``allclose()`` and ``isclose()`` functions.

The statistics package tests include an implementation, used for its
unit tests.

One can also find discussion and sample implementations on Stack
Overflow, and other help sites.

These existing implementations indicate that this is a common need,
and not trivial to write oneself, making it a candidate for the
standard library.


Proposed Implementation
=======================

NOTE: this PEP is the result of an extended discussion on the
python-ideas list [1]_.

The new function will have the following signature::

  is_close_to(actual, expected, tol=1e-8, abs_tol=0.0)

``actual``: is the value that has been computed, measured, etc.

``expected``: is the "known" value.

``tol``: is the relative tolerance -- it is the amount of error
allowed, relative to the magnitude of the expected value.

``abs_tol``: is an minimum absolute tolerance level -- useful for
comparisons near zero.

Modulo error checking, etc, the function will return the result of::

    abs(expected-actual) <= max(tol*expected, abs_tol)


Handling of non-finite numbers
------------------------------

The IEEE 754 special values of NaN, inf, and -inf will be handled
according to IEEE rules. Specifically, NaN is not considered close to
any other value, including NaN. inf and -inf are only considered close
to themselves.


Non-float types
---------------

The primary use-case is expected to be floating point numbers.
However, users may want to compare other numeric types similarly. In
theory, it should work for any type that supports ``abs()``,
comparisons, and subtraction.  The code will be written and tested to
accommodate these types:

* ``Decimal``: for Decimal, the tolerance must be set to a Decimal type.

* ``int``

* ``Fraction``
 
* ``complex``: for complex, ``abs(z)`` will be used for scaling and
  comparison.


Behavior near zero
------------------

Relative comparison is problematic if either value is zero. In this
case, the difference is relative to zero, and thus will always be
smaller than the prescribed tolerance. To handle this case, an
optional parameter, ``abs_tol`` (default 0.0) can be used to set a
minimum tolerance to be used in the case of very small relative
tolerance. That is, the values will be considered close if::

    abs(a-b) <= abs(tol*expected) or abs(a-b) <= abs_tol

If the user sets the rel_tol parameter to 0.0, then only the absolute
tolerance will effect the result, so this function provides an
absolute tolerance check as well.

A sample implementation is available (as of Jan 22, 2015) on gitHub:

https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py


Relative Difference
===================

There are essentially two ways to think about how close two numbers
are to each-other: absolute difference: simply ``abs(a-b)``, and
relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute
difference is trivial enough that this proposal focuses on the
relative difference.

Usually, the scale factor is some function of the values under
consideration, for instance: 

1) The absolute value of one of the input values

2) The maximum absolute value of the two

3) The minimum absolute value of the two.

4) The arithmetic mean of the two


Symmetry
--------

A relative comparison can be either symmetric or non-symmetric. For a
symmetric algorithm:

``is_close_to(a,b)`` is always equal to ``is_close_to(b,a)``

This is an appealing consistency -- it mirrors the symmetry of
equality, and is less likely to confuse people. However, often the
question at hand is:

"Is this computed or measured value within some tolerance of a known
value?"

In this case, the user wants the relative tolerance to be specifically
scaled against the known value. It is also easier for the user to
reason about.

This proposal uses this asymmetric test to allow this specific
definition of relative tolerance.

Example:

For the question: "Is the value of a within 10% of b?", Using b to
scale the percent error clearly defines the result.

However, as this approach is not symmetric, a may be within 10% of b,
but b is not within 10% of a. Consider the case::

  a =  9.0
  b = 10.0

The difference between a and b is 1.0. 10% of a is 0.9, so b is not
within 10% of a. But 10% of b is 1.0, so a is within 10% of b. 

Casual users might reasonably expect that if a is close to b, then b
would also be close to a. However, in the common cases, the tolerance
is quite small and often poorly defined, i.e. 1e-8, defined to only
one significant figure, so the result will be very similar regardless
of the order of the values. And if the user does care about the
precise result, s/he can take care to always pass in the two
parameters in sorted order.

This proposed implementation uses asymmetric criteria with the scaling
value clearly identified.


Expected Uses
=============

The primary expected use case is various forms of testing -- "are the
results computed near what I expect as a result?" This sort of test
may or may not be part of a formal unit testing suite.

The function might be used also to determine if a measured value is
within an expected value.


Inappropriate uses
------------------

One use case for floating point comparison is testing the accuracy of
a numerical algorithm. However, in this case, the numerical analyst
ideally would be doing careful error propagation analysis, and should
understand exactly what to test for. It is also likely that ULP (Unit
in the Last Place) comparison may be called for. While this function
may prove useful in such situations, It is not intended to be used in
that way.


Other Approaches
================

``unittest.TestCase.assertAlmostEqual``
---------------------------------------

(https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual)

Tests that values are approximately (or not approximately) equal by
computing the difference, rounding to the given number of decimal
places (default 7), and comparing to zero.

This method was not selected for this proposal, as the use of decimal
digits is a specific, not generally useful or flexible test.

numpy ``is_close()``
--------------------

http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isclose.html

The numpy package provides the vectorized functions is_close() and
all_close, for similar use cases as this proposal:

``isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)``

      Returns a boolean array where two arrays are element-wise equal
      within a tolerance.

      The tolerance values are positive, typically very small numbers.
      The relative difference (rtol * abs(b)) and the absolute
      difference atol are added together to compare against the
      absolute difference between a and b

In this approach, the absolute and relative tolerance are added
together, rather than the ``or`` method used in this proposal. This is
computationally more simple, and if relative tolerance is larger than
the absolute tolerance, then the addition will have no effect. But if
the absolute and relative tolerances are of similar magnitude, then
the allowed difference will be about twice as large as expected.

Also, if the value passed in are small compared to the absolute
tolerance, then the relative tolerance will be completely swamped,
perhaps unexpectedly.

This is why, in this proposal, the absolute tolerance defaults to zero
-- the user will be required to choose a value appropriate for the
values at hand.


Boost floating-point comparison
-------------------------------

The Boost project ( [3]_ ) provides a floating point comparison
function. Is is a symetric approach, with both "weak" (larger of the
two relative errors) and "strong" (smaller of the two relative errors)
options.

It was decided that a method that clearly defined which value was used
to scale the relative error would be more appropriate for the standard
library.

References
==========

.. [1] Python-ideas list discussion thread
   (https://mail.python.org/pipermail/python-ideas/2015-January/030947.html)

.. [2] Wikipedaia page on relative difference
   (http://en.wikipedia.org/wiki/Relative_change_and_difference)

.. [3] Boost project floating-point comparison algorithms
   (http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8