PEP: 564 Title: Add new time functions with nanosecond resolution Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-October-2017 Python-Version: 3.7 Abstract ======== Add six new "nanosecond" variant of existing functions to the ``time`` module: ``clock_gettime_ns()``, ``clock_settime_ns()``, ``monotonic_ns()``, ``perf_counter_ns()``, ``process_time_ns()`` and ``time_ns()``. Similar to the existing functions without the ``_ns`` suffix, they have nanosecond resolution: use a number of nanoseconds as a Python int. The ``time.time_ns()`` resolution measured in Python is 3 times better than the ``time.time()`` resolution on Linux and Windows. Rationale ========= Float type limited to 104 days ------------------------------ The clocks resolution of desktop and latop computers is getting closer to nanosecond resolution. More and more clocks have a frequency in MHz, up to GHz for the CPU TSC clock. The Python ``time.time()`` function returns the current time as a floatting point number which is usually a 64-bit binary floatting number (in the IEEE 754 format). The problem is that the float type starts to lose nanoseconds after 104 days. Conversion from nanoseconds (``int``) to seconds (``float``) and then back to nanoseconds (``int``) to check if conversions lose precision:: # no precision loss >>> x = 2 ** 52 + 1; int(float(x * 1e-9) * 1e9) - x 0 # precision loss! (1 nanosecond) >>> x = 2 ** 53 + 1; int(float(x * 1e-9) * 1e9) - x -1 >>> print(datetime.timedelta(seconds=2 ** 53 / 1e9)) 104 days, 5:59:59.254741 ``time.time()`` returns seconds elapsed since the UNIX epoch: January 1st, 1970. This function loses precision since May 1970 (47 years ago):: >>> import datetime >>> unix_epoch = datetime.datetime(1970, 1, 1) >>> print(unix_epoch + datetime.timedelta(seconds=2**53 / 1e9)) 1970-04-15 05:59:59.254741 Previous rejected PEP --------------------- Five years ago, the PEP 410 proposed a large and complex change in all Python functions returning time to support nanosecond resolution using the ``decimal.Decimal`` type. The PEP was rejected for different reasons: * The idea of adding a new optional parameter to change the result type was rejected. It's an uncommon (and bad?) programming practice in Python. * It was not clear if hardware clocks really had a resolution of 1 nanosecond, especially at the Python level. * The ``decimal.Decimal`` type is uncommon in Python and so requires to adapt code to handle it. Issues caused by precision loss ------------------------------- Example 1: measure time delta in long running process ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A server is running for longer than 104 days. A clock is read before and after running a function to measure its performance to detect performance issues at runtime. Such benchmark only lose precision because of the float type used by clocks, not because of the clock resolution. On Python microbenchmarks, it is common to see function calls taking less than 100 ns. A difference of a single nanosecond becomes significant. Example 2: compare time with different resolution ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Two programs "A" and "B" are runing on the same system and use the system clock. The program A reads the system clock with nanosecond resolution and writes the timestamp with nanosecond resolution. The program B reads the timestamp with nanosecond resolution, but compares it to the system clock read with a worse resolution. To simplify the example, let's say that it reads the clock with second resolution. If that case, there is a window of 1 second while the program B can see the timestamp written by A as "in the future". Nowadays, more and more databases and filesystems support storing time with nanosecond resolution. .. note:: This issue was already fixed for file modification time by adding the ``st_mtime_ns`` field to the ``os.stat()`` result, and by accepting nanoseconds in ``os.utime()``. This PEP proposes to generalize the fix. CPython enhancements of the last 5 years ---------------------------------------- Since the PEP 410 was rejected: * The ``os.stat_result`` structure got 3 new fields for timestamps as nanoseconds (Python ``int``): ``st_atime_ns``, ``st_ctime_ns`` and ``st_mtime_ns``. * The PEP 418 was accepted, Python 3.3 got 3 new clocks: ``time.monotonic()``, ``time.perf_counter()`` and ``time.process_time()``. * The CPython private "pytime" C API handling time now uses a new ``_PyTime_t`` type: simple 64-bit signed integer (C ``int64_t``). The ``_PyTime_t`` unit is an implementation detail and not part of the API. The unit is currently ``1 nanosecond``. Existing Python APIs using nanoseconds as int --------------------------------------------- The ``os.stat_result`` structure has 3 fields for timestamps as nanoseconds (``int``): ``st_atime_ns``, ``st_ctime_ns`` and ``st_mtime_ns``. The ``ns`` parameter of the ``os.utime()`` function accepts a ``(atime_ns: int, mtime_ns: int)`` tuple: nanoseconds. Changes ======= New functions ------------- This PEP adds six new functions to the ``time`` module: * ``time.clock_gettime_ns(clock_id)`` * ``time.clock_settime_ns(clock_id, time: int)`` * ``time.monotonic_ns()`` * ``time.perf_counter_ns()`` * ``time.process_time_ns()`` * ``time.time_ns()`` These functions are similar to the version without the ``_ns`` suffix, but use nanoseconds as Python ``int``. For example, ``time.monotonic_ns() == int(time.monotonic() * 1e9)`` if ``monotonic()`` value is small enough to not lose precision. These functions are needed because they handle large timestamps, like time.time() which uses the UNIX epoch as reference, and so their version without the ``_ns`` suffix are likely to lose precision at the nanosecond resolution. Unchanged functions ------------------- Since the ``time.clock()`` function was deprecated in Python 3.3, no ``time.clock_ns()`` is added. Python has other functions handling time. No nanosecond variant was proposed because their internal resolution is greater or equal to 1 us, or because their maximum value is a small enough to not lose precision. For example, the maximum value of ``clock_getres()`` should be 1 second. Example of unchanged functions: * ``os`` module: ``sched_rr_get_interval()``, ``times()``, ``wait3()`` and ``wait4()`` * ``resource`` module: ``ru_utime`` and ``ru_stime`` fields of ``getrusage()`` * ``signal`` module: ``getitimer()``, ``setitimer()`` * ``time`` module: ``clock_getres()`` See also the `Annex: Clocks Resolution in Python`_. A new nanosecond flavor of these functions may be added later if an operating system adds a new function providing better resolution. Alternatives and discussion =========================== Sub-nanosecond resolution ------------------------- ``time.time_ns()`` API is not "future-proof": if clocks resolutions increase, new Python functions may be needed. In practive, the resolution of 1 nanosecond is currently enough for all structures used by all operating systems functions. Hardware clock with a resolution better than 1 nanosecond already exists. For example, the frequency of a CPU TSC clock is the CPU base frequency: the resolution is around 0.3 ns for a CPU running at 3 GHz. Users who have access to such hardware and really need sub-nanosecond resolution can easily extend Python for their needs. Such rare use case don't justify to design the Python standard library to support sub-nanosecond resolution. For the CPython implementation, nanosecond resolution is convenient: the standard and well supported ``int64_t`` type can be used to store time. It supports a time delta between -292 years and 292 years. Using the UNIX epoch as reference, this type supports time since year 1677 to year 2262:: >>> 1970 - 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25) 1677.728976954687 >>> 1970 + 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25) 2262.271023045313 Modify time.time() result type ------------------------------ It was proposed to modify ``time.time()`` to return a different float type with better precision. The PEP 410 proposed to use ``decimal.Decimal`` which already exists and supports arbitray precision, but it was rejected. Apart ``decimal.Decimal``, no portable ``float`` type with better precision is currently available in Python. Changing the builtin Python ``float`` type is out of the scope of this PEP. Moreover, changing existing functions to return a new type introduces a risk of breaking the backward compatibility even the new type is designed carefully. Different types --------------- Many ideas of new types were proposed to support larger or arbitrary precision: fractions, structures or 2-tuple using integers, fixed-precision floating point number, etc. See also the PEP 410 for a previous long discussion on other types. Adding a new type requires more effort to support it, than reusing the existing ``int`` type. The standard library, third party code and applications would have to be modified to support it. The Python ``int`` type is well known, well supported, ease to manipulate, and supports all arithmetic operations like: ``dt = t2 - t1``. Moreover, using nanoseconds as integer is not new in Python, it's already used for ``os.stat_result`` and ``os.utime(ns=(atime_ns, mtime_ns))``. .. note:: If the Python ``float`` type becomes larger (ex: decimal128 or float128), the ``time.time()`` precision will increase as well. Different API ------------- The ``time.time(ns=False)`` API was proposed to avoid adding new functions. It's an uncommon (and bad?) programming practice in Python to change the result type depending on a parameter. Different options were proposed to allow the user to choose the time resolution. If each Python module uses a different resolution, it can become difficult to handle different resolutions, instead of just seconds (``time.time()`` returning ``float``) and nanoseconds (``time.time_ns()`` returning ``int``). Moreover, as written above, there is no need for resolution better than 1 nanosecond in practive in the Python standard library. New time_ns module ------------------ Add a new ``time_ns`` module which contains the six new functions: * ``time_ns.clock_gettime(clock_id)`` * ``time_ns.clock_settime(clock_id, time: int)`` * ``time_ns.monotonic()`` * ``time_ns.perf_counter()`` * ``time_ns.process_time()`` * ``time_ns.time()`` The first question is if the ``time_ns`` should expose exactly the same API (constants, functions, etc.) than the ``time`` module. It can be painful to maintain two flavors of the ``time`` module. How users use suppose to make a choice between these two modules? If tomorrow, other nanosecond variant are needed in the ``os`` module, will we have to add a new ``os_ns`` module as well? There are functions related to time in many modules: ``time``, ``os``, ``signal``, ``resource``, ``select``, etc. Another idea is to add a ``time.ns`` submodule or a nested-namespace to get the ``time.ns.time()`` syntax. Annex: Clocks Resolution in Python ================================== This annex contains the resolution of clocks measured in Python, and not the resolution announced by the operating system or the resolution of the internal structure used by the operating system. Script ------ Example of script ot measure the smallest difference between two ``time.time()`` and ``time.time_ns()`` reads ignoring differences of zero:: import math import time LOOPS = 10 ** 6 print("time.time_ns(): %s" % time.time_ns()) print("time.time(): %s" % time.time()) min_dt = [abs(time.time_ns() - time.time_ns()) for _ in range(LOOPS)] min_dt = min(filter(bool, min_dt)) print("min time_ns() delta: %s ns" % min_dt) min_dt = [abs(time.time() - time.time()) for _ in range(LOOPS)] min_dt = min(filter(bool, min_dt)) print("min time() delta: %s ns" % math.ceil(min_dt * 1e9)) Linux ----- Clocks resolution measured in Python on Fedora 26 (kernel 4.12): ==================== ========== Function Resolution ==================== ========== clock() 1 us monotonic() 81 ns monotonic_ns() 84 ns perf_counter() 82 ns perf_counter_ns() 84 ns process_time() 2 ns process_time_ns() 1 ns resource.getrusage() 1 us time() **239 ns** time_ns() **84 ns** times().elapsed 10 ms times().user 10 ms ==================== ========== Notes on resolutions: * ``clock()`` frequency is ``CLOCKS_PER_SECOND`` which is 1,000,000 Hz (1 MHz): resolution of 1 us. * ``times()`` frequency is ``os.sysconf("SC_CLK_TCK")`` (or the ``HZ`` constant) which is equal to 100 Hz: resolution of 10 ms. * ``resource.getrusage()``, ``os.wait3()`` and ``os.wait4()`` use the ``ru_usage`` structure. The type of the ``ru_usage.ru_utime`` and ``ru_usage.ru_stime`` fields is the ``timeval`` structure which has a resolution of 1 us. Windows ------- Clocks resolution measured in Python on Windows 8.1: ================= ============= Function Resolution ================= ============= monotonic() 15 ms monotonic_ns() 15 ms perf_counter() 100 ns perf_counter_ns() 100 ns process_time() 15.6 ms process_time_ns() 15.6 ms time() **894.1 us** time_ns() **318 us** ================= ============= The frequency of ``perf_counter()`` and ``perf_counter_ns()`` comes from ``QueryPerformanceFrequency()``. The frequency is usually 10 MHz: resolution of 100 ns. In old Windows versions, the frequency was sometimes 3,579,545 Hz (3.6 MHz): resolution of 279 ns. Analysis -------- The resolution of ``time.time_ns()`` is much better than ``time.time()``: **84 ns (2.8x better) vs 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows**. The ``time.time()`` resolution will becomes larger (worse) next years since every day adds 864,00,000,000,000 nanoseconds to the system clock which increases the precision loss. The difference between ``time.perf_counter()``, ``time.monotonic clock()``, ``time.process_time()`` and their nanosecond variant is not visible in this quick script since the script runs less than 1 minute, and the uptime of the computer used to run the script was smaller than 1 week. A significant difference should be seen with an uptime of at least 104 days. ``resource.getrusage()`` and ``times()`` have a resolution greater or equal to 1 microsecond, and so don't need a variant with nanosecond resolution. .. note:: Internally, Python starts ``monotonic()`` and ``perf_counter()`` clocks at zero on some platforms which indirectly reduce the precision loss. Links ===== * `bpo-31784: Implementation of the PEP 564 `_ Copyright ========= This document has been placed in the public domain.