PEP: 418 Title: Add monotonic and high-resolution time functions Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-March-2012 Python-Version: 3.3 Abstract ======== Add time.monotonic(fallback=True) and time.highres() functions to Python 3.3. Rationale ========= Use cases: * Display the current time to a human (e.g. display a calendar or draw a wall clock): use system clock, i.e. time.time() or datetime.datetime.now(). * Benchmark, profiling, timeout: time.highres(). * Event scheduler: time.monotonic(), or time.monotonic(fallback=False). Functions ========= To fulfill the use cases, the functions' properties are: * time.time(): system clock, "wall clock". * time.highres(): clock with the best accuracy. * time.monotonic(fallback=True): monotonic clock. If no monotonic clock is available, falls back to system clock by default, or raises an OSError if *fallback* is False. time.monotonic(fallback=True) cannot go backward. time.time() ----------- The system time is the "wall clock". It can be set manually by the system administrator or automatically by a NTP daemon. It can jump backward and forward. It is not monotonic. It is available on all platforms and cannot fail. Pseudo-code [#pseudo]_:: if os.name == "nt": def time(): return _time.GetSystemTimeAsFileTime() else: def time(): if hasattr(time, "clock_gettime"): try: # resolution = 1 nanosecond return time.clock_gettime(time.CLOCK_REALTIME) except OSError: # CLOCK_REALTIME is not supported (unlikely) pass if hasattr(_time, "gettimeofday"): try: # resolution = 1 microsecond return _time.gettimeofday() except OSError: # gettimeofday() should not fail pass if hasattr(_time, "ftime"): # resolution = 1 millisecond return _time.ftime() else: # resolution = 1 second return _time.time() time.monotonic(fallback=True) ----------------------------- Clock that cannot go backward, its rate is as steady as possible. Its rate may be adjusted by NTP. The reference point of the returned value is undefined so only the difference of consecutive calls is valid. By default, it falls back to the system clock if no monotonic clock is available or if the monotonic clock failed, and so it cannot fail. If fallback is False, it raises OSError if the monotonic clock failed and NotImplementedError if the platform does not provide a monotonic clock (e.g. on GNU/Hurd). The elapsed time may or may not include time the system spends in sleep or hibernation; this depends on the operating system. Pseudo-code [#pseudo]_:: if os.name == 'nt': # GetTickCount64() requires Windows Vista, Server 2008 or later if hasattr(time, '_GetTickCount64'): def monotonic(fallback=True): return _time.GetTickCount64() else: def monotonic(fallback=True): ticks = _time.GetTickCount() if ticks < monotonic.last: # Integer overflow detected monotonic.delta += 2**32 monotonic.last = ticks return ticks + monotonic.delta monotonic.last = 0 monotonic.delta = 0 elif os.name == 'mac': def monotonic(fallback=True): if monotonic.factor is None: factor = _time.mach_timebase_info() monotonic.factor = timebase[0] / timebase[1] return _time.mach_absolute_time() * monotonic.factor monotonic.factor = None elif os.name.startswith('sunos'): def monotonic(fallback=True): if monotonic.use_clock_highres: try: time.clock_gettime(time.CLOCK_HIGHRES) except OSError: monotonic.use_clock_highres = False if monotonic.use_gethrtime: try: return time.gethrtime() except OSError: if not fallback: raise monotonic.use_gethrtime = False return time.time() monotonic.use_clock_highres = (hasattr(time, 'clock_gettime') and hasattr(time, 'CLOCK_HIGHRES')) monotonic.use_gethrtime = True elif hasattr(time, "clock_gettime"): def monotonic(fallback=True): while monotonic.clocks: try: clk_id = monotonic.clocks[0] return time.clock_gettime(clk_id) except OSError: # CLOCK_MONOTONIC_RAW requires a Linux kernel >= 2.6.28 if len(monotonic.clocks) == 1 and not fallback: raise del monotonic.clocks[0] return time.time() monotonic.clocks = [] if hasattr(time, 'CLOCK_MONOTONIC_RAW'): monotonic.clocks.append(time.CLOCK_MONOTONIC_RAW) if hasattr(time, 'CLOCK_HIGHRES'): monotonic.clocks.append(time.CLOCK_HIGHRES) monotonic.clocks.append(time.CLOCK_MONOTONIC) else: def monotonic(fallback=True): if not fallback: raise NotImplementedError("you platform does not provide any monotonic clock") return time.time() On Windows, QueryPerformanceCounter() is not used even though it has a better resolution than GetTickCount(). It is not reliable and has too many issues. .. note:: time.monotonic() detects GetTickCount() integer overflow (32 bits, roll-over after 49.7 days): it increases a delta by 2\ :sup:`32` each time than an overflow is detected. The delta is stored in the process-local state and so the value of time.monotonic() may be different in two Python processes. time.highres() -------------- Clock with the best available resolution. It is available on all platforms and cannot fail. Pseudo-code:: def highres(): if highres.use_performance_counter: try: return _time.QueryPerformanceCounter() except OSError: # QueryPerformanceFrequency() may fail, if the installed # hardware does not support a high-resolution performance # counter for example highres.use_performance_counter = False if highres.use_monotonic: # Monotonic clock is preferred over system clock try: return time.monotonic() except OSError: highres.use_monotonic = False return time.time() highres.use_performance_counter = (os.name == 'nt') highres.use_monotonic = hasattr(time, 'monotonic') Hardware clocks =============== * HPET: An HPET chip consists of a 64-bit up-counter (main counter) counting at least at 10 MHz and a set of up to 256 comparators (at least 3). Each HPET can have up to 32 timers. * TSC (Time Stamp Counter): Historically, the TSC increased with every internal processor clock cycle, but now the rate is usually constant (even if the processor changes frequency) and usually equals the maximum processor frequency. The instructor RDTSC can be used to read this counter. * ACPI PMTMR (power management timer): ACPI 24-bit timer with a frequency of 3.5 MHz (3,579,545 Hz). HPET can cause around 3 seconds of drift per day. * Cyclone: The Cyclone timer uses a 32-bit counter on IBM Extended X-Architecture (EXA) chipsets which include computers that use the IBM "Summit" series chipsets (ex: x440). This is available in IA32 and IA64 architectures. * PIT (programmable interrupt timer): Intel 8253/8254 chipsets with a configurable frequency in range 18.2 Hz - 1.2 MHz. It is a 16-bit counter. * RTC (Real-time clock). Most RTCs use a crystal oscillator with a frequency of 32,768 Hz Operating system clocks ======================= Monotonic clocks ---------------- ========================= =============== =============== ================ ==================== Name Resolution Accuracy Adjusted by NTP? Action on suspend ========================= =============== =============== ================ ==================== CLOCK_MONOTONIC_RAW 1 ns (*) No Stopped gethrtime 1 ns (*) No Not stopped CLOCK_HIGHRES 1 ns (*) No ? CLOCK_MONOTONIC 1 ns (*) Yes on Linux Stopped on Linux mach_absolute_time() 1 ns (*) No ? QueryPerformanceCounter() \- 0.3 ns - 5 ns No Accuracy issue GetTickCount[64]() 1 ms 1 ms - 15 ms No Include suspend time timeGetTime() 1 ms 1 ms - 15 ms No ? ========================= =============== =============== ================ ==================== (*) The accuracy of monotonic clocks depends on the operating system and the hardware clock. The resolution is the smallest difference between two timestamps supported by the format used by the clock. For example, clock_gettime() uses a timespec structure which has two integer fields, tv_sec and tv_nsec, so the resolution is 1 nanosecond. The accuracy is the effective smallest difference of two timestamps of the clock. It does not reflect the stability the clock rate. For example, QueryPerformanceCounter() has a good accuracy but is known to not have a steady rate. mach_absolute_time ^^^^^^^^^^^^^^^^^^ Mac OS X provides a monotonic clock: mach_absolute_time(). It is based on absolute elapsed time delta since system boot. It is not adjusted and cannot be set. mach_timebase_info() gives a fraction to convert the clock value to a number of nanoseconds. According to the documentation (`Technical Q&A QA1398 `_), mach_timebase_info() is always equal to one and never fails, even if the function may fail according to its prototype. mach_absolute_time() stops during a sleep on a PowerPC CPU, but not on an Intel CPU: `Different behaviour of mach_absolute_time() on i386/ppc `_. mach_absolute_time() has a resolution of 1 nanosecond. CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW represent monotonic time since some unspecified starting point. They cannot be set. Documentation: refer to the manual page of your operating system. Examples: * `FreeBSD clock_gettime() manual page `_ * `Linux clock_gettime() manual page `_ CLOCK_MONOTONIC is available at least on the following operating systems: * DragonFly BSD, FreeBSD >= 5.0, OpenBSD, NetBSD * Linux * Solaris The following operating systems don't support CLOCK_MONOTONIC: * GNU/Hurd (see `open issues/ clock_gettime `_) * Mac OS X * Windows CLOCK_MONOTONIC_RAW is specific to Linux. It is similar to CLOCK_MONOTONIC, but provides access to a raw hardware-based time that is not subject to NTP adjustments. CLOCK_MONOTONIC_RAW requires Linux 2.6.28 or later. On Linux, NTP may adjust the CLOCK_MONOTONIC rate, but it cannot jump backward. If available, CLOCK_MONOTONIC_RAW should be used instead of CLOCK_MONOTONIC to avoid the NTP adjustment. CLOCK_MONOTONIC stops while the machine is suspended. clock_gettime() fails if the system does not support the specified clock, even if the standard C library supports it. For example, CLOCK_MONOTONIC_RAW requires a kernel version 2.6.28 or later. clock_getres() gives the clock resolution. It is 1 nanosecond on Linux. .. note:: clock_gettime() requires to link the program against the rt (real-time) library. Windows: QueryPerformanceCounter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ High-resolution performance counter. It is monotonic. QueryPerformanceFrequency() gives its frequency. It has a much higher resolution, but has lower long term accuracy than GetTickCount() and timeGetTime() clocks. For example, it will drift compared to the low precision clocks. Documentation: * `MSDN: QueryPerformanceCounter() documentation `_ * `MSDN: QueryPerformanceFrequency() documentation `_ Hardware clocks used by QueryPerformanceCounter: * Windows XP: RDTSC instruction of Intel processors, the clock frequency is the frequency of the processor (between 200 MHz and 3 GHz, usually greater than 1 GHz nowadays). * Windows 2000: ACPI power management timer, frequency = 3,549,545 Hz. It can be forced through the "/usepmtimer" flag in boot.ini. .. * Windows 95/98: 8245 PIT chipset, frequency = 1,193,181 Hz QueryPerformanceFrequency() should only be called once: the frequency will not change while the system is running. It fails if the installed hardware does not support a high-resolution performance counter. QueryPerformanceCounter() cannot be adjusted: `SetSystemTimeAdjustment() `_ only adjusts the system time. Bugs: * The performance counter value may unexpectedly leap forward because of a hardware bug, see `KB274323`_. * On VirtualBox, QueryPerformanceCounter() does not increment the high part every time the low part overflows, see `Monotonic timers `_ (2009). * VirtualBox had a bug in its HPET virtualized device: QueryPerformanceCounter() did jump forward by approx. 42 seconds (`issue #8707 `_). * Windows XP had a bug (see `KB896256`_): on a multiprocessor computer, QueryPerformanceCounter() returned a different value for each processor. The bug was fixed in Windows XP SP2. * Issues with processor with variable frequency: the frequency is changed depending on the workload to reduce memory consumption. .. _KB896256: http://support.microsoft.com/?id=896256 .. _KB274323: http://support.microsoft.com/?id=274323 Windows: GetTickCount(), GetTickCount64() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ GetTickCount() and GetTickCount64() are monotonic, cannot fail and are not adjusted by SetSystemTimeAdjustment(). MSDN documentation: `GetTickCount() `_, `GetTickCount64() `_. The elapsed time retrieved by GetTickCount() or GetTickCount64() includes time the system spends in sleep or hibernation. GetTickCount64() was added to Windows Vista and Windows Server 2008. The clock resolution is 1 millisecond. Its accuracy is usually around 15 ms. It is possible to improve the accuracy using the `undocumented NtSetTimerResolution() function `_. There are applications using this undocumented function, example: `Timer Resolution `_. WaitForSingleObject() use the same timer than GetTickCount() with the same resolution. GetTickCount() has an accuracy of 55 ms on Windows 9x. Windows: timeGetTime ^^^^^^^^^^^^^^^^^^^^ The timeGetTime function retrieves the system time, in milliseconds. The system time is the time elapsed since Windows was started. Read the `timeGetTime() documentation `_. The return type of timeGetTime() is a 32-bit unsigned integer. As GetTickCount(), timeGetTime() rolls over after 2^32 milliseconds (49.7 days). The default precision of the timeGetTime function can be five milliseconds or more, depending on the machine. timeBeginPeriod() can be used to increase the precision of timeGetTime() up to 1 millisecond, but it negatively affects power consumption. .. note:: timeGetTime() and timeBeginPeriod() are part the Windows multimedia library and so require to link the program against winmm or to dynamically load the library. Solaris: CLOCK_HIGHRES ^^^^^^^^^^^^^^^^^^^^^^ The Solaris OS has a CLOCK_HIGHRES timer that attempts to use an optimal hardware source, and may give close to nanosecond resolution. CLOCK_HIGHRES is the nonadjustable, high-resolution clock. For timers created with a clockid_t value of CLOCK_HIGHRES, the system will attempt to use an optimal hardware source. Solaris: gethrtime ^^^^^^^^^^^^^^^^^^ The gethrtime() function returns the current high-resolution real time. Time is expressed as nanoseconds since some arbitrary time in the past; it is not correlated in any way to the time of day, and thus is not subject to resetting or drifting by way of adjtime() or settimeofday(). The hires timer is ideally suited to performance measurement tasks, where cheap, accurate interval timing is required. The linearity of gethrtime() is not preserved accross cpr suspend-resume cycle (`Bug 4272663 `_). Read the `gethrtime() manual page of Solaris 11 `_. On Solaris, gethrtime() is the same as clock_gettime(CLOCK_MONOTONIC). System time clocks ------------------ ========================= =============== =============== Name Resolution Accuracy ========================= =============== =============== CLOCK_REALTIME 1 ns (*) GetSystemTimeAsFileTime 100 ns 1 ms - 15 ms gettimeofday() 1 µs (*) ftime() 1 ms (*) time() 1 sec 1 sec ========================= =============== =============== (*) The accuracy of system clocks depends on the operating system and the hardware clock. On Windows, the accuracy is in the range 1 ms - 15 ms. Windows: GetSystemTimeAsFileTime ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The system time can be read using GetSystemTimeAsFileTime(), ftime() and time(). The system time resolution can be read using GetSystemTimeAdjustment(). The accuracy is usually between 1 millisecond and 15 milliseconds. Resolution: * GetSystemTimeAsFileTime(): 100 nanoseconds * ftime(): 1 millisecond * time(): 1 second The system time can be set using SetSystemTime(). System time on UNIX ^^^^^^^^^^^^^^^^^^^ gettimeofday(), ftime(), time() and clock_gettime(CLOCK_REALTIME) return the system clock. Resolution: * clock_gettime(): clock_getres(CLOCK_REALTIME), 1 nanosecond on Linux * gettimeofday(): 1 microsecond * ftime(): 1 millisecond * time(): 1 second The system time can be set using settimeofday() or clock_settime(CLOCK_REALTIME). Process and thread time ----------------------- The process and thread time cannot be set. They are not monotonic: the clocks stop while the process/thread is idle. Process ^^^^^^^ * Windows: GetProcessTimes() * clock_gettime(CLOCK_PROCESS_CPUTIME_ID): High-resolution per-process timer from the CPU. * clock(): * Windows: The elapsed wall-clock time since the start of the process (elapsed time in seconds times CLOCKS_PER_SEC). It can fail. * UNIX: returns an approximation of processor time used by the program. * times() * getrusage(): ru_utime and ru_stime fields Resolution: * clock() rate is CLOCKS_PER_SEC. It was called CLK_TCK in Microsoft C before 6.0. On Linux 3, clock() has a resolution of 1 microsecond. * The clock resolution can be read using clock_getres(). clock_getres(CLOCK_REALTIME) is 1 nanosecond on Linux. * GetProcessTimes(): call GetSystemTimeAdjustment(). Thread ^^^^^^ * Windows: GetThreadTimes() * clock_gettime(CLOCK_THREAD_CPUTIME_ID): Thread-specific CPU-time clock. Resolution: * CLOCK_THREAD_CPUTIME_ID: call clock_getres(). 1 nanosecond on Linux. * GetThreadTimes(): call GetSystemTimeAdjustment() See also pthread_getcpuclockid(). Windows: QueryUnbiasedInterruptTime ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Gets the current unbiased interrupt time from the biased interrupt time and the current sleep bias amount. This time is not affected by power management sleep transitions. The elapsed time retrieved by the QueryUnbiasedInterruptTime function includes only time that the system spends in the working state. QueryUnbiasedInterruptTime() is not monotonic. QueryUnbiasedInterruptTime() was introduced in Windows 7. Linux timers ------------ There were 4 implementations of the time in the Linux kernel: UTIME (1996), timer wheel (1997), HRT (2001) and hrtimers (2007). The latter is the result of the "high-res-timers" project started by George Anzinger in 2001, with contributions by Thomas Gleixner and Douglas Niehaus. hrtimers implementation was merged into Linux 2.6.21, released in 2007. hrtimers supports various clock sources. It sets a priority to each source to decide which one will be used. * TSC (Time Stamp Counter): Internal processor clock incremented at each processor cycle. Its frequency is the processor frequency and so usually higher than 1 GHz. Its priority is 300 by default, but falls to 0 if the processor frequency changes and the counter becomes unstable. * HPET: An HPET chip consists of a 64-bit up-counter (main counter) counting at least at 10 MHz and a set of up to 256 comparators (at least 3). Each HPET can have up to 32 timers. * PIT (programmable interrupt timer): Intel 8253/8254 chipsets with a configurable frequency in range 18.2 Hz - 1.2 MHz. Linux uses the frequency 1,193,181.8 Hz. It is a 16-bit counter. * PMTMR (power management timer): ACPI 24-bit timer with a frequency of 3.5 MHz (3,579,545 Hz). Its priority is 200 by default, but changes to 110 if the chipset is broken and need a software workaround. HPET can cause around 3 seconds of drift per day. * Cyclone: The Cyclone timer uses a 32-bit counter on IBM Extended X-Architecture (EXA) chipsets which include computers that use the IBM "Summit" series chipsets (ex: x440). This is available in IA32 and IA64 architectures. High-resolution timers are not supported on all hardware architectures. They are at least provided on x86/x86_64, ARM and PowerPC. The list of available clock sources can be read in /sys/devices/system/clocksource/clocksource0/available_clocksource. It is possible to force a clocksource at runtime by writing its name into /sys/devices/system/clocksource/clocksource0/current_clocksource. /proc/timer_list contains the list of all hardware timers. Read also the `time(7) manual page `_: "overview of time and timers". Alternatives: API design ======================== Other names for new functions ----------------------------- time.highres(): * time.hires(): "hires" can be read as "to hire" as in "he hires a car to go on holiday", rather than a "HIgh-RESolution clock". * time.timer(): "it would be too easy to confuse with (or misspell as) time.time()" time.monotonic(): * time.steady(): no OS provides a clock advancing at a steady rate, so "steady" should be avoided. * time.try_monotonic(): it is a clear and obvious solution for the use-case of "I prefer the monotonic clock, if it is available, otherwise I'll take my chances with a best-effect clock." * time.wallclock(): it is not the system time aka the "wall clock", but a monotonic clock with an unspecified starting point One function, no flag --------------------- time.monotonic() returns (time: float, is_monotonic: bool). An alternative is to use a function attribute: time.monotonic.is_monotonic. The attribute value would be None before the first call to time.monotonic(). Working around operating system bugs? ===================================== Should Python ensure manually that a monotonic clock is truly monotonic by computing the maximum with the clock value and the previous value? Since it's relatively straightforward to cache the last value returned using a static variable, it might be interesting to use this to make sure that the values returned are indeed monotonic. * Virtual machines provide less reliable clocks. * QueryPerformanceCounter() has known bugs (only one is not fixed yet) Python may only work around a specific known operating system bug: `KB274323`_ contains a code example to workaround the bug (use GetTickCount() to detect QueryPerformanceCounter() leap). Footnotes ========= .. [#pseudo] "_time" is an hypothetical module only used for the example. The time module is implemented in C and so there is no need for such module. Links ===== Related Python issues: * `Issue #12822: NewGIL should use CLOCK_MONOTONIC if possible. `_ * `Issue #14222: Use time.steady() to implement timeout `_ * `Issue #14397: Use GetTickCount/GetTickCount64 instead of QueryPerformanceCounter for monotonic clock `_ * `Issue #14428: Implementation of the PEP 418 `_ Libraries exposing monotonic clocks: * `Java: System.nanoTime `_ * `Qt library: QElapsedTimer `_ * `glib library: g_get_monotonic_time () `_ uses GetTickCount64()/GetTickCount() on Windows, clock_gettime(CLOCK_MONOTONIC) on UNIX or falls back to the system clock * `python-monotonic-time `_ (`github `_) * `monotonic_clock `_ * `Perl: Time::HiRes `_ exposes clock_gettime(CLOCK_MONOTONIC) * `Ruby: AbsoluteTime.now `_: use clock_gettime(CLOCK_MONOTONIC), mach_absolute_time() or gettimeofday(). "AbsoluteTime.monotonic?" method indicates if AbsoluteTime.now is monotonic or not. Time: * `hrtimers - subsystem for high-resolution kernel timers `_ * `C++ Timeout Specification `_ * `Windows: Game Timing and Multicore Processors `_ * `Implement a Continuously Updating, High-Resolution Time Provider for Windows `_ * `clockspeed `_ uses a hardware tick counter to compensate for a persistently fast or slow system clock * `Retrieving system time `_ lists hardware clocks and time functions with their resolution and epoch or range * On Windows, the JavaScript runtime of Firefox interpolates GetSystemTimeAsFileTime() with QueryPerformanceCounter() to get an higher resolution. See the `Bug 363258 - bad millisecond resolution for (new Date).getTime() / Date.now() on Windows `_. * `When microseconds matter `_: How the IBM High Resolution Time Stamp Facility accurately measures itty bits of time Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: