diff --git a/pep-0539.txt b/pep-0539.txt index d7bba3677..bdb986656 100644 --- a/pep-0539.txt +++ b/pep-0539.txt @@ -16,7 +16,9 @@ Abstract The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, -while deprecating the existing API. +while deprecating the existing API. The new API is named "Thread Specific +Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the origin of +the name). Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, ``pythread.h``, is not @@ -54,8 +56,10 @@ These would be superseded by a new set of analogous functions:: PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key) PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value) PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key) - PyAPI_FUNC(void) PyThread_tss_delete_value(Py_tss_t *key) - PyAPI_FUNC(void) PyThread_ReInitTSS(void) + +``PyThread_delete_key_value`` and ``PyThread_ReInitTLS`` don't provide +on the new TSS API because these functions are for the CPython's own TLS +implementation that was removed [8]_. The specification also adds a few new features: @@ -64,10 +68,10 @@ The specification also adds a few new features: typedef struct { bool _is_initialized; - NATIVE_TLS_KEY_T _key; + NATIVE_TSS_KEY_T _key; } Py_tss_t; - where ``NATIVE_TLS_KEY_T`` is a macro whose value depends on the + where ``NATIVE_TSS_KEY_T`` is a macro whose value depends on the underlying native TLS implementation (e.g. ``pthread_key_t``). * A constant default value for ``Py_tss_t`` variables, @@ -82,9 +86,10 @@ The specification also adds a few new features: The first two are needed for dynamic (de-)allocation of a `Py_tss_t`, particularly in extension modules built with `Py_LIMITED_API`, where static allocation of this type is not possible due to its implementation - being opaque at build time. ``PyThread_tss_is_created`` returns ``true`` - if the given ``Py_tss_t`` has been initialized (i.e. by - ``PyThread_tss_create``). + being opaque at build time. The value returned by ``PyThread_tss_alloc`` + is the same state (not yet initialized key) as statically allocated with + ``Py_tss_NEEDS_INIT``. ``PyThread_tss_is_created`` returns ``true`` if the + given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``). The new ``PyThread_tss_`` functions are almost exactly analogous to their original counterparts with a minor difference: Whereas @@ -92,7 +97,7 @@ original counterparts with a minor difference: Whereas ``int``, ``PyThread_tss_create`` takes a ``Py_tss_t*`` as an argument and returns an ``int`` status code. The behavior of ``PyThread_tss_create`` is undefined if the value pointed to by the ``key`` argument is not initialized -by ``Py_tss_NEEDS_INIT``. The returned status status code is zero on success +by ``Py_tss_NEEDS_INIT``. The returned status code is zero on success and non-zero on failure. The meanings of non-zero status codes are not otherwise defined by this specification. @@ -135,6 +140,18 @@ The rest of the API is used analogously to the old API:: assert(!PyThread_tss_is_created(&tss_key)); +Platform Support Changes +======================== + +A new "Native thread implementation" section will be added to PEP 11 that +states: + +* As of CPython 3.7, in the case of enabling thread support, all platforms are + required to provide at least one of native thread implementation (as of + pthreads or Windows) to implement TSS API. Any TSS API problems that occur + in the implementation without native thread will be closed as "won't fix". + + Motivation ========== @@ -144,15 +161,15 @@ TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an ``int``, and so it has been to the time of writing. This used CPython's own TLS implementation, -the current generation of which can still be found, largely unchanged, in +but the current generation of which hasn't been used, largely unchanged, in Python/thread.c. Support for implementation of the API on top of native -thread implementations (NT and pthreads) was added much later, and the -built-in implementation may still be used on other platforms. +thread implementations (pthreads and Windows) was added much later, and the +own implementation has been no longer necessary and removed [9]_. The problem with the choice of ``int`` to represent a TLS key, is that while -it was fine for CPython's internal TLS implementation, and happens to be -compatible with NT (which uses ``DWORD`` for the analogous data), it is not -compatible with the POSIX standard for the pthreads API, which defines +it was fine for CPython's own TLS implementation, and happens to be +compatible with Windows (which uses ``DWORD`` for the analogous data), it is +not compatible with the POSIX standard for the pthreads API, which defines ``pthread_key_t`` as an opaque type not further defined by the standard (as with ``Py_tss_t`` described above). This leaves it up to the underlying implementation how a ``pthread_key_t`` value is used to look up @@ -185,8 +202,7 @@ Rationale for Proposed Solution =============================== The use of an opaque type (``Py_tss_t``) to key TLS values allows the API to -be compatible, at least in this regard, with CPython's internal TLS -implementation, as well as all present (NT and posix) and future (C11?) +be compatible, with all present (posix and Windows) and future (C11?) native TLS implementations supported by CPython, as it allows the definition of ``Py_tss_t`` to depend on the underlying implementation. @@ -204,8 +220,8 @@ by the fact that not all native TLS implementations define a sentinel value for uninitialized TLS keys. For example, on Windows a TLS key is represented by a ``DWORD`` (``unsigned int``) and its value must be treated as opaque [3]_. So there is no unsigned integer value that can be safely -used to represent an uninititalized TLS key on Windows. Likewise, POSIX -does not specify a sentintel for an uninitialized ``pthread_key_t``, instead +used to represent an uninitialized TLS key on Windows. Likewise, POSIX +does not specify a sentinel for an uninitialized ``pthread_key_t``, instead relying on the ``pthread_once`` interface to ensure that a given TLS key is initialized only once per-process. Therefore, the ``Py_tss_t`` type contains an explicit ``._is_initialized`` that can indicate the key's @@ -241,13 +257,14 @@ Rejected Ideas Python should not be hobbled on affected platforms despite them being otherwise perfectly capable of running multi-threaded Python. -* Affected platforms should not define ``Py_HAVE_NATIVE_TLS``: This is a more - acceptable alternative to the previous idea, and in fact there is a patch to - do just that [4]_. However, CPython's internal TLS implementation being - "slower and clunkier" in general than native implementations still needlessly - hobbles performance on affected platforms. At least one other module - (``tracemalloc``) is also broken if Python is built without - ``Py_HAVE_NATIVE_TLS``. +* Affected platforms should use CPython's own TLS implementation instead of + native TLS implementation: This is a more acceptable alternative to the + previous idea, and in fact there had been a patch to do just that [4]_. + However, the own implementation being "slower and clunkier" in general + than native implementations still needlessly hobbles performance on affected + platforms. At least one other module (``tracemalloc``) is also broken if + Python is built without native implementation. And this idea cannot be + adopted because the own implementation was removed. * Keep the existing API, but work around the issue by providing a mapping from ``pthread_key_t`` values to ``int`` values. A couple attempts were made at @@ -263,7 +280,13 @@ Implementation ============== An initial version of a patch [7]_ is available on the bug tracker for this -issue. +issue. Since the migration to Github, it's being developed in the +``pep539-tss-api`` feature branch [10]_ in Masayuki Yamamoto's fork of the +CPython repository on Github. A work-in-progress PR is available at [11]_. + +This reference implementation covers not only the enhancement request in API +features, but also the client codes fix needed to replace the existing TLS API +with the new TSS API. Copyright @@ -282,3 +305,7 @@ References and Footnotes .. [5] http://bugs.python.org/file44269/issue25658-1.patch .. [6] http://bugs.python.org/file44303/key-constant-time.diff .. [7] http://bugs.python.org/file46379/pythread-tss-3.patch +.. [8] http://bugs.python.org/issue25658#msg298342 +.. [9] http://bugs.python.org/issue30832 +.. [10] https://github.com/python/cpython/compare/master...ma8ma:pep539-tss-api +.. [11] https://github.com/python/cpython/pull/1362