From 86ca2506f321e206f2395bb09b40e5aca7f17ce8 Mon Sep 17 00:00:00 2001 From: Erik Bray Date: Fri, 8 Sep 2017 00:12:06 +0200 Subject: [PATCH] PEP 539: Update for draft implementation feedback * Don't use `bool` types in the implementation, per discussion at https://github.com/python/cpython/pull/1362#discussion_r136357583 * Update for removal of --without-threads * miscellaneous other clarifications and updates --- pep-0539.txt | 143 ++++++++++++++++++++++++++------------------------- 1 file changed, 73 insertions(+), 70 deletions(-) diff --git a/pep-0539.txt b/pep-0539.txt index a9905a7db..7444c5c8f 100644 --- a/pep-0539.txt +++ b/pep-0539.txt @@ -16,15 +16,15 @@ Abstract The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, -while deprecating the existing API. The new API is named "Thread Specific -Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the origin of -the name). +while deprecating the existing API. The new API is named the "Thread +Specific Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the +origin of the name). Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, ``pythread.h``, is not -included in ``Python.h`` either directly or indirectly), this proposal probably -only affects CPython, but might also affect other interpreter implementations -(PyPy?) that implement parts of the CPython API. +included in ``Python.h`` either directly or indirectly), this proposal +probably only affects CPython, but might also affect other interpreter +implementations (PyPy?) that implement parts of the CPython API. This is motivated primarily by the fact that the old API uses ``int`` to represent TLS keys across all platforms, which is neither POSIX-compliant, @@ -63,7 +63,7 @@ The specification also adds a few new features: depend on the underlying TLS implementation. It is defined:: typedef struct { - bool _is_initialized; + int _is_initialized; NATIVE_TSS_KEY_T _key; } Py_tss_t; @@ -77,25 +77,26 @@ The specification also adds a few new features: PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void) PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key) - PyAPI_FUNC(bool) PyThread_tss_is_created(Py_tss_t *key) + PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key) The first two are needed for dynamic (de-)allocation of a ``Py_tss_t``, particularly in extension modules built with ``Py_LIMITED_API``, where static allocation of this type is not possible due to its implementation - being opaque at build time. A value returned by ``PyThread_tss_alloc`` - is the same state as initialized by ``Py_tss_NEEDS_INIT``, or ``NULL`` for - dynamic allocation failure. The behavior of ``PyThread_tss_free`` involves - calling ``PyThread_tss_delete`` preventively, or is no-op if the value - pointed to by the ``key`` argument is ``NULL``. ``PyThread_tss_is_created`` - returns ``true`` if the given ``Py_tss_t`` has been initialized (i.e. by - ``PyThread_tss_create``). + being opaque at build time. A value returned by ``PyThread_tss_alloc`` is + in the same state as a value initialized with ``Py_tss_NEEDS_INIT``, or + ``NULL`` in the case of dynamic allocation failure. The behavior of + ``PyThread_tss_free`` involves calling ``PyThread_tss_delete`` + preventively, or is a no-op if the value pointed to by the ``key`` + argument is ``NULL``. ``PyThread_tss_is_created`` returns non-zero if the + given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``). The new TSS API does not provide functions which correspond to ``PyThread_delete_key_value`` and ``PyThread_ReInitTLS``, because these -functions are for the removed CPython's own TLS implementation, that is the -existing API behavior has become as follows: ``PyThread_delete_key_value(key)`` -is equal to ``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` -is no-op [8]_. +functions were needed only for CPython's now defunct built-in TLS +implementation; that is the existing behavior of these functions is treated +as follows: ``PyThread_delete_key_value(key)`` is equalivalent to +``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` is a +no-op [8]_. The new ``PyThread_tss_`` functions are almost exactly analogous to their original counterparts with a few minor differences: Whereas @@ -116,17 +117,17 @@ any size. This is especially necessary for extension modules built with value pointed to by the ``key`` argument is ``NULL``. Moreover, because of the use of ``Py_tss_t`` instead of ``int``, there are -additional behaviors which the existing API design would be carried over into -new API: The TSS key creation and deletion are parts of "do-if-needed" flow -and these features are silently skipped if already done--Calling -``PyThread_tss_create`` with an initialized key does nothing and returns -success soon. This is also the case of calling ``PyThread_tss_delete`` with -an uninitialized key. +behaviors in the new API which differ from the existing API with regard to +key creation and deletion. ``PyThread_tss_create`` can be called repeatedly +on the same key--calling it on an already initialized key is a no-op and +immediately returns success. Similarly for calling ``PyThread_tss_delete`` +with an uninitialized key. The behavior of ``PyThread_tss_delete`` is defined to change the key's -initialization state to "uninitialized" in order to restart the CPython -interpreter without terminating the process (e.g. embedding Python in an -application) [12]_. +initialization state to "uninitialized"--this allows, for example, +statically allocated keys to be reset to a sensible state when restarting +the CPython interpreter without terminating the process (e.g. embedding +Python in an application) [12]_. The old ``PyThread_*_key*`` functions will be marked as deprecated in the documentation, but will not generate runtime deprecation warnings. @@ -150,20 +151,20 @@ Features - create key - create key - set value - set value - get value - get value - delete value - (set ``NULL`` instead) [8]_ - - reinitialize keys (for - (unnecessary) [8]_ - after fork) + - reinitialize keys (after - (unnecessary) [8]_ + fork) - dynamically (de-)allocate key - check key's initialization state Default Value (``-1`` as key creation ``Py_tss_NEEDS_INIT`` failure) -Requirement native thread native thread +Requirement native threads native threads (since CPython 3.7 [9]_) -Restriction Not support platform where Unable to statically allocate - native TLS key is defined in key when ``Py_LIMITED_API`` - a way that cannot be safely is defined. - cast to ``int``. +Restriction No support for platforms Unable to statically allocate + where native TLS key is keys when ``Py_LIMITED_API`` + defined in a way that cannot is defined. + be safely cast to ``int``. ================= ============================= ============================= Example @@ -209,10 +210,10 @@ Platform Support Changes A new "Native Thread Implementation" section will be added to PEP 11 that states: -* As of CPython 3.7, in the case of enabling thread support, all platforms are - required to provide at least one of native thread implementation (as of - pthreads or Windows) to implement TSS API. Any TSS API problems that occur - in the implementation without native thread will be closed as "won't fix". +* As of CPython 3.7, all platforms are required to provide a native thread + implementation (such as pthreads or Windows) to implement the TSS + API. Any TSS API problems that occur in an implementation without native + threads will be closed as "won't fix". Motivation @@ -223,11 +224,11 @@ TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an ``int``, and so it has -been to the time of writing. This used CPython's own TLS implementation, -but the current generation of which hasn't been used, largely unchanged, in -Python/thread.c. Support for implementation of the API on top of native -thread implementations (pthreads and Windows) was added much later, and the -own implementation has been no longer necessary and removed [9]_. +been to the time of writing. This used CPython's own TLS implementation +which long remained unused, largely unchanged, in Python/thread.c. Support +for implementation of the API on top of native thread implementations +(pthreads and Windows) was added much later, and the built-in implementation +has been deemed no longer necessary and has since been removed [9]_. The problem with the choice of ``int`` to represent a TLS key, is that while it was fine for CPython's own TLS implementation, and happens to be @@ -265,16 +266,17 @@ Rationale for Proposed Solution =============================== The use of an opaque type (``Py_tss_t``) to key TLS values allows the API to -be compatible, with all present (POSIX and Windows) and future (C11?) -native TLS implementations supported by CPython, as it allows the definition -of ``Py_tss_t`` to depend on the underlying implementation. +be compatible with all present (POSIX and Windows) and future (C11?) native +TLS implementations supported by CPython, as it allows the definition of +``Py_tss_t`` to depend on the underlying implementation. Since the existing TLS API has been available in *the limited API* [13]_ for -some platforms (e.g. Linux), CPython makes an effort to provide the new TSS API -at that level likewise. Note, however, that ``Py_tss_t`` definition becomes to -be an opaque struct when ``Py_LIMITED_API`` is defined, because exposing -``NATIVE_TSS_KEY_T`` as part of the limited API would prevent us from switching -native thread implementation without rebuilding extension module. +some platforms (e.g. Linux), CPython makes an effort to provide the new TSS +API at that level likewise. Note, however, that the ``Py_tss_t`` definition +becomes to be an opaque struct when ``Py_LIMITED_API`` is defined, because +exposing ``NATIVE_TSS_KEY_T`` as part of the limited API would prevent us +from switching native thread implementation without rebuilding extension +modules. A new API must be introduced, rather than changing the function signatures of the current API, in order to maintain backwards compatibility. The new API @@ -316,29 +318,29 @@ Rejected Ideas wishing to be supported by CPython should follow the requirements of PEP-11. As explained above, while this would be a fair argument if CPython were being to asked to make changes to support particular quirks - or features of a specific platform, in this case it is quirk of CPython + or features of a specific platform, in this case it is a quirk of CPython that prevents it from being used to its full potential on otherwise POSIX-compliant platforms. The fact that the current implementation happens to work on Linux is a happy accident, and there's no guarantee that this will never change. * Affected platforms should just configure Python ``--without-threads``: - This is a possible temporary workaround to the issue, but only that. - Python should not be hobbled on affected platforms despite them being - otherwise perfectly capable of running multi-threaded Python. + this is no longer an option as the ``--without-threads`` option has + been removed for Python 3.7 [16]_. -* Affected platforms should use CPython's own TLS implementation instead of - native TLS implementation: This is a more acceptable alternative to the - previous idea, and in fact there had been a patch to do just that [4]_. - However, the own implementation being "slower and clunkier" in general - than native implementations still needlessly hobbles performance on affected - platforms. At least one other module (``tracemalloc``) is also broken if - Python is built without native implementation. And this idea cannot be - adopted because the own implementation was removed. +* Affected platforms should use CPython's built-in TLS implementation + instead of a native TLS implementation: This is a more acceptable + alternative to the previous idea, and in fact there had been a patch to do + just that [4]_. However, the built-in implementation being "slower and + clunkier" in general than native implementations still needlessly hobbles + performance on affected platforms. At least one other module + (``tracemalloc``) is also broken if Python is built without a native TLS + implementation. This idea also cannot be adopted because the built-in + implementation has since been removed. * Keep the existing API, but work around the issue by providing a mapping from ``pthread_key_t`` values to ``int`` values. A couple attempts were made at - this ([5]_, [6]_), but this only injects needless complexity and overhead + this ([5]_, [6]_), but this injects needless complexity and overhead into performance-critical code on platforms that are not currently affected by this issue (such as Linux). Even if use of this workaround were made conditional on platform compatibility, it introduces platform-specific code @@ -350,13 +352,13 @@ Implementation ============== An initial version of a patch [7]_ is available on the bug tracker for this -issue. Since the migration to Github, it's being developed in the +issue. Since the migration to GitHub, its development has continued in the ``pep539-tss-api`` feature branch [10]_ in Masayuki Yamamoto's fork of the -CPython repository on Github. A work-in-progress PR is available at [11]_. +CPython repository on GitHub. A work-in-progress PR is available at [11]_. -This reference implementation covers not only the enhancement request in API -features, but also the client codes fix needed to replace the existing TLS API -with the new TSS API. +This reference implementation covers not only the new API implementation +features, but also the client code updates needed to replace the existing +TLS API with the new TSS API. Copyright @@ -384,3 +386,4 @@ References and Footnotes (https://www.python.org/dev/peps/pep-0384/) .. [14] http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create.html .. [15] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=404 +.. [16] https://bugs.python.org/issue31370