From f9a66fe5118b5e08e1a7c2f546188afa9db4d0c2 Mon Sep 17 00:00:00 2001 From: Mariatta Date: Mon, 6 Feb 2017 13:14:46 -0800 Subject: [PATCH] Convert PEPs 213, 241, 291, 298, 311 (#199) Further progress on issue #4 PEP 213: - Fix typo in the word miniscule -> minuscule PEP 241 PEP 291 PEP 298 PEP 311 --- pep-0213.txt | 351 +++++++++++++++++++++++++----------------------- pep-0241.txt | 355 ++++++++++++++++++++++++++----------------------- pep-0291.txt | 218 ++++++++++++++++-------------- pep-0298.txt | 307 +++++++++++++++++++++--------------------- pep-0311.txt | 366 +++++++++++++++++++++++++++------------------------ 5 files changed, 845 insertions(+), 752 deletions(-) diff --git a/pep-0213.txt b/pep-0213.txt index 0bb532739..4f8acf132 100644 --- a/pep-0213.txt +++ b/pep-0213.txt @@ -5,231 +5,248 @@ Last-Modified: $Date$ Author: paul@prescod.net (Paul Prescod) Status: Deferred Type: Standards Track +Content-Type: text/x-rst Created: 21-Jul-2000 Python-Version: 2.1 Post-History: Introduction +============ - It is possible (and even relatively common) in Python code and - in extension modules to "trap" when an instance's client code - attempts to set an attribute and execute code instead. In other - words, it is possible to allow users to use attribute assignment/ - retrieval/deletion syntax even though the underlying implementation - is doing some computation rather than directly modifying a - binding. +It is possible (and even relatively common) in Python code and +in extension modules to "trap" when an instance's client code +attempts to set an attribute and execute code instead. In other +words, it is possible to allow users to use attribute assignment/ +retrieval/deletion syntax even though the underlying implementation +is doing some computation rather than directly modifying a +binding. - This PEP describes a feature that makes it easier, more efficient - and safer to implement these handlers for Python instances. +This PEP describes a feature that makes it easier, more efficient +and safer to implement these handlers for Python instances. Justification +============= - Scenario 1: +Scenario 1 +---------- - You have a deployed class that works on an attribute named - "stdout". After a while, you think it would be better to - check that stdout is really an object with a "write" method - at the moment of assignment. Rather than change to a - setstdout method (which would be incompatible with deployed - code) you would rather trap the assignment and check the - object's type. +You have a deployed class that works on an attribute named +"stdout". After a while, you think it would be better to +check that stdout is really an object with a "write" method +at the moment of assignment. Rather than change to a +setstdout method (which would be incompatible with deployed +code) you would rather trap the assignment and check the +object's type. - Scenario 2: +Scenario 2 +---------- - You want to be as compatible as possible with an object - model that has a concept of attribute assignment. It could - be the W3C Document Object Model or a particular COM - interface (e.g. the PowerPoint interface). In that case - you may well want attributes in the model to show up as - attributes in the Python interface, even though the - underlying implementation may not use attributes at all. +You want to be as compatible as possible with an object +model that has a concept of attribute assignment. It could +be the W3C Document Object Model or a particular COM +interface (e.g. the PowerPoint interface). In that case +you may well want attributes in the model to show up as +attributes in the Python interface, even though the +underlying implementation may not use attributes at all. - Scenario 3: +Scenario 3 +---------- - A user wants to make an attribute read-only. +A user wants to make an attribute read-only. - In short, this feature allows programmers to separate the - interface of their module from the underlying implementation - for whatever purpose. Again, this is not a new feature but - merely a new syntax for an existing convention. +In short, this feature allows programmers to separate the +interface of their module from the underlying implementation +for whatever purpose. Again, this is not a new feature but +merely a new syntax for an existing convention. Current Solution +================ - To make some attributes read-only: +To make some attributes read-only:: - class foo: + class foo: def __setattr__( self, name, val ): - if name=="readonlyattr": - raise TypeError - elif name=="readonlyattr2": - raise TypeError - ... - else: - self.__dict__["name"]=val + if name=="readonlyattr": + raise TypeError + elif name=="readonlyattr2": + raise TypeError + ... + else: + self.__dict__["name"]=val - This has the following problems: +This has the following problems: - 1. The creator of the method must be intimately aware of whether - somewhere else in the class hiearchy __setattr__ has also been - trapped for any particular purpose. If so, she must specifically - call that method rather than assigning to the dictionary. There - are many different reasons to overload __setattr__ so there is a - decent potential for clashes. For instance object database - implementations often overload setattr for an entirely unrelated - purpose. +1. The creator of the method must be intimately aware of whether + somewhere else in the class hierarchy ``__setattr__`` has also been + trapped for any particular purpose. If so, she must specifically + call that method rather than assigning to the dictionary. There + are many different reasons to overload ``__setattr__`` so there is a + decent potential for clashes. For instance object database + implementations often overload setattr for an entirely unrelated + purpose. - 2. The string-based switch statement forces all attribute handlers - to be specified in one place in the code. They may then dispatch - to task-specific methods (for modularity) but this could cause - performance problems. +2. The string-based switch statement forces all attribute handlers + to be specified in one place in the code. They may then dispatch + to task-specific methods (for modularity) but this could cause + performance problems. - 3. Logic for the setting, getting and deleting must live in - __getattr__, __setattr__ and __delattr__. Once again, this can be - mitigated through an extra level of method call but this is - inefficient. +3. Logic for the setting, getting and deleting must live in + ``__getattr__``, ``__setattr__`` and ``__delattr__``. Once again, this can + be mitigated through an extra level of method call but this is + inefficient. Proposed Syntax - - Special methods should declare themselves with declarations of the - following form: +=============== - class x: - def __attr_XXX__(self, op, val ): - if op=="get": - return someComputedValue(self.internal) - elif op=="set": - self.internal=someComputedValue(val) - elif op=="del": - del self.internal +Special methods should declare themselves with declarations of the +following form:: - Client code looks like this: + class x: + def __attr_XXX__(self, op, val ): + if op=="get": + return someComputedValue(self.internal) + elif op=="set": + self.internal=someComputedValue(val) + elif op=="del": + del self.internal - fooval=x.foo - x.foo=fooval+5 - del x.foo + +Client code looks like this:: + + fooval=x.foo + x.foo=fooval+5 + del x.foo Semantics - - Attribute references of all three kinds should call the method. - The op parameter can be "get"/"set"/"del". Of course this string - will be interned so the actual checks for the string will be - very fast. - - It is disallowed to actually have an attribute named XXX in the - same instance as a method named __attr_XXX__. +========= - An implementation of __attr_XXX__ takes precedence over an - implementation of __getattr__ based on the principle that - __getattr__ is supposed to be invoked only after finding an - appropriate attribute has failed. +Attribute references of all three kinds should call the method. +The op parameter can be "get"/"set"/"del". Of course this string +will be interned so the actual checks for the string will be +very fast. - An implementation of __attr_XXX__ takes precedence over an - implementation of __setattr__ in order to be consistent. The - opposite choice seems fairly feasible also, however. The same - goes for __del_y__. +It is disallowed to actually have an attribute named XXX in the +same instance as a method named __attr_XXX__. + +An implementation of __attr_XXX__ takes precedence over an +implementation of ``__getattr__`` based on the principle that +``__getattr__`` is supposed to be invoked only after finding an +appropriate attribute has failed. + +An implementation of __attr_XXX__ takes precedence over an +implementation of ``__setattr__`` in order to be consistent. The +opposite choice seems fairly feasible also, however. The same +goes for __del_y__. Proposed Implementation - - There is a new object type called an attribute access handler. - Objects of this type have the following attributes: +======================= - name (e.g. XXX, not __attr__XXX__ - method (pointer to a method object - - In PyClass_New, methods of the appropriate form will be detected and - converted into objects (just like unbound method objects). These are - stored in the class __dict__ under the name XXX. The original method - is stored as an unbound method under its original name. +There is a new object type called an attribute access handler. +Objects of this type have the following attributes:: - If there are any attribute access handlers in an instance at all, - a flag is set. Let's call it "I_have_computed_attributes" for - now. Derived classes inherit the flag from base classes. Instances - inherit the flag from classes. - - A get proceeds as usual until just before the object is returned. - In addition to the current check whether the returned object is a - method it would also check whether a returned object is an access - handler. If so, it would invoke the getter method and return - the value. To remove an attribute access handler you could directly - fiddle with the dictionary. - - A set proceeds by checking the "I_have_computed_attributes" flag. If - it is not set, everything proceeds as it does today. If it is set - then we must do a dictionary get on the requested object name. If it - returns an attribute access handler then we call the setter function - with the value. If it returns any other object then we discard the - result and continue as we do today. Note that having an attribute - access handler will mildly affect attribute "setting" performance for - all sets on a particular instance, but no more so than today, using - __setattr__. Gets are more efficient than they are today with - __getattr__. - - The I_have_computed_attributes flag is intended to eliminate the - performance degradation of an extra "get" per "set" for objects not - using this feature. Checking this flag should have miniscule - performance implications for all objects. + name (e.g. XXX, not __attr__XXX__) + method (pointer to a method object) - The implementation of delete is analogous to the implementation - of set. +In PyClass_New, methods of the appropriate form will be detected and +converted into objects (just like unbound method objects). These are +stored in the class ``__dict__`` under the name XXX. The original method +is stored as an unbound method under its original name. + +If there are any attribute access handlers in an instance at all, +a flag is set. Let's call it "I_have_computed_attributes" for +now. Derived classes inherit the flag from base classes. Instances +inherit the flag from classes. + +A get proceeds as usual until just before the object is returned. +In addition to the current check whether the returned object is a +method it would also check whether a returned object is an access +handler. If so, it would invoke the getter method and return +the value. To remove an attribute access handler you could directly +fiddle with the dictionary. + +A set proceeds by checking the "I_have_computed_attributes" flag. If +it is not set, everything proceeds as it does today. If it is set +then we must do a dictionary get on the requested object name. If it +returns an attribute access handler then we call the setter function +with the value. If it returns any other object then we discard the +result and continue as we do today. Note that having an attribute +access handler will mildly affect attribute "setting" performance for +all sets on a particular instance, but no more so than today, using +``__setattr__``. Gets are more efficient than they are today with +``__getattr__``. + +The I_have_computed_attributes flag is intended to eliminate the +performance degradation of an extra "get" per "set" for objects not +using this feature. Checking this flag should have minuscule +performance implications for all objects. + +The implementation of delete is analogous to the implementation +of set. Caveats +======= - 1. You might note that I have not proposed any logic to keep - the I_have_computed_attributes flag up to date as attributes - are added and removed from the instance's dictionary. This is - consistent with current Python. If you add a __setattr__ method - to an object after it is in use, that method will not behave as - it would if it were available at "compile" time. The dynamism is - arguably not worth the extra implementation effort. This snippet - demonstrates the current behavior: +1. You might note that I have not proposed any logic to keep + the I_have_computed_attributes flag up to date as attributes + are added and removed from the instance's dictionary. This is + consistent with current Python. If you add a ``__setattr__`` method + to an object after it is in use, that method will not behave as + it would if it were available at "compile" time. The dynamism is + arguably not worth the extra implementation effort. This snippet + demonstrates the current behavior:: - >>> def prn(*args):print args - >>> class a: - ... __setattr__=prn - >>> a().foo=5 - (<__main__.a instance at 882890>, 'foo', 5) + >>> def prn(*args):print args + >>> class a: - >>> class b: pass - >>> bi=b() - >>> bi.__setattr__=prn - >>> b.foo=5 + ... __setattr__=prn + >>> a().foo=5 + (<__main__.a instance at 882890>, 'foo', 5) - 2. Assignment to __dict__["XXX"] can overwrite the attribute - access handler for __attr_XXX__. Typically the access handlers will - store information away in private __XXX variables + >>> class b: pass + >>> bi=b() + >>> bi.__setattr__=prn + >>> b.foo=5 - 3. An attribute access handler that attempts to call setattr or getattr - on the object itself can cause an infinite loop (as with __getattr__) - Once again, the solution is to use a special (typically private) - variable such as __XXX. + +2. Assignment to __dict__["XXX"] can overwrite the attribute + access handler for __attr_XXX__. Typically the access handlers will + store information away in private __XXX variables + + +3. An attribute access handler that attempts to call setattr or getattr + on the object itself can cause an infinite loop (as with ``__getattr__``) + Once again, the solution is to use a special (typically private) + variable such as __XXX. Note +==== - The descriptor mechanism described in PEP 252 is powerful enough - to support this more directly. A 'getset' constructor may be - added to the language making this possible: +The descriptor mechanism described in PEP 252 is powerful enough +to support this more directly. A 'getset' constructor may be +added to the language making this possible:: - class C: - def get_x(self): - return self.__x - def set_x(self, v): - self.__x = v - x = getset(get_x, set_x) + class C: + def get_x(self): + return self.__x + def set_x(self, v): + self.__x = v + x = getset(get_x, set_x) - Additional syntactic sugar might be added, or a naming convention - could be recognized. +Additional syntactic sugar might be added, or a naming convention +could be recognized. - -Local Variables: -mode: indented-text -indent-tabs-mode: nil -End: + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + End: diff --git a/pep-0241.txt b/pep-0241.txt index c7db81794..abf703e39 100644 --- a/pep-0241.txt +++ b/pep-0241.txt @@ -5,232 +5,261 @@ Last-Modified: $Date$ Author: A.M. Kuchling Status: Final Type: Standards Track +Content-Type: text/x-rst Created: 12-Mar-2001 Post-History: 19-Mar-2001 -Introduction - This PEP describes a mechanism for adding metadata to Python - packages. It includes specifics of the field names, and their - semantics and usage. +Introduction +============ + +This PEP describes a mechanism for adding metadata to Python +packages. It includes specifics of the field names, and their +semantics and usage. Including Metadata in Packages +============================== - The Distutils 'sdist' command will be modified to extract the - metadata fields from the arguments and write them to a file in the - generated zipfile or tarball. This file will be named PKG-INFO - and will be placed in the top directory of the source - distribution (where the README, INSTALL, and other files usually - go). +The Distutils 'sdist' command will be modified to extract the +metadata fields from the arguments and write them to a file in the +generated zipfile or tarball. This file will be named PKG-INFO +and will be placed in the top directory of the source +distribution (where the README, INSTALL, and other files usually +go). - Developers may not provide their own PKG-INFO file. The "sdist" - command will, if it detects an existing PKG-INFO file, terminate - with an appropriate error message. This should prevent confusion - caused by the PKG-INFO and setup.py files being out of sync. +Developers may not provide their own PKG-INFO file. The "sdist" +command will, if it detects an existing PKG-INFO file, terminate +with an appropriate error message. This should prevent confusion +caused by the PKG-INFO and setup.py files being out of sync. + +The PKG-INFO file format is a single set of RFC-822 headers +parseable by the rfc822.py module. The field names listed in the +following section are used as the header names. There's no +extension mechanism in this simple format; the Catalog and Distutils +SIGs will aim at getting a more flexible format ready for Python 2.2. - The PKG-INFO file format is a single set of RFC-822 headers - parseable by the rfc822.py module. The field names listed in the - following section are used as the header names. There's no - extension mechanism in this simple format; the Catalog and Distutils - SIGs will aim at getting a more flexible format ready for Python 2.2. - Fields +====== - This section specifies the names and semantics of each of the - supported metadata fields. - - Fields marked with "(Multiple use)" may be specified multiple - times in a single PKG-INFO file. Other fields may only occur - once in a PKG-INFO file. Fields marked with "(optional)" are - not required to appear in a valid PKG-INFO file, all other - fields must be present. +This section specifies the names and semantics of each of the +supported metadata fields. - Metadata-Version +Fields marked with "(Multiple use)" may be specified multiple +times in a single PKG-INFO file. Other fields may only occur +once in a PKG-INFO file. Fields marked with "(optional)" are +not required to appear in a valid PKG-INFO file, all other +fields must be present. - Version of the file format; currently "1.0" is the only - legal value here. +Metadata-Version +---------------- - Example: +Version of the file format; currently "1.0" is the only +legal value here. - Metadata-Version: 1.0 +Example:: - Name + Metadata-Version: 1.0 - The name of the package. +Name +---- - Example: +The name of the package. - Name: BeagleVote - - Version +Example:: - A string containing the package's version number. This - field should be parseable by one of the Version classes - (StrictVersion or LooseVersion) in the distutils.version - module. + Name: BeagleVote - Example: +Version +------- - Version: 1.0a2 - - Platform (multiple use) +A string containing the package's version number. This +field should be parseable by one of the Version classes +(StrictVersion or LooseVersion) in the distutils.version +module. - A comma-separated list of platform specifications, summarizing - the operating systems supported by the package. The major - supported platforms are listed below, but this list is - necessarily incomplete. +Example:: - POSIX, MacOS, Windows, BeOS, PalmOS. + Version: 1.0a2 - Binary distributions will use the Supported-Platform field in - their metadata to specify the OS and CPU for which the binary - package was compiled. The semantics of the Supported-Platform - are not specified in this PEP. +Platform (multiple use) +----------------------- - Example: +A comma-separated list of platform specifications, summarizing +the operating systems supported by the package. The major +supported platforms are listed below, but this list is +necessarily incomplete. - Platform: POSIX, Windows - - Summary +:: - A one-line summary of what the package does. + POSIX, MacOS, Windows, BeOS, PalmOS. - Example: +Binary distributions will use the Supported-Platform field in +their metadata to specify the OS and CPU for which the binary +package was compiled. The semantics of the Supported-Platform +are not specified in this PEP. - Summary: A module for collecting votes from beagles. - - Description (optional) +Example:: - A longer description of the package that can run to several - paragraphs. (Software that deals with metadata should not - assume any maximum size for this field, though one hopes that - people won't include their instruction manual as the - long-description.) + Platform: POSIX, Windows - Example: - - Description: This module collects votes from beagles - in order to determine their electoral wishes. - Do NOT try to use this module with basset hounds; - it makes them grumpy. - - Keywords (optional) +Summary +------- - A list of additional keywords to be used to assist searching - for the package in a larger catalog. +A one-line summary of what the package does. - Example: +Example:: - Keywords: dog puppy voting election - - Home-page (optional) + Summary: A module for collecting votes from beagles. - A string containing the URL for the package's home page. +Description (optional) +---------------------- - Example: +A longer description of the package that can run to several +paragraphs. (Software that deals with metadata should not +assume any maximum size for this field, though one hopes that +people won't include their instruction manual as the +long-description.) - Home-page: http://www.example.com/~cschultz/bvote/ - - Author (optional) +Example:: - A string containing at a minimum the author's name. Contact - information can also be added, separating each line with - newlines. + Description: This module collects votes from beagles + in order to determine their electoral wishes. + Do NOT try to use this module with basset hounds; + it makes them grumpy. - Example: +Keywords (optional) +------------------- - Author: C. Schultz - Universal Features Syndicate - Los Angeles, CA - - Author-email +A list of additional keywords to be used to assist searching +for the package in a larger catalog. - A string containing the author's e-mail address. It can contain - a name and e-mail address in the legal forms for a RFC-822 - 'From:' header. It's not optional because cataloging systems - can use the e-mail portion of this field as a unique key - representing the author. A catalog might provide authors the - ability to store their GPG key, personal home page, and other - additional metadata *about the author*, and optionally the - ability to associate several e-mail addresses with the same - person. Author-related metadata fields are not covered by this - PEP. +Example:: - Example: + Keywords: dog puppy voting election - Author-email: "C. Schultz" - - License - - A string selected from a short list of choices, specifying the - license covering the package. Some licenses result in the - software being freely redistributable, so packagers and - resellers can automatically know that they're free to - redistribute the software. Other licenses will require - a careful reading by a human to determine how the software can be - repackaged and resold. +Home-page (optional) +-------------------- - The choices are: +A string containing the URL for the package's home page. - Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", - Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown, - nocommercial, nosell, nosource, shareware, other +Example:: - Definitions of some of the licenses are: + Home-page: http://www.example.com/~cschultz/bvote/ - DFSG The license conforms to the Debian Free Software - Guidelines, but does not use one of the other - DFSG conforming licenses listed here. - More information is available at: - http://www.debian.org/social_contract#guidelines +Author (optional) +----------------- - Python Python 1.6 or higher license. Version 1.5.2 and - earlier are under the MIT license. +A string containing at a minimum the author's name. Contact +information can also be added, separating each line with +newlines. - public domain Software is public domain, not copyrighted. - unknown Status is not known - nocommercial Free private use but commercial use not permitted - nosell Free use but distribution for profit by arrangement - nosource Freely distributable but no source code - shareware Payment is requested if software is used - other General category for other non-DFSG licenses +Example:: - Some of these licenses can be interpreted to mean the software is - freely redistributable. The list of redistributable licenses is: + Author: C. Schultz + Universal Features Syndicate + Los Angeles, CA - Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", - Mozilla PL, "public domain", Python, Qt PL, Zope PL, - nosource, shareware +Author-email +------------ - Note that being redistributable does not mean a package - qualifies as free software, 'nosource' and 'shareware' being - examples. +A string containing the author's e-mail address. It can contain +a name and e-mail address in the legal forms for a RFC-822 +'From:' header. It's not optional because cataloging systems +can use the e-mail portion of this field as a unique key +representing the author. A catalog might provide authors the +ability to store their GPG key, personal home page, and other +additional metadata *about the author*, and optionally the +ability to associate several e-mail addresses with the same +person. Author-related metadata fields are not covered by this +PEP. - Example: +Example:: + + Author-email: "C. Schultz" + +License +------- + +A string selected from a short list of choices, specifying the +license covering the package. Some licenses result in the +software being freely redistributable, so packagers and +resellers can automatically know that they're free to +redistribute the software. Other licenses will require +a careful reading by a human to determine how the software can be +repackaged and resold. + +The choices are:: + + Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", + Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown, + nocommercial, nosell, nosource, shareware, other + +Definitions of some of the licenses are: + +============= =================================================== +DFSG The license conforms to the Debian Free Software + Guidelines, but does not use one of the other + DFSG conforming licenses listed here. + More information is available at: + http://www.debian.org/social_contract#guidelines + +Python Python 1.6 or higher license. Version 1.5.2 and + earlier are under the MIT license. + +public domain Software is public domain, not copyrighted. + +unknown Status is not known + +nocommercial Free private use but commercial use not permitted + +nosell Free use but distribution for profit by arrangement + +nosource Freely distributable but no source code + +shareware Payment is requested if software is used + +other General category for other non-DFSG licenses +============= =================================================== + +Some of these licenses can be interpreted to mean the software is +freely redistributable. The list of redistributable licenses is:: + + Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", + Mozilla PL, "public domain", Python, Qt PL, Zope PL, + nosource, shareware + +Note that being redistributable does not mean a package +qualifies as free software, 'nosource' and 'shareware' being +examples. + +Example:: + + License: MIT - License: MIT - Acknowledgements +================ - Many changes and rewrites to this document were suggested by the - readers of the Distutils SIG. In particular, Sean Reifschneider - often contributed actual text for inclusion in this PEP. - - The list of licenses was compiled using the SourceForge license - list and the CTAN license list compiled by Graham Williams; Carey - Evans also offered several useful suggestions on this list. +Many changes and rewrites to this document were suggested by the +readers of the Distutils SIG. In particular, Sean Reifschneider +often contributed actual text for inclusion in this PEP. + +The list of licenses was compiled using the SourceForge license +list and the CTAN license list compiled by Graham Williams; Carey +Evans also offered several useful suggestions on this list. Copyright +========= - This document has been placed in the public domain. +This document has been placed in the public domain. - -Local Variables: -mode: indented-text -indent-tabs-mode: nil -End: + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + End: diff --git a/pep-0291.txt b/pep-0291.txt index e0ab731c3..817090367 100644 --- a/pep-0291.txt +++ b/pep-0291.txt @@ -5,147 +5,161 @@ Last-Modified: $Date$ Author: nnorwitz@gmail.com (Neal Norwitz) Status: Final Type: Informational +Content-Type: text/x-rst Created: 06-Jun-2002 Python-Version: 2.3 Post-History: Abstract +======== - This PEP describes the packages and modules in the Python 2 - standard library which should remain backward compatible with - previous versions of Python. If a package is not listed here, - then it need only remain compatible with the version of Python it - is distributed with. +This PEP describes the packages and modules in the Python 2 +standard library which should remain backward compatible with +previous versions of Python. If a package is not listed here, +then it need only remain compatible with the version of Python it +is distributed with. - This PEP has no bearing on the Python 3 standard library. +This PEP has no bearing on the Python 3 standard library. Rationale +========= - Authors have various reasons why packages and modules should - continue to work with previous versions of Python. In order to - maintain backward compatibility for these modules while moving the - rest of the standard library forward, it is necessary to know - which modules can be modified and which should use old and - possibly deprecated features. +Authors have various reasons why packages and modules should +continue to work with previous versions of Python. In order to +maintain backward compatibility for these modules while moving the +rest of the standard library forward, it is necessary to know +which modules can be modified and which should use old and +possibly deprecated features. - Generally, authors should attempt to keep changes backward - compatible with the previous released version of Python in order - to make bug fixes easier to backport. +Generally, authors should attempt to keep changes backward +compatible with the previous released version of Python in order +to make bug fixes easier to backport. - In addition to a package or module being listed in this PEP, - authors must add a comment at the top of each file documenting - the compatibility requirement. +In addition to a package or module being listed in this PEP, +authors must add a comment at the top of each file documenting +the compatibility requirement. - When a major version of Python is released, a Subversion branch is - created for continued maintenance and bug fix releases. A package - version on a branch may have a different compatibility requirement - than the same package on the trunk (i.e. current bleeding-edge - development). Where appropriate, these branch compatibilities are - listed below. +When a major version of Python is released, a Subversion branch is +created for continued maintenance and bug fix releases. A package +version on a branch may have a different compatibility requirement +than the same package on the trunk (i.e. current bleeding-edge +development). Where appropriate, these branch compatibilities are +listed below. Features to Avoid +================= - The following list contains common features to avoid in order - to maintain backward compatibility with each version of Python. - This list is not complete! It is only meant as a general guide. +The following list contains common features to avoid in order +to maintain backward compatibility with each version of Python. +This list is not complete! It is only meant as a general guide. - Note that the features below were implemented in the version - following the one listed. For example, features listed next to - 1.5.2 were implemented in 2.0. +Note that the features below were implemented in the version +following the one listed. For example, features listed next to +1.5.2 were implemented in 2.0. - Version Features to Avoid - ------- ----------------- - 1.5.2 string methods, Unicode, list comprehensions, - augmented assignment (eg, +=), zip(), import x as y, - dict.setdefault(), print >> f, - calling f(*args, **kw), plus all features below +======= ====================================================== +Version Features to Avoid +======= ====================================================== +1.5.2 string methods, Unicode, list comprehensions, + augmented assignment (eg, +=), zip(), import x as y, + dict.setdefault(), print >> f, + calling f(\*args, \**kw), plus all features below - 2.0 nested scopes, rich comparisons, - function attributes, plus all features below +2.0 nested scopes, rich comparisons, + function attributes, plus all features below - 2.1 use of object or new-style classes, iterators, - using generators, nested scopes, or // - without from __future__ import ... statement, - isinstance(X, TYP) where TYP is a tuple of types, - plus all features below +2.1 use of object or new-style classes, iterators, + using generators, nested scopes, or // + without from __future__ import ... statement, + isinstance(X, TYP) where TYP is a tuple of types, + plus all features below - 2.2 bool, True, False, basestring, enumerate(), - {}.pop(), PendingDeprecationWarning, - Universal Newlines, plus all features below - plus all features below +2.2 bool, True, False, basestring, enumerate(), + {}.pop(), PendingDeprecationWarning, + Universal Newlines, plus all features below + plus all features below - 2.3 generator expressions, multi-line imports, - decorators, int/long unification, set/frozenset, - reversed(), sorted(), "".rsplit(), - plus all features below +2.3 generator expressions, multi-line imports, + decorators, int/long unification, set/frozenset, + reversed(), sorted(), "".rsplit(), + plus all features below - 2.4 with statement, conditional expressions, - combined try/except/finally, relative imports, - yield expressions or generator.throw/send/close(), - plus all features below +2.4 with statement, conditional expressions, + combined try/except/finally, relative imports, + yield expressions or generator.throw/send/close(), + plus all features below - 2.5 with statement without from __future__ import, - io module, str.format(), except as, - bytes, b'' literals, property.setter/deleter +2.5 with statement without from __future__ import, + io module, str.format(), except as, + bytes, b'' literals, property.setter/deleter +======= ====================================================== Backward Compatible Packages, Modules, and Tools +================================================ - Package/Module Maintainer(s) Python Version Notes - -------------- ------------- -------------- ----- - 2to3 Benjamin Peterson 2.5 - bsddb Greg Smith 2.1 - Barry Warsaw - compiler Jeremy Hylton 2.1 - decimal Raymond Hettinger 2.3 [2] - distutils Tarek Ziade 2.3 - email Barry Warsaw 2.1 / 2.3 [1] - pkgutil Phillip Eby 2.3 - platform Marc-Andre Lemburg 1.5.2 - pybench Marc-Andre Lemburg 1.5.2 [3] - sre Fredrik Lundh 2.1 - subprocess Peter Astrand 2.2 - wsgiref Phillip J. Eby 2.1 - xml (PyXML) Martin v. Loewis 2.0 - xmlrpclib Fredrik Lundh 2.1 - - Tool Maintainer(s) Python Version - ---- ------------- -------------- - None +============== ================== ============== ===== +Package/Module Maintainer(s) Python Version Notes +============== ================== ============== ===== +2to3 Benjamin Peterson 2.5 +bsddb - Greg Smith 2.1 + - Barry Warsaw +compiler Jeremy Hylton 2.1 +decimal Raymond Hettinger 2.3 [2] +distutils Tarek Ziade 2.3 +email Barry Warsaw 2.1 / 2.3 [1] +pkgutil Phillip Eby 2.3 +platform Marc-Andre Lemburg 1.5.2 +pybench Marc-Andre Lemburg 1.5.2 [3] +sre Fredrik Lundh 2.1 +subprocess Peter Astrand 2.2 +wsgiref Phillip J. Eby 2.1 +xml (PyXML) Martin v. Loewis 2.0 +xmlrpclib Fredrik Lundh 2.1 +============== ================== ============== ===== - Notes - ----- +==== ============= ============== +Tool Maintainer(s) Python Version +==== ============= ============== +None +==== ============= ============== - [1] The email package version 2 was distributed with Python up to - Python 2.3, and this must remain Python 2.1 compatible. email - package version 3 will be distributed with Python 2.4 and will - need to remain compatible only with Python 2.3. - [2] Specification updates will be treated as bugfixes and backported. - Python 2.3 compatibility will be kept for at least Python 2.4. - The decision will be revisited for Python 2.5 and not changed - unless compelling advantages arise. +Notes +----- - [3] pybench lives under the Tools/ directory. Compatibility with - older Python versions is needed in order to be able to compare - performance between Python versions. New features may still - be used in new tests, which may then be configured to fail - gracefully on import by the tool in older Python versions. +1. The email package version 2 was distributed with Python up to + Python 2.3, and this must remain Python 2.1 compatible. email + package version 3 will be distributed with Python 2.4 and will + need to remain compatible only with Python 2.3. + +2. Specification updates will be treated as bugfixes and backported. + Python 2.3 compatibility will be kept for at least Python 2.4. + The decision will be revisited for Python 2.5 and not changed + unless compelling advantages arise. + +3. pybench lives under the Tools/ directory. Compatibility with + older Python versions is needed in order to be able to compare + performance between Python versions. New features may still + be used in new tests, which may then be configured to fail + gracefully on import by the tool in older Python versions. Copyright +========= - This document has been placed in the public domain. +This document has been placed in the public domain. - -Local Variables: -mode: indented-text -indent-tabs-mode: nil -sentence-end-double-space: t -fill-column: 70 -End: + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + End: diff --git a/pep-0298.txt b/pep-0298.txt index ce74d173b..cf4bc8a97 100644 --- a/pep-0298.txt +++ b/pep-0298.txt @@ -5,229 +5,240 @@ Last-Modified: $Date$ Author: Thomas Heller Status: Withdrawn Type: Standards Track +Content-Type: text/x-rst Created: 26-Jul-2002 Python-Version: 2.3 Post-History: 30-Jul-2002, 1-Aug-2002 Abstract +======== - This PEP proposes an extension to the buffer interface called the - 'locked buffer interface'. +This PEP proposes an extension to the buffer interface called the +'locked buffer interface'. - The locked buffer interface avoids the flaws of the 'old' buffer - interface [1] as defined in Python versions up to and including - 2.2, and has the following semantics: +The locked buffer interface avoids the flaws of the 'old' buffer +interface [1]_ as defined in Python versions up to and including +2.2, and has the following semantics: - The lifetime of the retrieved pointer is clearly defined and - controlled by the client. +- The lifetime of the retrieved pointer is clearly defined and + controlled by the client. - The buffer size is returned as a 'size_t' data type, which - allows access to large buffers on platforms where sizeof(int) - != sizeof(void *). +- The buffer size is returned as a 'size_t' data type, which + allows access to large buffers on platforms where ``sizeof(int) + != sizeof(void *)``. - (Guido comments: This second sounds like a change we could also - make to the "old" buffer interface, if we introduce another flag - bit that's *not* part of the default flags.) +(Guido comments: This second sounds like a change we could also +make to the "old" buffer interface, if we introduce another flag +bit that's *not* part of the default flags.) Specification +============= - The locked buffer interface exposes new functions which return the - size and the pointer to the internal memory block of any python - object which chooses to implement this interface. +The locked buffer interface exposes new functions which return the +size and the pointer to the internal memory block of any python +object which chooses to implement this interface. - Retrieving a buffer from an object puts this object in a locked - state during which the buffer may not be freed, resized, or - reallocated. +Retrieving a buffer from an object puts this object in a locked +state during which the buffer may not be freed, resized, or +reallocated. - The object must be unlocked again by releasing the buffer if it's - no longer used by calling another function in the locked buffer - interface. If the object never resizes or reallocates the buffer - during its lifetime, this function may be NULL. Failure to call - this function (if it is != NULL) is a programming error and may - have unexpected results. +The object must be unlocked again by releasing the buffer if it's +no longer used by calling another function in the locked buffer +interface. If the object never resizes or reallocates the buffer +during its lifetime, this function may be NULL. Failure to call +this function (if it is != NULL) is a programming error and may +have unexpected results. - The locked buffer interface omits the memory segment model which - is present in the old buffer interface - only a single memory - block can be exposed. +The locked buffer interface omits the memory segment model which +is present in the old buffer interface - only a single memory +block can be exposed. - The memory blocks can be accessed without holding the global - interpreter lock. +The memory blocks can be accessed without holding the global +interpreter lock. Implementation +============== - Define a new flag in Include/object.h: +Define a new flag in Include/object.h:: - /* PyBufferProcs contains bf_acquirelockedreadbuffer, - bf_acquirelockedwritebuffer, and bf_releaselockedbuffer */ - #define Py_TPFLAGS_HAVE_LOCKEDBUFFER (1L<<15) + /* PyBufferProcs contains bf_acquirelockedreadbuffer, + bf_acquirelockedwritebuffer, and bf_releaselockedbuffer */ + #define Py_TPFLAGS_HAVE_LOCKEDBUFFER (1L<<15) - This flag would be included in Py_TPFLAGS_DEFAULT: +This flag would be included in ``Py_TPFLAGS_DEFAULT``:: - #define Py_TPFLAGS_DEFAULT ( \ - .... - Py_TPFLAGS_HAVE_LOCKEDBUFFER | \ - .... - 0) + #define Py_TPFLAGS_DEFAULT ( \ + .... + Py_TPFLAGS_HAVE_LOCKEDBUFFER | \ + .... + 0) - Extend the PyBufferProcs structure by new fields in - Include/object.h: +Extend the ``PyBufferProcs`` structure by new fields in +Include/object.h:: - typedef size_t (*acquirelockedreadbufferproc)(PyObject *, - const void **); - typedef size_t (*acquirelockedwritebufferproc)(PyObject *, - void **); - typedef void (*releaselockedbufferproc)(PyObject *); + typedef size_t (*acquirelockedreadbufferproc)(PyObject *, + const void **); + typedef size_t (*acquirelockedwritebufferproc)(PyObject *, + void **); + typedef void (*releaselockedbufferproc)(PyObject *); - typedef struct { - getreadbufferproc bf_getreadbuffer; - getwritebufferproc bf_getwritebuffer; - getsegcountproc bf_getsegcount; - getcharbufferproc bf_getcharbuffer; - /* locked buffer interface functions */ - acquirelockedreadbufferproc bf_acquirelockedreadbuffer; - acquirelockedwritebufferproc bf_acquirelockedwritebuffer; - releaselockedbufferproc bf_releaselockedbuffer; - } PyBufferProcs; + typedef struct { + getreadbufferproc bf_getreadbuffer; + getwritebufferproc bf_getwritebuffer; + getsegcountproc bf_getsegcount; + getcharbufferproc bf_getcharbuffer; + /* locked buffer interface functions */ + acquirelockedreadbufferproc bf_acquirelockedreadbuffer; + acquirelockedwritebufferproc bf_acquirelockedwritebuffer; + releaselockedbufferproc bf_releaselockedbuffer; + } PyBufferProcs; - The new fields are present if the Py_TPFLAGS_HAVE_LOCKEDBUFFER - flag is set in the object's type. +The new fields are present if the ``Py_TPFLAGS_HAVE_LOCKEDBUFFER`` +flag is set in the object's type. - The Py_TPFLAGS_HAVE_LOCKEDBUFFER flag implies the - Py_TPFLAGS_HAVE_GETCHARBUFFER flag. +The ``Py_TPFLAGS_HAVE_LOCKEDBUFFER`` flag implies the +``Py_TPFLAGS_HAVE_GETCHARBUFFER`` flag. - The acquirelockedreadbufferproc and acquirelockedwritebufferproc - functions return the size in bytes of the memory block on success, - and fill in the passed void * pointer on success. If these - functions fail - either because an error occurs or no memory block - is exposed - they must set the void * pointer to NULL and raise an - exception. The return value is undefined in these cases and - should not be used. +The ``acquirelockedreadbufferproc`` and ``acquirelockedwritebufferproc`` +functions return the size in bytes of the memory block on success, +and fill in the passed void \* pointer on success. If these +functions fail - either because an error occurs or no memory block +is exposed - they must set the void \* pointer to NULL and raise an +exception. The return value is undefined in these cases and +should not be used. - If calls to these functions succeed, eventually the buffer must be - released by a call to the releaselockedbufferproc, supplying the - original object as argument. The releaselockedbufferproc cannot - fail. For objects that actually maintain an internal lock count - it would be a fatal error if the releaselockedbufferproc function - would be called too often, leading to a negative lock count. +If calls to these functions succeed, eventually the buffer must be +released by a call to the ``releaselockedbufferproc``, supplying the +original object as argument. The ``releaselockedbufferproc`` cannot +fail. For objects that actually maintain an internal lock count +it would be a fatal error if the ``releaselockedbufferproc`` function +would be called too often, leading to a negative lock count. - Similar to the 'old' buffer interface, any of these functions may - be set to NULL, but it is strongly recommended to implement the - releaselockedbufferproc function (even if it does nothing) if any - of the acquireread/writelockedbufferproc functions are - implemented, to discourage extension writers from checking for a - NULL value and not calling it. +Similar to the 'old' buffer interface, any of these functions may +be set to NULL, but it is strongly recommended to implement the +``releaselockedbufferproc`` function (even if it does nothing) if any +of the ``acquireread``/``writelockedbufferproc`` functions are +implemented, to discourage extension writers from checking for a +NULL value and not calling it. - These functions aren't supposed to be called directly, they are - called through convenience functions declared in - Include/abstract.h: +These functions aren't supposed to be called directly, they are +called through convenience functions declared in +Include/abstract.h:: - int PyObject_AquireLockedReadBuffer(PyObject *obj, - const void **buffer, - size_t *buffer_len); + int PyObject_AquireLockedReadBuffer(PyObject *obj, + const void **buffer, + size_t *buffer_len); - int PyObject_AcquireLockedWriteBuffer(PyObject *obj, - void **buffer, - size_t *buffer_len); + int PyObject_AcquireLockedWriteBuffer(PyObject *obj, + void **buffer, + size_t *buffer_len); - void PyObject_ReleaseLockedBuffer(PyObject *obj); + void PyObject_ReleaseLockedBuffer(PyObject *obj); - The former two functions return 0 on success, set buffer to the - memory location and buffer_len to the length of the memory block - in bytes. On failure, or if the locked buffer interface is not - implemented by obj, they return -1 and set an exception. +The former two functions return 0 on success, set buffer to the +memory location and buffer_len to the length of the memory block +in bytes. On failure, or if the locked buffer interface is not +implemented by obj, they return -1 and set an exception. - The latter function doesn't return anything, and cannot fail. +The latter function doesn't return anything, and cannot fail. Backward Compatibility +====================== - The size of the PyBufferProcs structure changes if this proposal - is implemented, but the type's tp_flags slot can be used to - determine if the additional fields are present. +The size of the ``PyBufferProcs`` structure changes if this proposal +is implemented, but the type's ``tp_flags`` slot can be used to +determine if the additional fields are present. Reference Implementation +======================== - An implementation has been uploaded to the SourceForge patch - manager as http://www.python.org/sf/652857. +An implementation has been uploaded to the SourceForge patch +manager as http://www.python.org/sf/652857. Additional Notes/Comments +========================= - Python strings, unicode strings, mmap objects, and array objects - would expose the locked buffer interface. +Python strings, unicode strings, mmap objects, and array objects +would expose the locked buffer interface. - mmap and array objects would actually enter a locked state while - the buffer is active, this is not needed for strings and unicode - objects. Resizing locked array objects is not allowed and will - raise an exception. Whether closing a locked mmap object is an - error or will only be deferred until the lock count reaches zero - is an implementation detail. +mmap and array objects would actually enter a locked state while +the buffer is active, this is not needed for strings and unicode +objects. Resizing locked array objects is not allowed and will +raise an exception. Whether closing a locked mmap object is an +error or will only be deferred until the lock count reaches zero +is an implementation detail. - Guido recommends: +Guido recommends - But I'm still very concerned that if most built-in types - (e.g. strings, bytes) don't implement the release - functionality, it's too easy for an extension to seem to work - while forgetting to release the buffer. + But I'm still very concerned that if most built-in types + (e.g. strings, bytes) don't implement the release + functionality, it's too easy for an extension to seem to work + while forgetting to release the buffer. - I recommend that at least some built-in types implement the - acquire/release functionality with a counter, and assert that - the counter is zero when the object is deleted -- if the - assert fails, someone DECREF'ed their reference to the object - without releasing it. (The rule should be that you must own a - reference to the object while you've acquired the object.) + I recommend that at least some built-in types implement the + acquire/release functionality with a counter, and assert that + the counter is zero when the object is deleted -- if the + assert fails, someone DECREF'ed their reference to the object + without releasing it. (The rule should be that you must own a + reference to the object while you've acquired the object.) - For strings that might be impractical because the string - object would have to grow 4 bytes to hold the counter; but the - new bytes object (PEP 296) could easily implement the counter, - and the array object too -- that way there will be plenty of - opportunity to test proper use of the protocol. + For strings that might be impractical because the string + object would have to grow 4 bytes to hold the counter; but the + new bytes object (PEP 296) could easily implement the counter, + and the array object too -- that way there will be plenty of + opportunity to test proper use of the protocol. Community Feedback +================== - Greg Ewing doubts the locked buffer interface is needed at all, he - thinks the normal buffer interface could be used if the pointer is - (re)fetched each time it's used. This seems to be dangerous, - because even innocent looking calls to the Python API like - Py_DECREF() may trigger execution of arbitrary Python code. +Greg Ewing doubts the locked buffer interface is needed at all, he +thinks the normal buffer interface could be used if the pointer is +(re)fetched each time it's used. This seems to be dangerous, +because even innocent looking calls to the Python API like +``Py_DECREF()`` may trigger execution of arbitrary Python code. - The first version of this proposal didn't have the release - function, but it turned out that this would have been too - restrictive: mmap and array objects wouldn't have been able to - implement it, because mmap objects can be closed anytime if not - locked, and array objects could resize or reallocate the buffer. +The first version of this proposal didn't have the release +function, but it turned out that this would have been too +restrictive: mmap and array objects wouldn't have been able to +implement it, because mmap objects can be closed anytime if not +locked, and array objects could resize or reallocate the buffer. - This PEP will probably be rejected because nobody except the - author needs it. +This PEP will probably be rejected because nobody except the +author needs it. References +========== - [1] The buffer interface - http://mail.python.org/pipermail/python-dev/2000-October/009974.html +.. [1] The buffer interface + http://mail.python.org/pipermail/python-dev/2000-October/009974.html - [2] The Buffer Problem - http://www.python.org/dev/peps/pep-0296/ +.. [2] The Buffer Problem + http://www.python.org/dev/peps/pep-0296/ Copyright +========= - This document has been placed in the public domain. +This document has been placed in the public domain. - -Local Variables: -mode: indented-text -indent-tabs-mode: nil -sentence-end-double-space: t -fill-column: 70 -End: + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + End: diff --git a/pep-0311.txt b/pep-0311.txt index e9392ed4b..b4c4b3f4b 100644 --- a/pep-0311.txt +++ b/pep-0311.txt @@ -5,230 +5,252 @@ Last-Modified: $Date$ Author: Mark Hammond Status: Final Type: Standards Track -Content-Type: text/plain +Content-Type: text/x-rst Created: 05-Feb-2003 Post-History: 05-Feb-2003 14-Feb-2003 19-Apr-2003 Abstract +======== - This PEP proposes a simplified API for access to the Global - Interpreter Lock (GIL) for Python extension modules. - Specifically, it provides a solution for authors of complex - multi-threaded extensions, where the current state of Python - (i.e., the state of the GIL is unknown. +This PEP proposes a simplified API for access to the Global +Interpreter Lock (GIL) for Python extension modules. +Specifically, it provides a solution for authors of complex +multi-threaded extensions, where the current state of Python +(i.e., the state of the GIL is unknown. - This PEP proposes a new API, for platforms built with threading - support, to manage the Python thread state. An implementation - strategy is proposed, along with an initial, platform independent - implementation. +This PEP proposes a new API, for platforms built with threading +support, to manage the Python thread state. An implementation +strategy is proposed, along with an initial, platform independent +implementation. Rationale +========= - The current Python interpreter state API is suitable for simple, - single-threaded extensions, but quickly becomes incredibly complex - for non-trivial, multi-threaded extensions. +The current Python interpreter state API is suitable for simple, +single-threaded extensions, but quickly becomes incredibly complex +for non-trivial, multi-threaded extensions. - Currently Python provides two mechanisms for dealing with the GIL: +Currently Python provides two mechanisms for dealing with the GIL: - - Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. - These macros are provided primarily to allow a simple Python - extension that already owns the GIL to temporarily release it - while making an "external" (ie, non-Python), generally - expensive, call. Any existing Python threads that are blocked - waiting for the GIL are then free to run. While this is fine - for extensions making calls from Python into the outside world, - it is no help for extensions that need to make calls into Python - when the thread state is unknown. +- ``Py_BEGIN_ALLOW_THREADS`` and ``Py_END_ALLOW_THREADS`` macros. + These macros are provided primarily to allow a simple Python + extension that already owns the GIL to temporarily release it + while making an "external" (ie, non-Python), generally + expensive, call. Any existing Python threads that are blocked + waiting for the GIL are then free to run. While this is fine + for extensions making calls from Python into the outside world, + it is no help for extensions that need to make calls into Python + when the thread state is unknown. - - PyThreadState and PyInterpreterState APIs. - These API functions allow an extension/embedded application to - acquire the GIL, but suffer from a serious boot-strapping - problem - they require you to know the state of the Python - interpreter and of the GIL before they can be used. One - particular problem is for extension authors that need to deal - with threads never before seen by Python, but need to call - Python from this thread. It is very difficult, delicate and - error prone to author an extension where these "new" threads - always know the exact state of the GIL, and therefore can - reliably interact with this API. +- ``PyThreadState`` and ``PyInterpreterState`` APIs. + These API functions allow an extension/embedded application to + acquire the GIL, but suffer from a serious boot-strapping + problem - they require you to know the state of the Python + interpreter and of the GIL before they can be used. One + particular problem is for extension authors that need to deal + with threads never before seen by Python, but need to call + Python from this thread. It is very difficult, delicate and + error prone to author an extension where these "new" threads + always know the exact state of the GIL, and therefore can + reliably interact with this API. - For these reasons, the question of how such extensions should - interact with Python is quickly becoming a FAQ. The main impetus - for this PEP, a thread on python-dev [1], immediately identified - the following projects with this exact issue: +For these reasons, the question of how such extensions should +interact with Python is quickly becoming a FAQ. The main impetus +for this PEP, a thread on python-dev [1]_, immediately identified +the following projects with this exact issue: - - The win32all extensions - - Boost - - ctypes - - Python-GTK bindings - - Uno - - PyObjC - - Mac toolbox - - PyXPCOM +- The win32all extensions +- Boost +- ctypes +- Python-GTK bindings +- Uno +- PyObjC +- Mac toolbox +- PyXPCOM - Currently, there is no reasonable, portable solution to this - problem, forcing each extension author to implement their own - hand-rolled version. Further, the problem is complex, meaning - many implementations are likely to be incorrect, leading to a - variety of problems that will often manifest simply as "Python has - hung". +Currently, there is no reasonable, portable solution to this +problem, forcing each extension author to implement their own +hand-rolled version. Further, the problem is complex, meaning +many implementations are likely to be incorrect, leading to a +variety of problems that will often manifest simply as "Python has +hung". - While the biggest problem in the existing thread-state API is the - lack of the ability to query the current state of the lock, it is - felt that a more complete, simplified solution should be offered - to extension authors. Such a solution should encourage authors to - provide error-free, complex extension modules that take full - advantage of Python's threading mechanisms. +While the biggest problem in the existing thread-state API is the +lack of the ability to query the current state of the lock, it is +felt that a more complete, simplified solution should be offered +to extension authors. Such a solution should encourage authors to +provide error-free, complex extension modules that take full +advantage of Python's threading mechanisms. Limitations and Exclusions +========================== - This proposal identifies a solution for extension authors with - complex multi-threaded requirements, but that only require a - single "PyInterpreterState". There is no attempt to cater for - extensions that require multiple interpreter states. At the time - of writing, no extension has been identified that requires - multiple PyInterpreterStates, and indeed it is not clear if that - facility works correctly in Python itself. +This proposal identifies a solution for extension authors with +complex multi-threaded requirements, but that only require a +single "PyInterpreterState". There is no attempt to cater for +extensions that require multiple interpreter states. At the time +of writing, no extension has been identified that requires +multiple PyInterpreterStates, and indeed it is not clear if that +facility works correctly in Python itself. - This API will not perform automatic initialization of Python, or - initialize Python for multi-threaded operation. Extension authors - must continue to call Py_Initialize(), and for multi-threaded - applications, PyEval_InitThreads(). The reason for this is that - the first thread to call PyEval_InitThreads() is nominated as the - "main thread" by Python, and so forcing the extension author to - specify the main thread (by forcing her to make this first call) - removes ambiguity. As Py_Initialize() must be called before - PyEval_InitThreads(), and as both of these functions currently - support being called multiple times, the burden this places on - extension authors is considered reasonable. +This API will not perform automatic initialization of Python, or +initialize Python for multi-threaded operation. Extension authors +must continue to call ``Py_Initialize()``, and for multi-threaded +applications, ``PyEval_InitThreads()``. The reason for this is that +the first thread to call ``PyEval_InitThreads()`` is nominated as the +"main thread" by Python, and so forcing the extension author to +specify the main thread (by forcing her to make this first call) +removes ambiguity. As ``Py_Initialize()`` must be called before +``PyEval_InitThreads()``, and as both of these functions currently +support being called multiple times, the burden this places on +extension authors is considered reasonable. - It is intended that this API be all that is necessary to acquire - the Python GIL. Apart from the existing, standard - Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is - assumed that no additional thread state API functions will be used - by the extension. Extensions with such complicated requirements - are free to continue to use the existing thread state API. +It is intended that this API be all that is necessary to acquire +the Python GIL. Apart from the existing, standard +``Py_BEGIN_ALLOW_THREADS`` and ``Py_END_ALLOW_THREADS`` macros, it is +assumed that no additional thread state API functions will be used +by the extension. Extensions with such complicated requirements +are free to continue to use the existing thread state API. Proposal +======== - This proposal recommends a new API be added to Python to simplify - the management of the GIL. This API will be available on all - platforms built with WITH_THREAD defined. +This proposal recommends a new API be added to Python to simplify +the management of the GIL. This API will be available on all +platforms built with ``WITH_THREAD`` defined. - The intent is that assuming Python has correctly been initialized, - an extension author be able to use a small, well-defined "prologue - dance", at any time and on any thread, which will ensure Python - is ready to be used on that thread. After the extension has - finished with Python, it must also perform an "epilogue dance" to - release any resources previously acquired. Ideally, these dances - can be expressed in a single line. +The intent is that assuming Python has correctly been initialized, +an extension author be able to use a small, well-defined "prologue +dance", at any time and on any thread, which will ensure Python +is ready to be used on that thread. After the extension has +finished with Python, it must also perform an "epilogue dance" to +release any resources previously acquired. Ideally, these dances +can be expressed in a single line. - Specifically, the following new APIs are proposed: +Specifically, the following new APIs are proposed:: - /* Ensure that the current thread is ready to call the Python - C API, regardless of the current state of Python, or of its - thread lock. This may be called as many times as desired - by a thread so long as each call is matched with a call to - PyGILState_Release(). In general, other thread-state APIs may - be used between _Ensure() and _Release() calls, so long as the - thread-state is restored to its previous state before the Release(). - For example, normal use of the Py_BEGIN_ALLOW_THREADS/ - Py_END_ALLOW_THREADS macros are acceptable. - - The return value is an opaque "handle" to the thread state when - PyGILState_Acquire() was called, and must be passed to - PyGILState_Release() to ensure Python is left in the same state. Even - though recursive calls are allowed, these handles can *not* be - shared - each unique call to PyGILState_Ensure must save the handle - for its call to PyGILState_Release. - - When the function returns, the current thread will hold the GIL. - - Failure is a fatal error. - */ - PyAPI_FUNC(PyGILState_STATE) PyGILState_Ensure(void); + /* Ensure that the current thread is ready to call the Python + C API, regardless of the current state of Python, or of its + thread lock. This may be called as many times as desired + by a thread so long as each call is matched with a call to + PyGILState_Release(). In general, other thread-state APIs may + be used between _Ensure() and _Release() calls, so long as the + thread-state is restored to its previous state before the Release(). + For example, normal use of the Py_BEGIN_ALLOW_THREADS/ + Py_END_ALLOW_THREADS macros are acceptable. - /* Release any resources previously acquired. After this call, Python's - state will be the same as it was prior to the corresponding - PyGILState_Acquire call (but generally this state will be unknown to - the caller, hence the use of the GILState API.) - - Every call to PyGILState_Ensure must be matched by a call to - PyGILState_Release on the same thread. - */ - PyAPI_FUNC(void) PyGILState_Release(PyGILState_STATE); + The return value is an opaque "handle" to the thread state when + PyGILState_Acquire() was called, and must be passed to + PyGILState_Release() to ensure Python is left in the same state. Even + though recursive calls are allowed, these handles can *not* be + shared - each unique call to PyGILState_Ensure must save the handle + for its call to PyGILState_Release. - Common usage will be: + When the function returns, the current thread will hold the GIL. - void SomeCFunction(void) - { - /* ensure we hold the lock */ - PyGILState_STATE state = PyGILState_Ensure(); - /* Use the Python API */ - ... - /* Restore the state of Python */ - PyGILState_Release(state); - } + Failure is a fatal error. + */ + PyAPI_FUNC(PyGILState_STATE) PyGILState_Ensure(void); + + /* Release any resources previously acquired. After this call, Python's + state will be the same as it was prior to the corresponding + PyGILState_Acquire call (but generally this state will be unknown to + the caller, hence the use of the GILState API.) + + Every call to PyGILState_Ensure must be matched by a call to + PyGILState_Release on the same thread. + */ + PyAPI_FUNC(void) PyGILState_Release(PyGILState_STATE); + +Common usage will be:: + + void SomeCFunction(void) + { + /* ensure we hold the lock */ + PyGILState_STATE state = PyGILState_Ensure(); + /* Use the Python API */ + ... + /* Restore the state of Python */ + PyGILState_Release(state); + } Design and Implementation +========================= - The general operation of PyGILState_Ensure() will be: - - assert Python is initialized. - - Get a PyThreadState for the current thread, creating and saving - if necessary. - - remember the current state of the lock (owned/not owned) - - If the current state does not own the GIL, acquire it. - - Increment a counter for how many calls to - PyGILState_Ensure have been made on the current thread. - - return +The general operation of ``PyGILState_Ensure()`` will be: - The general operation of PyGILState_Release() will be: +- assert Python is initialized. - - assert our thread currently holds the lock. - - If old state indicates lock was previously unlocked, release GIL. - - Decrement the PyGILState_Ensure counter for the thread. - - If counter == 0: - - release and delete the PyThreadState. - - forget the ThreadState as being owned by the thread. - - return +- Get a ``PyThreadState`` for the current thread, creating and saving + if necessary. - It is assumed that it is an error if two discrete PyThreadStates - are used for a single thread. Comments in pystate.h ("State - unique per thread") support this view, although it is never - directly stated. Thus, this will require some implementation of - Thread Local Storage. Fortunately, a platform independent - implementation of Thread Local Storage already exists in the - Python source tree, in the SGI threading port. This code will be - integrated into the platform independent Python core, but in such - a way that platforms can provide a more optimal implementation if - desired. +- remember the current state of the lock (owned/not owned) + +- If the current state does not own the GIL, acquire it. + +- Increment a counter for how many calls to ``PyGILState_Ensure`` have been + made on the current thread. + +- return + +The general operation of ``PyGILState_Release()`` will be: + +- assert our thread currently holds the lock. + +- If old state indicates lock was previously unlocked, release GIL. + +- Decrement the ``PyGILState_Ensure`` counter for the thread. + +- If counter == 0: + + - release and delete the ``PyThreadState``. + + - forget the ``ThreadState`` as being owned by the thread. + +- return + +It is assumed that it is an error if two discrete ``PyThreadStates`` +are used for a single thread. Comments in ``pystate.h`` ("State +unique per thread") support this view, although it is never +directly stated. Thus, this will require some implementation of +Thread Local Storage. Fortunately, a platform independent +implementation of Thread Local Storage already exists in the +Python source tree, in the SGI threading port. This code will be +integrated into the platform independent Python core, but in such +a way that platforms can provide a more optimal implementation if +desired. Implementation +============== - An implementation of this proposal can be found at - http://www.python.org/sf/684256 +An implementation of this proposal can be found at +http://www.python.org/sf/684256 References +========== - [1] http://mail.python.org/pipermail/python-dev/2002-December/031424.html +.. [1] David Abrahams, Extension modules, Threading, and the GIL + http://mail.python.org/pipermail/python-dev/2002-December/031424.html Copyright +========= - This document has been placed in the public domain. +This document has been placed in the public domain. - -Local Variables: -mode: indented-text -indent-tabs-mode: nil -sentence-end-double-space: t -fill-column: 70 -End: + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + End: