498 lines
16 KiB
Plaintext
498 lines
16 KiB
Plaintext
PEP: 319
|
||
Title: Python Synchronize/Asynchronize Block
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Michel Pelletier <michel@users.sourceforge.net>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Created: 24-Feb-2003
|
||
Python-Version: 2.4?
|
||
Post-History:
|
||
|
||
|
||
Abstract
|
||
|
||
This PEP proposes adding two new keywords to Python, `synchronize'
|
||
and 'asynchronize'.
|
||
|
||
|
||
The `synchronize' Keyword
|
||
|
||
The concept of code synchronization in Python is too low-level.
|
||
To synchronize code a programmer must be aware of the details of
|
||
the following pseudo-code pattern:
|
||
|
||
initialize_lock()
|
||
|
||
...
|
||
|
||
acquire_lock()
|
||
try:
|
||
change_shared_data()
|
||
finally:
|
||
release_lock()
|
||
|
||
This synchronized block pattern is not the only pattern (more
|
||
discussed below) but it is very common. This PEP proposes
|
||
replacing the above code with the following equivalent:
|
||
|
||
synchronize:
|
||
change_shared_data()
|
||
|
||
The advantages of this scheme are simpler syntax and less room for
|
||
user error. Currently users are required to write code about
|
||
acquiring and releasing thread locks in 'try/finally' blocks;
|
||
errors in this code can cause notoriously difficult concurrent
|
||
thread locking issues.
|
||
|
||
|
||
The `asynchronize' Keyword
|
||
|
||
While executing a `synchronize' block of code a programmer may
|
||
want to "drop back" to running asynchronously momentarily to run
|
||
blocking input/output routines or something else that might take a
|
||
indeterminate amount of time and does not require synchronization.
|
||
This code usually follows the pattern:
|
||
|
||
initialize_lock()
|
||
|
||
...
|
||
|
||
acquire_lock()
|
||
try:
|
||
change_shared_data()
|
||
release_lock() # become async
|
||
do_blocking_io()
|
||
acquire_lock() # sync again
|
||
change_shared_data2()
|
||
|
||
finally:
|
||
release_lock()
|
||
|
||
The asynchronous section of the code is not very obvious visually,
|
||
so it is marked up with comments. Using the proposed
|
||
'asynchronize' keyword this code becomes much cleaner, easier to
|
||
understand, and less prone to error:
|
||
|
||
synchronize:
|
||
change_shared_data()
|
||
|
||
asynchronize:
|
||
do_blocking_io()
|
||
|
||
change_shared_data2()
|
||
|
||
Encountering an `asynchronize' keyword inside a non-synchronized
|
||
block can raise either an error or issue a warning (as all code
|
||
blocks are implicitly asynchronous anyway). It is important to
|
||
note that the above example is *not* the same as:
|
||
|
||
synchronize:
|
||
change_shared_data()
|
||
|
||
do_blocking_io()
|
||
|
||
synchronize:
|
||
change_shared_data2()
|
||
|
||
Because both synchronized blocks of code may be running inside the
|
||
same iteration of a loop, Consider:
|
||
|
||
while in_main_loop():
|
||
synchronize:
|
||
change_shared_data()
|
||
|
||
asynchronize:
|
||
do_blocking_io()
|
||
|
||
change_shared_data2()
|
||
|
||
Many threads may be looping through this code. Without the
|
||
'asynchronize' keyword one thread cannot stay in the loop and
|
||
release the lock at the same time while blocking IO is going on.
|
||
This pattern of releasing locks inside a main loop to do blocking
|
||
IO is used extensively inside the CPython interpreter itself.
|
||
|
||
|
||
Synchronization Targets
|
||
|
||
As proposed the `synchronize' and `asynchronize' keywords
|
||
synchronize a block of code. However programmers may want to
|
||
specify a target object that threads synchronize on. Any object
|
||
can be a synchronization target.
|
||
|
||
Consider a two-way queue object: two different objects are used by
|
||
the same `synchronize' code block to synchronize both queues
|
||
separately in the 'get' method:
|
||
|
||
class TwoWayQueue:
|
||
def __init__(self):
|
||
self.front = []
|
||
self.rear = []
|
||
|
||
def putFront(self, item):
|
||
self.put(item, self.front)
|
||
|
||
def getFront(self):
|
||
item = self.get(self.front)
|
||
return item
|
||
|
||
def putRear(self, item):
|
||
self.put(item, self.rear)
|
||
|
||
def getRear(self):
|
||
item = self.get(self.rear)
|
||
return item
|
||
|
||
def put(self, item, queue):
|
||
synchronize queue:
|
||
queue.append(item)
|
||
|
||
def get(self, queue):
|
||
synchronize queue:
|
||
item = queue[0]
|
||
del queue[0]
|
||
return item
|
||
|
||
Here is the equivalent code in Python as it is now without a
|
||
`synchronize' keyword:
|
||
|
||
import thread
|
||
|
||
class LockableQueue:
|
||
|
||
def __init__(self):
|
||
self.queue = []
|
||
self.lock = thread.allocate_lock()
|
||
|
||
class TwoWayQueue:
|
||
def __init__(self):
|
||
self.front = LockableQueue()
|
||
self.rear = LockableQueue()
|
||
|
||
def putFront(self, item):
|
||
self.put(item, self.front)
|
||
|
||
def getFront(self):
|
||
item = self.get(self.front)
|
||
return item
|
||
|
||
def putRear(self, item):
|
||
self.put(item, self.rear)
|
||
|
||
def getRear(self):
|
||
item = self.get(self.rear)
|
||
return item
|
||
|
||
def put(self, item, queue):
|
||
queue.lock.acquire()
|
||
try:
|
||
queue.append(item)
|
||
finally:
|
||
queue.lock.release()
|
||
|
||
def get(self, queue):
|
||
queue.lock.acquire()
|
||
try:
|
||
item = queue[0]
|
||
del queue[0]
|
||
return item
|
||
finally:
|
||
queue.lock.release()
|
||
|
||
The last example had to define an extra class to associate a lock
|
||
with the queue where the first example the `synchronize' keyword
|
||
does this association internally and transparently.
|
||
|
||
|
||
Other Patterns that Synchronize
|
||
|
||
There are some situations where the `synchronize' and
|
||
`asynchronize' keywords cannot entirely replace the use of lock
|
||
methods like `acquire' and `release'. Some examples are if the
|
||
programmer wants to provide arguments for `acquire' or if a lock
|
||
is acquired in one code block but released in another, as shown
|
||
below.
|
||
|
||
Here is a class from Zope modified to use both the `synchronize'
|
||
and `asynchronize' keywords and also uses a pool of explicit locks
|
||
that are acquired and released in different code blocks and thus
|
||
don't use `synchronize':
|
||
|
||
import thread
|
||
from ZServerPublisher import ZServerPublisher
|
||
|
||
class ZRendevous:
|
||
|
||
def __init__(self, n=1):
|
||
pool=[]
|
||
self._lists=pool, [], []
|
||
|
||
synchronize:
|
||
while n > 0:
|
||
l=thread.allocate_lock()
|
||
l.acquire()
|
||
pool.append(l)
|
||
thread.start_new_thread(ZServerPublisher,
|
||
(self.accept,))
|
||
n=n-1
|
||
|
||
def accept(self):
|
||
synchronize:
|
||
pool, requests, ready = self._lists
|
||
while not requests:
|
||
l=pool[-1]
|
||
del pool[-1]
|
||
ready.append(l)
|
||
|
||
asynchronize:
|
||
l.acquire()
|
||
|
||
pool.append(l)
|
||
|
||
r=requests[0]
|
||
del requests[0]
|
||
return r
|
||
|
||
def handle(self, name, request, response):
|
||
synchronize:
|
||
pool, requests, ready = self._lists
|
||
requests.append((name, request, response))
|
||
if ready:
|
||
l=ready[-1]
|
||
del ready[-1]
|
||
l.release()
|
||
|
||
Here is the original class as found in the
|
||
'Zope/ZServer/PubCore/ZRendevous.py' module. The "convenience" of
|
||
the '_a' and '_r' shortcut names obscure the code:
|
||
|
||
import thread
|
||
from ZServerPublisher import ZServerPublisher
|
||
|
||
class ZRendevous:
|
||
|
||
def __init__(self, n=1):
|
||
sync=thread.allocate_lock()
|
||
self._a=sync.acquire
|
||
self._r=sync.release
|
||
pool=[]
|
||
self._lists=pool, [], []
|
||
self._a()
|
||
try:
|
||
while n > 0:
|
||
l=thread.allocate_lock()
|
||
l.acquire()
|
||
pool.append(l)
|
||
thread.start_new_thread(ZServerPublisher,
|
||
(self.accept,))
|
||
n=n-1
|
||
finally: self._r()
|
||
|
||
def accept(self):
|
||
self._a()
|
||
try:
|
||
pool, requests, ready = self._lists
|
||
while not requests:
|
||
l=pool[-1]
|
||
del pool[-1]
|
||
ready.append(l)
|
||
self._r()
|
||
l.acquire()
|
||
self._a()
|
||
pool.append(l)
|
||
|
||
r=requests[0]
|
||
del requests[0]
|
||
return r
|
||
finally: self._r()
|
||
|
||
def handle(self, name, request, response):
|
||
self._a()
|
||
try:
|
||
pool, requests, ready = self._lists
|
||
requests.append((name, request, response))
|
||
if ready:
|
||
l=ready[-1]
|
||
del ready[-1]
|
||
l.release()
|
||
finally: self._r()
|
||
|
||
In particular the asynchronize section of the `accept' method is
|
||
not very obvious. To beginner programmers, `synchronize' and
|
||
`asynchronize' remove many of the problems encountered when
|
||
juggling multiple `acquire' and `release' methods on different
|
||
locks in different `try/finally' blocks.
|
||
|
||
|
||
Formal Syntax
|
||
|
||
Python syntax is defined in a modified BNF grammar notation
|
||
described in the Python Language Reference [1]. This section
|
||
describes the proposed synchronization syntax using this grammar:
|
||
|
||
synchronize_stmt: 'synchronize' [test] ':' suite
|
||
asynchronize_stmt: 'asynchronize' [test] ':' suite
|
||
compound_stmt: ... | synchronized_stmt | asynchronize_stmt
|
||
|
||
(The '...' indicates other compound statements elided).
|
||
|
||
|
||
Proposed Implementation
|
||
|
||
The author of this PEP has not explored an implementation yet.
|
||
There are several implementation issues that must be resolved.
|
||
The main implementation issue is what exactly gets locked and
|
||
unlocked during a synchronized block.
|
||
|
||
During an unqualified synchronized block (the use of the
|
||
`synchronize' keyword without an target argument) a lock could be
|
||
created and associated with the synchronized code block object.
|
||
Any threads that are to execute the block must first acquire the
|
||
code block lock.
|
||
|
||
When an `asynchronize' keyword is encountered in a `synchronize'
|
||
block the code block lock is unlocked before the inner block is
|
||
executed and re-locked when the inner block terminates.
|
||
|
||
When a synchronized block target is specified the object is
|
||
associated with a lock. How this is implemented cleanly is
|
||
probably the highest risk of this proposal. Java Virtual Machines
|
||
typically associate a special hidden lock object with target
|
||
object and use it to synchronized the block around the target
|
||
only.
|
||
|
||
|
||
Backward Compatibility
|
||
|
||
Backward compatibility is solved with the new `from __future__'
|
||
Python syntax [2], and the new warning framework [3] to evolve the
|
||
Python language into phasing out any conflicting names that use
|
||
the new keywords `synchronize' and `asynchronize'. To use the
|
||
syntax now, a developer could use the statement:
|
||
|
||
from __future__ import threadsync # or whatever
|
||
|
||
In addition, any code that uses the keyword `synchronize' or
|
||
`asynchronize' as an identifier will be issued a warning from
|
||
Python. After the appropriate period of time, the syntax would
|
||
become standard, the above import statement would do nothing, and
|
||
any identifiers named `synchronize' or `asynchronize' would raise
|
||
an exception.
|
||
|
||
|
||
PEP 310 Reliable Acquisition/Release Pairs
|
||
|
||
PEP 310 [4] proposes the 'with' keyword that can serve the same
|
||
function as 'synchronize' (but no facility for 'asynchronize').
|
||
The pattern:
|
||
|
||
initialize_lock()
|
||
|
||
with the_lock:
|
||
change_shared_data()
|
||
|
||
is equivalent to the proposed:
|
||
|
||
synchronize the_lock:
|
||
change_shared_data()
|
||
|
||
PEP 310 must synchronize on an exsiting lock, while this PEP
|
||
proposes that unqualified 'synchronize' statements synchronize on
|
||
a global, internal, transparent lock in addition to qualifiled
|
||
'synchronize' statements. The 'with' statement also requires lock
|
||
initialization, while the 'synchronize' statment can synchronize
|
||
on any target object *including* locks.
|
||
|
||
While limited in this fashion, the 'with' statment is more
|
||
abstract and serves more purposes than synchronization. For
|
||
example, transactions could be used with the 'with' keyword:
|
||
|
||
initialize_transaction()
|
||
|
||
with my_transaction:
|
||
do_in_transaction()
|
||
|
||
# when the block terminates, the transaction is commited.
|
||
|
||
The 'synchronize' and 'asynchronize' keywords cannot serve this or
|
||
any other general acquire/release pattern other than thread
|
||
synchronization.
|
||
|
||
|
||
How Java Does It
|
||
|
||
Java defines a 'synchronized' keyword (note the grammatical tense
|
||
different between the Java keyword and this PEP's 'synchronize')
|
||
which must be qualified on any object. The syntax is:
|
||
|
||
synchronized (Expression) Block
|
||
|
||
Expression must yeild a valid object (null raises an error and
|
||
exceptions during 'Expression' terminate the 'synchronized' block
|
||
for the same reason) upon which 'Block' is synchronized.
|
||
|
||
|
||
How Jython Does It
|
||
|
||
Jython uses a 'synchronize' class with the static method
|
||
'make_synchronized' that accepts one callable argument and returns
|
||
a newly created, synchronized, callable "wrapper" around the
|
||
argument.
|
||
|
||
|
||
Summary of Proposed Changes to Python
|
||
|
||
Adding new `synchronize' and `asynchronize' keywords to the
|
||
language.
|
||
|
||
|
||
Risks
|
||
|
||
This PEP proposes adding two keywords to the Python language. This
|
||
may break code.
|
||
|
||
There is no implementation to test.
|
||
|
||
It's not the most important problem facing Python programmers
|
||
today (although it is a fairly notorious one).
|
||
|
||
The equivalent Java keyword is the past participle 'synchronized'.
|
||
This PEP proposes the present tense, 'synchronize' as being more
|
||
in spirit with Python (there being less distinction between
|
||
compile-time and run-time in Python than Java).
|
||
|
||
|
||
Dissenting Opinion
|
||
|
||
This PEP has not been discussed on python-dev.
|
||
|
||
|
||
References
|
||
|
||
[1] The Python Language Reference
|
||
http://www.python.org/doc/current/ref/ref.html
|
||
|
||
[2] PEP 236, Back to the __future__, Peters
|
||
http://python.sourceforge.net/peps/pep-0236.html
|
||
|
||
[3] PEP 230, Warning Framework, van Rossum
|
||
http://python.sourceforge.net/peps/pep-0236.html
|
||
|
||
[4] PEP 310, Reliable Acquisition/Release Pairs, Hudson, Moore
|
||
http://www.python.org/peps/pep-0310.html
|
||
|
||
|
||
Copyright
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
End:
|