507 lines
15 KiB
ReStructuredText
507 lines
15 KiB
ReStructuredText
PEP: 319
|
|
Title: Python Synchronize/Asynchronize Block
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Michel Pelletier <michel@users.sourceforge.net>
|
|
Status: Rejected
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 24-Feb-2003
|
|
Python-Version: 2.4
|
|
Post-History:
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes adding two new keywords to Python, 'synchronize'
|
|
and 'asynchronize'.
|
|
|
|
Pronouncement
|
|
=============
|
|
|
|
This PEP is rejected in favor of :pep:`343`.
|
|
|
|
The 'synchronize' Keyword
|
|
The concept of code synchronization in Python is too low-level.
|
|
To synchronize code a programmer must be aware of the details of
|
|
the following pseudo-code pattern::
|
|
|
|
initialize_lock()
|
|
|
|
...
|
|
|
|
acquire_lock()
|
|
try:
|
|
change_shared_data()
|
|
finally:
|
|
release_lock()
|
|
|
|
This synchronized block pattern is not the only pattern (more
|
|
discussed below) but it is very common. This PEP proposes
|
|
replacing the above code with the following equivalent::
|
|
|
|
synchronize:
|
|
change_shared_data()
|
|
|
|
The advantages of this scheme are simpler syntax and less room for
|
|
user error. Currently users are required to write code about
|
|
acquiring and releasing thread locks in 'try/finally' blocks;
|
|
errors in this code can cause notoriously difficult concurrent
|
|
thread locking issues.
|
|
|
|
|
|
The 'asynchronize' Keyword
|
|
While executing a 'synchronize' block of code a programmer may
|
|
want to "drop back" to running asynchronously momentarily to run
|
|
blocking input/output routines or something else that might take an
|
|
indeterminate amount of time and does not require synchronization.
|
|
This code usually follows the pattern::
|
|
|
|
initialize_lock()
|
|
|
|
...
|
|
|
|
acquire_lock()
|
|
try:
|
|
change_shared_data()
|
|
release_lock() # become async
|
|
do_blocking_io()
|
|
acquire_lock() # sync again
|
|
change_shared_data2()
|
|
|
|
finally:
|
|
release_lock()
|
|
|
|
The asynchronous section of the code is not very obvious visually,
|
|
so it is marked up with comments. Using the proposed
|
|
'asynchronize' keyword this code becomes much cleaner, easier to
|
|
understand, and less prone to error::
|
|
|
|
synchronize:
|
|
change_shared_data()
|
|
|
|
asynchronize:
|
|
do_blocking_io()
|
|
|
|
change_shared_data2()
|
|
|
|
Encountering an 'asynchronize' keyword inside a non-synchronized
|
|
block can raise either an error or issue a warning (as all code
|
|
blocks are implicitly asynchronous anyway). It is important to
|
|
note that the above example is **not** the same as::
|
|
|
|
synchronize:
|
|
change_shared_data()
|
|
|
|
do_blocking_io()
|
|
|
|
synchronize:
|
|
change_shared_data2()
|
|
|
|
Because both synchronized blocks of code may be running inside the
|
|
same iteration of a loop, Consider::
|
|
|
|
while in_main_loop():
|
|
synchronize:
|
|
change_shared_data()
|
|
|
|
asynchronize:
|
|
do_blocking_io()
|
|
|
|
change_shared_data2()
|
|
|
|
Many threads may be looping through this code. Without the
|
|
'asynchronize' keyword one thread cannot stay in the loop and
|
|
release the lock at the same time while blocking IO is going on.
|
|
This pattern of releasing locks inside a main loop to do blocking
|
|
IO is used extensively inside the CPython interpreter itself.
|
|
|
|
|
|
Synchronization Targets
|
|
=======================
|
|
|
|
As proposed the 'synchronize' and 'asynchronize' keywords
|
|
synchronize a block of code. However programmers may want to
|
|
specify a target object that threads synchronize on. Any object
|
|
can be a synchronization target.
|
|
|
|
Consider a two-way queue object: two different objects are used by
|
|
the same 'synchronize' code block to synchronize both queues
|
|
separately in the 'get' method::
|
|
|
|
class TwoWayQueue:
|
|
def __init__(self):
|
|
self.front = []
|
|
self.rear = []
|
|
|
|
def putFront(self, item):
|
|
self.put(item, self.front)
|
|
|
|
def getFront(self):
|
|
item = self.get(self.front)
|
|
return item
|
|
|
|
def putRear(self, item):
|
|
self.put(item, self.rear)
|
|
|
|
def getRear(self):
|
|
item = self.get(self.rear)
|
|
return item
|
|
|
|
def put(self, item, queue):
|
|
synchronize queue:
|
|
queue.append(item)
|
|
|
|
def get(self, queue):
|
|
synchronize queue:
|
|
item = queue[0]
|
|
del queue[0]
|
|
return item
|
|
|
|
Here is the equivalent code in Python as it is now without a
|
|
'synchronize' keyword::
|
|
|
|
import thread
|
|
|
|
class LockableQueue:
|
|
|
|
def __init__(self):
|
|
self.queue = []
|
|
self.lock = thread.allocate_lock()
|
|
|
|
class TwoWayQueue:
|
|
def __init__(self):
|
|
self.front = LockableQueue()
|
|
self.rear = LockableQueue()
|
|
|
|
def putFront(self, item):
|
|
self.put(item, self.front)
|
|
|
|
def getFront(self):
|
|
item = self.get(self.front)
|
|
return item
|
|
|
|
def putRear(self, item):
|
|
self.put(item, self.rear)
|
|
|
|
def getRear(self):
|
|
item = self.get(self.rear)
|
|
return item
|
|
|
|
def put(self, item, queue):
|
|
queue.lock.acquire()
|
|
try:
|
|
queue.append(item)
|
|
finally:
|
|
queue.lock.release()
|
|
|
|
def get(self, queue):
|
|
queue.lock.acquire()
|
|
try:
|
|
item = queue[0]
|
|
del queue[0]
|
|
return item
|
|
finally:
|
|
queue.lock.release()
|
|
|
|
The last example had to define an extra class to associate a lock
|
|
with the queue where the first example the 'synchronize' keyword
|
|
does this association internally and transparently.
|
|
|
|
|
|
Other Patterns that Synchronize
|
|
===============================
|
|
|
|
There are some situations where the 'synchronize' and
|
|
'asynchronize' keywords cannot entirely replace the use of lock
|
|
methods like ``acquire`` and ``release``. Some examples are if the
|
|
programmer wants to provide arguments for ``acquire`` or if a lock
|
|
is acquired in one code block but released in another, as shown
|
|
below.
|
|
|
|
Here is a class from Zope modified to use both the 'synchronize'
|
|
and 'asynchronize' keywords and also uses a pool of explicit locks
|
|
that are acquired and released in different code blocks and thus
|
|
don't use 'synchronize'::
|
|
|
|
import thread
|
|
from ZServerPublisher import ZServerPublisher
|
|
|
|
class ZRendevous:
|
|
|
|
def __init__(self, n=1):
|
|
pool=[]
|
|
self._lists=pool, [], []
|
|
|
|
synchronize:
|
|
while n > 0:
|
|
l=thread.allocate_lock()
|
|
l.acquire()
|
|
pool.append(l)
|
|
thread.start_new_thread(ZServerPublisher,
|
|
(self.accept,))
|
|
n=n-1
|
|
|
|
def accept(self):
|
|
synchronize:
|
|
pool, requests, ready = self._lists
|
|
while not requests:
|
|
l=pool[-1]
|
|
del pool[-1]
|
|
ready.append(l)
|
|
|
|
asynchronize:
|
|
l.acquire()
|
|
|
|
pool.append(l)
|
|
|
|
r=requests[0]
|
|
del requests[0]
|
|
return r
|
|
|
|
def handle(self, name, request, response):
|
|
synchronize:
|
|
pool, requests, ready = self._lists
|
|
requests.append((name, request, response))
|
|
if ready:
|
|
l=ready[-1]
|
|
del ready[-1]
|
|
l.release()
|
|
|
|
Here is the original class as found in the
|
|
'Zope/ZServer/PubCore/ZRendevous.py' module. The "convenience" of
|
|
the '_a' and '_r' shortcut names obscure the code::
|
|
|
|
import thread
|
|
from ZServerPublisher import ZServerPublisher
|
|
|
|
class ZRendevous:
|
|
|
|
def __init__(self, n=1):
|
|
sync=thread.allocate_lock()
|
|
self._a=sync.acquire
|
|
self._r=sync.release
|
|
pool=[]
|
|
self._lists=pool, [], []
|
|
self._a()
|
|
try:
|
|
while n > 0:
|
|
l=thread.allocate_lock()
|
|
l.acquire()
|
|
pool.append(l)
|
|
thread.start_new_thread(ZServerPublisher,
|
|
(self.accept,))
|
|
n=n-1
|
|
finally: self._r()
|
|
|
|
def accept(self):
|
|
self._a()
|
|
try:
|
|
pool, requests, ready = self._lists
|
|
while not requests:
|
|
l=pool[-1]
|
|
del pool[-1]
|
|
ready.append(l)
|
|
self._r()
|
|
l.acquire()
|
|
self._a()
|
|
pool.append(l)
|
|
|
|
r=requests[0]
|
|
del requests[0]
|
|
return r
|
|
finally: self._r()
|
|
|
|
def handle(self, name, request, response):
|
|
self._a()
|
|
try:
|
|
pool, requests, ready = self._lists
|
|
requests.append((name, request, response))
|
|
if ready:
|
|
l=ready[-1]
|
|
del ready[-1]
|
|
l.release()
|
|
finally: self._r()
|
|
|
|
In particular the asynchronize section of the ``accept`` method is
|
|
not very obvious. To beginner programmers, 'synchronize' and
|
|
'asynchronize' remove many of the problems encountered when
|
|
juggling multiple ``acquire`` and ``release`` methods on different
|
|
locks in different ``try/finally`` blocks.
|
|
|
|
|
|
Formal Syntax
|
|
=============
|
|
|
|
Python syntax is defined in a modified BNF grammar notation
|
|
described in the Python Language Reference [1]_. This section
|
|
describes the proposed synchronization syntax using this grammar::
|
|
|
|
synchronize_stmt: 'synchronize' [test] ':' suite
|
|
asynchronize_stmt: 'asynchronize' [test] ':' suite
|
|
compound_stmt: ... | synchronized_stmt | asynchronize_stmt
|
|
|
|
(The '...' indicates other compound statements elided).
|
|
|
|
|
|
Proposed Implementation
|
|
=======================
|
|
|
|
The author of this PEP has not explored an implementation yet.
|
|
There are several implementation issues that must be resolved.
|
|
The main implementation issue is what exactly gets locked and
|
|
unlocked during a synchronized block.
|
|
|
|
During an unqualified synchronized block (the use of the
|
|
'synchronize' keyword without a target argument) a lock could be
|
|
created and associated with the synchronized code block object.
|
|
Any threads that are to execute the block must first acquire the
|
|
code block lock.
|
|
|
|
When an 'asynchronize' keyword is encountered in a 'synchronize'
|
|
block the code block lock is unlocked before the inner block is
|
|
executed and re-locked when the inner block terminates.
|
|
|
|
When a synchronized block target is specified the object is
|
|
associated with a lock. How this is implemented cleanly is
|
|
probably the highest risk of this proposal. Java Virtual Machines
|
|
typically associate a special hidden lock object with target
|
|
object and use it to synchronized the block around the target
|
|
only.
|
|
|
|
|
|
Backward Compatibility
|
|
======================
|
|
|
|
Backward compatibility is solved with the new ``from __future__``
|
|
Python syntax (:pep:`236`), and the new warning framework (:pep:`230`)
|
|
to evolve the
|
|
Python language into phasing out any conflicting names that use
|
|
the new keywords 'synchronize' and 'asynchronize'. To use the
|
|
syntax now, a developer could use the statement::
|
|
|
|
from __future__ import threadsync # or whatever
|
|
|
|
In addition, any code that uses the keyword 'synchronize' or
|
|
'asynchronize' as an identifier will be issued a warning from
|
|
Python. After the appropriate period of time, the syntax would
|
|
become standard, the above import statement would do nothing, and
|
|
any identifiers named 'synchronize' or 'asynchronize' would raise
|
|
an exception.
|
|
|
|
|
|
PEP 310 Reliable Acquisition/Release Pairs
|
|
==========================================
|
|
|
|
:pep:`310` proposes the 'with' keyword that can serve the same
|
|
function as 'synchronize' (but no facility for 'asynchronize').
|
|
The pattern::
|
|
|
|
initialize_lock()
|
|
|
|
with the_lock:
|
|
change_shared_data()
|
|
|
|
is equivalent to the proposed::
|
|
|
|
synchronize the_lock:
|
|
change_shared_data()
|
|
|
|
:pep:`310` must synchronize on an existing lock, while this PEP
|
|
proposes that unqualified 'synchronize' statements synchronize on
|
|
a global, internal, transparent lock in addition to qualified
|
|
'synchronize' statements. The 'with' statement also requires lock
|
|
initialization, while the 'synchronize' statement can synchronize
|
|
on any target object **including** locks.
|
|
|
|
While limited in this fashion, the 'with' statement is more
|
|
abstract and serves more purposes than synchronization. For
|
|
example, transactions could be used with the 'with' keyword::
|
|
|
|
initialize_transaction()
|
|
|
|
with my_transaction:
|
|
do_in_transaction()
|
|
|
|
# when the block terminates, the transaction is committed.
|
|
|
|
The 'synchronize' and 'asynchronize' keywords cannot serve this or
|
|
any other general acquire/release pattern other than thread
|
|
synchronization.
|
|
|
|
|
|
How Java Does It
|
|
================
|
|
|
|
Java defines a 'synchronized' keyword (note the grammatical tense
|
|
different between the Java keyword and this PEP's 'synchronize')
|
|
which must be qualified on any object. The syntax is::
|
|
|
|
synchronized (Expression) Block
|
|
|
|
Expression must yield a valid object (null raises an error and
|
|
exceptions during 'Expression' terminate the 'synchronized' block
|
|
for the same reason) upon which 'Block' is synchronized.
|
|
|
|
|
|
How Jython Does It
|
|
==================
|
|
|
|
Jython uses a 'synchronize' class with the static method
|
|
'make_synchronized' that accepts one callable argument and returns
|
|
a newly created, synchronized, callable "wrapper" around the
|
|
argument.
|
|
|
|
|
|
Summary of Proposed Changes to Python
|
|
=====================================
|
|
|
|
Adding new 'synchronize' and 'asynchronize' keywords to the
|
|
language.
|
|
|
|
|
|
Risks
|
|
=====
|
|
|
|
This PEP proposes adding two keywords to the Python language. This
|
|
may break code.
|
|
|
|
There is no implementation to test.
|
|
|
|
It's not the most important problem facing Python programmers
|
|
today (although it is a fairly notorious one).
|
|
|
|
The equivalent Java keyword is the past participle 'synchronized'.
|
|
This PEP proposes the present tense, 'synchronize' as being more
|
|
in spirit with Python (there being less distinction between
|
|
compile-time and run-time in Python than Java).
|
|
|
|
|
|
Dissenting Opinion
|
|
==================
|
|
|
|
This PEP has not been discussed on python-dev.
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [1] The Python Language Reference
|
|
http://docs.python.org/reference/
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
End:
|