reSTify PEP 371 (#325)
This commit is contained in:
parent
0f7b74537f
commit
dd454efb3a
76
pep-0371.txt
76
pep-0371.txt
|
@ -6,24 +6,25 @@ Author: Jesse Noller <jnoller@gmail.com>,
|
||||||
Richard Oudkerk <r.m.oudkerk@googlemail.com>
|
Richard Oudkerk <r.m.oudkerk@googlemail.com>
|
||||||
Status: Final
|
Status: Final
|
||||||
Type: Standards Track
|
Type: Standards Track
|
||||||
Content-Type: text/plain
|
Content-Type: text/x-rst
|
||||||
Created: 06-May-2008
|
Created: 06-May-2008
|
||||||
Python-Version: 2.6 / 3.0
|
Python-Version: 2.6 / 3.0
|
||||||
Post-History:
|
Post-History:
|
||||||
|
|
||||||
|
|
||||||
Abstract
|
Abstract
|
||||||
|
========
|
||||||
|
|
||||||
This PEP proposes the inclusion of the pyProcessing [1] package
|
This PEP proposes the inclusion of the ``pyProcessing`` [1]_ package
|
||||||
into the Python standard library, renamed to "multiprocessing".
|
into the Python standard library, renamed to "multiprocessing".
|
||||||
|
|
||||||
The processing package mimics the standard library threading
|
The ``processing`` package mimics the standard library ``threading``
|
||||||
module functionality to provide a process-based approach to
|
module functionality to provide a process-based approach to
|
||||||
threaded programming allowing end-users to dispatch multiple
|
threaded programming allowing end-users to dispatch multiple
|
||||||
tasks that effectively side-step the global interpreter lock.
|
tasks that effectively side-step the global interpreter lock.
|
||||||
|
|
||||||
The package also provides server and client functionality
|
The package also provides server and client functionality
|
||||||
(processing.Manager) to provide remote sharing and management of
|
(``processing.Manager``) to provide remote sharing and management of
|
||||||
objects and tasks so that applications may not only leverage
|
objects and tasks so that applications may not only leverage
|
||||||
multiple cores on the local machine, but also distribute objects
|
multiple cores on the local machine, but also distribute objects
|
||||||
and tasks across a cluster of networked machines.
|
and tasks across a cluster of networked machines.
|
||||||
|
@ -33,10 +34,11 @@ Abstract
|
||||||
capabilities of the package.
|
capabilities of the package.
|
||||||
|
|
||||||
Rationale
|
Rationale
|
||||||
|
=========
|
||||||
|
|
||||||
The current CPython interpreter implements the Global Interpreter
|
The current CPython interpreter implements the Global Interpreter
|
||||||
Lock (GIL) and barring work in Python 3000 or other versions
|
Lock (GIL) and barring work in Python 3000 or other versions
|
||||||
currently planned [2], the GIL will remain as-is within the
|
currently planned [2]_, the GIL will remain as-is within the
|
||||||
CPython interpreter for the foreseeable future. While the GIL
|
CPython interpreter for the foreseeable future. While the GIL
|
||||||
itself enables clean and easy to maintain C code for the
|
itself enables clean and easy to maintain C code for the
|
||||||
interpreter and extensions base, it is frequently an issue for
|
interpreter and extensions base, it is frequently an issue for
|
||||||
|
@ -63,7 +65,7 @@ Rationale
|
||||||
desirable than using lightweight threads, especially on those
|
desirable than using lightweight threads, especially on those
|
||||||
platforms where process creation is fast and optimized.
|
platforms where process creation is fast and optimized.
|
||||||
|
|
||||||
For example, a simple threaded application:
|
For example, a simple threaded application::
|
||||||
|
|
||||||
from threading import Thread as worker
|
from threading import Thread as worker
|
||||||
|
|
||||||
|
@ -75,7 +77,7 @@ Rationale
|
||||||
t.join()
|
t.join()
|
||||||
|
|
||||||
The pyprocessing package mirrored the API so well, that with a
|
The pyprocessing package mirrored the API so well, that with a
|
||||||
simple change of the import to:
|
simple change of the import to::
|
||||||
|
|
||||||
from processing import process as worker
|
from processing import process as worker
|
||||||
|
|
||||||
|
@ -93,13 +95,14 @@ Rationale
|
||||||
code, while the threading module is not.
|
code, while the threading module is not.
|
||||||
|
|
||||||
The "Distributed" Problem
|
The "Distributed" Problem
|
||||||
|
=========================
|
||||||
|
|
||||||
In the discussion on Python-Dev about the inclusion of this
|
In the discussion on Python-Dev about the inclusion of this
|
||||||
package [3] there was confusion about the intentions this PEP with
|
package [3]_ there was confusion about the intentions this PEP with
|
||||||
an attempt to solve the "Distributed" problem - frequently
|
an attempt to solve the "Distributed" problem - frequently
|
||||||
comparing the functionality of this package with other solutions
|
comparing the functionality of this package with other solutions
|
||||||
like MPI-based communication [4], CORBA, or other distributed
|
like MPI-based communication [4]_, CORBA, or other distributed
|
||||||
object approaches [5].
|
object approaches [5]_.
|
||||||
|
|
||||||
The "distributed" problem is large and varied. Each programmer
|
The "distributed" problem is large and varied. Each programmer
|
||||||
working within this domain has either very strong opinions about
|
working within this domain has either very strong opinions about
|
||||||
|
@ -122,6 +125,7 @@ The "Distributed" Problem
|
||||||
recommend that approach.
|
recommend that approach.
|
||||||
|
|
||||||
Performance Comparison
|
Performance Comparison
|
||||||
|
======================
|
||||||
|
|
||||||
As we all know - there are "lies, damned lies, and benchmarks".
|
As we all know - there are "lies, damned lies, and benchmarks".
|
||||||
These speed comparisons, while aimed at showcasing the performance
|
These speed comparisons, while aimed at showcasing the performance
|
||||||
|
@ -130,12 +134,13 @@ Performance Comparison
|
||||||
for those platforms with sluggish process forking timing.
|
for those platforms with sluggish process forking timing.
|
||||||
|
|
||||||
All benchmarks were run using the following:
|
All benchmarks were run using the following:
|
||||||
|
|
||||||
* 4 Core Intel Xeon CPU @ 3.00GHz
|
* 4 Core Intel Xeon CPU @ 3.00GHz
|
||||||
* 16 GB of RAM
|
* 16 GB of RAM
|
||||||
* Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)
|
* Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)
|
||||||
* pyProcessing 0.52
|
* pyProcessing 0.52
|
||||||
|
|
||||||
All of the code for this can be downloaded from:
|
All of the code for this can be downloaded from
|
||||||
http://jessenoller.com/code/bench-src.tgz
|
http://jessenoller.com/code/bench-src.tgz
|
||||||
|
|
||||||
The basic method of execution for these benchmarks is in the
|
The basic method of execution for these benchmarks is in the
|
||||||
|
@ -149,7 +154,7 @@ Performance Comparison
|
||||||
picking the best run of that 100 iterations via the timeit module.
|
picking the best run of that 100 iterations via the timeit module.
|
||||||
|
|
||||||
First, to identify the overhead of the spawning of the workers, we
|
First, to identify the overhead of the spawning of the workers, we
|
||||||
execute a function which is simply a pass statement (empty):
|
execute a function which is simply a pass statement (empty)::
|
||||||
|
|
||||||
cmd: python run_benchmarks.py empty_func.py
|
cmd: python run_benchmarks.py empty_func.py
|
||||||
Importing empty_func
|
Importing empty_func
|
||||||
|
@ -175,7 +180,7 @@ Performance Comparison
|
||||||
version of the code.
|
version of the code.
|
||||||
|
|
||||||
The second test calculates 50000 Fibonacci numbers inside of each
|
The second test calculates 50000 Fibonacci numbers inside of each
|
||||||
thread (isolated and shared nothing):
|
thread (isolated and shared nothing)::
|
||||||
|
|
||||||
cmd: python run_benchmarks.py fibonacci.py
|
cmd: python run_benchmarks.py fibonacci.py
|
||||||
Importing fibonacci
|
Importing fibonacci
|
||||||
|
@ -197,7 +202,7 @@ Performance Comparison
|
||||||
processes (8 procs) 0.417899 seconds
|
processes (8 procs) 0.417899 seconds
|
||||||
|
|
||||||
The third test calculates the sum of all primes below 100000,
|
The third test calculates the sum of all primes below 100000,
|
||||||
again sharing nothing.
|
again sharing nothing::
|
||||||
|
|
||||||
cmd: run_benchmarks.py crunch_primes.py
|
cmd: run_benchmarks.py crunch_primes.py
|
||||||
Importing crunch_primes
|
Importing crunch_primes
|
||||||
|
@ -229,7 +234,7 @@ Performance Comparison
|
||||||
a steep improvement in the threading module approach versus a
|
a steep improvement in the threading module approach versus a
|
||||||
single-threaded approach. In this case, each worker is opening a
|
single-threaded approach. In this case, each worker is opening a
|
||||||
descriptor to lorem.txt, randomly seeking within it and writing
|
descriptor to lorem.txt, randomly seeking within it and writing
|
||||||
lines to /dev/null:
|
lines to /dev/null::
|
||||||
|
|
||||||
cmd: python run_benchmarks.py file_io.py
|
cmd: python run_benchmarks.py file_io.py
|
||||||
Importing file_io
|
Importing file_io
|
||||||
|
@ -257,7 +262,7 @@ Performance Comparison
|
||||||
Finally, we will run a socket-based test to show network I/O
|
Finally, we will run a socket-based test to show network I/O
|
||||||
performance. This function grabs a URL from a server on the LAN
|
performance. This function grabs a URL from a server on the LAN
|
||||||
that is a simple error page from tomcat. It gets the page 100
|
that is a simple error page from tomcat. It gets the page 100
|
||||||
times. The network is silent, and a 10G connection:
|
times. The network is silent, and a 10G connection::
|
||||||
|
|
||||||
cmd: python run_benchmarks.py url_get.py
|
cmd: python run_benchmarks.py url_get.py
|
||||||
Importing url_get
|
Importing url_get
|
||||||
|
@ -285,12 +290,12 @@ Performance Comparison
|
||||||
pyprocessing is fairly close.
|
pyprocessing is fairly close.
|
||||||
|
|
||||||
One item of note however, is that there is an implicit overhead
|
One item of note however, is that there is an implicit overhead
|
||||||
within the pyprocessing package's Queue implementation due to the
|
within the pyprocessing package's ``Queue`` implementation due to the
|
||||||
object serialization.
|
object serialization.
|
||||||
|
|
||||||
Alec Thomas provided a short example based on the
|
Alec Thomas provided a short example based on the
|
||||||
run_benchmarks.py script to demonstrate this overhead versus the
|
run_benchmarks.py script to demonstrate this overhead versus the
|
||||||
default Queue implementation:
|
default ``Queue`` implementation::
|
||||||
|
|
||||||
cmd: run_bench_queue.py
|
cmd: run_bench_queue.py
|
||||||
non_threaded (1 iters) 0.010546 seconds
|
non_threaded (1 iters) 0.010546 seconds
|
||||||
|
@ -314,6 +319,7 @@ Performance Comparison
|
||||||
included in the package's documentation.
|
included in the package's documentation.
|
||||||
|
|
||||||
Maintenance
|
Maintenance
|
||||||
|
===========
|
||||||
|
|
||||||
Richard M. Oudkerk - the author of the pyprocessing package has
|
Richard M. Oudkerk - the author of the pyprocessing package has
|
||||||
agreed to maintain the package within Python SVN. Jesse Noller
|
agreed to maintain the package within Python SVN. Jesse Noller
|
||||||
|
@ -321,9 +327,10 @@ Maintenance
|
||||||
package.
|
package.
|
||||||
|
|
||||||
API Naming
|
API Naming
|
||||||
|
==========
|
||||||
|
|
||||||
While the aim of the package's API is designed to closely mimic that of
|
While the aim of the package's API is designed to closely mimic that of
|
||||||
the threading and Queue modules as of python 2.x, those modules are not
|
the threading and ``Queue`` modules as of python 2.x, those modules are not
|
||||||
PEP 8 compliant. It has been decided that instead of adding the package
|
PEP 8 compliant. It has been decided that instead of adding the package
|
||||||
"as is" and therefore perpetuating the non-PEP 8 compliant naming, we
|
"as is" and therefore perpetuating the non-PEP 8 compliant naming, we
|
||||||
will rename all APIs, classes, etc to be fully PEP 8 compliant.
|
will rename all APIs, classes, etc to be fully PEP 8 compliant.
|
||||||
|
@ -343,6 +350,7 @@ API Naming
|
||||||
again have matching APIs.
|
again have matching APIs.
|
||||||
|
|
||||||
Timing/Schedule
|
Timing/Schedule
|
||||||
|
===============
|
||||||
|
|
||||||
Some concerns have been raised about the timing/lateness of this
|
Some concerns have been raised about the timing/lateness of this
|
||||||
PEP for the 2.6 and 3.0 releases this year, however it is felt by
|
PEP for the 2.6 and 3.0 releases this year, however it is felt by
|
||||||
|
@ -356,23 +364,25 @@ Timing/Schedule
|
||||||
constrained to the actual package itself.
|
constrained to the actual package itself.
|
||||||
|
|
||||||
Open Issues
|
Open Issues
|
||||||
|
===========
|
||||||
|
|
||||||
* Confirm no "default" remote connection capabilities, if needed
|
* Confirm no "default" remote connection capabilities, if needed
|
||||||
enable the remote security mechanisms by default for those
|
enable the remote security mechanisms by default for those
|
||||||
classes which offer remote capabilities.
|
classes which offer remote capabilities.
|
||||||
|
|
||||||
* Some of the API (Queue methods qsize(), task_done() and join())
|
* Some of the API (``Queue`` methods ``qsize()``, ``task_done()`` and ``join()``)
|
||||||
either need to be added, or the reason for their exclusion needs
|
either need to be added, or the reason for their exclusion needs
|
||||||
to be identified and documented clearly.
|
to be identified and documented clearly.
|
||||||
|
|
||||||
Closed Issues
|
Closed Issues
|
||||||
|
=============
|
||||||
|
|
||||||
* The PyGILState bug patch submitted in issue 1683 by roudkerk
|
* The ``PyGILState`` bug patch submitted in issue 1683 by roudkerk
|
||||||
must be applied for the package unit tests to work.
|
must be applied for the package unit tests to work.
|
||||||
|
|
||||||
* Existing documentation has to be moved to ReST formatting.
|
* Existing documentation has to be moved to ReST formatting.
|
||||||
|
|
||||||
* Reliance on ctypes: The pyprocessing package's reliance on
|
* Reliance on ctypes: The ``pyprocessing`` package's reliance on
|
||||||
ctypes prevents the package from functioning on platforms where
|
ctypes prevents the package from functioning on platforms where
|
||||||
ctypes is not supported. This is not a restriction of this
|
ctypes is not supported. This is not a restriction of this
|
||||||
package, but rather of ctypes.
|
package, but rather of ctypes.
|
||||||
|
@ -388,40 +398,42 @@ Closed Issues
|
||||||
default behavior of the package to spawn processes using the
|
default behavior of the package to spawn processes using the
|
||||||
current executable name rather than the Python interpreter. Note
|
current executable name rather than the Python interpreter. Note
|
||||||
that Mark Hammond has suggested a factory-style interface for
|
that Mark Hammond has suggested a factory-style interface for
|
||||||
this[7].
|
this [7]_.
|
||||||
|
|
||||||
References
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
[1] PyProcessing home page
|
.. [1] PyProcessing home page
|
||||||
http://pyprocessing.berlios.de/
|
http://pyprocessing.berlios.de/
|
||||||
|
|
||||||
[2] See Adam Olsen's "safe threading" project
|
.. [2] See Adam Olsen's "safe threading" project
|
||||||
http://code.google.com/p/python-safethread/
|
http://code.google.com/p/python-safethread/
|
||||||
|
|
||||||
[3] See: Addition of "pyprocessing" module to standard lib.
|
.. [3] See: Addition of "pyprocessing" module to standard lib.
|
||||||
https://mail.python.org/pipermail/python-dev/2008-May/079417.html
|
https://mail.python.org/pipermail/python-dev/2008-May/079417.html
|
||||||
|
|
||||||
[4] http://mpi4py.scipy.org/
|
.. [4] http://mpi4py.scipy.org/
|
||||||
|
|
||||||
[5] See "Cluster Computing"
|
.. [5] See "Cluster Computing"
|
||||||
http://wiki.python.org/moin/ParallelProcessing
|
http://wiki.python.org/moin/ParallelProcessing
|
||||||
|
|
||||||
[6] The original run_benchmark.py code was published in Python
|
.. [6] The original run_benchmark.py code was published in Python
|
||||||
Magazine in December 2007: "Python Threads and the Global
|
Magazine in December 2007: "Python Threads and the Global
|
||||||
Interpreter Lock" by Jesse Noller. It has been modified for
|
Interpreter Lock" by Jesse Noller. It has been modified for
|
||||||
this PEP.
|
this PEP.
|
||||||
|
|
||||||
[7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34
|
.. [7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34
|
||||||
|
|
||||||
[8] Addition Python-Dev discussion
|
.. [8] Addition Python-Dev discussion
|
||||||
https://mail.python.org/pipermail/python-dev/2008-June/080011.html
|
https://mail.python.org/pipermail/python-dev/2008-June/080011.html
|
||||||
|
|
||||||
Copyright
|
Copyright
|
||||||
|
=========
|
||||||
|
|
||||||
This document has been placed in the public domain.
|
This document has been placed in the public domain.
|
||||||
|
|
||||||
|
|
||||||
|
..
|
||||||
Local Variables:
|
Local Variables:
|
||||||
mode: indented-text
|
mode: indented-text
|
||||||
indent-tabs-mode: nil
|
indent-tabs-mode: nil
|
||||||
|
|
Loading…
Reference in New Issue