reSTify PEP 371 (#325)
This commit is contained in:
Huang Huang
Brett Cannon
parent
0f7b74537f
commit
dd454efb3a
76
pep-0371.txt
76
pep-0371.txt
|
|
@ -6,24 +6,25 @@ Author: Jesse Noller <jnoller@gmail.com>,
|
|||
Richard Oudkerk <r.m.oudkerk@googlemail.com>
|
||||
Status: Final
|
||||
Type: Standards Track
|
||||
Content-Type: text/plain
|
||||
Content-Type: text/x-rst
|
||||
Created: 06-May-2008
|
||||
Python-Version: 2.6 / 3.0
|
||||
Post-History:
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes the inclusion of the pyProcessing [1] package
|
||||
This PEP proposes the inclusion of the ``pyProcessing`` [1]_ package
|
||||
into the Python standard library, renamed to "multiprocessing".
|
||||
|
||||
The processing package mimics the standard library threading
|
||||
The ``processing`` package mimics the standard library ``threading``
|
||||
module functionality to provide a process-based approach to
|
||||
threaded programming allowing end-users to dispatch multiple
|
||||
tasks that effectively side-step the global interpreter lock.
|
||||
|
||||
The package also provides server and client functionality
|
||||
(processing.Manager) to provide remote sharing and management of
|
||||
(``processing.Manager``) to provide remote sharing and management of
|
||||
objects and tasks so that applications may not only leverage
|
||||
multiple cores on the local machine, but also distribute objects
|
||||
and tasks across a cluster of networked machines.
|
||||
|
|
@ -33,10 +34,11 @@ Abstract
|
|||
capabilities of the package.
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
The current CPython interpreter implements the Global Interpreter
|
||||
Lock (GIL) and barring work in Python 3000 or other versions
|
||||
currently planned [2], the GIL will remain as-is within the
|
||||
currently planned [2]_, the GIL will remain as-is within the
|
||||
CPython interpreter for the foreseeable future. While the GIL
|
||||
itself enables clean and easy to maintain C code for the
|
||||
interpreter and extensions base, it is frequently an issue for
|
||||
|
|
@ -63,7 +65,7 @@ Rationale
|
|||
desirable than using lightweight threads, especially on those
|
||||
platforms where process creation is fast and optimized.
|
||||
|
||||
For example, a simple threaded application:
|
||||
For example, a simple threaded application::
|
||||
|
||||
from threading import Thread as worker
|
||||
|
||||
|
|
@ -75,7 +77,7 @@ Rationale
|
|||
t.join()
|
||||
|
||||
The pyprocessing package mirrored the API so well, that with a
|
||||
simple change of the import to:
|
||||
simple change of the import to::
|
||||
|
||||
from processing import process as worker
|
||||
|
||||
|
|
@ -93,13 +95,14 @@ Rationale
|
|||
code, while the threading module is not.
|
||||
|
||||
The "Distributed" Problem
|
||||
=========================
|
||||
|
||||
In the discussion on Python-Dev about the inclusion of this
|
||||
package [3] there was confusion about the intentions this PEP with
|
||||
package [3]_ there was confusion about the intentions this PEP with
|
||||
an attempt to solve the "Distributed" problem - frequently
|
||||
comparing the functionality of this package with other solutions
|
||||
like MPI-based communication [4], CORBA, or other distributed
|
||||
object approaches [5].
|
||||
like MPI-based communication [4]_, CORBA, or other distributed
|
||||
object approaches [5]_.
|
||||
|
||||
The "distributed" problem is large and varied. Each programmer
|
||||
working within this domain has either very strong opinions about
|
||||
|
|
@ -122,6 +125,7 @@ The "Distributed" Problem
|
|||
recommend that approach.
|
||||
|
||||
Performance Comparison
|
||||
======================
|
||||
|
||||
As we all know - there are "lies, damned lies, and benchmarks".
|
||||
These speed comparisons, while aimed at showcasing the performance
|
||||
|
|
@ -130,12 +134,13 @@ Performance Comparison
|
|||
for those platforms with sluggish process forking timing.
|
||||
|
||||
All benchmarks were run using the following:
|
||||
|
||||
* 4 Core Intel Xeon CPU @ 3.00GHz
|
||||
* 16 GB of RAM
|
||||
* Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)
|
||||
* pyProcessing 0.52
|
||||
|
||||
All of the code for this can be downloaded from:
|
||||
All of the code for this can be downloaded from
|
||||
http://jessenoller.com/code/bench-src.tgz
|
||||
|
||||
The basic method of execution for these benchmarks is in the
|
||||
|
|
@ -149,7 +154,7 @@ Performance Comparison
|
|||
picking the best run of that 100 iterations via the timeit module.
|
||||
|
||||
First, to identify the overhead of the spawning of the workers, we
|
||||
execute a function which is simply a pass statement (empty):
|
||||
execute a function which is simply a pass statement (empty)::
|
||||
|
||||
cmd: python run_benchmarks.py empty_func.py
|
||||
Importing empty_func
|
||||
|
|
@ -175,7 +180,7 @@ Performance Comparison
|
|||
version of the code.
|
||||
|
||||
The second test calculates 50000 Fibonacci numbers inside of each
|
||||
thread (isolated and shared nothing):
|
||||
thread (isolated and shared nothing)::
|
||||
|
||||
cmd: python run_benchmarks.py fibonacci.py
|
||||
Importing fibonacci
|
||||
|
|
@ -197,7 +202,7 @@ Performance Comparison
|
|||
processes (8 procs) 0.417899 seconds
|
||||
|
||||
The third test calculates the sum of all primes below 100000,
|
||||
again sharing nothing.
|
||||
again sharing nothing::
|
||||
|
||||
cmd: run_benchmarks.py crunch_primes.py
|
||||
Importing crunch_primes
|
||||
|
|
@ -229,7 +234,7 @@ Performance Comparison
|
|||
a steep improvement in the threading module approach versus a
|
||||
single-threaded approach. In this case, each worker is opening a
|
||||
descriptor to lorem.txt, randomly seeking within it and writing
|
||||
lines to /dev/null:
|
||||
lines to /dev/null::
|
||||
|
||||
cmd: python run_benchmarks.py file_io.py
|
||||
Importing file_io
|
||||
|
|
@ -257,7 +262,7 @@ Performance Comparison
|
|||
Finally, we will run a socket-based test to show network I/O
|
||||
performance. This function grabs a URL from a server on the LAN
|
||||
that is a simple error page from tomcat. It gets the page 100
|
||||
times. The network is silent, and a 10G connection:
|
||||
times. The network is silent, and a 10G connection::
|
||||
|
||||
cmd: python run_benchmarks.py url_get.py
|
||||
Importing url_get
|
||||
|
|
@ -285,12 +290,12 @@ Performance Comparison
|
|||
pyprocessing is fairly close.
|
||||
|
||||
One item of note however, is that there is an implicit overhead
|
||||
within the pyprocessing package's Queue implementation due to the
|
||||
within the pyprocessing package's ``Queue`` implementation due to the
|
||||
object serialization.
|
||||
|
||||
Alec Thomas provided a short example based on the
|
||||
run_benchmarks.py script to demonstrate this overhead versus the
|
||||
default Queue implementation:
|
||||
default ``Queue`` implementation::
|
||||
|
||||
cmd: run_bench_queue.py
|
||||
non_threaded (1 iters) 0.010546 seconds
|
||||
|
|
@ -314,6 +319,7 @@ Performance Comparison
|
|||
included in the package's documentation.
|
||||
|
||||
Maintenance
|
||||
===========
|
||||
|
||||
Richard M. Oudkerk - the author of the pyprocessing package has
|
||||
agreed to maintain the package within Python SVN. Jesse Noller
|
||||
|
|
@ -321,9 +327,10 @@ Maintenance
|
|||
package.
|
||||
|
||||
API Naming
|
||||
==========
|
||||
|
||||
While the aim of the package's API is designed to closely mimic that of
|
||||
the threading and Queue modules as of python 2.x, those modules are not
|
||||
the threading and ``Queue`` modules as of python 2.x, those modules are not
|
||||
PEP 8 compliant. It has been decided that instead of adding the package
|
||||
"as is" and therefore perpetuating the non-PEP 8 compliant naming, we
|
||||
will rename all APIs, classes, etc to be fully PEP 8 compliant.
|
||||
|
|
@ -343,6 +350,7 @@ API Naming
|
|||
again have matching APIs.
|
||||
|
||||
Timing/Schedule
|
||||
===============
|
||||
|
||||
Some concerns have been raised about the timing/lateness of this
|
||||
PEP for the 2.6 and 3.0 releases this year, however it is felt by
|
||||
|
|
@ -356,23 +364,25 @@ Timing/Schedule
|
|||
constrained to the actual package itself.
|
||||
|
||||
Open Issues
|
||||
===========
|
||||
|
||||
* Confirm no "default" remote connection capabilities, if needed
|
||||
enable the remote security mechanisms by default for those
|
||||
classes which offer remote capabilities.
|
||||
|
||||
* Some of the API (Queue methods qsize(), task_done() and join())
|
||||
* Some of the API (``Queue`` methods ``qsize()``, ``task_done()`` and ``join()``)
|
||||
either need to be added, or the reason for their exclusion needs
|
||||
to be identified and documented clearly.
|
||||
|
||||
Closed Issues
|
||||
=============
|
||||
|
||||
* The PyGILState bug patch submitted in issue 1683 by roudkerk
|
||||
* The ``PyGILState`` bug patch submitted in issue 1683 by roudkerk
|
||||
must be applied for the package unit tests to work.
|
||||
|
||||
* Existing documentation has to be moved to ReST formatting.
|
||||
|
||||
* Reliance on ctypes: The pyprocessing package's reliance on
|
||||
* Reliance on ctypes: The ``pyprocessing`` package's reliance on
|
||||
ctypes prevents the package from functioning on platforms where
|
||||
ctypes is not supported. This is not a restriction of this
|
||||
package, but rather of ctypes.
|
||||
|
|
@ -388,40 +398,42 @@ Closed Issues
|
|||
default behavior of the package to spawn processes using the
|
||||
current executable name rather than the Python interpreter. Note
|
||||
that Mark Hammond has suggested a factory-style interface for
|
||||
this[7].
|
||||
this [7]_.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[1] PyProcessing home page
|
||||
.. [1] PyProcessing home page
|
||||
http://pyprocessing.berlios.de/
|
||||
|
||||
[2] See Adam Olsen's "safe threading" project
|
||||
.. [2] See Adam Olsen's "safe threading" project
|
||||
http://code.google.com/p/python-safethread/
|
||||
|
||||
[3] See: Addition of "pyprocessing" module to standard lib.
|
||||
.. [3] See: Addition of "pyprocessing" module to standard lib.
|
||||
https://mail.python.org/pipermail/python-dev/2008-May/079417.html
|
||||
|
||||
[4] http://mpi4py.scipy.org/
|
||||
.. [4] http://mpi4py.scipy.org/
|
||||
|
||||
[5] See "Cluster Computing"
|
||||
.. [5] See "Cluster Computing"
|
||||
http://wiki.python.org/moin/ParallelProcessing
|
||||
|
||||
[6] The original run_benchmark.py code was published in Python
|
||||
.. [6] The original run_benchmark.py code was published in Python
|
||||
Magazine in December 2007: "Python Threads and the Global
|
||||
Interpreter Lock" by Jesse Noller. It has been modified for
|
||||
this PEP.
|
||||
|
||||
[7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34
|
||||
.. [7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34
|
||||
|
||||
[8] Addition Python-Dev discussion
|
||||
.. [8] Addition Python-Dev discussion
|
||||
https://mail.python.org/pipermail/python-dev/2008-June/080011.html
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
|
|
|
|||
Loading…
Reference in New Issue