reSTify PEP 371 (#325)

This commit is contained in:
Huang Huang 2017-08-17 06:47:03 +08:00 committed by Brett Cannon
parent 0f7b74537f
commit dd454efb3a
1 changed files with 321 additions and 309 deletions

View File

@ -6,24 +6,25 @@ Author: Jesse Noller <jnoller@gmail.com>,
Richard Oudkerk <r.m.oudkerk@googlemail.com> Richard Oudkerk <r.m.oudkerk@googlemail.com>
Status: Final Status: Final
Type: Standards Track Type: Standards Track
Content-Type: text/plain Content-Type: text/x-rst
Created: 06-May-2008 Created: 06-May-2008
Python-Version: 2.6 / 3.0 Python-Version: 2.6 / 3.0
Post-History: Post-History:
Abstract Abstract
========
This PEP proposes the inclusion of the pyProcessing [1] package This PEP proposes the inclusion of the ``pyProcessing`` [1]_ package
into the Python standard library, renamed to "multiprocessing". into the Python standard library, renamed to "multiprocessing".
The processing package mimics the standard library threading The ``processing`` package mimics the standard library ``threading``
module functionality to provide a process-based approach to module functionality to provide a process-based approach to
threaded programming allowing end-users to dispatch multiple threaded programming allowing end-users to dispatch multiple
tasks that effectively side-step the global interpreter lock. tasks that effectively side-step the global interpreter lock.
The package also provides server and client functionality The package also provides server and client functionality
(processing.Manager) to provide remote sharing and management of (``processing.Manager``) to provide remote sharing and management of
objects and tasks so that applications may not only leverage objects and tasks so that applications may not only leverage
multiple cores on the local machine, but also distribute objects multiple cores on the local machine, but also distribute objects
and tasks across a cluster of networked machines. and tasks across a cluster of networked machines.
@ -33,10 +34,11 @@ Abstract
capabilities of the package. capabilities of the package.
Rationale Rationale
=========
The current CPython interpreter implements the Global Interpreter The current CPython interpreter implements the Global Interpreter
Lock (GIL) and barring work in Python 3000 or other versions Lock (GIL) and barring work in Python 3000 or other versions
currently planned [2], the GIL will remain as-is within the currently planned [2]_, the GIL will remain as-is within the
CPython interpreter for the foreseeable future. While the GIL CPython interpreter for the foreseeable future. While the GIL
itself enables clean and easy to maintain C code for the itself enables clean and easy to maintain C code for the
interpreter and extensions base, it is frequently an issue for interpreter and extensions base, it is frequently an issue for
@ -63,7 +65,7 @@ Rationale
desirable than using lightweight threads, especially on those desirable than using lightweight threads, especially on those
platforms where process creation is fast and optimized. platforms where process creation is fast and optimized.
For example, a simple threaded application: For example, a simple threaded application::
from threading import Thread as worker from threading import Thread as worker
@ -75,7 +77,7 @@ Rationale
t.join() t.join()
The pyprocessing package mirrored the API so well, that with a The pyprocessing package mirrored the API so well, that with a
simple change of the import to: simple change of the import to::
from processing import process as worker from processing import process as worker
@ -93,13 +95,14 @@ Rationale
code, while the threading module is not. code, while the threading module is not.
The "Distributed" Problem The "Distributed" Problem
=========================
In the discussion on Python-Dev about the inclusion of this In the discussion on Python-Dev about the inclusion of this
package [3] there was confusion about the intentions this PEP with package [3]_ there was confusion about the intentions this PEP with
an attempt to solve the "Distributed" problem - frequently an attempt to solve the "Distributed" problem - frequently
comparing the functionality of this package with other solutions comparing the functionality of this package with other solutions
like MPI-based communication [4], CORBA, or other distributed like MPI-based communication [4]_, CORBA, or other distributed
object approaches [5]. object approaches [5]_.
The "distributed" problem is large and varied. Each programmer The "distributed" problem is large and varied. Each programmer
working within this domain has either very strong opinions about working within this domain has either very strong opinions about
@ -122,6 +125,7 @@ The "Distributed" Problem
recommend that approach. recommend that approach.
Performance Comparison Performance Comparison
======================
As we all know - there are "lies, damned lies, and benchmarks". As we all know - there are "lies, damned lies, and benchmarks".
These speed comparisons, while aimed at showcasing the performance These speed comparisons, while aimed at showcasing the performance
@ -130,12 +134,13 @@ Performance Comparison
for those platforms with sluggish process forking timing. for those platforms with sluggish process forking timing.
All benchmarks were run using the following: All benchmarks were run using the following:
* 4 Core Intel Xeon CPU @ 3.00GHz * 4 Core Intel Xeon CPU @ 3.00GHz
* 16 GB of RAM * 16 GB of RAM
* Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6) * Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)
* pyProcessing 0.52 * pyProcessing 0.52
All of the code for this can be downloaded from: All of the code for this can be downloaded from
http://jessenoller.com/code/bench-src.tgz http://jessenoller.com/code/bench-src.tgz
The basic method of execution for these benchmarks is in the The basic method of execution for these benchmarks is in the
@ -149,7 +154,7 @@ Performance Comparison
picking the best run of that 100 iterations via the timeit module. picking the best run of that 100 iterations via the timeit module.
First, to identify the overhead of the spawning of the workers, we First, to identify the overhead of the spawning of the workers, we
execute a function which is simply a pass statement (empty): execute a function which is simply a pass statement (empty)::
cmd: python run_benchmarks.py empty_func.py cmd: python run_benchmarks.py empty_func.py
Importing empty_func Importing empty_func
@ -175,7 +180,7 @@ Performance Comparison
version of the code. version of the code.
The second test calculates 50000 Fibonacci numbers inside of each The second test calculates 50000 Fibonacci numbers inside of each
thread (isolated and shared nothing): thread (isolated and shared nothing)::
cmd: python run_benchmarks.py fibonacci.py cmd: python run_benchmarks.py fibonacci.py
Importing fibonacci Importing fibonacci
@ -197,7 +202,7 @@ Performance Comparison
processes (8 procs) 0.417899 seconds processes (8 procs) 0.417899 seconds
The third test calculates the sum of all primes below 100000, The third test calculates the sum of all primes below 100000,
again sharing nothing. again sharing nothing::
cmd: run_benchmarks.py crunch_primes.py cmd: run_benchmarks.py crunch_primes.py
Importing crunch_primes Importing crunch_primes
@ -229,7 +234,7 @@ Performance Comparison
a steep improvement in the threading module approach versus a a steep improvement in the threading module approach versus a
single-threaded approach. In this case, each worker is opening a single-threaded approach. In this case, each worker is opening a
descriptor to lorem.txt, randomly seeking within it and writing descriptor to lorem.txt, randomly seeking within it and writing
lines to /dev/null: lines to /dev/null::
cmd: python run_benchmarks.py file_io.py cmd: python run_benchmarks.py file_io.py
Importing file_io Importing file_io
@ -257,7 +262,7 @@ Performance Comparison
Finally, we will run a socket-based test to show network I/O Finally, we will run a socket-based test to show network I/O
performance. This function grabs a URL from a server on the LAN performance. This function grabs a URL from a server on the LAN
that is a simple error page from tomcat. It gets the page 100 that is a simple error page from tomcat. It gets the page 100
times. The network is silent, and a 10G connection: times. The network is silent, and a 10G connection::
cmd: python run_benchmarks.py url_get.py cmd: python run_benchmarks.py url_get.py
Importing url_get Importing url_get
@ -285,12 +290,12 @@ Performance Comparison
pyprocessing is fairly close. pyprocessing is fairly close.
One item of note however, is that there is an implicit overhead One item of note however, is that there is an implicit overhead
within the pyprocessing package's Queue implementation due to the within the pyprocessing package's ``Queue`` implementation due to the
object serialization. object serialization.
Alec Thomas provided a short example based on the Alec Thomas provided a short example based on the
run_benchmarks.py script to demonstrate this overhead versus the run_benchmarks.py script to demonstrate this overhead versus the
default Queue implementation: default ``Queue`` implementation::
cmd: run_bench_queue.py cmd: run_bench_queue.py
non_threaded (1 iters) 0.010546 seconds non_threaded (1 iters) 0.010546 seconds
@ -314,6 +319,7 @@ Performance Comparison
included in the package's documentation. included in the package's documentation.
Maintenance Maintenance
===========
Richard M. Oudkerk - the author of the pyprocessing package has Richard M. Oudkerk - the author of the pyprocessing package has
agreed to maintain the package within Python SVN. Jesse Noller agreed to maintain the package within Python SVN. Jesse Noller
@ -321,9 +327,10 @@ Maintenance
package. package.
API Naming API Naming
==========
While the aim of the package's API is designed to closely mimic that of While the aim of the package's API is designed to closely mimic that of
the threading and Queue modules as of python 2.x, those modules are not the threading and ``Queue`` modules as of python 2.x, those modules are not
PEP 8 compliant. It has been decided that instead of adding the package PEP 8 compliant. It has been decided that instead of adding the package
"as is" and therefore perpetuating the non-PEP 8 compliant naming, we "as is" and therefore perpetuating the non-PEP 8 compliant naming, we
will rename all APIs, classes, etc to be fully PEP 8 compliant. will rename all APIs, classes, etc to be fully PEP 8 compliant.
@ -343,6 +350,7 @@ API Naming
again have matching APIs. again have matching APIs.
Timing/Schedule Timing/Schedule
===============
Some concerns have been raised about the timing/lateness of this Some concerns have been raised about the timing/lateness of this
PEP for the 2.6 and 3.0 releases this year, however it is felt by PEP for the 2.6 and 3.0 releases this year, however it is felt by
@ -356,23 +364,25 @@ Timing/Schedule
constrained to the actual package itself. constrained to the actual package itself.
Open Issues Open Issues
===========
* Confirm no "default" remote connection capabilities, if needed * Confirm no "default" remote connection capabilities, if needed
enable the remote security mechanisms by default for those enable the remote security mechanisms by default for those
classes which offer remote capabilities. classes which offer remote capabilities.
* Some of the API (Queue methods qsize(), task_done() and join()) * Some of the API (``Queue`` methods ``qsize()``, ``task_done()`` and ``join()``)
either need to be added, or the reason for their exclusion needs either need to be added, or the reason for their exclusion needs
to be identified and documented clearly. to be identified and documented clearly.
Closed Issues Closed Issues
=============
* The PyGILState bug patch submitted in issue 1683 by roudkerk * The ``PyGILState`` bug patch submitted in issue 1683 by roudkerk
must be applied for the package unit tests to work. must be applied for the package unit tests to work.
* Existing documentation has to be moved to ReST formatting. * Existing documentation has to be moved to ReST formatting.
* Reliance on ctypes: The pyprocessing package's reliance on * Reliance on ctypes: The ``pyprocessing`` package's reliance on
ctypes prevents the package from functioning on platforms where ctypes prevents the package from functioning on platforms where
ctypes is not supported. This is not a restriction of this ctypes is not supported. This is not a restriction of this
package, but rather of ctypes. package, but rather of ctypes.
@ -388,40 +398,42 @@ Closed Issues
default behavior of the package to spawn processes using the default behavior of the package to spawn processes using the
current executable name rather than the Python interpreter. Note current executable name rather than the Python interpreter. Note
that Mark Hammond has suggested a factory-style interface for that Mark Hammond has suggested a factory-style interface for
this[7]. this [7]_.
References References
==========
[1] PyProcessing home page .. [1] PyProcessing home page
http://pyprocessing.berlios.de/ http://pyprocessing.berlios.de/
[2] See Adam Olsen's "safe threading" project .. [2] See Adam Olsen's "safe threading" project
http://code.google.com/p/python-safethread/ http://code.google.com/p/python-safethread/
[3] See: Addition of "pyprocessing" module to standard lib. .. [3] See: Addition of "pyprocessing" module to standard lib.
https://mail.python.org/pipermail/python-dev/2008-May/079417.html https://mail.python.org/pipermail/python-dev/2008-May/079417.html
[4] http://mpi4py.scipy.org/ .. [4] http://mpi4py.scipy.org/
[5] See "Cluster Computing" .. [5] See "Cluster Computing"
http://wiki.python.org/moin/ParallelProcessing http://wiki.python.org/moin/ParallelProcessing
[6] The original run_benchmark.py code was published in Python .. [6] The original run_benchmark.py code was published in Python
Magazine in December 2007: "Python Threads and the Global Magazine in December 2007: "Python Threads and the Global
Interpreter Lock" by Jesse Noller. It has been modified for Interpreter Lock" by Jesse Noller. It has been modified for
this PEP. this PEP.
[7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34 .. [7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34
[8] Addition Python-Dev discussion .. [8] Addition Python-Dev discussion
https://mail.python.org/pipermail/python-dev/2008-June/080011.html https://mail.python.org/pipermail/python-dev/2008-June/080011.html
Copyright Copyright
=========
This document has been placed in the public domain. This document has been placed in the public domain.
..
Local Variables: Local Variables:
mode: indented-text mode: indented-text
indent-tabs-mode: nil indent-tabs-mode: nil