iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/pep-0371/index.html

535 lines
44 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 371 – Addition of the multiprocessing package to the standard library | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0371/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 371 – Addition of the multiprocessing package to the standard library | peps.python.org'>
<meta property="og:description" content="This PEP proposes the inclusion of the pyProcessing 1 package into the Python standard library, renamed to “multiprocessing”.">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0371/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP proposes the inclusion of the pyProcessing 1 package into the Python standard library, renamed to “multiprocessing”.">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 371</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 371 – Addition of the multiprocessing package to the standard library</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Jesse Noller &lt;jnoller&#32;&#97;t&#32;gmail.com&gt;,
Richard Oudkerk &lt;r.m.oudkerk&#32;&#97;t&#32;googlemail.com&gt;</dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-even">Created<span class="colon">:</span></dt>
<dd class="field-even">06-May-2008</dd>
<dt class="field-odd">Python-Version<span class="colon">:</span></dt>
<dd class="field-odd">2.6, 3.0</dd>
<dt class="field-even">Post-History<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2008-June/080011.html" title="Python-Dev message">03-Jun-2008</a></dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#the-distributed-problem">The “Distributed” Problem</a></li>
<li><a class="reference internal" href="#performance-comparison">Performance Comparison</a></li>
<li><a class="reference internal" href="#maintenance">Maintenance</a></li>
<li><a class="reference internal" href="#api-naming">API Naming</a></li>
<li><a class="reference internal" href="#timing-schedule">Timing/Schedule</a></li>
<li><a class="reference internal" href="#open-issues">Open Issues</a></li>
<li><a class="reference internal" href="#closed-issues">Closed Issues</a></li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP proposes the inclusion of the <code class="docutils literal notranslate"><span class="pre">pyProcessing</span></code> <a class="footnote-reference brackets" href="#id8" id="id1">[1]</a> package
into the Python standard library, renamed to “multiprocessing”.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">processing</span></code> package mimics the standard library <code class="docutils literal notranslate"><span class="pre">threading</span></code>
module functionality to provide a process-based approach to
threaded programming allowing end-users to dispatch multiple
tasks that effectively side-step the global interpreter lock.</p>
<p>The package also provides server and client functionality
(<code class="docutils literal notranslate"><span class="pre">processing.Manager</span></code>) to provide remote sharing and management of
objects and tasks so that applications may not only leverage
multiple cores on the local machine, but also distribute objects
and tasks across a cluster of networked machines.</p>
<p>While the distributed capabilities of the package are beneficial,
the primary focus of this PEP is the core threading-like API and
capabilities of the package.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>The current CPython interpreter implements the Global Interpreter
Lock (GIL) and barring work in Python 3000 or other versions
currently planned <a class="footnote-reference brackets" href="#id9" id="id2">[2]</a>, the GIL will remain as-is within the
CPython interpreter for the foreseeable future. While the GIL
itself enables clean and easy to maintain C code for the
interpreter and extensions base, it is frequently an issue for
those Python programmers who are leveraging multi-core machines.</p>
<p>The GIL itself prevents more than a single thread from running
within the interpreter at any given point in time, effectively
removing Python’s ability to take advantage of multi-processor
systems.</p>
<p>The pyprocessing package offers a method to side-step the GIL
allowing applications within CPython to take advantage of
multi-core architectures without asking users to completely change
their programming paradigm (i.e.: dropping threaded programming
for another “concurrent” approach - Twisted, Actors, etc).</p>
<p>The Processing package offers CPython a “known API” which mirrors
albeit in a <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliant manner, that of the threading API,
with known semantics and easy scalability.</p>
<p>In the future, the package might not be as relevant should the
CPython interpreter enable “true” threading, however for some
applications, forking an OS process may sometimes be more
desirable than using lightweight threads, especially on those
platforms where process creation is fast and optimized.</p>
<p>For example, a simple threaded application:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span> <span class="k">as</span> <span class="n">worker</span>
<span class="k">def</span> <span class="nf">afunc</span><span class="p">(</span><span class="n">number</span><span class="p">):</span>
<span class="nb">print</span> <span class="n">number</span> <span class="o">*</span> <span class="mi">3</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">worker</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">afunc</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">4</span><span class="p">,))</span>
<span class="n">t</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">t</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>The pyprocessing package mirrored the API so well, that with a
simple change of the import to:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">processing</span> <span class="kn">import</span> <span class="n">process</span> <span class="k">as</span> <span class="n">worker</span>
</pre></div>
</div>
<p>The code would now execute through the processing.process class.
Obviously, with the renaming of the API to <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliance there
would be additional renaming which would need to occur within
user applications, however minor.</p>
<p>This type of compatibility means that, with a minor (in most cases)
change in code, users’ applications will be able to leverage all
cores and processors on a given machine for parallel execution.
In many cases the pyprocessing package is even faster than the
normal threading approach for I/O bound programs. This of course,
takes into account that the pyprocessing package is in optimized C
code, while the threading module is not.</p>
</section>
<section id="the-distributed-problem">
<h2><a class="toc-backref" href="#the-distributed-problem" role="doc-backlink">The “Distributed” Problem</a></h2>
<p>In the discussion on Python-Dev about the inclusion of this
package <a class="footnote-reference brackets" href="#id10" id="id3">[3]</a> there was confusion about the intentions this PEP with
an attempt to solve the “Distributed” problem - frequently
comparing the functionality of this package with other solutions
like MPI-based communication <a class="footnote-reference brackets" href="#id11" id="id4">[4]</a>, CORBA, or other distributed
object approaches <a class="footnote-reference brackets" href="#id12" id="id5">[5]</a>.</p>
<p>The “distributed” problem is large and varied. Each programmer
working within this domain has either very strong opinions about
their favorite module/method or a highly customized problem for
which no existing solution works.</p>
<p>The acceptance of this package does not preclude or recommend that
programmers working on the “distributed” problem not examine other
solutions for their problem domain. The intent of including this
package is to provide entry-level capabilities for local
concurrency and the basic support to spread that concurrency
across a network of machines - although the two are not tightly
coupled, the pyprocessing package could in fact, be used in
conjunction with any of the other solutions including MPI/etc.</p>
<p>If necessary - it is possible to completely decouple the local
concurrency abilities of the package from the
network-capable/shared aspects of the package. Without serious
concerns or cause however, the author of this PEP does not
recommend that approach.</p>
</section>
<section id="performance-comparison">
<h2><a class="toc-backref" href="#performance-comparison" role="doc-backlink">Performance Comparison</a></h2>
<p>As we all know - there are “lies, damned lies, and benchmarks”.
These speed comparisons, while aimed at showcasing the performance
of the pyprocessing package, are by no means comprehensive or
applicable to all possible use cases or environments. Especially
for those platforms with sluggish process forking timing.</p>
<p>All benchmarks were run using the following:</p>
<ul class="simple">
<li>4 Core Intel Xeon CPU &#64; 3.00GHz</li>
<li>16 GB of RAM</li>
<li>Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)</li>
<li>pyProcessing 0.52</li>
</ul>
<p>All of the code for this can be downloaded from
<a class="reference external" href="http://jessenoller.com/code/bench-src.tgz">http://jessenoller.com/code/bench-src.tgz</a></p>
<p>The basic method of execution for these benchmarks is in the
run_benchmarks.py <a class="footnote-reference brackets" href="#id13" id="id6">[6]</a> script, which is simply a wrapper to execute a
target function through a single threaded (linear), multi-threaded
(via threading), and multi-process (via pyprocessing) function for
a static number of iterations with increasing numbers of execution
loops and/or threads.</p>
<p>The run_benchmarks.py script executes each function 100 times,
picking the best run of that 100 iterations via the timeit module.</p>
<p>First, to identify the overhead of the spawning of the workers, we
execute a function which is simply a pass statement (empty):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">python</span> <span class="n">run_benchmarks</span><span class="o">.</span><span class="n">py</span> <span class="n">empty_func</span><span class="o">.</span><span class="n">py</span>
<span class="n">Importing</span> <span class="n">empty_func</span>
<span class="n">Starting</span> <span class="n">tests</span> <span class="o">...</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.000001</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.000796</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.000714</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.000002</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.001963</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.001466</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.000002</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.003986</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.002701</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.000003</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.007990</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.005512</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>As you can see, process forking via the pyprocessing package is
faster than the speed of building and then executing the threaded
version of the code.</p>
<p>The second test calculates 50000 Fibonacci numbers inside of each
thread (isolated and shared nothing):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">python</span> <span class="n">run_benchmarks</span><span class="o">.</span><span class="n">py</span> <span class="n">fibonacci</span><span class="o">.</span><span class="n">py</span>
<span class="n">Importing</span> <span class="n">fibonacci</span>
<span class="n">Starting</span> <span class="n">tests</span> <span class="o">...</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.195548</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.197909</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.201175</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.397540</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.397637</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.204265</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.795333</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.797262</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.206990</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">1.591680</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">1.596824</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.417899</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>The third test calculates the sum of all primes below 100000,
again sharing nothing:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">run_benchmarks</span><span class="o">.</span><span class="n">py</span> <span class="n">crunch_primes</span><span class="o">.</span><span class="n">py</span>
<span class="n">Importing</span> <span class="n">crunch_primes</span>
<span class="n">Starting</span> <span class="n">tests</span> <span class="o">...</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.495157</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.522320</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.523757</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">1.052048</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">1.154726</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.524603</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">2.104733</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">2.455215</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.530688</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">4.217455</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">5.109192</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">1.077939</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>The reason why tests two and three focused on pure numeric
crunching is to showcase how the current threading implementation
does hinder non-I/O applications. Obviously, these tests could be
improved to use a queue for coordination of results and chunks of
work but that is not required to show the performance of the
package and core processing.process module.</p>
<p>The next test is an I/O bound test. This is normally where we see
a steep improvement in the threading module approach versus a
single-threaded approach. In this case, each worker is opening a
descriptor to lorem.txt, randomly seeking within it and writing
lines to /dev/null:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">python</span> <span class="n">run_benchmarks</span><span class="o">.</span><span class="n">py</span> <span class="n">file_io</span><span class="o">.</span><span class="n">py</span>
<span class="n">Importing</span> <span class="n">file_io</span>
<span class="n">Starting</span> <span class="n">tests</span> <span class="o">...</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.057750</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.089992</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.090817</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.180256</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.329961</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.096683</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.370841</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">1.103678</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.101535</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.749571</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">2.437204</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.203438</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>As you can see, pyprocessing is still faster on this I/O operation
than using multiple threads. And using multiple threads is slower
than the single threaded execution itself.</p>
<p>Finally, we will run a socket-based test to show network I/O
performance. This function grabs a URL from a server on the LAN
that is a simple error page from tomcat. It gets the page 100
times. The network is silent, and a 10G connection:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">python</span> <span class="n">run_benchmarks</span><span class="o">.</span><span class="n">py</span> <span class="n">url_get</span><span class="o">.</span><span class="n">py</span>
<span class="n">Importing</span> <span class="n">url_get</span>
<span class="n">Starting</span> <span class="n">tests</span> <span class="o">...</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.124774</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.120478</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.121404</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.239574</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.146138</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.138366</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.479159</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.200985</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.188847</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.960621</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.659298</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.298625</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>We finally see threaded performance surpass that of
single-threaded execution, but the pyprocessing package is still
faster when increasing the number of workers. If you stay with
one or two threads/workers, then the timing between threads and
pyprocessing is fairly close.</p>
<p>One item of note however, is that there is an implicit overhead
within the pyprocessing package’s <code class="docutils literal notranslate"><span class="pre">Queue</span></code> implementation due to the
object serialization.</p>
<p>Alec Thomas provided a short example based on the
run_benchmarks.py script to demonstrate this overhead versus the
default <code class="docutils literal notranslate"><span class="pre">Queue</span></code> implementation:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">cmd</span><span class="p">:</span> <span class="n">run_bench_queue</span><span class="o">.</span><span class="n">py</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.010546</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">1</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.015164</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">1</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.066167</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.020768</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">2</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.041635</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">2</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.084270</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.041718</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">4</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.086394</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">4</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.144176</span> <span class="n">seconds</span>
<span class="n">non_threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">iters</span><span class="p">)</span> <span class="mf">0.083488</span> <span class="n">seconds</span>
<span class="n">threaded</span> <span class="p">(</span><span class="mi">8</span> <span class="n">threads</span><span class="p">)</span> <span class="mf">0.184254</span> <span class="n">seconds</span>
<span class="n">processes</span> <span class="p">(</span><span class="mi">8</span> <span class="n">procs</span><span class="p">)</span> <span class="mf">0.302999</span> <span class="n">seconds</span>
</pre></div>
</div>
<p>Additional benchmarks can be found in the pyprocessing package’s
source distribution’s examples/ directory. The examples will be
included in the package’s documentation.</p>
</section>
<section id="maintenance">
<h2><a class="toc-backref" href="#maintenance" role="doc-backlink">Maintenance</a></h2>
<p>Richard M. Oudkerk - the author of the pyprocessing package has
agreed to maintain the package within Python SVN. Jesse Noller
has volunteered to also help maintain/document and test the
package.</p>
</section>
<section id="api-naming">
<h2><a class="toc-backref" href="#api-naming" role="doc-backlink">API Naming</a></h2>
<p>While the aim of the package’s API is designed to closely mimic that of
the threading and <code class="docutils literal notranslate"><span class="pre">Queue</span></code> modules as of python 2.x, those modules are not
<a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliant. It has been decided that instead of adding the package
“as is” and therefore perpetuating the non-<a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliant naming, we
will rename all APIs, classes, etc to be fully <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliant.</p>
<p>This change does affect the ease-of-drop in replacement for those using
the threading module, but that is an acceptable side-effect in the view
of the authors, especially given that the threading module’s own API
will change.</p>
<p>Issue 3042 in the tracker proposes that for Python 2.6 there will be
two APIs for the threading module - the current one, and the <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a>
compliant one. Warnings about the upcoming removal of the original
java-style API will be issued when -3 is invoked.</p>
<p>In Python 3000, the threading API will become <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> compliant, which
means that the multiprocessing module and the threading module will
again have matching APIs.</p>
</section>
<section id="timing-schedule">
<h2><a class="toc-backref" href="#timing-schedule" role="doc-backlink">Timing/Schedule</a></h2>
<p>Some concerns have been raised about the timing/lateness of this
PEP for the 2.6 and 3.0 releases this year, however it is felt by
both the authors and others that the functionality this package
offers surpasses the risk of inclusion.</p>
<p>However, taking into account the desire not to destabilize
Python-core, some refactoring of pyprocessing’s code “into”
Python-core can be withheld until the next 2.x/3.x releases. This
means that the actual risk to Python-core is minimal, and largely
constrained to the actual package itself.</p>
</section>
<section id="open-issues">
<h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2>
<ul class="simple">
<li>Confirm no “default” remote connection capabilities, if needed
enable the remote security mechanisms by default for those
classes which offer remote capabilities.</li>
<li>Some of the API (<code class="docutils literal notranslate"><span class="pre">Queue</span></code> methods <code class="docutils literal notranslate"><span class="pre">qsize()</span></code>, <code class="docutils literal notranslate"><span class="pre">task_done()</span></code> and <code class="docutils literal notranslate"><span class="pre">join()</span></code>)
either need to be added, or the reason for their exclusion needs
to be identified and documented clearly.</li>
</ul>
</section>
<section id="closed-issues">
<h2><a class="toc-backref" href="#closed-issues" role="doc-backlink">Closed Issues</a></h2>
<ul class="simple">
<li>The <code class="docutils literal notranslate"><span class="pre">PyGILState</span></code> bug patch submitted in issue 1683 by roudkerk
must be applied for the package unit tests to work.</li>
<li>Existing documentation has to be moved to ReST formatting.</li>
<li>Reliance on ctypes: The <code class="docutils literal notranslate"><span class="pre">pyprocessing</span></code> package’s reliance on
ctypes prevents the package from functioning on platforms where
ctypes is not supported. This is not a restriction of this
package, but rather of ctypes.</li>
<li>DONE: Rename top-level package from “pyprocessing” to
“multiprocessing”.</li>
<li>DONE: Also note that the default behavior of process spawning
does not make it compatible with use within IDLE as-is, this
will be examined as a bug-fix or “setExecutable” enhancement.</li>
<li>DONE: Add in “multiprocessing.setExecutable()” method to override the
default behavior of the package to spawn processes using the
current executable name rather than the Python interpreter. Note
that Mark Hammond has suggested a factory-style interface for
this <a class="footnote-reference brackets" href="#id14" id="id7">[7]</a>.</li>
</ul>
</section>
<section id="references">
<h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="id8" role="doc-footnote">
<dt class="label" id="id8">[<a href="#id1">1</a>]</dt>
<dd>The 2008 era PyProcessing project (the pyprocessing name was since repurposed)
<a class="reference external" href="https://web.archive.org/web/20080914113946/https://pyprocessing.berlios.de/">https://web.archive.org/web/20080914113946/https://pyprocessing.berlios.de/</a></aside>
<aside class="footnote brackets" id="id9" role="doc-footnote">
<dt class="label" id="id9">[<a href="#id2">2</a>]</dt>
<dd>See Adam Olsen’s “safe threading” project
<a class="reference external" href="https://code.google.com/archive/p/python-safethread/">https://code.google.com/archive/p/python-safethread/</a></aside>
<aside class="footnote brackets" id="id10" role="doc-footnote">
<dt class="label" id="id10">[<a href="#id3">3</a>]</dt>
<dd>See: Addition of “pyprocessing” module to standard lib.
<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2008-May/079417.html">https://mail.python.org/pipermail/python-dev/2008-May/079417.html</a></aside>
<aside class="footnote brackets" id="id11" role="doc-footnote">
<dt class="label" id="id11">[<a href="#id4">4</a>]</dt>
<dd><a class="reference external" href="https://mpi4py.readthedocs.io/">https://mpi4py.readthedocs.io/</a></aside>
<aside class="footnote brackets" id="id12" role="doc-footnote">
<dt class="label" id="id12">[<a href="#id5">5</a>]</dt>
<dd>See “Cluster Computing”
<a class="reference external" href="https://wiki.python.org/moin/ParallelProcessing#Cluster_Computing">https://wiki.python.org/moin/ParallelProcessing#Cluster_Computing</a></aside>
<aside class="footnote brackets" id="id13" role="doc-footnote">
<dt class="label" id="id13">[<a href="#id6">6</a>]</dt>
<dd>The original run_benchmark.py code was published in Python
Magazine in December 2007: “Python Threads and the Global
Interpreter Lock” by Jesse Noller. It has been modified for
this PEP.</aside>
<aside class="footnote brackets" id="id14" role="doc-footnote">
<dt class="label" id="id14">[<a href="#id7">7</a>]</dt>
<dd><a class="reference external" href="http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34">http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34</a></aside>
</aside>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0371.rst">https://github.com/python/peps/blob/main/peps/pep-0371.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0371.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a></li>
<li><a class="reference internal" href="#the-distributed-problem">The “Distributed” Problem</a></li>
<li><a class="reference internal" href="#performance-comparison">Performance Comparison</a></li>
<li><a class="reference internal" href="#maintenance">Maintenance</a></li>
<li><a class="reference internal" href="#api-naming">API Naming</a></li>
<li><a class="reference internal" href="#timing-schedule">Timing/Schedule</a></li>
<li><a class="reference internal" href="#open-issues">Open Issues</a></li>
<li><a class="reference internal" href="#closed-issues">Closed Issues</a></li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0371.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink