141 lines
6.0 KiB
Plaintext
141 lines
6.0 KiB
Plaintext
PEP: 3145
|
|
Title: Asynchronous I/O For subprocess.Popen
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
|
|
Status: Withdrawn
|
|
Type: Standards Track
|
|
Content-Type: text/plain
|
|
Created: 04-Aug-2009
|
|
Python-Version: 3.2
|
|
Post-History:
|
|
|
|
Abstract:
|
|
|
|
In its present form, the subprocess.Popen implementation is prone to
|
|
dead-locking and blocking of the parent Python script while waiting on data
|
|
from the child process. This PEP proposes to make
|
|
subprocess.Popen more asynchronous to help alleviate these
|
|
problems.
|
|
|
|
|
|
PEP Deferral:
|
|
|
|
Further exploration of the concepts covered in this PEP has been deferred
|
|
at least until after PEP 3156 has been resolved.
|
|
|
|
|
|
PEP Withdrawal:
|
|
|
|
This can be dealt with in the bug tracker.
|
|
A specific proposal is attached to http://bugs.python.org/issue18823
|
|
|
|
|
|
Motivation:
|
|
|
|
A search for "python asynchronous subprocess" will turn up numerous
|
|
accounts of people wanting to execute a child process and communicate with
|
|
it from time to time reading only the data that is available instead of
|
|
blocking to wait for the program to produce data [1] [2] [3]. The current
|
|
behavior of the subprocess module is that when a user sends or receives
|
|
data via the stdin, stderr and stdout file objects, dead locks are common
|
|
and documented [4] [5]. While communicate can be used to alleviate some of
|
|
the buffering issues, it will still cause the parent process to block while
|
|
attempting to read data when none is available to be read from the child
|
|
process.
|
|
|
|
Rationale:
|
|
|
|
There is a documented need for asynchronous, non-blocking functionality in
|
|
subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the
|
|
utility of the Python standard library that can be used on Unix based and
|
|
Windows builds of Python. Practically every I/O object in Python has a
|
|
file-like wrapper of some sort. Sockets already act as such and for
|
|
strings there is StringIO. Popen can be made to act like a file by simply
|
|
using the methods attached to the subprocess.Popen.stderr, stdout and
|
|
stdin file-like objects. But when using the read and write methods of
|
|
those options, you do not have the benefit of asynchronous I/O. In the
|
|
proposed solution the wrapper wraps the asynchronous methods to mimic a
|
|
file object.
|
|
|
|
Reference Implementation:
|
|
|
|
I have been maintaining a Google Code repository that contains all of my
|
|
changes including tests and documentation [9] as well as blog detailing
|
|
the problems I have come across in the development process [10].
|
|
|
|
I have been working on implementing non-blocking asynchronous I/O in the
|
|
subprocess.Popen module as well as a wrapper class for subprocess.Popen
|
|
that makes it so that an executed process can take the place of a file by
|
|
duplicating all of the methods and attributes that file objects have.
|
|
|
|
There are two base functions that have been added to the subprocess.Popen
|
|
class: Popen.send and Popen._recv, each with two separate implementations,
|
|
one for Windows and one for Unix based systems. The Windows
|
|
implementation uses ctypes to access the functions needed to control pipes
|
|
in the kernel 32 DLL in an asynchronous manner. On Unix based systems,
|
|
the Python interface for file control serves the same purpose. The
|
|
different implementations of Popen.send and Popen._recv have identical
|
|
arguments to make code that uses these functions work across multiple
|
|
platforms.
|
|
|
|
When calling the Popen._recv function, it requires the pipe name be
|
|
passed as an argument so there exists the Popen.recv function that passes
|
|
selects stdout as the pipe for Popen._recv by default. Popen.recv_err
|
|
selects stderr as the pipe by default. Popen.recv and Popen.recv_err
|
|
are much easier to read and understand than Popen._recv('stdout' ...) and
|
|
Popen._recv('stderr' ...) respectively.
|
|
|
|
Since the Popen._recv function does not wait on data to be produced
|
|
before returning a value, it may return empty bytes. Popen.asyncread
|
|
handles this issue by returning all data read over a given time
|
|
interval.
|
|
|
|
The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
|
|
allow a process to act like a file so that there are no blocking issues
|
|
that can arise from using the stdout and stdin file objects produced from
|
|
a subprocess.Popen call.
|
|
|
|
|
|
References:
|
|
|
|
[1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
|
|
http://mail.python.org/pipermail/python-bugs-list/2006-December/
|
|
036524.html
|
|
|
|
[2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
|
|
http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
|
|
|
|
[3] How can I run an external command asynchronously from Python? - Stack
|
|
Overflow
|
|
http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
|
|
command-asynchronously-from-python
|
|
|
|
[4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
|
|
http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
|
|
|
|
[5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
|
|
http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
|
|
|
|
[6] Issue 1191964: asynchronous Subprocess - Python tracker
|
|
http://bugs.python.org/issue1191964
|
|
|
|
[7] Module to allow Asynchronous subprocess use on Windows and Posix
|
|
platforms - ActiveState Code
|
|
http://code.activestate.com/recipes/440554/
|
|
|
|
[8] subprocess.rst - subprocdev - Project Hosting on Google Code
|
|
http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
|
|
|
|
[9] subprocdev - Project Hosting on Google Code
|
|
http://code.google.com/p/subprocdev
|
|
|
|
[10] Python Subprocess Dev
|
|
http://subdev.blogspot.com/
|
|
|
|
Copyright:
|
|
|
|
This P.E.P. is licensed under the Open Publication License;
|
|
http://www.opencontent.org/openpub/.
|
|
|