diff --git a/pep-3145.txt b/pep-3145.txt index df2028faf..e2e57291d 100644 --- a/pep-3145.txt +++ b/pep-3145.txt @@ -5,136 +5,150 @@ Last-Modified: $Date$ Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Status: Withdrawn Type: Standards Track -Content-Type: text/plain +Content-Type: text/x-rst Created: 04-Aug-2009 Python-Version: 3.2 Post-History: -Abstract: - In its present form, the subprocess.Popen implementation is prone to - dead-locking and blocking of the parent Python script while waiting on data - from the child process. This PEP proposes to make - subprocess.Popen more asynchronous to help alleviate these - problems. +Abstract +======== + +In its present form, the subprocess.Popen implementation is prone to +dead-locking and blocking of the parent Python script while waiting on data +from the child process. This PEP proposes to make +subprocess.Popen more asynchronous to help alleviate these +problems. -PEP Deferral: +PEP Deferral +============ - Further exploration of the concepts covered in this PEP has been deferred - at least until after PEP 3156 has been resolved. +Further exploration of the concepts covered in this PEP has been deferred +at least until after PEP 3156 has been resolved. -PEP Withdrawal: +PEP Withdrawal +============== - This can be dealt with in the bug tracker. - A specific proposal is attached to http://bugs.python.org/issue18823 +This can be dealt with in the bug tracker. A specific proposal is +attached to [11]_. -Motivation: +Motivation +========== - A search for "python asynchronous subprocess" will turn up numerous - accounts of people wanting to execute a child process and communicate with - it from time to time reading only the data that is available instead of - blocking to wait for the program to produce data [1] [2] [3]. The current - behavior of the subprocess module is that when a user sends or receives - data via the stdin, stderr and stdout file objects, dead locks are common - and documented [4] [5]. While communicate can be used to alleviate some of - the buffering issues, it will still cause the parent process to block while - attempting to read data when none is available to be read from the child - process. +A search for "python asynchronous subprocess" will turn up numerous +accounts of people wanting to execute a child process and communicate with +it from time to time reading only the data that is available instead of +blocking to wait for the program to produce data [1]_ [2]_ [3]_. The current +behavior of the subprocess module is that when a user sends or receives +data via the stdin, stderr and stdout file objects, dead locks are common +and documented [4]_ [5]_. While communicate can be used to alleviate some of +the buffering issues, it will still cause the parent process to block while +attempting to read data when none is available to be read from the child +process. -Rationale: - There is a documented need for asynchronous, non-blocking functionality in - subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the - utility of the Python standard library that can be used on Unix based and - Windows builds of Python. Practically every I/O object in Python has a - file-like wrapper of some sort. Sockets already act as such and for - strings there is StringIO. Popen can be made to act like a file by simply - using the methods attached to the subprocess.Popen.stderr, stdout and - stdin file-like objects. But when using the read and write methods of - those options, you do not have the benefit of asynchronous I/O. In the - proposed solution the wrapper wraps the asynchronous methods to mimic a - file object. +Rationale +========= -Reference Implementation: +There is a documented need for asynchronous, non-blocking functionality in +subprocess.Popen [6]_ [7]_ [2]_ [3]_. Inclusion of the code would improve the +utility of the Python standard library that can be used on Unix based and +Windows builds of Python. Practically every I/O object in Python has a +file-like wrapper of some sort. Sockets already act as such and for +strings there is StringIO. Popen can be made to act like a file by simply +using the methods attached to the subprocess.Popen.stderr, stdout and +stdin file-like objects. But when using the read and write methods of +those options, you do not have the benefit of asynchronous I/O. In the +proposed solution the wrapper wraps the asynchronous methods to mimic a +file object. - I have been maintaining a Google Code repository that contains all of my - changes including tests and documentation [9] as well as blog detailing - the problems I have come across in the development process [10]. - I have been working on implementing non-blocking asynchronous I/O in the - subprocess.Popen module as well as a wrapper class for subprocess.Popen - that makes it so that an executed process can take the place of a file by - duplicating all of the methods and attributes that file objects have. +Reference Implementation +======================== - There are two base functions that have been added to the subprocess.Popen - class: Popen.send and Popen._recv, each with two separate implementations, - one for Windows and one for Unix-based systems. The Windows - implementation uses ctypes to access the functions needed to control pipes - in the kernel 32 DLL in an asynchronous manner. On Unix based systems, - the Python interface for file control serves the same purpose. The - different implementations of Popen.send and Popen._recv have identical - arguments to make code that uses these functions work across multiple - platforms. +I have been maintaining a Google Code repository that contains all of my +changes including tests and documentation [9]_ as well as blog detailing +the problems I have come across in the development process [10]_. - When calling the Popen._recv function, it requires the pipe name be - passed as an argument so there exists the Popen.recv function that passes - selects stdout as the pipe for Popen._recv by default. Popen.recv_err - selects stderr as the pipe by default. Popen.recv and Popen.recv_err - are much easier to read and understand than Popen._recv('stdout' ...) and - Popen._recv('stderr' ...) respectively. +I have been working on implementing non-blocking asynchronous I/O in the +subprocess.Popen module as well as a wrapper class for subprocess.Popen +that makes it so that an executed process can take the place of a file by +duplicating all of the methods and attributes that file objects have. - Since the Popen._recv function does not wait on data to be produced - before returning a value, it may return empty bytes. Popen.asyncread - handles this issue by returning all data read over a given time - interval. +There are two base functions that have been added to the subprocess.Popen +class: Popen.send and Popen._recv, each with two separate implementations, +one for Windows and one for Unix-based systems. The Windows +implementation uses ctypes to access the functions needed to control pipes +in the kernel 32 DLL in an asynchronous manner. On Unix based systems, +the Python interface for file control serves the same purpose. The +different implementations of Popen.send and Popen._recv have identical +arguments to make code that uses these functions work across multiple +platforms. - The ProcessIOWrapper class uses the asyncread and asyncwrite functions to - allow a process to act like a file so that there are no blocking issues - that can arise from using the stdout and stdin file objects produced from - a subprocess.Popen call. - +When calling the Popen._recv function, it requires the pipe name be +passed as an argument so there exists the Popen.recv function that passes +selects stdout as the pipe for Popen._recv by default. Popen.recv_err +selects stderr as the pipe by default. Popen.recv and Popen.recv_err +are much easier to read and understand than Popen._recv('stdout' ...) and +Popen._recv('stderr' ...) respectively. -References: +Since the Popen._recv function does not wait on data to be produced +before returning a value, it may return empty bytes. Popen.asyncread +handles this issue by returning all data read over a given time +interval. - [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess - http://mail.python.org/pipermail/python-bugs-list/2006-December/ - 036524.html +The ``ProcessIOWrapper`` class uses the ``asyncread`` and ``asyncwrite`` functions to +allow a process to act like a file so that there are no blocking issues +that can arise from using the stdout and stdin file objects produced from +a subprocess.Popen call. - [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess - http://ivory.idyll.org/blog/feb-07/problems-with-subprocess - [3] How can I run an external command asynchronously from Python? - Stack - Overflow - http://stackoverflow.com/questions/636561/how-can-i-run-an-external- - command-asynchronously-from-python +References +========== - [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation - http://docs.python.org/library/subprocess.html#subprocess.Popen.wait +.. [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess + http://mail.python.org/pipermail/python-bugs-list/2006-December/ + 036524.html - [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation - http://docs.python.org/library/subprocess.html#subprocess.Popen.kill +.. [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess + http://ivory.idyll.org/blog/feb-07/problems-with-subprocess - [6] Issue 1191964: asynchronous Subprocess - Python tracker - http://bugs.python.org/issue1191964 +.. [3] How can I run an external command asynchronously from Python? - Stack + Overflow + http://stackoverflow.com/questions/636561/how-can-i-run-an-external- + command-asynchronously-from-python - [7] Module to allow Asynchronous subprocess use on Windows and Posix - platforms - ActiveState Code - http://code.activestate.com/recipes/440554/ +.. [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation + http://docs.python.org/library/subprocess.html#subprocess.Popen.wait - [8] subprocess.rst - subprocdev - Project Hosting on Google Code - http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437 +.. [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation + http://docs.python.org/library/subprocess.html#subprocess.Popen.kill - [9] subprocdev - Project Hosting on Google Code - http://code.google.com/p/subprocdev +.. [6] Issue 1191964: asynchronous Subprocess - Python tracker + http://bugs.python.org/issue1191964 - [10] Python Subprocess Dev - http://subdev.blogspot.com/ +.. [7] Module to allow Asynchronous subprocess use on Windows and Posix + platforms - ActiveState Code + http://code.activestate.com/recipes/440554/ -Copyright: +.. [8] subprocess.rst - subprocdev - Project Hosting on Google Code + http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437 - This P.E.P. is licensed under the Open Publication License; - http://www.opencontent.org/openpub/. +.. [9] subprocdev - Project Hosting on Google Code + http://code.google.com/p/subprocdev + +.. [10] Python Subprocess Dev + http://subdev.blogspot.com/ + +.. [11] https://bugs.python.org/issue18823 -- Idle: use pipes instead of + sockets to talk with user subprocess + +Copyright +========= + +This P.E.P. is licensed under the Open Publication License; +http://www.opencontent.org/openpub/.