python-peps/pep-0397.txt

PEP: 397
Title: Python launcher for Windows
Version: $Revision$
Last-Modified: $Date$
Author: Mark Hammond <mhammond@skippinet.com.au>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 15-Mar-2011


Abstract

    This PEP describes a Python launcher for the Windows platform.  A 
    Python launcher is a single executable which uses a number of 
    heuristics to locate a Python executable and launch it with a
    specified command line.


Rationale

    Windows provides "file associations" so an executable can be associated
    with an extension, allowing for scripts to be executed directly in some
    contexts (eg., double-clicking the file in Windows Explorer.)  Until now,
    a strategy of "last installed Python wins" has been used and while not
    ideal, has generally been workable due to the conservative changes in
    Python 2.x releases.  As Python 3.x scripts are often syntactically
    incompatible with Python 2.x scripts, a different strategy must be used
    to allow files with a '.py' extension to use a different executable based
    on the Python version the script targets.  This will be done by borrowing
    the existing practices of another operating system - scripts will be able
    to nominate the version of Python they need by way of a "shebang" line, as
    described below.

    Unix-like operating systems (referred to simply as "Unix" in this
    PEP) allow scripts to be executed as if they were executable images
    by examining the script for a "shebang" line which specifies the
    actual executable to be used to run the script.  This is described in
    detail in the evecve(2) man page [1] and while user documentation will
    be created for this feature, for the purposes of this PEP that man
    page describes a valid shebang line.

    Additionally, these operating systems provide symbolic-links to
    Python executables in well-known directories. For example, many
    systems will have a link /usr/bin/python which references a
    particular version of Python installed under the operating-system.
    These symbolic links allow Python to be executed without regard for
    where Python it actually installed on the machine (eg., without
    requiring the path where Python is actually installed to be
    referenced in the shebang line or in the PATH.)  PEP 394 'The "python"
    command on Unix-Like Systems' [2] describes additional conventions
    for more fine-grained specification of a particular Python version.

    These 2 facilities combined allow for a portable and somewhat 
    predictable way of both starting Python interactively and for allowing
    Python scripts to execute.  This PEP describes an implementation of a 
    launcher which can offer the same benefits for Python on the Windows 
    platform and therefore allows the launcher to be the executable
    associated with '.py' files to support multiple Python versions
    concurrently.

    While this PEP offers the ability to use a shebang line which should
    work on both Windows and Unix, this is not the primary motivation for
    this PEP - the primary motivation is to allow a specific version to be
    specified without inventing new syntax or conventions to describe it.

An overview of the launcher.

    This PEP outlines the general functionality and key guidelines of a
    launcher for Windows.  It is accompanied by reference implementation,
    written in Python, which defines the detailed implementation.  Over
    time, changes to the implementation may be desired - if the changes 
    adhere to the guidelines in this PEP and have been made following 
    the standard Python development model this PEP need not change.  
    In other words, this PEP makes no attempt to describe in detail every 
    feature offered by the launcher but instead to offer guidelines the 
    launcher should adhere to.

    The actual implementation of the launcher will be written in C and
    will not link directly or indirectly with Python, but the Python
    based reference implementation should always remain the canonical
    description of the functionality.  Any changes proposed to the
    launcher functionality should first be made and agreed to in the
    Python implementation, after which time the C implementation can be
    changed to reflect the new reference.  Any deviations between the
    functionality of the reference implementation versus the C
    implementation due should imply the C implementation is in error
    (although some exceptions will need to be made due to the nature of the
    reference implementation being in Python)

    It is expected that 2 versions of the launcher will be needed - one
    which is a console program and one which is a "windows" (ie., GUI)
    program.  These 2 launchers correspond to the 'python.exe' and
    'pythonw.exe' executables which currently ship with Python.  The
    console launcher will be named 'py.exe' and the Windows one named
    'pyw.exe'.  The "windows" (ie., GUI) version of the launcher will attempt
    to locate and launch pythonw.exe even if a virtual shebang line nominates
    simply "python" - infact, the trailing 'w' notation will not be supported
    in the virtual shebang line at all.

    The launcher will be distributed with all future versions of Python
    and if possible, should be installed somewhere likely to already be
    on the system PATH (eg., the Windows System32) directory.  If installed,
    the "console" version of the launcher should be associated with .py files
    and the "windows" version associated with .pyw files.
    
    The launcher will not be tied to a specific version of Python - eg., a
    launcher distributed with Python 3.3 should be capable of locating and
    executing any Python 2.x and Python 3.x version.  Future versions of the
    launcher should remain backwards compatible with older versions, so later
    versions of Python can install an updated version of the launcher without
    impacting how the previously installed version of the launcher is used.

    The launcher will offer some conveniences for Python developers working
    interactively - for example, starting the launcher with no command-line
    arguments will launch the default Python with no command-line arguments.
    Further, command-line arguments will be supported to allow a specific
    Python version to be launched interactively - however, these conveniences
    will not detract from the primary purpose of launching scripts and will
    be easy to avoid if desired.

Guidelines for a Python launcher.

    The Python launcher described in this PEP will intentionally be
    constrained to the use-cases described in the Rationale section
    above.  It will not attempt to be a general purpose script launcher
    or shebang processor.

    The launcher should support for format of shebang lines as described
    in [1], including all restrictions listed.

    The launcher should support shebang lines commonly found on Unix.
    For example, a shebang line of '!# /usr/bin/python' should work even 
    though there is unlikely to be an executable in the relative Windows 
    directory "\usr\bin".  This means that many scripts can use a single
    shebang line and be likely to work on both Unix and Windows without
    modification.

    The launcher will support fully-qualified paths to executables.
    While this will make the script inherently non-portable, it is a
    feature offered by Unix and would be useful for Windows users in
    some cases.

    The launcher should be capable of supporting implementations other
    than CPython, such as jython and IronPython.  In particular, if there
    are common symbolic links used on Unix to specify such an
    implementation (such as "/usr/bin/jython"), the launcher need not 
    support such references on Windows.  However, if there is strong desire
    from the community to support these other implementations in a Windows
    specific way, any such proposals will be judged on their merit.  Even
    without specialized support for these alternate implementations, the
    ability to specify the fully-qualified path to an executable could be
    used as a fallback.

    On Unix, the user can control which specific version of Python is used
    by adjusting the links in /usr/bin to point to the desired version.  As
    the launcher on Windows will not use Windows links, environment variables
    will be used to override the semantics for determining exactly what
    version of Python will be used.  For example, while a shebang line of
    "/usr/bin/python2" will automatically locate a Python 2.x implementation,
    an environment variable can override exactly which Python 2.x
    implementation will be chosen.  Similarly for "/usr/bin/python" and
    "/usr/bin/python3".

    While the guidelines above are general, the launcher should make
    concessions for any Windows specific issues which increase the
    usefulness of the launcher on Windows.  In other words, the
    guidelines above will not prevent Windows specific functionality
    which makes the launcher significantly more useful for Windows
    users.

Shebang line parsing

    If the first command-line argument does not start with a dash ('-')
    character, an attempt will be made to open that argument as a file
    and parsed for a shebang line according to the rules in [1].  Once
    parsed, the command will be examined to see if it is a "virtual
    command".  A virtual command starts with either of the strings
    '/usr/bin/python', '/usr/bin/env python' or 'python' - while the
    latter will not work on Unix, it will be offered as a convenience
    for Windows users not familiar with the Unix conventions and with no
    desire to have the shebang line work on Unix.

    Commands which are virtual are not treated as containing a file-system
    path, but instead are treated as references to an installed Python.  
    Optionally, the virtual command may include a specific version, such
    as '/usr/bin/python2' or '/usr/bin/python3.2'.  If only a "major version"
    qualifier is found, the launcher will enumerate the installed Python 
    versions and use the latest minor release found for the major version,
    which is likely, although not guaranteed, to be the most recently 
    installed version in that family.  If major and minor version qualifiers 
    are found, that specific version will be located and used.

    If a virtual command has no version specifier, or if no shebang line
    is found, the launcher will attempt to locate a Python 2.x
    installation and use the latest found. If none are found, it will
    attempt to use the latest Python 3.x implementation found.

    The use of 'virtual' shebang lines will be encouraged as this should
    allow for portable shebang lines to be specified which work on
    multiple operating systems.

    If the shebang line is not a virtual one as described above, it is
    assumed to be a path to an executable - presumably a Python
    executable, but this will not be checked. No version qualifiers are
    parsed - if a specific version is required, then the path should
    reflect that. Relative path specifications will be considered
    relative to the directory of the launcher. Thus, a shebang line of
    '#! python' will look for a python executable in the same directory
    as the launcher.

    Non-virtual shebang lines should be discouraged as they make the
    script specific to a specific Windows installation. However, they
    are supported for maximum flexibility.

    If the first argument can not be opened as a file or if no valid
    shebang line can be found, a default Python interpreter will be
    located and the arguments passed to that.  However, if a valid
    shebang line is found but the process specified by that line can not
    be started, the default interpreter will not be started - the error
    to create the specified child process will be considered fatal and
    the launcher will display an appropriate message and terminate with
    a specific exit code.

    On 64bit Windows with both 32bit and 64bit implementations of the
    same (major.minor) Python version installed, the 64bit version will
    always be preferred.  This will be true for both 32bit and 64bit
    implementations of the launcher - a 32bit launcher will prefer to
    execute a 64bit Python installation of the same version if
    available.  This is so the behavior of the launcher can be predicted
    knowing only what versions are installed on the PC and without
    regard to the order in which they were installed.

Command-line handling

    Only the first command-line argument will be checked for a shebang line
    and only if that argument does not start with a '-'.

    If the only command-line argument is "-h" or "--help", the launcher will
    print a small banner and command-line usage, then pass the argument to
    the default Python.  This will have the effect of help for the launcher
    being printed followed by help for Python itself.  The output from the
    launcher will attempt to clearly indicate the extended help information
    is coming from the launcher and not Python.

    As a concession to interactively launching Python, the launcher will
    support the first command-line argument optionally being a version
    specifier in the form "-n[.n][:bits]" (where n is a single integer and
    bits is either '32' or '64') to nominate a specific version be used. For
    example, while "py.exe" may locate and launch the latest Python 2.x
    implementation installed, a command-line such as "py.exe -3" could specify
    the latest Python 3.x implementation be launched, while "py.exe -2.6:32"
    could specify a 32bit implementation Python 2.6 be located and launched.
    If a Python 2.x implementation is desired to be launched with the -3 flag,
    the command-line would need to be similar to "py.exe -2 -3" (or the
    specific version of Python could obviously be launched manually without
    use of this launcher.) Note that this feature can not be used with shebang
    processing as the file scanned for a shebang line and this argument must
    both be the first argument and therefore are mutually exclusive. 

    All other arguments will be passed untouched to the child Python process.

Process Launching

    Ideally, the launcher process would execute Python directly inside
    the same process, primarily so the parent of the launcher process could
    terminate the launcher and have the Python interpreter terminate.  If the
    launcher executes Python as a sub-process and the parent of the launcher
    terminates the launcher, the Python process will be unaffected.

    However, there are a number of practical problems associated with this
    approach.  Windows does not support the execv* family of Unix functions,
    so this could only be done by the launcher dynamically loading the Python
    DLL, but this would have a number of side-effects.  The most serious
    side effect of this is that the value of sys.executable would refer to the
    launcher instead of the Python implementation.  Many Python scripts use the
    value of sys.executable to launch child processes, and these scripts may
    fail to work as expected if the launcher is used - consider a "parent"
    script with a shebang line of '#! /usr/bin/python3' which attempts to
    launch a child script with no shebang line at all. Currently that
    child script would be launched using the exact same version running
    the parent script, but if sys.executable referred to the launcher the
    child would be likely executed using a Python 2.x version and would be
    likely to fail with a SyntaxError.  A solution for this would need to be
    found before this could be considered.

    The rules mentioned above regarding 64bit and 32bit programs would also be
    rendered impossible with this 'in-process' model - a 32bit launcher would
    be unable to load the 64bit version of Python and vice-versa.

    Given the impossibility of supporting the 64bit and 32bit requirements
    above, the launcher will execute Python in a child process, remaining
    alive while the child process is executing, then terminate with the same
    exit code as returned by the child.

    To address the concerns regarding the termination of the launcher not
    killing the child, the Win32 Job API will be used to arrange so that the
    child process is automatically killed when the parent is terminated
    (although children of that child process will continue as is the case
    now.) However, that functionality only works on Windows XP and later, so
    the launcher running on the Windows 2000 platform will not have this
    feature - killing the launcher on that platform will keep the child
    process alive.  This feature has been implemented in the reference
    implementation.

Alternative implementations and distributions of Python.

    While this PEP is primarily concerned with the python.org distributions of
    CPython, it does not preclude future enhancements to support both
    different distributions of CPython (eg. ActivePython, Enthought), nor
    different implementations of Python (eg. jython, IronPython.)
    Enhancements to support these alternatives should be proposed by
    (community or corporate) representatives of these alternatives and
    addressed via the normal Python development process and after relevant
    patches have been submitted for the reference implementation.

References

    [1] http://linux.die.net/man/2/execve

    [2] http://www.python.org/dev/peps/pep-0394/

Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: