python-peps/pep-0394.txt

PEP: 394
Title: The "python" Command on Unix-Like Systems
Version: $Revision$
Last-Modified: $Date$
Author: Kerrick Staley <mail@kerrickstaley.com>,
        Nick Coghlan <ncoghlan@gmail.com>,
        Barry Warsaw <barry@python.org>
Status: Active
Type: Informational
Content-Type: text/x-rst
Created: 02-Mar-2011
Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014
Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116594.html


Abstract
========

This PEP provides a convention to ensure that Python scripts can continue to
be portable across ``*nix`` systems, regardless of the default version of the
Python interpreter (i.e. the version invoked by the ``python`` command).

* ``python2`` will refer to some version of Python 2.x.
* ``python3`` will refer to some version of Python 3.x.
* for the time being, all distributions *should* ensure that ``python``
  refers to the same target as ``python2``.
* however, end users should be aware that ``python`` refers to ``python3``
  on at least Arch Linux (that change is what prompted the creation of this
  PEP), so ``python`` should be used in the shebang line only for scripts
  that are source compatible with both Python 2 and 3.
* in preparation for an eventual change in the default version of Python,
  Python 2 only scripts should either be updated to be source compatible
  with Python 3 or else to use ``python2`` in the shebang line.


Recommendation
==============

* Unix-like software distributions (including systems like Mac OS X and
  Cygwin) should install the ``python2`` command into the default path
  whenever a version of the Python 2 interpreter is installed, and the same
  for ``python3`` and the Python 3 interpreter.
* When  invoked, ``python2`` should run some version of the Python 2
  interpreter, and ``python3`` should run some version of the Python 3
  interpreter.
* The more general ``python`` command should be installed whenever
  any version of Python 2 is installed and should invoke the same version of
  Python as the ``python2`` command (however, note that some distributions
  have already chosen to have ``python`` implement the ``python3``
  command; see the `Rationale`_ and `Migration Notes`_ below).
* The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should
  likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``,
  with the original commands invoking these versions by default, but possibly
  invoking the Python 3.x versions instead if configured to do so by the
  system administrator.
* In order to tolerate differences across platforms, all new code that needs
  to invoke the Python interpreter should not specify ``python``, but rather
  should specify either ``python2`` or ``python3`` (or the more specific
  ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_).
  This distinction should be made in shebangs, when invoking from a shell
  script, when invoking via the system() call, or when invoking in any other
  context.
* One exception to this is scripts that are deliberately written to be source
  compatible with both Python 2.x and 3.x. Such scripts may continue to use
  ``python`` on their shebang line without affecting their portability.
* When reinvoking the interpreter from a Python script, querying
  ``sys.executable`` to avoid hardcoded assumptions regarding the
  interpreter location remains the preferred approach.

These recommendations are the outcome of the relevant python-dev discussions
in March and July 2011 ([1]_, [2]_), February 2012 ([4]_) and
September 2014 ([6]_).


Rationale
=========

This recommendation is needed as, even though the majority of distributions
still alias the ``python`` command to Python 2, some now alias it to
Python 3 ([5]_). As some of the former distributions did not provide a
``python2`` command by default, there was previously no way for Python 2 code
(or any code that invokes the Python 2 interpreter directly rather than via
``sys.executable``) to reliably run on all Unix-like systems without
modification, as the ``python`` command would invoke the wrong interpreter
version on some systems, and the ``python2`` command would fail completely
on others. The recommendations in this PEP provide a very simple mechanism
to restore cross-platform support, with minimal additional work required
on the part of distribution maintainers.


Future Changes to this Recommendation
=====================================

It is anticipated that there will eventually come a time where the third
party ecosystem surrounding Python 3 is sufficiently mature for this
recommendation to be updated to suggest that the ``python`` symlink
refer to ``python3`` rather than ``python2``.

This recommendation will be periodically reviewed over the next few years,
and updated when the core development team judges it appropriate. As a
point of reference, regular maintenance releases for the Python 2.7 series
will continue until at least 2020.


Migration Notes
===============

This section does not contain any official recommendations from the core
CPython developers. It's merely a collection of notes regarding various
aspects of migrating to Python 3 as the default version of Python for a
system. They will hopefully be helpful to any distributions considering
making such a change.

* The main barrier to a distribution switching the ``python`` command from
  ``python2`` to ``python3`` isn't breakage within the distribution, but
  instead breakage of private third party scripts developed by sysadmins
  and other users. Updating the ``python`` command to invoke ``python3``
  by default indicates that a distribution is willing to break such scripts
  with errors that are potentially quite confusing for users that aren't
  yet familiar with the backwards incompatible changes in Python 3. For
  example, while the change of ``print`` from a statement to a builtin
  function is relatively simple for automated converters to handle, the
  SyntaxError from attempting to use the Python 2 notation in versions of
  Python 3 prior to 3.4.2 is thoroughly confusing if you aren't already
  aware of the change::

      $ python3 -c 'print "Hello, world!"'
        File "<string>", line 1
          print "Hello, world!"
                              ^
      SyntaxError: invalid syntax

  (In Python 3.4.2+, that generic error message has been replaced with the
  more explicit "SyntaxError: Missing parentheses in call to 'print'")
* Avoiding breakage of such third party scripts is the key reason this
  PEP recommends that ``python`` continue to refer to ``python2`` for the
  time being. Until the conventions described in this PEP are more widely
  adopted, having ``python`` invoke ``python2`` will remain the recommended
  option.
* The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on
  which they invoke specific minor versions of the Python interpreter. It
  can be useful for distribution-specific packages to take advantage of these
  utilities if they exist, since it will prevent code breakage if the default
  minor version of a given major version is changed. However, scripts
  intending to be cross-platform should not rely on the presence of these
  utilities, but rather should be tested on several recent minor versions of
  the target major version, compensating, if necessary, for the small
  differences that exist between minor versions. This prevents the need for
  sysadmins to install many very similar versions of the interpreter.
* When the ``pythonX.X`` binaries are provided by a distribution, the
  ``python2`` and ``python3`` commands should refer to one of those files
  rather than being provided as a separate binary file.
* It is suggested that even distribution-specific packages follow the
  ``python2``/``python3`` convention, even in code that is not intended to
  operate on other distributions. This will reduce problems if the
  distribution later decides to change the version of the Python interpreter
  that the ``python`` command invokes, or if a sysadmin installs a custom
  ``python`` command with a different major version than the distribution
  default. Distributions can test whether they are fully following this
  convention by changing the ``python`` interpreter on a test box and checking
  to see if anything breaks.
* If the above point is adhered to and sysadmins are permitted to change the
  ``python`` command, then the ``python`` command should always be implemented
  as a link to the interpreter binary (or a link to a link) and not vice
  versa. That way, if a sysadmin does decide to replace the installed
  ``python`` file, they can do so without inadvertently deleting the
  previously installed binary.
* If the Python 2 interpreter becomes uncommon, scripts should nevertheless
  continue to use the ``python3`` convention rather that just ``python``. This
  will ease transition in the event that yet another major version of Python
  is released.
* If these conventions are adhered to, it will become the case that the
  ``python`` command is only executed in an interactive manner as a user
  convenience, or to run scripts that are source compatible with both Python
  2 and Python 3.


Backwards Compatibility
=======================

A potential problem can arise if a script adhering to the
``python2``/``python3`` convention is executed on a system not supporting
these commands. This is mostly a non-issue, since the sysadmin can simply
create these symbolic links and avoid further problems. It is a significantly
more obvious breakage than the sometimes cryptic errors that can arise when
attempting to execute a script containing Python 2 specific syntax with a
Python 3 interpreter.


Application to the CPython Reference Interpreter
================================================

While technically a new feature, the ``make install`` and ``make bininstall``
command in the 2.7 version of CPython were adjusted to create the
following chains of symbolic links in the relevant ``bin`` directory (the
final item listed in the chain is the actual installed binary, preceding
items are relative symbolic links)::

    python -> python2 -> python2.7
    python-config -> python2-config -> python2.7-config

Similar adjustments were made to the Mac OS X binary installer.

This feature first appeared in the default installation process in
CPython 2.7.3.

The installation commands in the CPython 3.x series already create the
appropriate symlinks. For example, CPython 3.2 creates::

    python3 -> python3.2
    idle3 -> idle3.2
    pydoc3 -> pydoc3.2
    python3-config -> python3.2-config

And CPython 3.3 creates::

    python3 -> python3.3
    idle3 -> idle3.3
    pydoc3 -> pydoc3.3
    python3-config -> python3.3-config
    pysetup3 -> pysetup3.3

The implementation progress of these features in the default installers was
managed on the tracker as issue #12627 ([3]_).


Impact on PYTHON* Environment Variables
=======================================

The choice of target for the ``python`` command implicitly affects a
distribution's expected interpretation of the various Python related
environment variables. The use of ``*.pth`` files in the relevant
``site-packages`` folder, the "per-user site packages" feature (see
``python -m site``) or more flexible tools such as ``virtualenv`` are all more
tolerant of the presence of multiple versions of Python on a system than the
direct use of ``PYTHONPATH``.


Exclusion of MS Windows
=======================

This PEP deliberately excludes any proposals relating to Microsoft Windows, as
devising an equivalent solution for Windows was deemed too complex to handle
here. PEP 397 and the related discussion on the python-dev mailing list
address this issue (like this PEP, the PEP 397 launcher invokes Python 2 by
default if versions of both Python 2 and 3 are installed on the system).


References
==========

.. [1] Support the /usr/bin/python2 symlink upstream (with bonus grammar class!)
   (http://mail.python.org/pipermail/python-dev/2011-March/108491.html)

.. [2] Rebooting \PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
   (http://mail.python.org/pipermail/python-dev/2011-July/112322.html)

.. [3] Implement \PEP 394 in the CPython Makefile
   (http://bugs.python.org/issue12627)

.. [4] \PEP 394 request for pronouncement (python2 symlink in \*nix systems)
   (http://mail.python.org/pipermail/python-dev/2012-February/116435.html)

.. [5] Arch Linux announcement that their "python" link now refers Python 3
   (https://www.archlinux.org/news/python-is-now-python-3/)

.. [6] PEP 394 - Clarification of what "python" command should invoke
   (https://mail.python.org/pipermail/python-dev/2014-September/136374.html)

Copyright
===========
This document has been placed in the public domain.