python-peps/pep-0405.txt

PEP: 405
Title: Python Virtual Environments
Version: $Revision$
Last-Modified: $Date$
Author: Carl Meyer <carl@oddbird.net>
BDFL-Delegate: Nick Coghlan
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 13-Jun-2011
Python-Version: 3.3
Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012


Abstract
========

This PEP proposes to add to Python a mechanism for lightweight
"virtual environments" with their own site directories, optionally
isolated from system site directories.  Each virtual environment has
its own Python binary (allowing creation of environments with various
Python versions) and can have its own independent set of installed
Python packages in its site directories, but shares the standard
library with the base installed Python.


Motivation
==========

The utility of Python virtual environments has already been well
established by the popularity of existing third-party
virtual-environment tools, primarily Ian Bicking's `virtualenv`_.
Virtual environments are already widely used for dependency management
and isolation, ease of installing and using Python packages without
system-administrator access, and automated testing of Python software
across multiple Python versions, among other uses.

Existing virtual environment tools suffer from lack of support from
the behavior of Python itself.  Tools such as `rvirtualenv`_, which do
not copy the Python binary into the virtual environment, cannot
provide reliable isolation from system site directories.  Virtualenv,
which does copy the Python binary, is forced to duplicate much of
Python's ``site`` module and manually symlink/copy an ever-changing
set of standard-library modules into the virtual environment in order
to perform a delicate boot-strapping dance at every startup.
(Virtualenv must copy the binary in order to provide isolation, as
Python dereferences a symlinked executable before searching for
``sys.prefix``.)

The ``PYTHONHOME`` environment variable, Python's only existing
built-in solution for virtual environments, requires
copying/symlinking the entire standard library into every environment.
Copying the whole standard library is not a lightweight solution, and
cross-platform support for symlinks remains inconsistent (even on
Windows platforms that do support them, creating them often requires
administrator privileges).

A virtual environment mechanism integrated with Python and drawing on
years of experience with existing third-party tools can be lower
maintenance, more reliable, and more easily available to all Python
users.

.. _virtualenv: http://www.virtualenv.org

.. _rvirtualenv: https://github.com/kvbik/rvirtualenv


Specification
=============

When the Python binary is executed, it attempts to determine its
prefix (which it stores in ``sys.prefix``), which is then used to find
the standard library and other key files, and by the ``site`` module
to determine the location of the site-package directories.  Currently
the prefix is found (assuming ``PYTHONHOME`` is not set) by first
walking up the filesystem tree looking for a marker file (``os.py``)
that signifies the presence of the standard library, and if none is
found, falling back to the build-time prefix hardcoded in the binary.

This PEP proposes to add a new first step to this search.  If a
``pyvenv.cfg`` file is found either adjacent to the Python executable,
or one directory above it, this file is scanned for lines of the form
``key = value``.  If a ``home`` key is found, this signifies that the
Python binary belongs to a virtual environment, and the value of the
``home`` key is the directory containing the Python executable used to
create this virtual environment.

In this case, prefix-finding continues as normal using the value of
the ``home`` key as the effective Python binary location, which finds
the prefix of the base installation.  ``sys.base_prefix`` is set to
this value, while ``sys.prefix`` is set to the directory containing
``pyvenv.cfg``.

(If ``pyvenv.cfg`` is not found or does not contain the ``home`` key,
prefix-finding continues normally, and ``sys.prefix`` will be equal to
``sys.base_prefix``.)

The ``site`` and ``sysconfig`` standard-library modules are modified
such that the standard library and header files are are found relative
to ``sys.base_prefix``, while site-package directories ("purelib" and
"platlib", in ``sysconfig`` terms) are still found relative to
``sys.prefix``.

(Also, ``sys.base_exec_prefix`` is added, and handled similarly with
regard to ``sys.exec_prefix``.)

Thus, a Python virtual environment in its simplest form would consist
of nothing more than a copy or symlink of the Python binary
accompanied by a ``pyvenv.cfg`` file and a site-packages directory.


Isolation from system site-packages
-----------------------------------

By default, a virtual environment is entirely isolated from the
system-level site-packages directories.

If the ``pyvenv.cfg`` file also contains a key
``include-system-site-packages`` with a value of ``true`` (not case
sensitive), the ``site`` module will also add the system site
directories to ``sys.path`` after the virtual environment site
directories.  Thus system-installed packages will still be importable,
but a package of the same name installed in the virtual environment
will take precedence.

:pep:`370` user-level site-packages are considered part of the system
site-packages for venv purposes: they are not available from an
isolated venv, but are available from an
``include-system-site-packages = true`` venv.


Creating virtual environments
-----------------------------

This PEP also proposes adding a new ``venv`` module to the standard
library which implements the creation of virtual environments.  This
module can be executed using the ``-m`` flag::

    python3 -m venv /path/to/new/virtual/environment

A ``pyvenv`` installed script is also provided to make this more
convenient::

    pyvenv /path/to/new/virtual/environment

Running this command creates the target directory (creating any parent
directories that don't exist already) and places a ``pyvenv.cfg`` file
in it with a ``home`` key pointing to the Python installation the
command was run from.  It also creates a ``bin/`` (or ``Scripts`` on
Windows) subdirectory containing a copy (or symlink) of the
``python3`` executable, and the ``pysetup3`` script from the
``packaging`` standard library module (to facilitate easy installation
of packages from PyPI into the new virtualenv).  And it creates an
(initially empty) ``lib/pythonX.Y/site-packages`` (or
``Lib\site-packages`` on Windows) subdirectory.

If the target directory already exists an error will be raised, unless
the ``--clear`` option was provided, in which case the target
directory will be deleted and virtual environment creation will
proceed as usual.

The created ``pyvenv.cfg`` file also includes the
``include-system-site-packages`` key, set to ``true`` if ``venv`` is
run with the ``--system-site-packages`` option, ``false`` by default.

Multiple paths can be given to ``pyvenv``, in which case an identical
virtualenv will be created, according to the given options, at each
provided path.

The ``venv`` module also provides "shell activation scripts" for POSIX
and Windows systems which simply add the virtual environment's ``bin``
(or ``Scripts``) directory to the front of the user's shell PATH.
This is not strictly necessary for use of a virtual environment (as an
explicit path to the venv's python binary or scripts can just as well
be used), but it is convenient.

In order to allow ``pysetup`` and other Python package managers to
install packages into the virtual environment the same way they would
install into a normal Python installation, and avoid special-casing
virtual environments in ``sysconfig`` beyond using ``sys.site_prefix``
in place of ``sys.prefix``, the internal virtual environment layout
mimics the layout of the Python installation itself on each platform.
So a typical virtual environment layout on a POSIX system would be::

    pyvenv.cfg
    bin/python3
    bin/python
    bin/pysetup3
    include/
    lib/python3.3/site-packages/

While on a Windows system::

    pyvenv.cfg
    Scripts/python.exe
    Scripts/python3.dll
    Scripts/pysetup3.exe
    Scripts/pysetup3-script.py
            ... other DLLs and pyds...
    Include/
    Lib/site-packages/

Third-party packages installed into the virtual environment will have
their Python modules placed in the ``site-packages`` directory, and
their executables placed in ``bin/`` or ``Scripts``.

.. note::

    On a normal Windows system-level installation, the Python binary
    itself wouldn't go inside the "Scripts/" subdirectory, as it does
    in the default venv layout.  This is useful in a virtual
    environment so that a user only has to add a single directory to
    their shell PATH in order to effectively "activate" the virtual
    environment.

.. note::

    On Windows, it is necessary to also copy or symlink DLLs and pyd
    files from compiled stdlib modules into the env, because if the
    venv is created from a non-system-wide Python installation,
    Windows won't be able to find the Python installation's copies of
    those files when Python is run from the venv.


Sysconfig install schemes and user-site
---------------------------------------

This approach explicitly chooses not to introduce a new sysconfig
install scheme for venvs. Rather, by modifying ``sys.prefix`` we
ensure that existing install schemes which base locations on
``sys.prefix`` will simply work in a venv. Installation to other
install schemes (for instance, the user-site schemes) whose paths are
not relative to ``sys.prefix``, will not be affected by a venv at all.

It may be feasible to create an alternative implementation of Python
virtual environments based on a virtual-specific sysconfig scheme, but
it would be less robust, as it would require more code to be aware of
whether it is operating within a virtual environment or not.


Copies versus symlinks
----------------------

The technique in this PEP works equally well in general with a copied
or symlinked Python binary (and other needed DLLs on Windows).
Symlinking is preferable where possible, because in the case of an
upgrade to the underlying Python installation, a Python executable
copied in a venv might become out-of-sync with the installed standard
library and require manual upgrade.

There are some cross-platform difficulties with symlinks:

* Not all Windows versions support symlinks, and even on those that
  do, creating them often requires administrator privileges.

* On OS X framework builds of Python, sys.executable is just a stub
  that executes the real Python binary.  Symlinking this stub does not
  work; it must be copied.  (Fortunately the stub is also small, and
  not changed by bugfix upgrades to Python, so copying it is not an
  issue).

Thus, this PEP proposes to symlink the binary on all platforms except
for Windows, and OS X framework builds. A ``--symlink`` option is
available to force the use of symlinks on Windows versions that
support them, if the appropriate permissions are available. (This
option has no effect on OS X framework builds, since symlinking can
never work there, and has no advantages).

On Windows, if ``--symlink`` is not used, this means that if the
underlying Python installation is upgraded, the Python binary and DLLs
in the venv should be updated, or there could be issues of mismatch
with the upgraded standard library. The pyvenv script accepts a
``--upgrade`` option for easily performing this upgrade on an existing
venv.


Include files
-------------

Current virtualenv handles include files in this way:

On POSIX systems where the installed Python's include files are found
in ``${base_prefix}/include/pythonX.X``, virtualenv creates
``${venv}/include/`` and symlink ``${base_prefix}/include/pythonX.X``
to ``${venv}/include/pythonX.X``. On Windows, where Python's include
files are found in ``{{ sys.prefix }}/Include`` and symlinks are not
reliably available, virtualenv copies ``{{ sys.prefix }}/Include`` to
``${venv}/Include``. This ensures that extension modules built and
installed within the virtualenv will always find the Python header
files they need in the expected location relative to ``sys.prefix``.

This solution is not ideal when an extension module installs its own
header files, as the default installation location for those header
files may be a symlink to a system directory that may not be
writable. One installer, pip, explicitly works around this by
installing header files to a nonstandard location
``${venv}/include/site/pythonX.X/``, as in Python there's currently no
standard abstraction for a site-specific include directory.

This PEP proposes a slightly different approach, though one with
essentially the same effect and the same set of advantages and
disadvantages. Rather than symlinking or copying include files into
the venv, we simply modify the sysconfig schemes so that header files
are always looked for relative to ``base_prefix`` rather than
``prefix``. (We also create an ``include/`` directory within the venv

Better handling of include files in distutils/packaging and, by
extension, pyvenv, is an area that may deserve its own future PEP. For
now, we propose that the behavior of virtualenv has thus far proved
itself to be at least "good enough" in practice.

[Open question: should pyvenv instead simply copy the behavior of
virtualenv entirely, to avoid introducing unexpected new issues here?]


API
---

The high-level method described above makes use of a simple API which
provides mechanisms for third-party virtual environment creators to
customize environment creation according to their needs.

The ``venv`` module contains an ``EnvBuilder`` class which accepts the
following keyword arguments on instantiation:

* ``system_site_packages`` - A Boolean value indicating that the
  system Python site-packages should be available to the environment.
  Defaults to ``False``.

* ``clear`` - A Boolean value which, if true, will delete any existing
  target directory instead of raising an exception.  Defaults to
  ``False``.

* ``symlinks`` - A Boolean value indicating whether to attempt to
  symlink the Python binary (and any necessary DLLs or other binaries,
  e.g. ``pythonw.exe``), rather than copying.  Defaults to ``False``.

The instantiated env-builder has a ``create`` method, which takes as
required argument the path (absolute or relative to the current
directory) of the target directory which is to contain the virtual
environment.  The ``create`` method either creates the environment in
the specified directory, or raises an appropriate exception.

The ``venv`` module also provides a module-level ``create`` function
as a convenience::

    def create(env_dir,
               system_site_packages=False, clear=False, use_symlinks=False):
        builder = EnvBuilder(
            system_site_packages=system_site_packages,
            clear=clear,
            use_symlinks=use_symlinks)
        builder.create(env_dir)

Creators of third-party virtual environment tools are free to use the
provided ``EnvBuilder`` class as a base class.

The ``create`` method of the ``EnvBuilder`` class illustrates the
hooks available for customization::

    def create(self, env_dir):
        """
        Create a virtualized Python environment in a directory.

        :param env_dir: The target directory to create an environment in.

        """
        env_dir = os.path.abspath(env_dir)
        context = self.create_directories(env_dir)
        self.create_configuration(context)
        self.setup_python(context)
        self.post_setup(context)

Each of the methods ``create_directories``, ``create_configuration``,
``setup_python``, and ``post_setup`` can be overridden.  The functions
of these methods are:

* ``create_directories`` - creates the environment directory and all
  necessary directories, and returns a context object.  This is just a
  holder for attributes (such as paths), for use by the other methods.

* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration
  file in the environment.

* ``setup_python`` - creates a copy of the Python executable (and,
  under Windows, DLLs) in the environment.

* ``post_setup`` - A (no-op by default) hook method which can be
  overridden in third party subclasses to pre-install packages or
  install scripts in the virtual environment.

In addition, ``EnvBuilder`` provides a utility method that can be
called from ``post_setup`` in subclasses to assist in installing
custom scripts into the virtual environment.  The method
``install_scripts`` accepts as arguments the ``context`` object (see
above) and a path to a directory.  The directory should contain
subdirectories "common", "posix", "nt", each containing scripts
destined for the bin directory in the environment.  The contents of
"common" and the directory corresponding to ``os.name`` are copied
after doing some text replacement of placeholders:

* ``__VENV_DIR__`` is replaced with absolute path of the environment
  directory.

* ``__VENV_NAME__`` is replaced with the environment name (final path
  segment of environment directory).

* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
  (either ``bin`` or ``Scripts``).

* ``__VENV_PYTHON__`` is replaced with the absolute path of the
  environment's executable.

The ``DistributeEnvBuilder`` subclass in the reference implementation
illustrates how the customization hook can be used in practice to
pre-install Distribute into the virtual environment.  It's not
envisaged that ``DistributeEnvBuilder`` will be actually added to
Python core, but it makes the reference implementation more
immediately useful for testing and exploratory purposes.


Backwards Compatibility
=======================

Splitting the meanings of ``sys.prefix``
----------------------------------------

Any virtual environment tool along these lines (which attempts to
isolate site-packages, while still making use of the base Python's
standard library with no need for it to be symlinked into the virtual
environment) is proposing a split between two different meanings
(among others) that are currently both wrapped up in ``sys.prefix``:
the answers to the questions "Where is the standard library?" and
"Where is the site-packages location where third-party modules should
be installed?"

This split could be handled by introducing a new ``sys`` attribute for
either the former prefix or the latter prefix.  Either option
potentially introduces some backwards-incompatibility with software
written to assume the other meaning for ``sys.prefix``.  (Such
software should preferably be using the APIs in the ``site`` and
``sysconfig`` modules to answer these questions rather than using
``sys.prefix`` directly, in which case there is no
backwards-compatibility issue, but in practice ``sys.prefix`` is
sometimes used.)

The `documentation`__ for ``sys.prefix`` describes it as "A string
giving the site-specific directory prefix where the platform
independent Python files are installed," and specifically mentions the
standard library and header files as found under ``sys.prefix``.  It
does not mention ``site-packages``.

__ http://docs.python.org/dev/library/sys.html#sys.prefix

Maintaining this documented definition would mean leaving
``sys.prefix`` pointing to the base system installation (which is
where the standard library and header files are found), and
introducing a new value in ``sys`` (something like
``sys.site_prefix``) to point to the prefix for ``site-packages``.
This would maintain the documented semantics of ``sys.prefix``, but
risk breaking isolation if third-party code uses ``sys.prefix`` rather
than ``sys.site_prefix`` or the appropriate ``site`` API to find
site-packages directories.

The most notable case is probably `setuptools`_ and its fork
`distribute`_, which mostly use ``distutils``and ``sysconfig`` APIs,
but do use ``sys.prefix`` directly to build up a list of site
directories for pre-flight checking where ``pth`` files can usefully
be placed.

Otherwise, a `Google Code Search`_ turns up what appears to be a
roughly even mix of usage between packages using ``sys.prefix`` to
build up a site-packages path and packages using it to e.g. eliminate
the standard-library from code-execution tracing.

Although it requires modifying the documented definition of
``sys.prefix``, this PEP prefers to have ``sys.prefix`` point to the
virtual environment (where ``site-packages`` is found), and introduce
``sys.base_prefix`` to point to the standard library and Python header
files. Rationale for this choice:

* It is preferable to err on the side of greater isolation of the
  virtual environment.

* Virtualenv already modifies ``sys.prefix`` to point at the virtual
  environment, and in practice this has not been a problem.

* No modification is required to setuptools/distribute.

.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
.. _distribute: http://packages.python.org/distribute/
.. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs


Impact on other Python implementations
--------------------------------------

The majority of this PEP's changes occur in the standard library, which is
shared by other Python implementations and should not present any
problem.

Other Python implementations will need to replicate the new
``sys.prefix``-finding behavior of the interpreter bootstrap, including
locating and parsing the ``pyvenv.cfg`` file, if it is present.


Reference Implementation
========================

The reference implementation is found in `a clone of the CPython
Mercurial repository`_.  To test it, build and install it (the virtual
environment tool currently does not run from a source tree).  From the
installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create
a virtual environment.

.. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: