545 lines
22 KiB
Plaintext
545 lines
22 KiB
Plaintext
PEP: 405
|
||
Title: Python Virtual Environments
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Carl Meyer <carl@oddbird.net>
|
||
BDFL-Delegate: Nick Coghlan
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 13-Jun-2011
|
||
Python-Version: 3.3
|
||
Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
This PEP proposes to add to Python a mechanism for lightweight
|
||
"virtual environments" with their own site directories, optionally
|
||
isolated from system site directories. Each virtual environment has
|
||
its own Python binary (allowing creation of environments with various
|
||
Python versions) and can have its own independent set of installed
|
||
Python packages in its site directories, but shares the standard
|
||
library with the base installed Python.
|
||
|
||
|
||
Motivation
|
||
==========
|
||
|
||
The utility of Python virtual environments has already been well
|
||
established by the popularity of existing third-party
|
||
virtual-environment tools, primarily Ian Bicking's `virtualenv`_.
|
||
Virtual environments are already widely used for dependency management
|
||
and isolation, ease of installing and using Python packages without
|
||
system-administrator access, and automated testing of Python software
|
||
across multiple Python versions, among other uses.
|
||
|
||
Existing virtual environment tools suffer from lack of support from
|
||
the behavior of Python itself. Tools such as `rvirtualenv`_, which do
|
||
not copy the Python binary into the virtual environment, cannot
|
||
provide reliable isolation from system site directories. Virtualenv,
|
||
which does copy the Python binary, is forced to duplicate much of
|
||
Python's ``site`` module and manually symlink/copy an ever-changing
|
||
set of standard-library modules into the virtual environment in order
|
||
to perform a delicate boot-strapping dance at every startup.
|
||
(Virtualenv must copy the binary in order to provide isolation, as
|
||
Python dereferences a symlinked executable before searching for
|
||
``sys.prefix``.)
|
||
|
||
The ``PYTHONHOME`` environment variable, Python's only existing
|
||
built-in solution for virtual environments, requires
|
||
copying/symlinking the entire standard library into every environment.
|
||
Copying the whole standard library is not a lightweight solution, and
|
||
cross-platform support for symlinks remains inconsistent (even on
|
||
Windows platforms that do support them, creating them often requires
|
||
administrator privileges).
|
||
|
||
A virtual environment mechanism integrated with Python and drawing on
|
||
years of experience with existing third-party tools can be lower
|
||
maintenance, more reliable, and more easily available to all Python
|
||
users.
|
||
|
||
.. _virtualenv: http://www.virtualenv.org
|
||
|
||
.. _rvirtualenv: https://github.com/kvbik/rvirtualenv
|
||
|
||
|
||
Specification
|
||
=============
|
||
|
||
When the Python binary is executed, it attempts to determine its
|
||
prefix (which it stores in ``sys.prefix``), which is then used to find
|
||
the standard library and other key files, and by the ``site`` module
|
||
to determine the location of the site-package directories. Currently
|
||
the prefix is found (assuming ``PYTHONHOME`` is not set) by first
|
||
walking up the filesystem tree looking for a marker file (``os.py``)
|
||
that signifies the presence of the standard library, and if none is
|
||
found, falling back to the build-time prefix hardcoded in the binary.
|
||
|
||
This PEP proposes to add a new first step to this search. If a
|
||
``pyvenv.cfg`` file is found either adjacent to the Python executable,
|
||
or one directory above it, this file is scanned for lines of the form
|
||
``key = value``. If a ``home`` key is found, this signifies that the
|
||
Python binary belongs to a virtual environment, and the value of the
|
||
``home`` key is the directory containing the Python executable used to
|
||
create this virtual environment.
|
||
|
||
In this case, prefix-finding continues as normal using the value of
|
||
the ``home`` key as the effective Python binary location, which finds
|
||
the prefix of the base installation. ``sys.base_prefix`` is set to
|
||
this value, while ``sys.prefix`` is set to the directory containing
|
||
``pyvenv.cfg``.
|
||
|
||
(If ``pyvenv.cfg`` is not found or does not contain the ``home`` key,
|
||
prefix-finding continues normally, and ``sys.prefix`` will be equal to
|
||
``sys.base_prefix``.)
|
||
|
||
The ``site`` and ``sysconfig`` standard-library modules are modified
|
||
such that the standard library and header files are are found relative
|
||
to ``sys.base_prefix``, while site-package directories ("purelib" and
|
||
"platlib", in ``sysconfig`` terms) are still found relative to
|
||
``sys.prefix``.
|
||
|
||
(Also, ``sys.base_exec_prefix`` is added, and handled similarly with
|
||
regard to ``sys.exec_prefix``.)
|
||
|
||
Thus, a Python virtual environment in its simplest form would consist
|
||
of nothing more than a copy or symlink of the Python binary
|
||
accompanied by a ``pyvenv.cfg`` file and a site-packages directory.
|
||
|
||
|
||
Isolation from system site-packages
|
||
-----------------------------------
|
||
|
||
By default, a virtual environment is entirely isolated from the
|
||
system-level site-packages directories.
|
||
|
||
If the ``pyvenv.cfg`` file also contains a key
|
||
``include-system-site-packages`` with a value of ``true`` (not case
|
||
sensitive), the ``site`` module will also add the system site
|
||
directories to ``sys.path`` after the virtual environment site
|
||
directories. Thus system-installed packages will still be importable,
|
||
but a package of the same name installed in the virtual environment
|
||
will take precedence.
|
||
|
||
:pep:`370` user-level site-packages are considered part of the system
|
||
site-packages for venv purposes: they are not available from an
|
||
isolated venv, but are available from an
|
||
``include-system-site-packages = true`` venv.
|
||
|
||
|
||
Creating virtual environments
|
||
-----------------------------
|
||
|
||
This PEP also proposes adding a new ``venv`` module to the standard
|
||
library which implements the creation of virtual environments. This
|
||
module can be executed using the ``-m`` flag::
|
||
|
||
python3 -m venv /path/to/new/virtual/environment
|
||
|
||
A ``pyvenv`` installed script is also provided to make this more
|
||
convenient::
|
||
|
||
pyvenv /path/to/new/virtual/environment
|
||
|
||
Running this command creates the target directory (creating any parent
|
||
directories that don't exist already) and places a ``pyvenv.cfg`` file
|
||
in it with a ``home`` key pointing to the Python installation the
|
||
command was run from. It also creates a ``bin/`` (or ``Scripts`` on
|
||
Windows) subdirectory containing a copy (or symlink) of the
|
||
``python3`` executable, and the ``pysetup3`` script from the
|
||
``packaging`` standard library module (to facilitate easy installation
|
||
of packages from PyPI into the new virtualenv). And it creates an
|
||
(initially empty) ``lib/pythonX.Y/site-packages`` (or
|
||
``Lib\site-packages`` on Windows) subdirectory.
|
||
|
||
If the target directory already exists an error will be raised, unless
|
||
the ``--clear`` option was provided, in which case the target
|
||
directory will be deleted and virtual environment creation will
|
||
proceed as usual.
|
||
|
||
The created ``pyvenv.cfg`` file also includes the
|
||
``include-system-site-packages`` key, set to ``true`` if ``venv`` is
|
||
run with the ``--system-site-packages`` option, ``false`` by default.
|
||
|
||
Multiple paths can be given to ``pyvenv``, in which case an identical
|
||
virtualenv will be created, according to the given options, at each
|
||
provided path.
|
||
|
||
The ``venv`` module also provides "shell activation scripts" for POSIX
|
||
and Windows systems which simply add the virtual environment's ``bin``
|
||
(or ``Scripts``) directory to the front of the user's shell PATH.
|
||
This is not strictly necessary for use of a virtual environment (as an
|
||
explicit path to the venv's python binary or scripts can just as well
|
||
be used), but it is convenient.
|
||
|
||
In order to allow ``pysetup`` and other Python package managers to
|
||
install packages into the virtual environment the same way they would
|
||
install into a normal Python installation, and avoid special-casing
|
||
virtual environments in ``sysconfig`` beyond using ``sys.site_prefix``
|
||
in place of ``sys.prefix``, the internal virtual environment layout
|
||
mimics the layout of the Python installation itself on each platform.
|
||
So a typical virtual environment layout on a POSIX system would be::
|
||
|
||
pyvenv.cfg
|
||
bin/python3
|
||
bin/python
|
||
bin/pysetup3
|
||
include/
|
||
lib/python3.3/site-packages/
|
||
|
||
While on a Windows system::
|
||
|
||
pyvenv.cfg
|
||
Scripts/python.exe
|
||
Scripts/python3.dll
|
||
Scripts/pysetup3.exe
|
||
Scripts/pysetup3-script.py
|
||
... other DLLs and pyds...
|
||
Include/
|
||
Lib/site-packages/
|
||
|
||
Third-party packages installed into the virtual environment will have
|
||
their Python modules placed in the ``site-packages`` directory, and
|
||
their executables placed in ``bin/`` or ``Scripts``.
|
||
|
||
.. note::
|
||
|
||
On a normal Windows system-level installation, the Python binary
|
||
itself wouldn't go inside the "Scripts/" subdirectory, as it does
|
||
in the default venv layout. This is useful in a virtual
|
||
environment so that a user only has to add a single directory to
|
||
their shell PATH in order to effectively "activate" the virtual
|
||
environment.
|
||
|
||
.. note::
|
||
|
||
On Windows, it is necessary to also copy or symlink DLLs and pyd
|
||
files from compiled stdlib modules into the env, because if the
|
||
venv is created from a non-system-wide Python installation,
|
||
Windows won't be able to find the Python installation's copies of
|
||
those files when Python is run from the venv.
|
||
|
||
|
||
Copies versus symlinks
|
||
----------------------
|
||
|
||
The technique in this PEP works equally well in general with a copied
|
||
or symlinked Python binary (and other needed DLLs on Windows). Some
|
||
users prefer a copied binary (for greater isolation from system
|
||
changes) and some prefer a symlinked one (so that e.g. security
|
||
updates automatically propagate to virtual environments).
|
||
|
||
There are some cross-platform difficulties with symlinks:
|
||
|
||
* Not all Windows versions support symlinks, and even on those that
|
||
do, creating them often requires administrator privileges.
|
||
|
||
* On OSX framework builds of Python, sys.executable is just a stub
|
||
that executes the real Python binary. Symlinking this stub does not
|
||
work with the implementation in this PEP; it must be copied.
|
||
(Fortunately the stub is also small, so copying it is not an issue).
|
||
|
||
Because of these issues, this PEP proposes to copy the Python binary
|
||
by default, to maintain cross-platform consistency in the default
|
||
behavior.
|
||
|
||
The ``pyvenv`` script accepts a ``--symlink`` option. If this option
|
||
is provided, the script will attempt to symlink instead of copy. If a
|
||
symlink fails (e.g. because they are not supported by the platform, or
|
||
additional privileges are needed), the script will warn the user and
|
||
fall back to a copy.
|
||
|
||
On OSX framework builds, where a symlink of the executable would
|
||
succeed but create a non-functional virtual environment, the script
|
||
will fail with an error message that symlinking is not supported on
|
||
OSX framework builds.
|
||
|
||
|
||
API
|
||
---
|
||
|
||
The high-level method described above makes use of a simple API which
|
||
provides mechanisms for third-party virtual environment creators to
|
||
customize environment creation according to their needs.
|
||
|
||
The ``venv`` module contains an ``EnvBuilder`` class which accepts the
|
||
following keyword arguments on instantiation:
|
||
|
||
* ``system_site_packages`` - A Boolean value indicating that the
|
||
system Python site-packages should be available to the environment.
|
||
Defaults to ``False``.
|
||
|
||
* ``clear`` - A Boolean value which, if true, will delete any existing
|
||
target directory instead of raising an exception. Defaults to
|
||
``False``.
|
||
|
||
* ``symlinks`` - A Boolean value indicating whether to attempt to
|
||
symlink the Python binary (and any necessary DLLs or other binaries,
|
||
e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``.
|
||
|
||
The instantiated env-builder has a ``create`` method, which takes as
|
||
required argument the path (absolute or relative to the current
|
||
directory) of the target directory which is to contain the virtual
|
||
environment. The ``create`` method either creates the environment in
|
||
the specified directory, or raises an appropriate exception.
|
||
|
||
The ``venv`` module also provides a module-level ``create`` function
|
||
as a convenience::
|
||
|
||
def create(env_dir,
|
||
system_site_packages=False, clear=False, use_symlinks=False):
|
||
builder = EnvBuilder(
|
||
system_site_packages=system_site_packages,
|
||
clear=clear,
|
||
use_symlinks=use_symlinks)
|
||
builder.create(env_dir)
|
||
|
||
Creators of third-party virtual environment tools are free to use the
|
||
provided ``EnvBuilder`` class as a base class.
|
||
|
||
The ``create`` method of the ``EnvBuilder`` class illustrates the
|
||
hooks available for customization::
|
||
|
||
def create(self, env_dir):
|
||
"""
|
||
Create a virtualized Python environment in a directory.
|
||
|
||
:param env_dir: The target directory to create an environment in.
|
||
|
||
"""
|
||
env_dir = os.path.abspath(env_dir)
|
||
context = self.create_directories(env_dir)
|
||
self.create_configuration(context)
|
||
self.setup_python(context)
|
||
self.post_setup(context)
|
||
|
||
Each of the methods ``create_directories``, ``create_configuration``,
|
||
``setup_python``, and ``post_setup`` can be overridden. The functions
|
||
of these methods are:
|
||
|
||
* ``create_directories`` - creates the environment directory and all
|
||
necessary directories, and returns a context object. This is just a
|
||
holder for attributes (such as paths), for use by the other methods.
|
||
|
||
* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration
|
||
file in the environment.
|
||
|
||
* ``setup_python`` - creates a copy of the Python executable (and,
|
||
under Windows, DLLs) in the environment.
|
||
|
||
* ``post_setup`` - A (no-op by default) hook method which can be
|
||
overridden in third party subclasses to pre-install packages or
|
||
install scripts in the virtual environment.
|
||
|
||
In addition, ``EnvBuilder`` provides a utility method that can be
|
||
called from ``post_setup`` in subclasses to assist in installing
|
||
custom scripts into the virtual environment. The method
|
||
``install_scripts`` accepts as arguments the ``context`` object (see
|
||
above) and a path to a directory. The directory should contain
|
||
subdirectories "common", "posix", "nt", each containing scripts
|
||
destined for the bin directory in the environment. The contents of
|
||
"common" and the directory corresponding to ``os.name`` are copied
|
||
after doing some text replacement of placeholders:
|
||
|
||
* ``__VENV_DIR__`` is replaced with absolute path of the environment
|
||
directory.
|
||
|
||
* ``__VENV_NAME__`` is replaced with the environment name (final path
|
||
segment of environment directory).
|
||
|
||
* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
|
||
(either ``bin`` or ``Scripts``).
|
||
|
||
* ``__VENV_PYTHON__`` is replaced with the absolute path of the
|
||
environment's executable.
|
||
|
||
The ``DistributeEnvBuilder`` subclass in the reference implementation
|
||
illustrates how the customization hook can be used in practice to
|
||
pre-install Distribute into the virtual environment. It's not
|
||
envisaged that ``DistributeEnvBuilder`` will be actually added to
|
||
Python core, but it makes the reference implementation more
|
||
immediately useful for testing and exploratory purposes.
|
||
|
||
|
||
Backwards Compatibility
|
||
=======================
|
||
|
||
Splitting the meanings of ``sys.prefix``
|
||
----------------------------------------
|
||
|
||
Any virtual environment tool along these lines (which attempts to
|
||
isolate site-packages, while still making use of the base Python's
|
||
standard library with no need for it to be symlinked into the virtual
|
||
environment) is proposing a split between two different meanings
|
||
(among others) that are currently both wrapped up in ``sys.prefix``:
|
||
the answers to the questions "Where is the standard library?" and
|
||
"Where is the site-packages location where third-party modules should
|
||
be installed?"
|
||
|
||
This split could be handled by introducing a new ``sys`` attribute for
|
||
either the former prefix or the latter prefix. Either option
|
||
potentially introduces some backwards-incompatibility with software
|
||
written to assume the other meaning for ``sys.prefix``. (Such
|
||
software should preferably be using the APIs in the ``site`` and
|
||
``sysconfig`` modules to answer these questions rather than using
|
||
``sys.prefix`` directly, in which case there is no
|
||
backwards-compatibility issue, but in practice ``sys.prefix`` is
|
||
sometimes used.)
|
||
|
||
The `documentation`__ for ``sys.prefix`` describes it as "A string
|
||
giving the site-specific directory prefix where the platform
|
||
independent Python files are installed," and specifically mentions the
|
||
standard library and header files as found under ``sys.prefix``. It
|
||
does not mention ``site-packages``.
|
||
|
||
__ http://docs.python.org/dev/library/sys.html#sys.prefix
|
||
|
||
Maintaining this documented definition would mean leaving
|
||
``sys.prefix`` pointing to the base system installation (which is
|
||
where the standard library and header files are found), and
|
||
introducing a new value in ``sys`` (something like
|
||
``sys.site_prefix``) to point to the prefix for ``site-packages``.
|
||
This would maintain the documented semantics of ``sys.prefix``, but
|
||
risk breaking isolation if third-party code uses ``sys.prefix`` rather
|
||
than ``sys.site_prefix`` or the appropriate ``site`` API to find
|
||
site-packages directories.
|
||
|
||
The most notable case is probably `setuptools`_ and its fork
|
||
`distribute`_, which mostly use ``distutils``and ``sysconfig`` APIs,
|
||
but do use ``sys.prefix`` directly to build up a list of site
|
||
directories for pre-flight checking where ``pth`` files can usefully
|
||
be placed.
|
||
|
||
Otherwise, a `Google Code Search`_ turns up what appears to be a
|
||
roughly even mix of usage between packages using ``sys.prefix`` to
|
||
build up a site-packages path and packages using it to e.g. eliminate
|
||
the standard-library from code-execution tracing.
|
||
|
||
Although it requires modifying the documented definition of
|
||
``sys.prefix``, this PEP prefers to have ``sys.prefix`` point to the
|
||
virtual environment (where ``site-packages`` is found), and introduce
|
||
``sys.base_prefix`` to point to the standard library and Python header
|
||
files. Rationale for this choice:
|
||
|
||
* It is preferable to err on the side of greater isolation of the
|
||
virtual environment.
|
||
|
||
* Virtualenv already modifies ``sys.prefix`` to point at the virtual
|
||
environment, and in practice this has not been a problem.
|
||
|
||
* No modification is required to setuptools/distribute.
|
||
|
||
.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
|
||
.. _distribute: http://packages.python.org/distribute/
|
||
.. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs
|
||
|
||
|
||
Open Questions
|
||
==============
|
||
|
||
What about include files?
|
||
-------------------------
|
||
|
||
For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in
|
||
``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default
|
||
in ``$VE/include/pythonX.Y``. With virtualenv, everything works
|
||
because the PythonX.Y include is symlinked, so everything that's
|
||
needed is in ``$VE/include``. At the moment the reference
|
||
implementation doesn't do anything with include files, besides
|
||
creating the include directory; this might need to change, to
|
||
copy/symlink ``$VE/include/pythonX.Y``.
|
||
|
||
As in Python there's no abstraction for a site-specific include
|
||
directory, other than for platform-specific stuff, then the user
|
||
expectation would seem to be that all include files anyone could ever
|
||
want should be found in one of just two locations, with sysconfig
|
||
labels "include" & "platinclude".
|
||
|
||
There's another issue: what if includes are Python-version-specific?
|
||
For example, SIP installs by default into ``$VE/include/pythonX.Y``
|
||
rather than ``$VE/include``, presumably because there's
|
||
version-specific stuff in there - but even if that's not the case with
|
||
SIP, it could be the case with some other package. And the problem
|
||
that gives is that you can't just symlink the ``include/pythonX.Y``
|
||
directory, but actually have to provide a writable directory and
|
||
symlink/copy the contents from the system ``include/pythonX.Y``. Of
|
||
course this is not hard to do, but it does seem inelegant. OTOH it's
|
||
really because there's no supporting concept in ``Python/sysconfig``.
|
||
|
||
|
||
Testability and Source Build Issues
|
||
-----------------------------------
|
||
|
||
Currently in the reference implementation, virtual environments must
|
||
be created with an installed Python, rather than a source build, as
|
||
the base installation. In order to be able to fully test the ``venv``
|
||
module in the Python regression test suite, some anomalies in how
|
||
sysconfig data is configured in source builds will need to be removed.
|
||
For example, ``sysconfig.get_paths()`` in a source build gives
|
||
(partial output)::
|
||
|
||
{
|
||
'include': '/home/vinay/tools/pythonv/Include',
|
||
'libdir': '/usr/lib ; or /usr/lib64 on a multilib system',
|
||
'platinclude': '/home/vinay/tools/pythonv',
|
||
'platlib': '/usr/local/lib/python3.3/site-packages',
|
||
'platstdlib': '/usr/local/lib/python3.3',
|
||
'purelib': '/usr/local/lib/python3.3/site-packages',
|
||
'stdlib': '/usr/local/lib/python3.3'
|
||
}
|
||
|
||
|
||
Need for ``install_name_tool`` on OSX?
|
||
--------------------------------------
|
||
|
||
`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode
|
||
developer tools, to modify the copied executable on OSX. We need
|
||
input from OSX developers on whether this is actually necessary in
|
||
this PEP's implementation of virtual environments, and if so, if there
|
||
is an alternative to ``install_name_tool`` that would allow ``venv``
|
||
to not require that Xcode is installed.
|
||
|
||
.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168
|
||
|
||
|
||
Other Python implementations?
|
||
-----------------------------
|
||
|
||
We should get feedback from Jython, IronPython, and PyPy about whether
|
||
there's anything in this PEP that they foresee as a difficulty for
|
||
their implementation.
|
||
|
||
|
||
Reference Implementation
|
||
========================
|
||
|
||
The in-progress reference implementation is found in `a clone of the
|
||
CPython Mercurial repository`_. To test it, build and install it (the
|
||
virtual environment tool currently does not run from a source tree).
|
||
From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv``
|
||
to create a virtual environment.
|
||
|
||
The reference implementation (like this PEP!) is a work in progress.
|
||
|
||
.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|
||
|