PEP: 405 Title: Python Virtual Environments Version: $Revision$ Last-Modified: $Date$ Author: Carl Meyer Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version: 3.3 Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012 Abstract ======== This PEP proposes to add to Python a mechanism for lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories, but shares the standard library with the base installed Python. Motivation ========== The utility of Python virtual environments has already been well established by the popularity of existing third-party virtual-environment tools, primarily Ian Bicking's `virtualenv`_. Virtual environments are already widely used for dependency management and isolation, ease of installing and using Python packages without system-administrator access, and automated testing of Python software across multiple Python versions, among other uses. Existing virtual environment tools suffer from lack of support from the behavior of Python itself. Tools such as `rvirtualenv`_, which do not copy the Python binary into the virtual environment, cannot provide reliable isolation from system site directories. Virtualenv, which does copy the Python binary, is forced to duplicate much of Python's ``site`` module and manually symlink/copy an ever-changing set of standard-library modules into the virtual environment in order to perform a delicate boot-strapping dance at every startup. (Virtualenv must copy the binary in order to provide isolation, as Python dereferences a symlinked executable before searching for ``sys.prefix``.) The ``PYTHONHOME`` environment variable, Python's only existing built-in solution for virtual environments, requires copying/symlinking the entire standard library into every environment. Copying the whole standard library is not a lightweight solution, and cross-platform support for symlinks remains inconsistent (even on Windows platforms that do support them, creating them often requires administrator privileges). A virtual environment mechanism integrated with Python and drawing on years of experience with existing third-party tools can be lower maintenance, more reliable, and more easily available to all Python users. .. _virtualenv: http://www.virtualenv.org .. _rvirtualenv: https://github.com/kvbik/rvirtualenv Specification ============= When the Python binary is executed, it attempts to determine its prefix (which it stores in ``sys.prefix``), which is then used to find the standard library and other key files, and by the ``site`` module to determine the location of the site-package directories. Currently the prefix is found (assuming ``PYTHONHOME`` is not set) by first walking up the filesystem tree looking for a marker file (``os.py``) that signifies the presence of the standard library, and if none is found, falling back to the build-time prefix hardcoded in the binary. This PEP proposes to add a new first step to this search. If a ``pyvenv.cfg`` file is found either adjacent to the Python executable, or one directory above it, this file is scanned for lines of the form ``key = value``. If a ``home`` key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the ``home`` key is the directory containing the Python executable used to create this virtual environment. In this case, prefix-finding continues as normal using the value of the ``home`` key as the effective Python binary location, which finds the prefix of the base installation. ``sys.base_prefix`` is set to this value, while ``sys.prefix`` is set to the directory containing ``pyvenv.cfg``. (If ``pyvenv.cfg`` is not found or does not contain the ``home`` key, prefix-finding continues normally, and ``sys.prefix`` will be equal to ``sys.base_prefix``.) The ``site`` and ``sysconfig`` standard-library modules are modified such that the standard library and header files are are found relative to ``sys.base_prefix``, while site-package directories ("purelib" and "platlib", in ``sysconfig`` terms) are still found relative to ``sys.prefix``. (Also, ``sys.base_exec_prefix`` is added, and handled similarly with regard to ``sys.exec_prefix``.) Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy or symlink of the Python binary accompanied by a ``pyvenv.cfg`` file and a site-packages directory. Isolation from system site-packages ----------------------------------- By default, a virtual environment is entirely isolated from the system-level site-packages directories. If the ``pyvenv.cfg`` file also contains a key ``include-system-site-packages`` with a value of ``true`` (not case sensitive), the ``site`` module will also add the system site directories to ``sys.path`` after the virtual environment site directories. Thus system-installed packages will still be importable, but a package of the same name installed in the virtual environment will take precedence. :pep:`370` user-level site-packages are considered part of the system site-packages for venv purposes: they are not available from an isolated venv, but are available from an ``include-system-site-packages = true`` venv. Creating virtual environments ----------------------------- This PEP also proposes adding a new ``venv`` module to the standard library which implements the creation of virtual environments. This module can be executed using the ``-m`` flag:: python3 -m venv /path/to/new/virtual/environment A ``pyvenv`` installed script is also provided to make this more convenient:: pyvenv /path/to/new/virtual/environment Running this command creates the target directory (creating any parent directories that don't exist already) and places a ``pyvenv.cfg`` file in it with a ``home`` key pointing to the Python installation the command was run from. It also creates a ``bin/`` (or ``Scripts`` on Windows) subdirectory containing a copy (or symlink) of the ``python3`` executable, and the ``pysetup3`` script from the ``packaging`` standard library module (to facilitate easy installation of packages from PyPI into the new virtualenv). And it creates an (initially empty) ``lib/pythonX.Y/site-packages`` (or ``Lib\site-packages`` on Windows) subdirectory. If the target directory already exists an error will be raised, unless the ``--clear`` option was provided, in which case the target directory will be deleted and virtual environment creation will proceed as usual. The created ``pyvenv.cfg`` file also includes the ``include-system-site-packages`` key, set to ``true`` if ``venv`` is run with the ``--system-site-packages`` option, ``false`` by default. Multiple paths can be given to ``pyvenv``, in which case an identical virtualenv will be created, according to the given options, at each provided path. The ``venv`` module also provides "shell activation scripts" for POSIX and Windows systems which simply add the virtual environment's ``bin`` (or ``Scripts``) directory to the front of the user's shell PATH. This is not strictly necessary for use of a virtual environment (as an explicit path to the venv's python binary or scripts can just as well be used), but it is convenient. In order to allow ``pysetup`` and other Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing virtual environments in ``sysconfig`` beyond using ``sys.site_prefix`` in place of ``sys.prefix``, the internal virtual environment layout mimics the layout of the Python installation itself on each platform. So a typical virtual environment layout on a POSIX system would be:: pyvenv.cfg bin/python3 bin/python bin/pysetup3 include/ lib/python3.3/site-packages/ While on a Windows system:: pyvenv.cfg Scripts/python.exe Scripts/python3.dll Scripts/pysetup3.exe Scripts/pysetup3-script.py ... other DLLs and pyds... Include/ Lib/site-packages/ Third-party packages installed into the virtual environment will have their Python modules placed in the ``site-packages`` directory, and their executables placed in ``bin/`` or ``Scripts``. .. note:: On a normal Windows system-level installation, the Python binary itself wouldn't go inside the "Scripts/" subdirectory, as it does in the default venv layout. This is useful in a virtual environment so that a user only has to add a single directory to their shell PATH in order to effectively "activate" the virtual environment. .. note:: On Windows, it is necessary to also copy or symlink DLLs and pyd files from compiled stdlib modules into the env, because if the venv is created from a non-system-wide Python installation, Windows won't be able to find the Python installation's copies of those files when Python is run from the venv. Copies versus symlinks ---------------------- The technique in this PEP works equally well in general with a copied or symlinked Python binary (and other needed DLLs on Windows). Some users prefer a copied binary (for greater isolation from system changes) and some prefer a symlinked one (so that e.g. security updates automatically propagate to virtual environments). There are some cross-platform difficulties with symlinks: * Not all Windows versions support symlinks, and even on those that do, creating them often requires administrator privileges. * On OSX framework builds of Python, sys.executable is just a stub that executes the real Python binary. Symlinking this stub does not work with the implementation in this PEP; it must be copied. (Fortunately the stub is also small, so copying it is not an issue). Because of these issues, this PEP proposes to copy the Python binary by default, to maintain cross-platform consistency in the default behavior. The ``pyvenv`` script accepts a ``--symlink`` option. If this option is provided, the script will attempt to symlink instead of copy. If a symlink fails (e.g. because they are not supported by the platform, or additional privileges are needed), the script will warn the user and fall back to a copy. On OSX framework builds, where a symlink of the executable would succeed but create a non-functional virtual environment, the script will fail with an error message that symlinking is not supported on OSX framework builds. API --- The high-level method described above makes use of a simple API which provides mechanisms for third-party virtual environment creators to customize environment creation according to their needs. The ``venv`` module contains an ``EnvBuilder`` class which accepts the following keyword arguments on instantiation: * ``system_site_packages`` - A Boolean value indicating that the system Python site-packages should be available to the environment. Defaults to ``False``. * ``clear`` - A Boolean value which, if true, will delete any existing target directory instead of raising an exception. Defaults to ``False``. * ``symlinks`` - A Boolean value indicating whether to attempt to symlink the Python binary (and any necessary DLLs or other binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``. The instantiated env-builder has a ``create`` method, which takes as required argument the path (absolute or relative to the current directory) of the target directory which is to contain the virtual environment. The ``create`` method either creates the environment in the specified directory, or raises an appropriate exception. The ``venv`` module also provides a module-level ``create`` function as a convenience:: def create(env_dir, system_site_packages=False, clear=False, use_symlinks=False): builder = EnvBuilder( system_site_packages=system_site_packages, clear=clear, use_symlinks=use_symlinks) builder.create(env_dir) Creators of third-party virtual environment tools are free to use the provided ``EnvBuilder`` class as a base class. The ``create`` method of the ``EnvBuilder`` class illustrates the hooks available for customization:: def create(self, env_dir): """ Create a virtualized Python environment in a directory. :param env_dir: The target directory to create an environment in. """ env_dir = os.path.abspath(env_dir) context = self.create_directories(env_dir) self.create_configuration(context) self.setup_python(context) self.post_setup(context) Each of the methods ``create_directories``, ``create_configuration``, ``setup_python``, and ``post_setup`` can be overridden. The functions of these methods are: * ``create_directories`` - creates the environment directory and all necessary directories, and returns a context object. This is just a holder for attributes (such as paths), for use by the other methods. * ``create_configuration`` - creates the ``pyvenv.cfg`` configuration file in the environment. * ``setup_python`` - creates a copy of the Python executable (and, under Windows, DLLs) in the environment. * ``post_setup`` - A (no-op by default) hook method which can be overridden in third party subclasses to pre-install packages or install scripts in the virtual environment. In addition, ``EnvBuilder`` provides a utility method that can be called from ``post_setup`` in subclasses to assist in installing custom scripts into the virtual environment. The method ``install_scripts`` accepts as arguments the ``context`` object (see above) and a path to a directory. The directory should contain subdirectories "common", "posix", "nt", each containing scripts destined for the bin directory in the environment. The contents of "common" and the directory corresponding to ``os.name`` are copied after doing some text replacement of placeholders: * ``__VENV_DIR__`` is replaced with absolute path of the environment directory. * ``__VENV_NAME__`` is replaced with the environment name (final path segment of environment directory). * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory (either ``bin`` or ``Scripts``). * ``__VENV_PYTHON__`` is replaced with the absolute path of the environment's executable. The ``DistributeEnvBuilder`` subclass in the reference implementation illustrates how the customization hook can be used in practice to pre-install Distribute into the virtual environment. It's not envisaged that ``DistributeEnvBuilder`` will be actually added to Python core, but it makes the reference implementation more immediately useful for testing and exploratory purposes. Backwards Compatibility ======================= Splitting the meanings of ``sys.prefix`` ---------------------------------------- Any virtual environment tool along these lines (which attempts to isolate site-packages, while still making use of the base Python's standard library with no need for it to be symlinked into the virtual environment) is proposing a split between two different meanings (among others) that are currently both wrapped up in ``sys.prefix``: the answers to the questions "Where is the standard library?" and "Where is the site-packages location where third-party modules should be installed?" This split could be handled by introducing a new ``sys`` attribute for either the former prefix or the latter prefix. Either option potentially introduces some backwards-incompatibility with software written to assume the other meaning for ``sys.prefix``. (Such software should preferably be using the APIs in the ``site`` and ``sysconfig`` modules to answer these questions rather than using ``sys.prefix`` directly, in which case there is no backwards-compatibility issue, but in practice ``sys.prefix`` is sometimes used.) The `documentation`__ for ``sys.prefix`` describes it as "A string giving the site-specific directory prefix where the platform independent Python files are installed," and specifically mentions the standard library and header files as found under ``sys.prefix``. It does not mention ``site-packages``. __ http://docs.python.org/dev/library/sys.html#sys.prefix Maintaining this documented definition would mean leaving ``sys.prefix`` pointing to the base system installation (which is where the standard library and header files are found), and introducing a new value in ``sys`` (something like ``sys.site_prefix``) to point to the prefix for ``site-packages``. This would maintain the documented semantics of ``sys.prefix``, but risk breaking isolation if third-party code uses ``sys.prefix`` rather than ``sys.site_prefix`` or the appropriate ``site`` API to find site-packages directories. The most notable case is probably `setuptools`_ and its fork `distribute`_, which mostly use ``distutils``and ``sysconfig`` APIs, but do use ``sys.prefix`` directly to build up a list of site directories for pre-flight checking where ``pth`` files can usefully be placed. Otherwise, a `Google Code Search`_ turns up what appears to be a roughly even mix of usage between packages using ``sys.prefix`` to build up a site-packages path and packages using it to e.g. eliminate the standard-library from code-execution tracing. Although it requires modifying the documented definition of ``sys.prefix``, this PEP prefers to have ``sys.prefix`` point to the virtual environment (where ``site-packages`` is found), and introduce ``sys.base_prefix`` to point to the standard library and Python header files. Rationale for this choice: * It is preferable to err on the side of greater isolation of the virtual environment. * Virtualenv already modifies ``sys.prefix`` to point at the virtual environment, and in practice this has not been a problem. * No modification is required to setuptools/distribute. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools .. _distribute: http://packages.python.org/distribute/ .. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs Open Questions ============== What about include files? ------------------------- For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in ``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default in ``$VE/include/pythonX.Y``. With virtualenv, everything works because the PythonX.Y include is symlinked, so everything that's needed is in ``$VE/include``. At the moment the reference implementation doesn't do anything with include files, besides creating the include directory; this might need to change, to copy/symlink ``$VE/include/pythonX.Y``. As in Python there's no abstraction for a site-specific include directory, other than for platform-specific stuff, then the user expectation would seem to be that all include files anyone could ever want should be found in one of just two locations, with sysconfig labels "include" & "platinclude". There's another issue: what if includes are Python-version-specific? For example, SIP installs by default into ``$VE/include/pythonX.Y`` rather than ``$VE/include``, presumably because there's version-specific stuff in there - but even if that's not the case with SIP, it could be the case with some other package. And the problem that gives is that you can't just symlink the ``include/pythonX.Y`` directory, but actually have to provide a writable directory and symlink/copy the contents from the system ``include/pythonX.Y``. Of course this is not hard to do, but it does seem inelegant. OTOH it's really because there's no supporting concept in ``Python/sysconfig``. Testability and Source Build Issues ----------------------------------- Currently in the reference implementation, virtual environments must be created with an installed Python, rather than a source build, as the base installation. In order to be able to fully test the ``venv`` module in the Python regression test suite, some anomalies in how sysconfig data is configured in source builds will need to be removed. For example, ``sysconfig.get_paths()`` in a source build gives (partial output):: { 'include': '/home/vinay/tools/pythonv/Include', 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', 'platinclude': '/home/vinay/tools/pythonv', 'platlib': '/usr/local/lib/python3.3/site-packages', 'platstdlib': '/usr/local/lib/python3.3', 'purelib': '/usr/local/lib/python3.3/site-packages', 'stdlib': '/usr/local/lib/python3.3' } Need for ``install_name_tool`` on OSX? -------------------------------------- `Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode developer tools, to modify the copied executable on OSX. We need input from OSX developers on whether this is actually necessary in this PEP's implementation of virtual environments, and if so, if there is an alternative to ``install_name_tool`` that would allow ``venv`` to not require that Xcode is installed. .. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 Other Python implementations? ----------------------------- We should get feedback from Jython, IronPython, and PyPy about whether there's anything in this PEP that they foresee as a difficulty for their implementation. Reference Implementation ======================== The in-progress reference implementation is found in `a clone of the CPython Mercurial repository`_. To test it, build and install it (the virtual environment tool currently does not run from a source tree). From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create a virtual environment. The reference implementation (like this PEP!) is a work in progress. .. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: