diff --git a/pep-0404.txt b/pep-0404.txt index e6e8f6b05..590ef4b43 100644 --- a/pep-0404.txt +++ b/pep-0404.txt @@ -37,22 +37,22 @@ across multiple Python versions, among other uses. Existing virtual environment tools suffer from lack of support from the behavior of Python itself. Tools such as `rvirtualenv`_, which do not copy the Python binary into the virtual environment, cannot -provide reliable isolation from system site directories. Virtualenv, +provide reliable isolation from system site directories. Virtualenv, which does copy the Python binary, is forced to duplicate much of Python's ``site`` module and manually symlink/copy an ever-changing set of standard-library modules into the virtual environment in order -to perform a delicate boot-strapping dance at every -startup. (Virtualenv copies the binary because symlinking it does not -provide isolation, as Python dereferences a symlinked executable -before searching for `sys.prefix`.) +to perform a delicate boot-strapping dance at every startup. +(Virtualenv copies the binary because symlinking it does not provide +isolation, as Python dereferences a symlinked executable before +searching for `sys.prefix`.) The ``PYTHONHOME`` environment variable, Python's only existing built-in solution for virtual environments, requires -copying/symlinking the entire standard library into every -environment. Copying the whole standard library is not a lightweight -solution, and cross-platform support for symlinks remains inconsistent -(even on Windows platforms that do support them, creating them often -requires administrator privileges). +copying/symlinking the entire standard library into every environment. +Copying the whole standard library is not a lightweight solution, and +cross-platform support for symlinks remains inconsistent (even on +Windows platforms that do support them, creating them often requires +administrator privileges). A virtual environment mechanism integrated with Python and drawing on years of experience with existing third-party tools can be lower @@ -79,7 +79,7 @@ found, falling back to the build-time prefix hardcoded in the binary. This PEP proposes to add a new first step to this search. If a ``pyvenv.cfg`` file is found either adjacent to the Python executable, or one directory above it, this file is scanned for lines of the form -``key = value``. If a ``home`` key is found, this signifies that the +``key = value``. If a ``home`` key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the ``home`` key is the directory containing the Python executable used to create this virtual environment. @@ -105,8 +105,7 @@ regard to ``sys.exec_prefix``.) Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy or symlink of the Python binary -accompanied by a ``pyvenv.cfg`` file and a site-packages -directory. +accompanied by a ``pyvenv.cfg`` file and a site-packages directory. Isolation from system site-packages @@ -173,9 +172,8 @@ install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing virtual environments in ``sysconfig`` beyond using ``sys.site_prefix`` in place of ``sys.prefix``, the internal virtual environment layout -mimics the layout of the Python installation itself on each -platform. So a typical virtual environment layout on a POSIX system -would be:: +mimics the layout of the Python installation itself on each platform. +So a typical virtual environment layout on a POSIX system would be:: pyvenv.cfg bin/python3 @@ -201,7 +199,7 @@ their executables placed in ``bin/`` or ``Scripts\``. On a normal Windows system-level installation, the Python binary itself wouldn't go inside the "Scripts/" subdirectory, as it does - in the default venv layout. This is useful in a virtual + in the default venv layout. This is useful in a virtual environment so that a user only has to add a single directory to their shell PATH in order to effectively "activate" the virtual environment. @@ -219,7 +217,7 @@ Copies versus symlinks ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). Some +or symlinked Python binary (and other needed DLLs on Windows). Some users prefer a copied binary (for greater isolation from system changes) and some prefer a symlinked one (so that e.g. security updates automatically propagate to virtual environments). @@ -230,16 +228,15 @@ There are some cross-platform difficulties with symlinks: do, creating them often requires administrator privileges. * On OSX framework builds of Python, sys.executable is just a stub - that executes the real Python binary. Symlinking this stub does not - work with the implementation in this PEP; it must be - copied. (Fortunately the stub is also small, so copying it is not an - issue). + that executes the real Python binary. Symlinking this stub does not + work with the implementation in this PEP; it must be copied. + (Fortunately the stub is also small, so copying it is not an issue). Because of these issues, this PEP proposes to copy the Python binary by default, to maintain cross-platform consistency in the default behavior. -The ``pyvenv`` script accepts a ``--symlink`` option. If this option +The ``pyvenv`` script accepts a ``--symlink`` option. If this option is provided, the script will attempt to symlink instead of copy. If a symlink fails (e.g. because they are not supported by the platform, or additional privileges are needed), the script will warn the user and @@ -259,25 +256,24 @@ provides mechanisms for third-party virtual environment creators to customize environment creation according to their needs. The ``venv`` module contains an ``EnvBuilder`` class which accepts the -following keyword arguments on instantiation:: +following keyword arguments on instantiation: - * ``system_site_packages`` - A Boolean value indicating that the - system Python site-packages should be available to the - environment (defaults to ``False``). +* ``system_site_packages`` - A Boolean value indicating that the + system Python site-packages should be available to the environment. + Defaults to ``False``. - * ``clear`` - A Boolean value which, if True, will delete any - existing target directory instead of raising an exception - (defaults to ``False``). +* ``clear`` - A Boolean value which, if true, will delete any existing + target directory instead of raising an exception. Defaults to + ``False``. - * ``use_symlinks`` - A Boolean value indicating whether to attempt - to symlink the Python binary (and any necessary DLLs or other - binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to - ``False``. +* ``use_symlinks`` - A Boolean value indicating whether to attempt to + symlink the Python binary (and any necessary DLLs or other binaries, + e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``. The returned env-builder is an object with a ``create`` method, which takes as required argument the path (absolute or relative to the current directory) of the target directory which is to contain the -virtual environment. The ``create`` method either creates the +virtual environment. The ``create`` method either creates the environment in the specified directory, or raises an appropriate exception. @@ -296,7 +292,7 @@ convenience:: builder.create(env_dir) The ``create`` method of the ``EnvBuilder`` class illustrates the -hooks available for customization: +hooks available for customization:: def create(self, env_dir): """ @@ -312,67 +308,65 @@ hooks available for customization: self.post_setup(context) Each of the methods ``create_directories``, ``create_configuration``, -``setup_python``, and ``post_setup`` can be -overridden. The functions of these methods are:: +``setup_python``, and ``post_setup`` can be overridden. The functions +of these methods are: - * ``create_directories`` - creates the environment directory and - all necessary directories, and returns a context object. This is - just a holder for attributes (such as paths), for use by the - other methods. +* ``create_directories`` - creates the environment directory and all + necessary directories, and returns a context object. This is just a + holder for attributes (such as paths), for use by the other methods. - * ``create_configuration`` - creates the ``pyvenv.cfg`` - configuration file in the environment. +* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration + file in the environment. - * ``setup_python`` - creates a copy of the Python executable (and, - under Windows, DLLs) in the environment. +* ``setup_python`` - creates a copy of the Python executable (and, + under Windows, DLLs) in the environment. - * ``post_setup`` - A (no-op by default) hook method which can be - overridden in third party implementations to pre-install packages - or install scripts in the virtual environment. +* ``post_setup`` - A (no-op by default) hook method which can be + overridden in third party implementations to pre-install packages or + install scripts in the virtual environment. In addition, ``EnvBuilder`` provides a utility method that can be called from ``post_setup`` in subclasses to assist in installing scripts into the virtual environment. The method ``install_scripts`` accepts as arguments the ``context`` object (see above) and a -bytestring. The bytestring should be a base64-encoded zip file +bytestring. The bytestring should be a base64-encoded zip file containing directories "common", "posix", "nt", each containing -scripts destined for the bin directory in the environment. The +scripts destined for the bin directory in the environment. The contents of "common" and the directory corresponding to ``os.name`` are copied after doing some text replacement of placeholders: - * ``__VENV_DIR__`` is replaced with absolute path of the - environment directory. +* ``__VENV_DIR__`` is replaced with absolute path of the environment + directory. - * ``__VENV_NAME__`` is replaced with the environment - name (final path segment of environment directory). +* ``__VENV_NAME__`` is replaced with the environment name (final path + segment of environment directory). - * ``__VENV_BIN_NAME__`` is replaced with the name of the bin - directory (either ``bin`` or ``Scripts``). - - * ``__VENV_PYTHON__`` is replaced with the absolute path of the - environment's executable. +* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory + (either ``bin`` or ``Scripts``). +* ``__VENV_PYTHON__`` is replaced with the absolute path of the + environment's executable. The ``DistributeEnvBuilder`` subclass in the reference implementation illustrates how the customization hook can be used in practice to pre-install Distribute and shell activation scripts into the virtual -environment. It's not envisaged that ``DistributeEnvBuilder`` will be +environment. It's not envisaged that ``DistributeEnvBuilder`` will be actually added to Python core, but it makes the reference implementation more immediately useful for testing and exploratory purposes. The "shell activation scripts" provided by ``DistributeEnvBuilder`` simply add the virtual environment's ``bin/`` (or ``Scripts\``) -directory to the front of the user's shell PATH. This is not strictly +directory to the front of the user's shell PATH. This is not strictly necessary for use of a virtual environment (as an explicit path to the venv's python binary or scripts can just as well be used), but it is convenient. This PEP does not propose that the ``venv`` module in core Python will -add such activation scripts by default, as they are -shell-specific. Adding activation scripts for the wide variety of -possible shells is an added maintenance burden, and is left to -third-party extension tools. +add such activation scripts by default, as they are shell-specific. +Adding activation scripts for the wide variety of possible shells is +an added maintenance burden, and is left to third-party extension +tools. No doubt the process of PEP review will show up any customization requirements which have not yet been considered. @@ -396,11 +390,12 @@ be installed?" This split could be handled by introducing a new ``sys`` attribute for either the former prefix or the latter prefix. Either option potentially introduces some backwards-incompatibility with software -written to assume the other meaning for ``sys.prefix``. (Such software -should preferably be using the APIs in the ``site`` and ``sysconfig`` -modules to answer these questions rather than using ``sys.prefix`` -directly, in which case there is no backwards-compatibility issue, but -in practice ``sys.prefix`` is sometimes used.) +written to assume the other meaning for ``sys.prefix``. (Such +software should preferably be using the APIs in the ``site`` and +``sysconfig`` modules to answer these questions rather than using +``sys.prefix`` directly, in which case there is no +backwards-compatibility issue, but in practice ``sys.prefix`` is +sometimes used.) The `documentation`__ for ``sys.prefix`` describes it as "A string giving the site-specific directory prefix where the platform @@ -413,11 +408,11 @@ __ http://docs.python.org/dev/library/sys.html#sys.prefix This PEP currently proposes to leave ``sys.prefix`` pointing to the base system installation (which is where the standard library and header files are found), and introduce a new value in ``sys`` -(``sys.site_prefix``) to point to the prefix for -``site-packages``. This maintains the documented semantics of -``sys.prefix``, but risks breaking isolation if third-party code uses -``sys.prefix`` rather than ``sys.site_prefix`` or the appropriate -``site`` API to find site-packages directories. +(``sys.site_prefix``) to point to the prefix for ``site-packages``. +This maintains the documented semantics of ``sys.prefix``, but risks +breaking isolation if third-party code uses ``sys.prefix`` rather than +``sys.site_prefix`` or the appropriate ``site`` API to find +site-packages directories. The most notable case is probably `setuptools`_ and its fork `distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but @@ -426,7 +421,7 @@ for pre-flight checking where ``pth`` files can usefully be placed. It would be trivial to modify these tools (currently only `distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and fall back to ``sys.prefix`` if it doesn't exist (for earlier versions -of Python). If Distribute is modified in this way and released before +of Python). If Distribute is modified in this way and released before Python 3.3 is released with the ``venv`` module, there would be no likely reason for an older version of Distribute to ever be installed in a virtual environment. @@ -434,9 +429,9 @@ in a virtual environment. In terms of other third-party usage, a `Google Code Search`_ turns up what appears to be a roughly even mix of usage between packages using ``sys.prefix`` to build up a site-packages path and packages using it -to e.g. eliminate the standard-library from code-execution -tracing. Either choice that's made here will require one or the other -of these uses to be updated. +to e.g. eliminate the standard-library from code-execution tracing. +Either choice that's made here will require one or the other of these +uses to be updated. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools .. _distribute: http://packages.python.org/distribute/ @@ -454,7 +449,7 @@ considerations in mind: * Inasmuch as "site" has a meaning in Python, it means a combination of Python version, standard library, and specific set of - site-packages. This is, fundamentally, what a venv is (although it + site-packages. This is, fundamentally, what a venv is (although it shares the standard library with its "base" site). * It is the Python ``site`` module which implements adding @@ -466,17 +461,17 @@ overloaded and of unclear meaning, and this usage will increase the overload. One proposed alternative is ``sys.venv_prefix``, which has the -advantage of being clearly related to the venv implementation. The +advantage of being clearly related to the venv implementation. The downside of this proposal is that it implies the attribute is only useful/relevant when in a venv and should be absent or ``None`` when -not in a venv. This imposes an unnecessary extra burden on code using -the attribute: ``sys.venv_prefix if sys.venv_prefix else -sys.prefix``. The prefix attributes are more usable and general if -they are always present and set, and split by meaning (stdlib vs -site-packages, roughly), rather than specifically tied to venv. Also, -third-party code should be encouraged to not know or care whether it -is running in a virtual environment or not; this option seems to work -against that goal. +not in a venv. This imposes an unnecessary extra burden on code using +the attribute: ``sys.venv_prefix if sys.venv_prefix else sys.prefix``. +The prefix attributes are more usable and general if they are always +present and set, and split by meaning (stdlib vs site-packages, +roughly), rather than specifically tied to venv. Also, third-party +code should be encouraged to not know or care whether it is running in +a virtual environment or not; this option seems to work against that +goal. Another option would be ``sys.local_prefix``, which has both the advantage and disadvantage, depending on perspective, that it @@ -489,11 +484,11 @@ Why not modify sys.prefix? As discussed above under `Backwards Compatibility`_, this PEP proposes to add ``sys.site_prefix`` as "the prefix relative to which -site-package directories are found". This maintains compatibility with -the documented meaning of ``sys.prefix`` (as the location relative to -which the standard library can be found), but means that code assuming -that site-packages directories are found relative to ``sys.prefix`` -will not respect the virtual environment correctly. +site-package directories are found". This maintains compatibility +with the documented meaning of ``sys.prefix`` (as the location +relative to which the standard library can be found), but means that +code assuming that site-packages directories are found relative to +``sys.prefix`` will not respect the virtual environment correctly. Since it is unable to modify ``distutils``/``sysconfig``, `virtualenv`_ is forced to instead re-point ``sys.prefix`` at the @@ -506,7 +501,7 @@ does this and it doesn't appear to have caused major problems with existing code. Another argument in favor of this is that it would be preferable to -err on the side of greater, rather than lesser, isolation. Changing +err on the side of greater, rather than lesser, isolation. Changing ``sys.prefix`` to point to the virtual environment and introducing a new ``sys.base_prefix`` attribute would err on the side of greater isolation in the face of existing code's use of ``sys.prefix``. @@ -515,13 +510,14 @@ isolation in the face of existing code's use of ``sys.prefix``. What about include files? ------------------------- -For example, ZeroMQ installs zmq.h and zmq_utils.h in $VE/include, -whereas SIP (part of PyQt4) installs sip.h by default in -$VE/include/pythonX.Y. With virtualenv, everything works because the -PythonX.Y include is symlinked, so everything that's needed is in -$VE/include. At the moment the reference implementation doesn't do -anything with include files, besides creating the include directory; -this might need to change, to copy/symlink $VE/include/pythonX.Y. +For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in +``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default +in ``$VE/include/pythonX.Y``. With virtualenv, everything works +because the PythonX.Y include is symlinked, so everything that's +needed is in ``$VE/include``. At the moment the reference +implementation doesn't do anything with include files, besides +creating the include directory; this might need to change, to +copy/symlink ``$VE/include/pythonX.Y``. As in Python there's no abstraction for a site-specific include directory, other than for platform-specific stuff, then the user @@ -530,26 +526,27 @@ want should be found in one of just two locations, with sysconfig labels "include" & "platinclude". There's another issue: what if includes are Python-version-specific? -For example, SIP installs by default into $VE/include/pythonX.Y rather -than $VE/include, presumably because there's version-specific stuff in -there - but even if that's not the case with SIP, it could be the case -with some other package. And the problem that gives is that you can't -just symlink the include/pythonX.Y directory, but actually have to -provide a writable directory and symlink/copy the contents from the -system include/pythonX.Y. Of course this is not hard to do, but it -does seem inelegant. OTOH it's really because there's no supporting -concept in Python/sysconfig. +For example, SIP installs by default into ``$VE/include/pythonX.Y`` +rather than ``$VE/include``, presumably because there's +version-specific stuff in there - but even if that's not the case with +SIP, it could be the case with some other package. And the problem +that gives is that you can't just symlink the ``include/pythonX.Y`` +directory, but actually have to provide a writable directory and +symlink/copy the contents from the system ``include/pythonX.Y``. Of +course this is not hard to do, but it does seem inelegant. OTOH it's +really because there's no supporting concept in ``Python/sysconfig``. Interface with packaging tools ------------------------------ Some work will be needed in packaging tools (Python 3.3 packaging, -Distribute) to support implementation of this PEP. For example: +Distribute) to support implementation of this PEP. For example: -* How Distribute and packaging use sys.prefix and/or sys.site_prefix. Clearly, - in practice we'll need to use Distribute for a while, until packages have - migrated over to usage of setup.cfg. +* How Distribute and packaging use ``sys.prefix`` and/or + ``sys.site_prefix``. Clearly, in practice we'll need to use + Distribute for a while, until packages have migrated over to usage + of setup.cfg. * How packaging and Distribute set up shebang lines in scripts which they install in virtual environments. @@ -560,10 +557,10 @@ Testability and Source Build Issues Currently in the reference implementation, virtual environments must be created with an installed Python, rather than a source build, as -the base installation. In order to be able to fully test the ``venv`` +the base installation. In order to be able to fully test the ``venv`` module in the Python regression test suite, some anomalies in how -sysconfig data is configured in source builds will need to be -removed. For example, sysconfig.get_paths() in a source build gives +sysconfig data is configured in source builds will need to be removed. +For example, ``sysconfig.get_paths()`` in a source build gives (partial output):: { @@ -581,11 +578,11 @@ Need for ``install_name_tool`` on OSX? -------------------------------------- `Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode -developer tools, to modify the copied executable on OSX. We need input -from OSX developers on whether this is actually necessary in this -PEP's implementation of virtual environments, and if so, if there is -an alternative to ``install_name_tool`` that would allow ``venv`` to -not require that Xcode is installed. +developer tools, to modify the copied executable on OSX. We need +input from OSX developers on whether this is actually necessary in +this PEP's implementation of virtual environments, and if so, if there +is an alternative to ``install_name_tool`` that would allow ``venv`` +to not require that Xcode is installed. .. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 @@ -595,9 +592,9 @@ Activation and Utility Scripts Virtualenv provides shell "activation" scripts as a user convenience, to put the virtual environment's Python binary first on the shell -PATH. This is a maintenance burden, as separate activation scripts -need to be provided and maintained for every supported shell. For this -reason, this PEP proposes to leave such scripts to be provided by +PATH. This is a maintenance burden, as separate activation scripts +need to be provided and maintained for every supported shell. For +this reason, this PEP proposes to leave such scripts to be provided by third-party extensions; virtual environments created by the core functionality would be used by directly invoking the environment's Python binary or scripts. @@ -634,9 +631,9 @@ Reference Implementation The in-progress reference implementation is found in `a clone of the CPython Mercurial repository`_. To test it, build and install it (the -virtual environment tool currently does not run from a source -tree). From the installed Python, run ``bin/pyvenv -/path/to/new/virtualenv`` to create a virtual environment. +virtual environment tool currently does not run from a source tree). +From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` +to create a virtual environment. The reference implementation (like this PEP!) is a work in progress.