Fix-up formatting.

This commit is contained in:
Georg Brandl 2011-10-30 12:40:47 +01:00
parent 89b311fab3
commit c127893abe
1 changed files with 127 additions and 130 deletions

View File

@ -37,22 +37,22 @@ across multiple Python versions, among other uses.
Existing virtual environment tools suffer from lack of support from
the behavior of Python itself. Tools such as `rvirtualenv`_, which do
not copy the Python binary into the virtual environment, cannot
provide reliable isolation from system site directories. Virtualenv,
provide reliable isolation from system site directories. Virtualenv,
which does copy the Python binary, is forced to duplicate much of
Python's ``site`` module and manually symlink/copy an ever-changing
set of standard-library modules into the virtual environment in order
to perform a delicate boot-strapping dance at every
startup. (Virtualenv copies the binary because symlinking it does not
provide isolation, as Python dereferences a symlinked executable
before searching for `sys.prefix`.)
to perform a delicate boot-strapping dance at every startup.
(Virtualenv copies the binary because symlinking it does not provide
isolation, as Python dereferences a symlinked executable before
searching for `sys.prefix`.)
The ``PYTHONHOME`` environment variable, Python's only existing
built-in solution for virtual environments, requires
copying/symlinking the entire standard library into every
environment. Copying the whole standard library is not a lightweight
solution, and cross-platform support for symlinks remains inconsistent
(even on Windows platforms that do support them, creating them often
requires administrator privileges).
copying/symlinking the entire standard library into every environment.
Copying the whole standard library is not a lightweight solution, and
cross-platform support for symlinks remains inconsistent (even on
Windows platforms that do support them, creating them often requires
administrator privileges).
A virtual environment mechanism integrated with Python and drawing on
years of experience with existing third-party tools can be lower
@ -79,7 +79,7 @@ found, falling back to the build-time prefix hardcoded in the binary.
This PEP proposes to add a new first step to this search. If a
``pyvenv.cfg`` file is found either adjacent to the Python executable,
or one directory above it, this file is scanned for lines of the form
``key = value``. If a ``home`` key is found, this signifies that the
``key = value``. If a ``home`` key is found, this signifies that the
Python binary belongs to a virtual environment, and the value of the
``home`` key is the directory containing the Python executable used to
create this virtual environment.
@ -105,8 +105,7 @@ regard to ``sys.exec_prefix``.)
Thus, a Python virtual environment in its simplest form would consist
of nothing more than a copy or symlink of the Python binary
accompanied by a ``pyvenv.cfg`` file and a site-packages
directory.
accompanied by a ``pyvenv.cfg`` file and a site-packages directory.
Isolation from system site-packages
@ -173,9 +172,8 @@ install packages into the virtual environment the same way they would
install into a normal Python installation, and avoid special-casing
virtual environments in ``sysconfig`` beyond using ``sys.site_prefix``
in place of ``sys.prefix``, the internal virtual environment layout
mimics the layout of the Python installation itself on each
platform. So a typical virtual environment layout on a POSIX system
would be::
mimics the layout of the Python installation itself on each platform.
So a typical virtual environment layout on a POSIX system would be::
pyvenv.cfg
bin/python3
@ -201,7 +199,7 @@ their executables placed in ``bin/`` or ``Scripts\``.
On a normal Windows system-level installation, the Python binary
itself wouldn't go inside the "Scripts/" subdirectory, as it does
in the default venv layout. This is useful in a virtual
in the default venv layout. This is useful in a virtual
environment so that a user only has to add a single directory to
their shell PATH in order to effectively "activate" the virtual
environment.
@ -219,7 +217,7 @@ Copies versus symlinks
----------------------
The technique in this PEP works equally well in general with a copied
or symlinked Python binary (and other needed DLLs on Windows). Some
or symlinked Python binary (and other needed DLLs on Windows). Some
users prefer a copied binary (for greater isolation from system
changes) and some prefer a symlinked one (so that e.g. security
updates automatically propagate to virtual environments).
@ -230,16 +228,15 @@ There are some cross-platform difficulties with symlinks:
do, creating them often requires administrator privileges.
* On OSX framework builds of Python, sys.executable is just a stub
that executes the real Python binary. Symlinking this stub does not
work with the implementation in this PEP; it must be
copied. (Fortunately the stub is also small, so copying it is not an
issue).
that executes the real Python binary. Symlinking this stub does not
work with the implementation in this PEP; it must be copied.
(Fortunately the stub is also small, so copying it is not an issue).
Because of these issues, this PEP proposes to copy the Python binary
by default, to maintain cross-platform consistency in the default
behavior.
The ``pyvenv`` script accepts a ``--symlink`` option. If this option
The ``pyvenv`` script accepts a ``--symlink`` option. If this option
is provided, the script will attempt to symlink instead of copy. If a
symlink fails (e.g. because they are not supported by the platform, or
additional privileges are needed), the script will warn the user and
@ -259,25 +256,24 @@ provides mechanisms for third-party virtual environment creators to
customize environment creation according to their needs.
The ``venv`` module contains an ``EnvBuilder`` class which accepts the
following keyword arguments on instantiation::
following keyword arguments on instantiation:
* ``system_site_packages`` - A Boolean value indicating that the
system Python site-packages should be available to the
environment (defaults to ``False``).
* ``system_site_packages`` - A Boolean value indicating that the
system Python site-packages should be available to the environment.
Defaults to ``False``.
* ``clear`` - A Boolean value which, if True, will delete any
existing target directory instead of raising an exception
(defaults to ``False``).
* ``clear`` - A Boolean value which, if true, will delete any existing
target directory instead of raising an exception. Defaults to
``False``.
* ``use_symlinks`` - A Boolean value indicating whether to attempt
to symlink the Python binary (and any necessary DLLs or other
binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to
``False``.
* ``use_symlinks`` - A Boolean value indicating whether to attempt to
symlink the Python binary (and any necessary DLLs or other binaries,
e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``.
The returned env-builder is an object with a ``create`` method, which
takes as required argument the path (absolute or relative to the
current directory) of the target directory which is to contain the
virtual environment. The ``create`` method either creates the
virtual environment. The ``create`` method either creates the
environment in the specified directory, or raises an appropriate
exception.
@ -296,7 +292,7 @@ convenience::
builder.create(env_dir)
The ``create`` method of the ``EnvBuilder`` class illustrates the
hooks available for customization:
hooks available for customization::
def create(self, env_dir):
"""
@ -312,67 +308,65 @@ hooks available for customization:
self.post_setup(context)
Each of the methods ``create_directories``, ``create_configuration``,
``setup_python``, and ``post_setup`` can be
overridden. The functions of these methods are::
``setup_python``, and ``post_setup`` can be overridden. The functions
of these methods are:
* ``create_directories`` - creates the environment directory and
all necessary directories, and returns a context object. This is
just a holder for attributes (such as paths), for use by the
other methods.
* ``create_directories`` - creates the environment directory and all
necessary directories, and returns a context object. This is just a
holder for attributes (such as paths), for use by the other methods.
* ``create_configuration`` - creates the ``pyvenv.cfg``
configuration file in the environment.
* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration
file in the environment.
* ``setup_python`` - creates a copy of the Python executable (and,
under Windows, DLLs) in the environment.
* ``setup_python`` - creates a copy of the Python executable (and,
under Windows, DLLs) in the environment.
* ``post_setup`` - A (no-op by default) hook method which can be
overridden in third party implementations to pre-install packages
or install scripts in the virtual environment.
* ``post_setup`` - A (no-op by default) hook method which can be
overridden in third party implementations to pre-install packages or
install scripts in the virtual environment.
In addition, ``EnvBuilder`` provides a utility method that can be
called from ``post_setup`` in subclasses to assist in installing
scripts into the virtual environment. The method ``install_scripts``
accepts as arguments the ``context`` object (see above) and a
bytestring. The bytestring should be a base64-encoded zip file
bytestring. The bytestring should be a base64-encoded zip file
containing directories "common", "posix", "nt", each containing
scripts destined for the bin directory in the environment. The
scripts destined for the bin directory in the environment. The
contents of "common" and the directory corresponding to ``os.name``
are copied after doing some text replacement of placeholders:
* ``__VENV_DIR__`` is replaced with absolute path of the
environment directory.
* ``__VENV_DIR__`` is replaced with absolute path of the environment
directory.
* ``__VENV_NAME__`` is replaced with the environment
name (final path segment of environment directory).
* ``__VENV_NAME__`` is replaced with the environment name (final path
segment of environment directory).
* ``__VENV_BIN_NAME__`` is replaced with the name of the bin
directory (either ``bin`` or ``Scripts``).
* ``__VENV_PYTHON__`` is replaced with the absolute path of the
environment's executable.
* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
(either ``bin`` or ``Scripts``).
* ``__VENV_PYTHON__`` is replaced with the absolute path of the
environment's executable.
The ``DistributeEnvBuilder`` subclass in the reference implementation
illustrates how the customization hook can be used in practice to
pre-install Distribute and shell activation scripts into the virtual
environment. It's not envisaged that ``DistributeEnvBuilder`` will be
environment. It's not envisaged that ``DistributeEnvBuilder`` will be
actually added to Python core, but it makes the reference
implementation more immediately useful for testing and exploratory
purposes.
The "shell activation scripts" provided by ``DistributeEnvBuilder``
simply add the virtual environment's ``bin/`` (or ``Scripts\``)
directory to the front of the user's shell PATH. This is not strictly
directory to the front of the user's shell PATH. This is not strictly
necessary for use of a virtual environment (as an explicit path to the
venv's python binary or scripts can just as well be used), but it is
convenient.
This PEP does not propose that the ``venv`` module in core Python will
add such activation scripts by default, as they are
shell-specific. Adding activation scripts for the wide variety of
possible shells is an added maintenance burden, and is left to
third-party extension tools.
add such activation scripts by default, as they are shell-specific.
Adding activation scripts for the wide variety of possible shells is
an added maintenance burden, and is left to third-party extension
tools.
No doubt the process of PEP review will show up any customization
requirements which have not yet been considered.
@ -396,11 +390,12 @@ be installed?"
This split could be handled by introducing a new ``sys`` attribute for
either the former prefix or the latter prefix. Either option
potentially introduces some backwards-incompatibility with software
written to assume the other meaning for ``sys.prefix``. (Such software
should preferably be using the APIs in the ``site`` and ``sysconfig``
modules to answer these questions rather than using ``sys.prefix``
directly, in which case there is no backwards-compatibility issue, but
in practice ``sys.prefix`` is sometimes used.)
written to assume the other meaning for ``sys.prefix``. (Such
software should preferably be using the APIs in the ``site`` and
``sysconfig`` modules to answer these questions rather than using
``sys.prefix`` directly, in which case there is no
backwards-compatibility issue, but in practice ``sys.prefix`` is
sometimes used.)
The `documentation`__ for ``sys.prefix`` describes it as "A string
giving the site-specific directory prefix where the platform
@ -413,11 +408,11 @@ __ http://docs.python.org/dev/library/sys.html#sys.prefix
This PEP currently proposes to leave ``sys.prefix`` pointing to the
base system installation (which is where the standard library and
header files are found), and introduce a new value in ``sys``
(``sys.site_prefix``) to point to the prefix for
``site-packages``. This maintains the documented semantics of
``sys.prefix``, but risks breaking isolation if third-party code uses
``sys.prefix`` rather than ``sys.site_prefix`` or the appropriate
``site`` API to find site-packages directories.
(``sys.site_prefix``) to point to the prefix for ``site-packages``.
This maintains the documented semantics of ``sys.prefix``, but risks
breaking isolation if third-party code uses ``sys.prefix`` rather than
``sys.site_prefix`` or the appropriate ``site`` API to find
site-packages directories.
The most notable case is probably `setuptools`_ and its fork
`distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but
@ -426,7 +421,7 @@ for pre-flight checking where ``pth`` files can usefully be placed.
It would be trivial to modify these tools (currently only
`distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and
fall back to ``sys.prefix`` if it doesn't exist (for earlier versions
of Python). If Distribute is modified in this way and released before
of Python). If Distribute is modified in this way and released before
Python 3.3 is released with the ``venv`` module, there would be no
likely reason for an older version of Distribute to ever be installed
in a virtual environment.
@ -434,9 +429,9 @@ in a virtual environment.
In terms of other third-party usage, a `Google Code Search`_ turns up
what appears to be a roughly even mix of usage between packages using
``sys.prefix`` to build up a site-packages path and packages using it
to e.g. eliminate the standard-library from code-execution
tracing. Either choice that's made here will require one or the other
of these uses to be updated.
to e.g. eliminate the standard-library from code-execution tracing.
Either choice that's made here will require one or the other of these
uses to be updated.
.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
.. _distribute: http://packages.python.org/distribute/
@ -454,7 +449,7 @@ considerations in mind:
* Inasmuch as "site" has a meaning in Python, it means a combination
of Python version, standard library, and specific set of
site-packages. This is, fundamentally, what a venv is (although it
site-packages. This is, fundamentally, what a venv is (although it
shares the standard library with its "base" site).
* It is the Python ``site`` module which implements adding
@ -466,17 +461,17 @@ overloaded and of unclear meaning, and this usage will increase the
overload.
One proposed alternative is ``sys.venv_prefix``, which has the
advantage of being clearly related to the venv implementation. The
advantage of being clearly related to the venv implementation. The
downside of this proposal is that it implies the attribute is only
useful/relevant when in a venv and should be absent or ``None`` when
not in a venv. This imposes an unnecessary extra burden on code using
the attribute: ``sys.venv_prefix if sys.venv_prefix else
sys.prefix``. The prefix attributes are more usable and general if
they are always present and set, and split by meaning (stdlib vs
site-packages, roughly), rather than specifically tied to venv. Also,
third-party code should be encouraged to not know or care whether it
is running in a virtual environment or not; this option seems to work
against that goal.
not in a venv. This imposes an unnecessary extra burden on code using
the attribute: ``sys.venv_prefix if sys.venv_prefix else sys.prefix``.
The prefix attributes are more usable and general if they are always
present and set, and split by meaning (stdlib vs site-packages,
roughly), rather than specifically tied to venv. Also, third-party
code should be encouraged to not know or care whether it is running in
a virtual environment or not; this option seems to work against that
goal.
Another option would be ``sys.local_prefix``, which has both the
advantage and disadvantage, depending on perspective, that it
@ -489,11 +484,11 @@ Why not modify sys.prefix?
As discussed above under `Backwards Compatibility`_, this PEP proposes
to add ``sys.site_prefix`` as "the prefix relative to which
site-package directories are found". This maintains compatibility with
the documented meaning of ``sys.prefix`` (as the location relative to
which the standard library can be found), but means that code assuming
that site-packages directories are found relative to ``sys.prefix``
will not respect the virtual environment correctly.
site-package directories are found". This maintains compatibility
with the documented meaning of ``sys.prefix`` (as the location
relative to which the standard library can be found), but means that
code assuming that site-packages directories are found relative to
``sys.prefix`` will not respect the virtual environment correctly.
Since it is unable to modify ``distutils``/``sysconfig``,
`virtualenv`_ is forced to instead re-point ``sys.prefix`` at the
@ -506,7 +501,7 @@ does this and it doesn't appear to have caused major problems with
existing code.
Another argument in favor of this is that it would be preferable to
err on the side of greater, rather than lesser, isolation. Changing
err on the side of greater, rather than lesser, isolation. Changing
``sys.prefix`` to point to the virtual environment and introducing a
new ``sys.base_prefix`` attribute would err on the side of greater
isolation in the face of existing code's use of ``sys.prefix``.
@ -515,13 +510,14 @@ isolation in the face of existing code's use of ``sys.prefix``.
What about include files?
-------------------------
For example, ZeroMQ installs zmq.h and zmq_utils.h in $VE/include,
whereas SIP (part of PyQt4) installs sip.h by default in
$VE/include/pythonX.Y. With virtualenv, everything works because the
PythonX.Y include is symlinked, so everything that's needed is in
$VE/include. At the moment the reference implementation doesn't do
anything with include files, besides creating the include directory;
this might need to change, to copy/symlink $VE/include/pythonX.Y.
For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in
``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default
in ``$VE/include/pythonX.Y``. With virtualenv, everything works
because the PythonX.Y include is symlinked, so everything that's
needed is in ``$VE/include``. At the moment the reference
implementation doesn't do anything with include files, besides
creating the include directory; this might need to change, to
copy/symlink ``$VE/include/pythonX.Y``.
As in Python there's no abstraction for a site-specific include
directory, other than for platform-specific stuff, then the user
@ -530,26 +526,27 @@ want should be found in one of just two locations, with sysconfig
labels "include" & "platinclude".
There's another issue: what if includes are Python-version-specific?
For example, SIP installs by default into $VE/include/pythonX.Y rather
than $VE/include, presumably because there's version-specific stuff in
there - but even if that's not the case with SIP, it could be the case
with some other package. And the problem that gives is that you can't
just symlink the include/pythonX.Y directory, but actually have to
provide a writable directory and symlink/copy the contents from the
system include/pythonX.Y. Of course this is not hard to do, but it
does seem inelegant. OTOH it's really because there's no supporting
concept in Python/sysconfig.
For example, SIP installs by default into ``$VE/include/pythonX.Y``
rather than ``$VE/include``, presumably because there's
version-specific stuff in there - but even if that's not the case with
SIP, it could be the case with some other package. And the problem
that gives is that you can't just symlink the ``include/pythonX.Y``
directory, but actually have to provide a writable directory and
symlink/copy the contents from the system ``include/pythonX.Y``. Of
course this is not hard to do, but it does seem inelegant. OTOH it's
really because there's no supporting concept in ``Python/sysconfig``.
Interface with packaging tools
------------------------------
Some work will be needed in packaging tools (Python 3.3 packaging,
Distribute) to support implementation of this PEP. For example:
Distribute) to support implementation of this PEP. For example:
* How Distribute and packaging use sys.prefix and/or sys.site_prefix. Clearly,
in practice we'll need to use Distribute for a while, until packages have
migrated over to usage of setup.cfg.
* How Distribute and packaging use ``sys.prefix`` and/or
``sys.site_prefix``. Clearly, in practice we'll need to use
Distribute for a while, until packages have migrated over to usage
of setup.cfg.
* How packaging and Distribute set up shebang lines in scripts which they
install in virtual environments.
@ -560,10 +557,10 @@ Testability and Source Build Issues
Currently in the reference implementation, virtual environments must
be created with an installed Python, rather than a source build, as
the base installation. In order to be able to fully test the ``venv``
the base installation. In order to be able to fully test the ``venv``
module in the Python regression test suite, some anomalies in how
sysconfig data is configured in source builds will need to be
removed. For example, sysconfig.get_paths() in a source build gives
sysconfig data is configured in source builds will need to be removed.
For example, ``sysconfig.get_paths()`` in a source build gives
(partial output)::
{
@ -581,11 +578,11 @@ Need for ``install_name_tool`` on OSX?
--------------------------------------
`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode
developer tools, to modify the copied executable on OSX. We need input
from OSX developers on whether this is actually necessary in this
PEP's implementation of virtual environments, and if so, if there is
an alternative to ``install_name_tool`` that would allow ``venv`` to
not require that Xcode is installed.
developer tools, to modify the copied executable on OSX. We need
input from OSX developers on whether this is actually necessary in
this PEP's implementation of virtual environments, and if so, if there
is an alternative to ``install_name_tool`` that would allow ``venv``
to not require that Xcode is installed.
.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168
@ -595,9 +592,9 @@ Activation and Utility Scripts
Virtualenv provides shell "activation" scripts as a user convenience,
to put the virtual environment's Python binary first on the shell
PATH. This is a maintenance burden, as separate activation scripts
need to be provided and maintained for every supported shell. For this
reason, this PEP proposes to leave such scripts to be provided by
PATH. This is a maintenance burden, as separate activation scripts
need to be provided and maintained for every supported shell. For
this reason, this PEP proposes to leave such scripts to be provided by
third-party extensions; virtual environments created by the core
functionality would be used by directly invoking the environment's
Python binary or scripts.
@ -634,9 +631,9 @@ Reference Implementation
The in-progress reference implementation is found in `a clone of the
CPython Mercurial repository`_. To test it, build and install it (the
virtual environment tool currently does not run from a source
tree). From the installed Python, run ``bin/pyvenv
/path/to/new/virtualenv`` to create a virtual environment.
virtual environment tool currently does not run from a source tree).
From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv``
to create a virtual environment.
The reference implementation (like this PEP!) is a work in progress.