PEP 704: Require virtual environments by default for installers (#2964)

Enforce the use of virtual environments and establish a conventional
name for the virtual environment directory.

Co-authored-by: Pradyun Gedam <pradyunsg@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Zak Johnson <me@zakj.net>

.github/CODEOWNERS

@@ -584,6 +584,7 @@ pep-0700.rst  @pfmoore
 pep-0701.rst  @pablogsal @isidentical @lysnikolaou
 pep-0702.rst  @jellezijlstra
 pep-0703.rst  @ambv
+pep-0704.rst  @brettcannon @pradyunsg
 # ...
 # pep-0754.txt
 # ...

pep-0704.rst

@@ -0,0 +1,172 @@
+PEP: 704
+Title: Require virtual environments by default for package installers
+Author: Pradyun Gedam <pradyunsg@gmail.com>
+Sponsor: Brett Cannon <brett@python.org>
+PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
+Discussions-To: https://discuss.python.org/t/22846
+Status: Draft
+Type: Standards Track
+Topic: Packaging
+Content-Type: text/x-rst
+Created: 16-Jan-2023
+Post-History: `16-Jan-2023 <https://discuss.python.org/t/22846>`__
+
+
+Abstract
+========
+
+This PEP recommends that package installers like ``pip``  require a virtual environment by default on Python 3.13+.
+
+
+Motivation
+==========
+
+Python virtual environments are an essential part of the development workflow for Python. However, they require extra effort since they are an opt-in feature, and requires users to either:
+
+- take explicit steps to activate/deactivate a virtual environment
+- use :samp:`{<path-to-venv>}/{<bin-path>}/{<executable>}` to run files
+
+For new users, things will seemingly work correctly when virtual environments are not used -— until they don't. Further, activating a virtual environment uses slightly different syntax and mechanisms on different platforms. This complicates the introduction to virtual environments, since now information and context about how/why they are useful needs to explained to justify adding additional steps to the workflow.
+
+It also creates a scope for mistakes, since users need to remember to activate the virtual environment before running an installer like ``pip`` or configure those installers to error out. On certain Linux distributions, forgetting to do so can result in the installer modifying files that are owned by the operating system (which is partially mitigated by :pep:`668` for distributions that opt-in to marking their environments accordingly).
+
+
+Rationale
+=========
+
+Changing the default behaviour of installers like ``pip`` to require a virtual environment to be active would:
+
+- make it easier for new users to get started with Python (since there's a consistent experience and virtual environments are understood as a thing you're required to use)
+- reduce the scope for accidental installation issues for all users by default (by explicitly flagging when you're not using a virtual environment).
+
+Setting up a convention of placing the virtual environment in-tree in a directory named ``.venv`` removes a decision point for common workflows and creates a clear convention within the ecosystem.
+
+Specification
+=============
+
+Requiring a virtual environment by default
+------------------------------------------
+
+When a user runs an installer without an active virtual environment, the installer SHOULD print an error message and exit with a non-zero exit code.
+
+The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named ``.venv``, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
+
+See :ref:`704-implementation` for more details.
+
+Opting out of virtual environments
+----------------------------------
+
+The installer SHOULD also provide a explicit opt-in to disable this requirement, allowing an end user to use it outside of a virtual environment. If the installer does not provide this functionality, it SHOULD mention this in the error message and documentation page.
+
+Consistent timeline for the change
+----------------------------------
+
+Installers MAY choose to implement this default behaviour on any Python versions, but SHOULD implement it on Python 3.13 or newer.
+
+
+Backwards Compatibility
+=======================
+
+This PEP is backwards incompatible with workflows where users are using installers outside of virtual environments. Such users will be prompted with an error message and will need to either:
+
+- explicitly opt-in to running the installer outside of a virtual environment, or
+- create and use a virtual environment
+
+Users who are already using virtual environments will not be affected by this change.
+
+Workflow tools (which manage virtual environments for the user, under the hood) should be unaffected, since they should already be using a virtual environment for running the installer.
+
+
+Security Implications
+=====================
+
+This PEP does not introduce any new security implications.
+
+
+How to Teach This
+=================
+
+This PEP requires that new users create and use a virtual environment to get started with using Python packages. This is, however, a best practice, as `demonstrated <https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-virtual-environments>`__ by the section on "basics of how to install Python packages" in the Python Packaging User Guide, which explains how/what virtual environments are before discussing using ``pip``.
+
+
+Reference Implementation
+========================
+
+There is no reference implementation for this PEP. However, the proposed behaviour is largely already implemented in ``pip`` and can be activated by setting the ``PIP_REQUIRE_VENV`` environment variable to ``1``. (Leaving it unset results in the proposed opt-in behaviour of not requiring a virtual environment for installation.)
+
+
+.. _704-implementation:
+
+Implementation Notes
+====================
+
+Detecting an active virtual environment
+---------------------------------------
+
+:pep:`As discussed in PEP 668 <668#backwards-compatibility>`, the logic for robustly detecting a virtual environment is something like::
+
+    def is_virtual_environment():
+        return sys.base_prefix != sys.prefix or hasattr(sys, "real_prefix")
+
+Documentation on using a virtual environment
+--------------------------------------------
+
+Package installers are expected to provide a link to a documentation page in the error message.
+
+Ideally, such a documentation page would explain what virtual environments are, why they are required, and how to create and activate a virtual environment using ``venv``. It should include instructions for the most common shells and platforms.
+
+Such a documentation page should be made available in the `Python Packaging User Guide <https://packaging.python.org>`__ to reduce duplicated effort across installers for covering this topic.
+
+Rejected Ideas
+==============
+
+Do not specify a name for the virtual environment directory
+-----------------------------------------------------------
+
+Using a consistent name for the virtual environment directory is important for a few reasons:
+
+1. It makes it easier for users to find the virtual environment directory, and to activate it.
+2. It removes a decision point for new users, since they do not need to decide on a name for the virtual environment directory.
+3. It creates a clear convention within the ecosystem, which makes it easier for users to find documentation.
+4. It ensures consistency across different tools, so that differences in the error messages do not confuse users.
+
+Use a different name for the virtual environment directory
+----------------------------------------------------------
+
+Functionally, the directory name does not matter much as long as there is a single consistent suggestion.
+
+The name ``.venv`` was picked since it:
+
+1. does not conflict with any valid Python import name
+2. does not conflict ``venv`` module in the standard library
+3. has pre-existing usage in the Python community
+4. has support for auto-detection in common text editors
+5. can be typed without modifier keys on common keyboard layouts
+
+Do not couple tooling behaviour with a Python version
+-----------------------------------------------------
+
+This PEP creates a coupling between the behaviour of installers and the Python version.
+
+This is already a rollout mechanism being used for behaviour changes in the installation tooling. For example, ``pip`` on Python 3.11 will use ``importlib.metadata`` instead of ``pkg_resources`` for parsing/fetching package metadata, and ``sysconfig`` instead of ``distutils.sysconfig`` for getting the paths to unpack wheels into.
+
+The difference with those cases is that they're supposed to be largely transparent to end users. This PEP is proposing a behaviour change that is not transparent to end users, and requires them to take action.
+
+The primary benefit of this is that it allows for redistributors to adapt their tooling in time for the new Python version and provides a clear and consistent point for change across the ecosystem. It also puts a clear deadline on when the default behaviour will consistently require a virtual environment by default (once Python 3.12 goes end-of-life).
+
+The primary issue with this approach is that it enforces a behaviour change on users when they upgrade to a new Python version, which can hamper the adoption of a new Python version. However, this is a migration/upgrade for existing users and it is a common expectation that *some* changes will be needed for migration/upgrades.
+
+The author of this PEP believes that the benefits of applying this consistently throughout the ecosystem with a deadline outweigh the drawbacks of enforcing a best-practice on users when they upgrade.
+
+
+Open Issues
+===========
+
+None.
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.