diff --git a/pep-0582.rst b/pep-0582.rst new file mode 100644 index 000000000..76cd9a660 --- /dev/null +++ b/pep-0582.rst @@ -0,0 +1,204 @@ +PEP: 582 +Title: Python local packages directory +Version: $Revision$ +Last-Modified: $Date$ +Author: Kushal Das , Steve Dower , + Donald Stufft , Nick Coghlan +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 16-May-2018 +Python-Version: 3.8 + + +Abstract +======== + +This PEP proposes to add to Python a mechanism to automatically recognize a +``__pypackages__`` directory and prefer importing packages installed in this +location over user or global site-packages. This will avoid the steps to create, +activate or deactivate "virtual environments". Python will use the +``__pypackages__`` from the base directory of the script when present. + + + +Motivation +========== + +Python virtual environments have become an essential part of development and +teaching workflow in the community, but at the same time, they create a barrier +to entry for many. The following are a few of the issues people run into while +being introduced to Python (or programming for the first time). + +- How virtual environments work is a lot of information for anyone new. It takes + a lot of extra time and effort to explain them. + +- Different platforms and shell environments require different sets of commands + to activate the virtual environments. Any workshop or teaching environment with + people coming with different operating systems installed on their laptops create a + lot of confusion among the participants. + +- Virtual environments need to be activated on each opened terminal. If someone + creates/opens a new terminal, that by default does not get the same environment + as in a previous terminal with virtual environment activated. + + +Specification +============= + +When the Python binary is executed, it attempts to determine its prefix (as +stored in ``sys.prefix``), which is then used to find the standard library and +other key files, and by the ``site`` module to determine the location of the +``site-package`` directories. Currently the prefix is found -- assuming +``PYTHONHOME`` is not set -- by first walking up the filesystem tree looking for +a marker file (``os.py``) that signifies the presence of the standard library, +and if none is found, falling back to the build-time prefix hard coded in the +binary. The result of this process is the contents of ``sys.path`` - a list of +locations that the Python import system will search for modules. + +This PEP proposes to add a new step in this process. If a ``__pypackages__`` +directory is found in the current working directory, then it will be included in +``sys.path`` after the current working directory and just before the system +site-packages. This way, if the Python executable starts in the given project +directory, it will automatically find all the dependencies inside of +``__pypackages__``. + +In case of Python scripts, Python will try to find ``__pypackages__`` in the +same directory as the script. If found (along with the current Python version +directory inside), then it will be used, otherwise Python will behave as it does +currently. + +If any package management tool finds the same ``__pypackages__`` directory in +the current working directory, it will install any packages there and also +create it if required based on Python version. + +Projects that use a source management system can include a ``__pypackages__`` +directory (empty or with e.g. a file like ``.gitignore``). After doing a fresh +check out the source code, a tool like ``pip`` can be used to install the +required dependencies directly into this directory. + +Example +------- + +The following shows an example project directory structure, and different ways +the Python executable and any script will behave. + +:: + + foo + __pypackages__ + 3.8 + lib + bottle + myscript.py + + /> python foo/myscript.py + sys.path[0] == 'foo' + sys.path[1] == 'foo/__pypackages__/3.8/lib' + + + cd foo + + foo> /usr/bin/ansible + #! /usr/bin/env python3 + foo> python /usr/bin/ansible + + foo> python myscript.py + + foo> python + sys.path[0] == '.' + sys.path[1] == './__pypackages__/3.8/lib' + + foo> python -m bottle + +We have a project directory called ``foo`` and it has a ``__pypackages__`` +inside of it. We have ``bottle`` installed in that +``__pypackages__/3.8/lib``, and have a ``myscript.py`` file inside of the +project directory. We have used whatever tool we generally use to install ``bottle`` +in that location. + +For invoking a script, Python will try to find a ``__pypackages__`` inside of +the directory that the script resides[1]_, ``/usr/bin``. The same will happen +in case of the last example, where we are executing ``/usr/bin/ansible`` from +inside of the ``foo`` directory. In both cases, it will **not** use the +``__pypackages__`` in the current working directory. + +Similarly, if we invoke ``myscript.py`` from the first example, it will use the +``__pypackages__ `` directory that was in the ``foo`` directory. + +If we go inside of the ``foo`` directory and start the Python executable (the +interpreter), it will find the ``__pypackages__`` directory inside of the +current working directory and use it in the ``sys.path``. The same happens if we +try to use the ``-m`` and use a module. In our example, ``bottle`` module will +be found inside of the ``__pypackages__`` directory. + +The above two examples are only cases where ``__pypackages__`` from current +working directory is used. + +In another example scenario, a trainer of a Python class can say "Today we are +going to learn how to use Twisted! To start, please checkout our example +project, go to that directory, and then run ``python3 -m pip install twisted``." + +That will install Twisted into a directory separate from ``python3``. There's no +need to discuss virtual environments, global versus user installs, etc. as the +install will be local by default. The trainer can then just keep telling them to +use ``python3`` without any activation step, etc. + + +.. [1]_: In the case of symlinks, it is the directory where the actual script + resides, not the symlink pointing to the script + + +Security Considerations +======================= + +While executing a Python script, it will not consider the ``__pypackages__`` in +the current directory, instead if there is a ``__pypackages__`` directory in the +same path of the script, that will be used. + +For example, if we execute ``python /usr/share/myproject/fancy.py`` from the +``/tmp`` dirctory and if there is a ``__pypackages__`` directory inside of +``/usr/share/myproject/`` directory, it will be used. Any potential +``__pypackages__`` directory in ``/tmp`` will be ignored. + + +Backwards Compatibility +======================= + +This does not affect any older version of Python implementation. + +Impact on other Python implementations +-------------------------------------- + +Other Python implementations will need to replicate the new behavior of the +interpreter bootstrap, including locating the ``__pypackages__`` directory and +adding it the ``sys.path`` just before site packages, if it is present. + + +Reference Implementation +======================== + +`Here `_ is a PoC +implementation (in the ``pypackages`` branch). + + +Rejected Ideas +============== + +``__pylocal__`` and ``python_modules``. + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 80 + coding: utf-8 + End: