python-peps/pep-0582.rst

PEP: 582
Title: Python local packages directory
Version: $Revision$
Last-Modified: $Date$
Author: Kushal Das <mail@kushaldas.in>, Steve Dower <steve.dower@python.org>,
        Donald Stufft <donald@stufft.io>, Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-582-python-local-packages-directory/963/
Status: Draft
Type: Standards Track
Topic: Packaging
Content-Type: text/x-rst
Created: 16-May-2018
Python-Version: 3.8


Abstract
========

This PEP proposes to add to Python a mechanism to automatically recognize a
``__pypackages__`` directory and prefer importing packages installed in this
location over user or global site-packages. This will avoid the steps to create,
activate or deactivate "virtual environments". Python will use the
``__pypackages__`` from the base directory of the script when present.



Motivation
==========

Python virtual environments have become an essential part of development and
teaching workflow in the community, but at the same time, they create a barrier
to entry for many. The following are a few of the issues people run into while
being introduced to Python (or programming for the first time).

- How virtual environments work is a lot of information for anyone new. It takes
  a lot of extra time and effort to explain them.

- Different platforms and shell environments require different sets of commands
  to activate the virtual environments. Any workshop or teaching environment with
  people coming with different operating systems installed on their laptops create a
  lot of confusion among the participants.

- Virtual environments need to be activated on each opened terminal. If someone
  creates/opens a new terminal, that by default does not get the same environment
  as in a previous terminal with virtual environment activated.


Specification
=============

When the Python binary is executed, it attempts to determine its prefix (as
stored in ``sys.prefix``), which is then used to find the standard library and
other key files, and by the ``site`` module to determine the location of the
``site-package`` directories.  Currently the prefix is found -- assuming
``PYTHONHOME`` is not set -- by first walking up the filesystem tree looking for
a marker file (``os.py``) that signifies the presence of the standard library,
and if none is found, falling back to the build-time prefix hard coded in the
binary. The result of this process is the contents of ``sys.path`` - a list of
locations that the Python import system will search for modules.

This PEP proposes to add a new step in this process. If a ``__pypackages__``
directory is found in the current working directory, then it will be included in
``sys.path`` after the current working directory and just before the system
site-packages. This way, if the Python executable starts in the given project
directory, it will automatically find all the dependencies inside of
``__pypackages__``.

In case of Python scripts, Python will try to find ``__pypackages__`` in the
same directory as the script. If found (along with the current Python version
directory inside), then it will be used, otherwise Python will behave as it does
currently.

If any package management tool finds the same ``__pypackages__`` directory in
the current working directory, it will install any packages there and also
create it if required based on Python version.

Projects that use a source management system can include a ``__pypackages__``
directory (empty or with e.g. a file like ``.gitignore``). After doing a fresh
check out the source code, a tool like ``pip`` can be used to install the
required dependencies directly into this directory.

Example
-------

The following shows an example project directory structure, and different ways
the Python executable and any script will behave.

::

    foo
        __pypackages__
            3.8
                lib
                    bottle
        myscript.py

    /> python foo/myscript.py
    sys.path[0] == 'foo'
    sys.path[1] == 'foo/__pypackages__/3.8/lib'


    cd foo

    foo> /usr/bin/ansible
        #! /usr/bin/env python3
    foo> python /usr/bin/ansible

    foo> python myscript.py

    foo> python
    sys.path[0] == '.'
    sys.path[1] == './__pypackages__/3.8/lib'

    foo> python -m bottle

We have a project directory called ``foo`` and it has a ``__pypackages__``
inside of it. We have ``bottle`` installed in that
``__pypackages__/3.8/lib``, and have a ``myscript.py`` file inside of the
project directory. We have used whatever tool we generally use to install ``bottle``
in that location.

For invoking a script, Python will try to find a ``__pypackages__`` inside of
the directory that the script resides[1]_, ``/usr/bin``.  The same will happen
in case of the last example, where we are executing ``/usr/bin/ansible`` from
inside of the ``foo`` directory. In both cases, it will **not** use the
``__pypackages__`` in the current working directory.

Similarly, if we invoke ``myscript.py`` from the first example, it will use the
``__pypackages__`` directory that was in the ``foo`` directory.

If we go inside of the ``foo`` directory and start the Python executable (the
interpreter), it will find the ``__pypackages__`` directory inside of the
current working directory and use it in the ``sys.path``. The same happens if we
try to use the ``-m`` and use a module. In our example, ``bottle`` module will
be found inside of the ``__pypackages__`` directory.

The above two examples are only cases where ``__pypackages__`` from current
working directory is used.

In another example scenario, a trainer of a Python class can say "Today we are
going to learn how to use Twisted! To start, please checkout our example
project, go to that directory, and then run ``python3 -m pip install twisted``."

That will install Twisted into a directory separate from ``python3``. There's no
need to discuss virtual environments, global versus user installs, etc. as the
install will be local by default. The trainer can then just keep telling them to
use ``python3`` without any activation step, etc.


.. [1]_: In the case of symlinks, it is the directory where the actual script
   resides, not the symlink pointing to the script


Security Considerations
=======================

While executing a Python script, it will not consider the ``__pypackages__`` in
the current directory, instead if there is a ``__pypackages__`` directory in the
same path of the script, that will be used.

For example, if we execute ``python /usr/share/myproject/fancy.py`` from the
``/tmp`` directory and  if there is a ``__pypackages__`` directory inside of
``/usr/share/myproject/`` directory, it will be used. Any potential
``__pypackages__`` directory in ``/tmp`` will be ignored.


Backwards Compatibility
=======================

This does not affect any older version of Python implementation.

Impact on other Python implementations
--------------------------------------

Other Python implementations will need to replicate the new behavior of the
interpreter bootstrap, including locating the ``__pypackages__`` directory and
adding it the ``sys.path`` just before site packages, if it is present.


Reference Implementation
========================

`Here <https://github.com/kushaldas/cpython/tree/pypackages>`_ is a PoC
implementation (in the ``pypackages`` branch).


Rejected Ideas
==============

``__pylocal__`` and ``python_modules``.


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 80
   coding: utf-8
   End: