206 lines
7.5 KiB
ReStructuredText
206 lines
7.5 KiB
ReStructuredText
PEP: 582
|
|
Title: Python local packages directory
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Kushal Das <mail@kushaldas.in>, Steve Dower <steve.dower@python.org>,
|
|
Donald Stufft <donald@stufft.io>, Nick Coghlan <ncoghlan@gmail.com>
|
|
Discussions-To: https://discuss.python.org/t/pep-582-python-local-packages-directory/963/
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 16-May-2018
|
|
Python-Version: 3.8
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes to add to Python a mechanism to automatically recognize a
|
|
``__pypackages__`` directory and prefer importing packages installed in this
|
|
location over user or global site-packages. This will avoid the steps to create,
|
|
activate or deactivate "virtual environments". Python will use the
|
|
``__pypackages__`` from the base directory of the script when present.
|
|
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
Python virtual environments have become an essential part of development and
|
|
teaching workflow in the community, but at the same time, they create a barrier
|
|
to entry for many. The following are a few of the issues people run into while
|
|
being introduced to Python (or programming for the first time).
|
|
|
|
- How virtual environments work is a lot of information for anyone new. It takes
|
|
a lot of extra time and effort to explain them.
|
|
|
|
- Different platforms and shell environments require different sets of commands
|
|
to activate the virtual environments. Any workshop or teaching environment with
|
|
people coming with different operating systems installed on their laptops create a
|
|
lot of confusion among the participants.
|
|
|
|
- Virtual environments need to be activated on each opened terminal. If someone
|
|
creates/opens a new terminal, that by default does not get the same environment
|
|
as in a previous terminal with virtual environment activated.
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
When the Python binary is executed, it attempts to determine its prefix (as
|
|
stored in ``sys.prefix``), which is then used to find the standard library and
|
|
other key files, and by the ``site`` module to determine the location of the
|
|
``site-package`` directories. Currently the prefix is found -- assuming
|
|
``PYTHONHOME`` is not set -- by first walking up the filesystem tree looking for
|
|
a marker file (``os.py``) that signifies the presence of the standard library,
|
|
and if none is found, falling back to the build-time prefix hard coded in the
|
|
binary. The result of this process is the contents of ``sys.path`` - a list of
|
|
locations that the Python import system will search for modules.
|
|
|
|
This PEP proposes to add a new step in this process. If a ``__pypackages__``
|
|
directory is found in the current working directory, then it will be included in
|
|
``sys.path`` after the current working directory and just before the system
|
|
site-packages. This way, if the Python executable starts in the given project
|
|
directory, it will automatically find all the dependencies inside of
|
|
``__pypackages__``.
|
|
|
|
In case of Python scripts, Python will try to find ``__pypackages__`` in the
|
|
same directory as the script. If found (along with the current Python version
|
|
directory inside), then it will be used, otherwise Python will behave as it does
|
|
currently.
|
|
|
|
If any package management tool finds the same ``__pypackages__`` directory in
|
|
the current working directory, it will install any packages there and also
|
|
create it if required based on Python version.
|
|
|
|
Projects that use a source management system can include a ``__pypackages__``
|
|
directory (empty or with e.g. a file like ``.gitignore``). After doing a fresh
|
|
check out the source code, a tool like ``pip`` can be used to install the
|
|
required dependencies directly into this directory.
|
|
|
|
Example
|
|
-------
|
|
|
|
The following shows an example project directory structure, and different ways
|
|
the Python executable and any script will behave.
|
|
|
|
::
|
|
|
|
foo
|
|
__pypackages__
|
|
3.8
|
|
lib
|
|
bottle
|
|
myscript.py
|
|
|
|
/> python foo/myscript.py
|
|
sys.path[0] == 'foo'
|
|
sys.path[1] == 'foo/__pypackages__/3.8/lib'
|
|
|
|
|
|
cd foo
|
|
|
|
foo> /usr/bin/ansible
|
|
#! /usr/bin/env python3
|
|
foo> python /usr/bin/ansible
|
|
|
|
foo> python myscript.py
|
|
|
|
foo> python
|
|
sys.path[0] == '.'
|
|
sys.path[1] == './__pypackages__/3.8/lib'
|
|
|
|
foo> python -m bottle
|
|
|
|
We have a project directory called ``foo`` and it has a ``__pypackages__``
|
|
inside of it. We have ``bottle`` installed in that
|
|
``__pypackages__/3.8/lib``, and have a ``myscript.py`` file inside of the
|
|
project directory. We have used whatever tool we generally use to install ``bottle``
|
|
in that location.
|
|
|
|
For invoking a script, Python will try to find a ``__pypackages__`` inside of
|
|
the directory that the script resides[1]_, ``/usr/bin``. The same will happen
|
|
in case of the last example, where we are executing ``/usr/bin/ansible`` from
|
|
inside of the ``foo`` directory. In both cases, it will **not** use the
|
|
``__pypackages__`` in the current working directory.
|
|
|
|
Similarly, if we invoke ``myscript.py`` from the first example, it will use the
|
|
``__pypackages__`` directory that was in the ``foo`` directory.
|
|
|
|
If we go inside of the ``foo`` directory and start the Python executable (the
|
|
interpreter), it will find the ``__pypackages__`` directory inside of the
|
|
current working directory and use it in the ``sys.path``. The same happens if we
|
|
try to use the ``-m`` and use a module. In our example, ``bottle`` module will
|
|
be found inside of the ``__pypackages__`` directory.
|
|
|
|
The above two examples are only cases where ``__pypackages__`` from current
|
|
working directory is used.
|
|
|
|
In another example scenario, a trainer of a Python class can say "Today we are
|
|
going to learn how to use Twisted! To start, please checkout our example
|
|
project, go to that directory, and then run ``python3 -m pip install twisted``."
|
|
|
|
That will install Twisted into a directory separate from ``python3``. There's no
|
|
need to discuss virtual environments, global versus user installs, etc. as the
|
|
install will be local by default. The trainer can then just keep telling them to
|
|
use ``python3`` without any activation step, etc.
|
|
|
|
|
|
.. [1]_: In the case of symlinks, it is the directory where the actual script
|
|
resides, not the symlink pointing to the script
|
|
|
|
|
|
Security Considerations
|
|
=======================
|
|
|
|
While executing a Python script, it will not consider the ``__pypackages__`` in
|
|
the current directory, instead if there is a ``__pypackages__`` directory in the
|
|
same path of the script, that will be used.
|
|
|
|
For example, if we execute ``python /usr/share/myproject/fancy.py`` from the
|
|
``/tmp`` directory and if there is a ``__pypackages__`` directory inside of
|
|
``/usr/share/myproject/`` directory, it will be used. Any potential
|
|
``__pypackages__`` directory in ``/tmp`` will be ignored.
|
|
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
This does not affect any older version of Python implementation.
|
|
|
|
Impact on other Python implementations
|
|
--------------------------------------
|
|
|
|
Other Python implementations will need to replicate the new behavior of the
|
|
interpreter bootstrap, including locating the ``__pypackages__`` directory and
|
|
adding it the ``sys.path`` just before site packages, if it is present.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
`Here <https://github.com/kushaldas/cpython/tree/pypackages>`_ is a PoC
|
|
implementation (in the ``pypackages`` branch).
|
|
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
``__pylocal__`` and ``python_modules``.
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 80
|
|
coding: utf-8
|
|
End:
|