PEP 662: Editable installs via virtual wheels (#1977)
This commit is contained in:
Bernát Gábor
GitHub
parent
c69fdbf5e1
commit
0bc330a6f1
|
|
@ -516,6 +516,7 @@ pep-0658.rst @brettcannon
|
|||
pep-0659.rst @markshannon
|
||||
pep-0660.rst @pfmoore
|
||||
pep-0661.rst @taleinat
|
||||
pep-0662.rst @brettcannon
|
||||
# ...
|
||||
# pep-0666.txt
|
||||
# ...
|
||||
|
|
|
|||
|
|
@ -0,0 +1,399 @@
|
|||
PEP: 662
|
||||
Title: Editable installs via virtual wheels
|
||||
Author: Bernát Gábor <gaborjbernat@gmail.com>
|
||||
Sponsor: Brett Cannon <brett@python.org>
|
||||
Discussions-To: https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat/9071
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 28-May-2021
|
||||
Post-History:
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This document describes extensions to the build backend and frontend
|
||||
communication (as introduced by :pep:`517`) to allow projects to be installed in
|
||||
editable mode by introducing virtual wheels.
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
During development, many Python users prefer to install their libraries so that
|
||||
changes to the underlying source code and resources are automatically reflected
|
||||
in subsequent interpreter invocations without an additional installation step.
|
||||
This mode is usually called "development mode" or "editable installs".
|
||||
Currently, there is no standardized way to accomplish this, as it was explicitly
|
||||
left out of :pep:`517` due to the complexity of the actual observed behaviors.
|
||||
|
||||
At the moment, users can achieve this in a few ways, neither of them being a
|
||||
standard:
|
||||
|
||||
- For just Python code by adding the relevant source directories to
|
||||
``sys.path`` (configurable from the command line interface via the
|
||||
``PYTHONPATH`` environment variable). Note in this case, the users have to
|
||||
install the project dependencies themselves, and entry points or project
|
||||
metadata are not generated.
|
||||
|
||||
- setuptools_ provides the `setup.py develop`_ mechanism: that installs a
|
||||
``pth`` file that injects the project root onto the ``sys.path`` at
|
||||
interpreter startup time, generates the project metadata, and also installs
|
||||
project dependencies. pip_ exposes calling this mechanism via the
|
||||
`pip install -e <project_directory>`_ command-line interface.
|
||||
|
||||
- flit_ provides the `flit install --symlink`_ command that symlinks the
|
||||
project files into the interpreters ``purelib`` folder, generates the
|
||||
project metadata, and also installs dependencies. Note, this allows
|
||||
supporting resource files too.
|
||||
|
||||
As these examples shows an editable install can be achieved in multiple ways
|
||||
and at the moment there's no standard way of doing it. Furthermore, it's not
|
||||
clear whose responsibility it is to achieve and define what an editable
|
||||
installation is:
|
||||
|
||||
1. allow the build backend to define and materialize it,
|
||||
2. allow the build frontend to define and materialize it,
|
||||
3. explicitly define and standardize one method from the possible options.
|
||||
|
||||
The author of this PEP believes there's no one size fits all solution here,
|
||||
each method of achieving editable effect has its pros and cons. Therefore
|
||||
this PEP rejects option three as it's unlikely for the community to agree on a
|
||||
single solution. Therefore, question remains as to whether the frontend or the
|
||||
build backend should own this responsibility. :pep:`660` proposes the build
|
||||
backend to own this, while the current PEP proposes the frontend.
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
:pep:`517` deferred "Editable installs" because this would have delayed further
|
||||
its adoption, and there wasn't an agreement on how editable installs should be
|
||||
achieved. Due to the popularity of the setuptools_ and pip_ projects, the status
|
||||
quo prevailed, and the backend could achieve editable mode by providing a
|
||||
``setup.py develop`` implementation, which the user could trigger via `pip
|
||||
install -e <project_directory>`_. By defining an editable interface between the
|
||||
build backend and frontend, we can eliminate the ``setup.py`` file and their
|
||||
current communication method.
|
||||
|
||||
Terminology and goals
|
||||
=====================
|
||||
|
||||
This PEP aims to delineate the frontend and the backend roles clearly and give
|
||||
the developers of each the maximum ability to provide valuable features to
|
||||
their users. In this proposal, the backend's role is to prepare the project for
|
||||
an editable installation, and then provide enough information to the frontend
|
||||
so that the frontend can manifest and enforce the editable installation.
|
||||
|
||||
The information the backend provides to the frontend is:
|
||||
|
||||
- the project metadata (as defined by :pep:`427` under ``.dist-info``),
|
||||
- the files to expose (formulated as a mapping of absolute source tree
|
||||
paths to relative target interpreter destination paths).
|
||||
|
||||
We refer to this set of information as the virtual wheel. This virtual wheel
|
||||
should contain all information a wheel contains, however it's not zipped and
|
||||
its installation will not be done by copying the files. The frontend's role is
|
||||
to take the virtual wheel and install the project in editable mode. The way it
|
||||
achieves this is entirely up to the frontend and is considered implementation
|
||||
detail.
|
||||
|
||||
The editable installation mode implies that the source code of the project
|
||||
being installed is available in a local directory. Once the project is
|
||||
installed in editable mode, some changes to the project code in the local
|
||||
source tree will become effective without the need for a new installation step.
|
||||
At a minimum, changes to the text of non-generated files that existed at the
|
||||
installation time should be reflected upon the subsequent import of the
|
||||
package.
|
||||
|
||||
Some kinds of changes, such as adding or modifying entry points or new
|
||||
dependencies, require a new installation step to become effective. These changes
|
||||
are typically made in build backend configuration files (such as
|
||||
``pyproject.toml``). This requirement is consistent with the general user
|
||||
expectation that such modifications will only become effective after
|
||||
re-installation.
|
||||
|
||||
While users expect editable installations to behave identically to standard
|
||||
installations, this may not always be possible and may be in tension with other
|
||||
user expectations. Depending on how a frontend implements the editable mode,
|
||||
some differences may be visible, such as the presence of additional files
|
||||
(compared to a typical installation), either in the source tree or the
|
||||
interpreter's installation path. Frontends should seek to minimize differences
|
||||
between the behavior of editable and standard installations and document known
|
||||
differences.
|
||||
|
||||
For reference, a non-editable installation works as follows:
|
||||
|
||||
#. The **developer** is using a tool, we'll call it here the **frontend**, to
|
||||
drive the project development (e.g., pip_). When the user wants to trigger a
|
||||
package build and installation of a project, they'll communicate with the
|
||||
**frontend**.
|
||||
|
||||
#. The frontend uses a **build frontend** to trigger the build of a wheel (e.g.,
|
||||
build_). The build frontend uses :pep:`517` to communicate with the **build
|
||||
backend** (e.g. setuptools_) - with the build backend installed into a
|
||||
:pep:`518` environment. Once invoked, the backend returns a wheel.
|
||||
|
||||
#. The frontend takes the wheel and feeds it to an **installer**
|
||||
(e.g.,`installer`_) to install the wheel into the target Python interpreter.
|
||||
|
||||
The Mechanism
|
||||
=============
|
||||
|
||||
This PEP adds two optional hooks to the :pep:`517` backend interface. One of the
|
||||
hooks is used to specify the build dependencies of an editable install. The
|
||||
other hook returns the necessary information via the build frontend the frontend
|
||||
needs to create an editable install.
|
||||
|
||||
``get_requires_for_build_editable``
|
||||
-----------------------------------
|
||||
|
||||
.. code::
|
||||
|
||||
def get_requires_for_build_editable(config_settings=None):
|
||||
...
|
||||
|
||||
This hook MUST return an additional sequence of strings containing :pep:`508`
|
||||
dependency specifications, above and beyond those specified in the
|
||||
``pyproject.toml`` file. The frontend must ensure that these dependencies are
|
||||
available in the build environment in which the ``build_editable`` hook is
|
||||
called.
|
||||
|
||||
If not defined, the default implementation is equivalent to returning ``[]``.
|
||||
|
||||
``build_editable``
|
||||
------------------
|
||||
|
||||
.. code::
|
||||
|
||||
def build_editable(config_settings=None):
|
||||
...
|
||||
|
||||
The function returns an object of type ``EditableInfo`` as defined below:
|
||||
|
||||
.. code::
|
||||
|
||||
from typing import Mapping, TypedDict
|
||||
|
||||
class SchemePaths(TypedDict, total=False):
|
||||
"""
|
||||
Files and folders that should be mapped:
|
||||
- key is the absolute source path
|
||||
- value is the relative path within the target interpreters prefix
|
||||
"""
|
||||
|
||||
purelib: Mapping[str, str]
|
||||
platlib: Mapping[str, str]
|
||||
headers: Mapping[str, str]
|
||||
scripts: Mapping[str, str]
|
||||
data: Mapping[str, str]
|
||||
|
||||
|
||||
class EditableInfo(TypedDict, total=True):
|
||||
version: int
|
||||
"""protocol version of the editable metadata, this PEP defines version 1"""
|
||||
|
||||
metadata_for_build_editable: str
|
||||
"""distribution information of the package as defined by PEP-491"""
|
||||
|
||||
paths: SchemePaths
|
||||
"""files to expose into the target interpreter"""
|
||||
|
||||
|
||||
The scheme paths map from project source absolute paths to target directory
|
||||
relative paths. We allow backends to change the project layout from the project
|
||||
source directory to what the interpreter will see by using the mapping.
|
||||
|
||||
For example if the backend returns ``"purelib": {"/me/project/src": ""}`` this
|
||||
would mean that expose all files and modules within ``/me/project/src`` at the
|
||||
root of the ``purelib`` path within the target interpreter.
|
||||
|
||||
Build frontend requirements
|
||||
---------------------------
|
||||
|
||||
The build frontend is responsible for setting up the environment for the build
|
||||
backend to generate the necessary information for an editable build. It's also
|
||||
responsible for communicating with the backend and receiving the
|
||||
``EditableInfo`` object. All recommendations from :pep:`517` for the build wheel
|
||||
hook applies here too.
|
||||
|
||||
Frontend requirements
|
||||
---------------------
|
||||
|
||||
The frontend is responsible for ensuring the ``.dist-info`` folder is available
|
||||
at runtime within the target interpreter for the ``importlib.metadata`` and
|
||||
``importlib.resources`` modules.
|
||||
|
||||
The frontend must ensure that all installation requirements specified in the
|
||||
distribution information files are installed as part of the editable
|
||||
installation into the target interpreter. Additionally, the user might also
|
||||
select additional ``extras`` groups that also should be installed as part of the
|
||||
editable installation.
|
||||
|
||||
The frontend also must generate entrypoints, which may be for the console or the
|
||||
GUI. Those entrypoints are defined by the distribution information files, which
|
||||
are generated during the editable installation process.
|
||||
|
||||
The frontend is responsible for generating the ``RECORD`` file based on the
|
||||
object the build backend returns and their chosen editable implementation. For
|
||||
this reason, the uninstallation of editables should not require any special
|
||||
treatment.
|
||||
|
||||
The frontend must create a ``direct_url.json`` file in the ``.dist-info``
|
||||
directory of the installed distribution, in compliance with PEP 610. The ``url``
|
||||
value must be a ``file://`` URL pointing to the project directory (i.e., the
|
||||
directory containing ``pyproject.toml``), and the ``dir_info`` value must be
|
||||
``{'editable': true}``.
|
||||
|
||||
The frontend must not rely on the ``prepare_metadata_for_build_wheel`` hook when
|
||||
installing in editable mode. It must instead invoke ``build_editable`` and use
|
||||
the ``.dist-info`` folder returned by that.
|
||||
|
||||
If the frontend concludes it cannot achieve an editable installation with the
|
||||
information provided by the build backend it should fail and raise an error to
|
||||
clarify to the user why not.
|
||||
|
||||
The frontend might implement one or more editable installation mechanisms and
|
||||
can leave it up to the user the choose one that its optimal to the use case
|
||||
of the user. For example, pip could add an editable mode flag, and allow the
|
||||
user to choose between ``pth`` files or symlinks (
|
||||
``pip install -e . --editable=pth`` vs ``pip install -e . --editable=symlink``).
|
||||
|
||||
Example editable implementations
|
||||
--------------------------------
|
||||
|
||||
To show how this PEP might be used, we'll now present a few case studies. Note
|
||||
the offered solutions are purely for illustrating purpose.
|
||||
|
||||
Add the source tree as is to the interpreter
|
||||
''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
This is one of the simplest implementations, it will add the source tree as is
|
||||
into the interpreters scheme paths, the virtual wheel might look like:
|
||||
|
||||
.. code::
|
||||
|
||||
{
|
||||
"metadata_for_build_editable": "<dir to dist-info>",
|
||||
{"scheme": "purelib": {"<project dir>": "<project dir>"}}
|
||||
}
|
||||
|
||||
The frontend then could either:
|
||||
|
||||
- Add the source directory onto the target interpreters ``sys.path`` during
|
||||
startup of it. This is done by creating a ``pth`` file into the target
|
||||
interpreters ``purelib`` folder. setuptools_ does this today and is what `pip
|
||||
install -e <project_directory>`_ translate too. This solution is fast and
|
||||
cross-platform compatible. However, this puts the entire source tree onto the
|
||||
system, potentially exposing modules that would not be available in a
|
||||
standard installation case.
|
||||
|
||||
- Symlink the folder, or the individual files within it. This method is what
|
||||
flit does via its `flit install --symlink`_. This solution requires the
|
||||
current platform to support symlinks. Still, it allows potentially to symlink
|
||||
individual files, which could solve the problem of including files that
|
||||
should be excluded from the source tree.
|
||||
|
||||
Using custom importers
|
||||
''''''''''''''''''''''
|
||||
|
||||
For a more robust and more dynamic collaboration between the build backend and
|
||||
the target interpreter, we can take advantage of the import system allowing the
|
||||
registration of custom importers. See :pep:`302` for more details and editables_
|
||||
as an example of this. The backend can generate a new importer during the
|
||||
editable build (or install it as an additional dependency) and register it at
|
||||
interpreter startup by adding a ``pth`` file.
|
||||
|
||||
.. code::
|
||||
|
||||
{
|
||||
"metadata_for_build_editable": "<dir to dist-info>",
|
||||
{
|
||||
"scheme": {
|
||||
"purelib": {
|
||||
"<project dir>/.editable/_register_importer.pth": "<project dir>/_register_importer.pth".
|
||||
"<project dir>/.editable/_editable_importer.py": "<project dir>/_editable_importer.py"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
The backend here registered a hook that is called whenever a new module is
|
||||
imported, allowing dynamic and on-demand functionality. Potential use cases
|
||||
where this is useful:
|
||||
|
||||
- Expose a source folder, but honor module excludes: the backend may generate
|
||||
an import hook that consults the exclusion table before allowing a source
|
||||
file loader to discover a file in the source directory or not.
|
||||
|
||||
- For a project, let there be two modules, ``A.py`` and ``B.py``. These are two
|
||||
separate files in the source directory; however, while building a wheel, they
|
||||
are merged into one mega file ``project.py``. In this case, with this PEP,
|
||||
the backend could generate an import hook that reads the source files at
|
||||
import time and merges them in memory before materializing it as a module.
|
||||
|
||||
- Automatically update out-of-date C-extensions: the backend may generate an
|
||||
import hook that checks the last modified timestamp for a C-extension source
|
||||
file. If it is greater than the current C-extension binary, trigger an update
|
||||
by calling the compiler before import.
|
||||
|
||||
Rejected ideas
|
||||
==============
|
||||
|
||||
This PEP competes with :pep:`660` and rejects that proposal because we think
|
||||
the mechanism of achieving an editable installation should be within the
|
||||
frontend rather than the build backend. Furthermore, this approach allows the
|
||||
ecosystem to use alternative means to accomplish the editable installation
|
||||
effect (e.g., insert path on ``sys.path`` or symlinks instead of just implying
|
||||
the loose wheel mode from the backend described by that PEP).
|
||||
|
||||
Prominently, :pep:`660` does not allow using symlinks to expose code and data
|
||||
files without also extending the wheel file standard with symlink support. It's
|
||||
not clear how the wheel format could be extended to support symlinks that refer
|
||||
not to files within the wheel itself, but files only available on the local
|
||||
disk. It's important to note that the backend itself (or backend generated
|
||||
code) must not generate these symlinks (e.g., at interpreter startup time) as
|
||||
that would conflict with the frontends book keeping of what files need to be
|
||||
uninstalled.
|
||||
|
||||
Finally, :pep:`660` adds support only for ``purelib`` and ``platlib`` files. It
|
||||
purposefully avoids supporting other types of information that the wheel format
|
||||
supports: ``include``, ``data`` and ``scripts``. With this path the frontend
|
||||
can support these on a best effort basis via the symlinks mechanism (though
|
||||
this feature is not universally available - on Windows require enablement). We
|
||||
believe its beneficial to add best effort support for these file types, rather
|
||||
than exclude the possibility of supporting them at all.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. _build: https://pypa-build.readthedocs.io
|
||||
|
||||
.. _editables: https://pypi.org/project/editables
|
||||
|
||||
.. _flit: https://flit.readthedocs.io/en/latest/index.html
|
||||
|
||||
.. _flit install --symlink: https://flit.readthedocs.io/en/latest/cmdline.html#cmdoption-flit-install-s
|
||||
|
||||
.. _installer: https://pypi.org/project/installer
|
||||
|
||||
.. _pip: https://pip.pypa.io
|
||||
|
||||
.. _pip install -e <project_directory>: https://pip.pypa.io/en/stable/cli/pip_install/#install-editable
|
||||
|
||||
.. _setup.py develop: https://setuptools.readthedocs.io/en/latest/userguide/commands.html#develop-deploy-the-project-source-in-development-mode
|
||||
|
||||
.. _setuptools: https://setuptools.readthedocs.io/en/latest/
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document is placed in the public domain or under the CC0-1.0-Universal
|
||||
license, whichever is more permissive.
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue