2021-05-20 14:58:22 -04:00
|
|
|
|
PEP: 660
|
2021-06-22 14:32:18 -04:00
|
|
|
|
Title: Editable installs for pyproject.toml based builds (wheel based)
|
2021-05-20 14:58:22 -04:00
|
|
|
|
Author: Daniel Holth <dholth@gmail.com>, Stéphane Bidoul <stephane.bidoul@gmail.com>
|
|
|
|
|
Sponsor: Paul Moore <p.f.moore@gmail.com>
|
|
|
|
|
Discussions-To: https://discuss.python.org/t/draft-pep-editable-installs-for-pep-517-style-build-backends/8510
|
2021-06-28 13:14:01 -04:00
|
|
|
|
Status: Accepted
|
2021-05-20 14:58:22 -04:00
|
|
|
|
Type: Standards Track
|
|
|
|
|
Content-Type: text/x-rst
|
|
|
|
|
Created: 30-Mar-2021
|
|
|
|
|
Post-History:
|
2022-04-20 05:53:08 -04:00
|
|
|
|
Resolution: https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450
|
|
|
|
|
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
|
========
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
This document describes a :pep:`517` style method for the installation of packages
|
2021-05-20 14:58:22 -04:00
|
|
|
|
in editable mode.
|
|
|
|
|
|
|
|
|
|
Motivation
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
Python programmers want to be able to develop packages without having to
|
|
|
|
|
install (i.e. copy) them into ``site-packages``, for example, by working in a
|
|
|
|
|
checkout of the source repository.
|
|
|
|
|
|
|
|
|
|
While this can be done by adding the relevant source directories to
|
|
|
|
|
``PYTHONPATH``, ``setuptools`` provides the ``setup.py develop`` mechanism that
|
|
|
|
|
makes the process easier, and also installs dependencies and entry points such
|
|
|
|
|
as console scripts. ``pip`` exposes this mechanism via its ``pip install
|
|
|
|
|
--editable`` option.
|
|
|
|
|
|
|
|
|
|
The installation of projects in such a way that the python code being
|
|
|
|
|
imported remains in the source directory is known as the *editable*
|
|
|
|
|
installation mode.
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
Now that :pep:`517` provides a mechanism to create alternatives to setuptools, and
|
2021-05-20 14:58:22 -04:00
|
|
|
|
decouple installation front ends from build backends, we need a new mechanism
|
|
|
|
|
to install packages in editable mode.
|
|
|
|
|
|
|
|
|
|
Rationale
|
|
|
|
|
=========
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
:pep:`517` deferred "Editable installs", meaning non-``setup.py``
|
2021-05-20 14:58:22 -04:00
|
|
|
|
distributions lacked that feature. The only way to retain ``editable`` installs
|
|
|
|
|
for these distributions was to provide a compatible ``setup.py develop``
|
2021-09-17 14:18:24 -04:00
|
|
|
|
implementation. By defining an editable hook other build frontends gain
|
2021-05-20 14:58:22 -04:00
|
|
|
|
parity with ``setup.py``.
|
|
|
|
|
|
|
|
|
|
Terminology and goals
|
|
|
|
|
=====================
|
|
|
|
|
|
|
|
|
|
The editable installation mode implies that the source code of the project
|
|
|
|
|
being installed is available in a local directory.
|
|
|
|
|
|
|
|
|
|
Once the project is installed in editable mode, users expect that changes to
|
|
|
|
|
the project *python* code in the local source tree become effective without the
|
|
|
|
|
need of a new installation step.
|
|
|
|
|
|
|
|
|
|
Some kind of changes, such as the addition or modification of entry points, or
|
|
|
|
|
the addition of new dependencies, require a new installation step to become
|
|
|
|
|
effective. These changes are typically made in build backend configuration
|
|
|
|
|
files (such as ``pyproject.toml``), so it is consistent with the general user
|
|
|
|
|
expectation that *python* source code is imported from the source tree.
|
|
|
|
|
|
|
|
|
|
The modification of non-python source code such a C extension modules obviously
|
|
|
|
|
require a compilation and/or installation step to become effective. The exact
|
|
|
|
|
steps to perform will remain specific to the build backend used.
|
|
|
|
|
|
|
|
|
|
When a project is installed in editable mode, users expect the installation to
|
2021-06-21 05:57:03 -04:00
|
|
|
|
behave identically as a regular installation. In particular the code must be
|
|
|
|
|
importable by other code, and metadata must be available to standard mechanisms
|
|
|
|
|
such as ``importlib.metadata``.
|
|
|
|
|
|
|
|
|
|
Depending on the way build backends implement this specification, some minor
|
|
|
|
|
differences may be visible such as the presence of additional files that are in
|
|
|
|
|
the source tree and would not be part of a regular install. Build backends are
|
|
|
|
|
encouraged to document such potential differences.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
The Mechanism
|
|
|
|
|
=============
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
This PEP adds three optional hooks to the :pep:`517` backend interface. These hooks
|
2021-06-22 14:32:18 -04:00
|
|
|
|
are used to build a wheel that, when installed, allows that distribution to be
|
2021-05-20 14:58:22 -04:00
|
|
|
|
imported from its source folder.
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
build_editable
|
|
|
|
|
--------------
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
|
2021-05-20 14:58:22 -04:00
|
|
|
|
...
|
|
|
|
|
|
|
|
|
|
Must build a ``.whl`` file, and place it in the specified ``wheel_directory``.
|
|
|
|
|
It must return the basename (not the full path) of the .whl file it creates, as
|
|
|
|
|
a unicode string.
|
|
|
|
|
|
|
|
|
|
May do an in-place build of the distribution as a side effect so that any
|
|
|
|
|
extension modules or other built artifacts are ready to be used.
|
|
|
|
|
|
|
|
|
|
The .whl file must comply with the Wheel binary file format specification (PEP
|
|
|
|
|
427). In particular it must contain a compliant .dist-info directory.
|
|
|
|
|
Metadata must be identical as the one that would have been produced by
|
|
|
|
|
``build_wheel`` or ``prepare_metadata_for_build_wheel``, except for
|
|
|
|
|
``Requires-Dist`` which may differ slightly as explained below.
|
|
|
|
|
|
|
|
|
|
Build-backends must produce wheels that have the same dependencies
|
|
|
|
|
(``Requires-Dist`` metadata) as wheels produced by the ``build_wheel`` hook,
|
|
|
|
|
with the exception that they can add dependencies necessary for their editable
|
|
|
|
|
mechanism to function at runtime (such as `editables`_).
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
The filename for the "editable" wheel needs to be :pep:`427` compliant too. It
|
2021-05-20 14:58:22 -04:00
|
|
|
|
does not need to use the same tags as ``build_wheel`` but it must be tagged as
|
|
|
|
|
compatible with the system.
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
If the build frontend has previously called ``prepare_metadata_for_build_editable``
|
|
|
|
|
and depends on the wheel resulting from this call to have metadata
|
|
|
|
|
matching this earlier call, then it should provide the path to the created
|
|
|
|
|
``.dist-info`` directory as the ``metadata_directory`` argument. If this
|
|
|
|
|
argument is provided, then ``build_editable`` MUST produce a wheel with identical
|
|
|
|
|
metadata. The directory passed in by the build frontend MUST be
|
|
|
|
|
identical to the directory created by ``prepare_metadata_for_build_editable``,
|
|
|
|
|
including any unrecognized files it created.
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
An "editable" wheel uses the wheel format not for distribution but as ephemeral
|
2021-05-20 14:58:22 -04:00
|
|
|
|
communication between the build system and the front end. This avoids having
|
|
|
|
|
the build backend install anything directly. This wheel must not be exposed
|
|
|
|
|
to end users, nor cached, nor distributed.
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
get_requires_for_build_editable
|
|
|
|
|
-------------------------------
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
def get_requires_for_build_editable(config_settings=None):
|
2021-05-20 14:58:22 -04:00
|
|
|
|
...
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
This hook MUST return an additional list of strings containing :pep:`508`
|
2021-05-20 14:58:22 -04:00
|
|
|
|
dependency specifications, above and beyond those specified in the
|
|
|
|
|
``pyproject.toml`` file, to be installed when calling the
|
2021-06-25 12:45:09 -04:00
|
|
|
|
``build_editable`` hooks.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
If not defined, the default implementation is equivalent to ``return []``.
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
prepare_metadata_for_build_editable
|
|
|
|
|
-----------------------------------
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
|
|
|
|
|
...
|
|
|
|
|
|
|
|
|
|
Must create a ``.dist-info`` directory containing wheel metadata
|
|
|
|
|
inside the specified ``metadata_directory`` (i.e., creates a directory
|
|
|
|
|
like ``{metadata_directory}/{package}-{version}.dist-info/``). This
|
|
|
|
|
directory MUST be a valid ``.dist-info`` directory as defined in the
|
|
|
|
|
wheel specification, except that it need not contain ``RECORD`` or
|
|
|
|
|
signatures. The hook MAY also create other files inside this
|
|
|
|
|
directory, and a build frontend MUST preserve, but otherwise ignore, such files;
|
|
|
|
|
the intention
|
|
|
|
|
here is that in cases where the metadata depends on build-time
|
|
|
|
|
decisions, the build backend may need to record these decisions in
|
|
|
|
|
some convenient format for re-use by the actual wheel-building step.
|
|
|
|
|
|
|
|
|
|
This must return the basename (not the full path) of the ``.dist-info``
|
|
|
|
|
directory it creates, as a unicode string.
|
|
|
|
|
|
|
|
|
|
If a build frontend needs this information and the method is
|
|
|
|
|
not defined, it should call ``build_editable`` and look at the resulting
|
|
|
|
|
metadata directly.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
What to put in the wheel
|
|
|
|
|
------------------------
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
Build backends must populate the generated wheel with files that when installed will result in an editable install.
|
|
|
|
|
Build backends may use different techniques to achieve the goals of an editable
|
2021-05-20 14:58:22 -04:00
|
|
|
|
install. This section provides examples and is not normative.
|
|
|
|
|
|
|
|
|
|
* Build backends may choose to place a ``.pth`` file at the root of the ``.whl`` file,
|
|
|
|
|
containing the root directory of the source tree. This approach is simple but
|
|
|
|
|
not very precise, although it may be considered good enough (especially when
|
|
|
|
|
using the ``src`` layout) and is similar to what ``setup.py develop``
|
|
|
|
|
currently does.
|
|
|
|
|
* The `editables`_ library shows how to build proxy modules that
|
|
|
|
|
provide a high quality editable installation. It accepts a list of modules
|
|
|
|
|
to include, and hide. When imported, these proxy modules replace themselves
|
|
|
|
|
with the code from the source tree. Path-based methods make all scripts under
|
|
|
|
|
a path importable, often including the project's own ``setup.py`` and other
|
|
|
|
|
scripts that would not be part of a normal installation. The proxy strategy
|
|
|
|
|
can achieve a higher level of fidelity than path-based methods.
|
2021-06-21 05:57:03 -04:00
|
|
|
|
* Symbolic links are another useful mechanism to realize editable installs.
|
|
|
|
|
Since, at the time this writing, the ``wheel`` specification does not support
|
|
|
|
|
symbolic links, they are not directly usable to set-up symbolic links in the
|
|
|
|
|
target environment. It is however possible for the backend to create a
|
|
|
|
|
symlink structure in some ``build`` directory of the source tree, and add
|
|
|
|
|
that directory to the python path via a ``.pth`` file in the "editable"
|
|
|
|
|
wheel. If some files linked in this manner depend on python implementation or
|
|
|
|
|
version, ABI or platform, care must be taken to generate the link structure
|
|
|
|
|
in different directories depending on compatibility tags, so the same project
|
|
|
|
|
tree can be installed in editable mode in multiple environments.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
|
|
|
|
Frontend requirements
|
|
|
|
|
---------------------
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
Frontends must install "editable" wheels in the same way as regular wheels.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
This also means uninstallation of editables does not require any special treatment.
|
|
|
|
|
|
|
|
|
|
Frontends must create a ``direct_url.json`` file in the ``.dist-info``
|
2022-01-21 06:03:51 -05:00
|
|
|
|
directory of the installed distribution, in compliance with :pep:`610`. The
|
2021-05-20 14:58:22 -04:00
|
|
|
|
``url`` value must be a ``file://`` url pointing to the project directory
|
|
|
|
|
(i.e. the directory containing ``pyproject.toml``), and the ``dir_info`` value
|
|
|
|
|
must be ``{'editable': true}``.
|
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
Frontends must execute ``get_requires_for_build_editable`` hooks in
|
2021-05-20 14:58:22 -04:00
|
|
|
|
an environment which contains the bootstrap requirements specified in the
|
|
|
|
|
``pyproject.toml`` file.
|
|
|
|
|
|
2021-07-14 14:01:22 -04:00
|
|
|
|
Frontends must execute the ``prepare_metadata_for_build_editable`` and
|
2021-06-25 12:45:09 -04:00
|
|
|
|
``build_editable`` hooks in an environment which contains the bootstrap
|
|
|
|
|
requirements from ``pyproject.toml`` and those specified by the
|
|
|
|
|
``get_requires_for_build_editable`` hook.
|
2021-05-20 14:58:22 -04:00
|
|
|
|
|
2021-06-25 12:45:09 -04:00
|
|
|
|
Frontends must not expose the wheel obtained from ``build_editable``
|
2021-05-20 14:58:22 -04:00
|
|
|
|
to end users. The wheel must be discarded after installation and must not be
|
|
|
|
|
cached nor distributed.
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
Limitations
|
|
|
|
|
===========
|
|
|
|
|
|
|
|
|
|
With regard to the wheel ``.data`` directory, this PEP focuses on making the
|
|
|
|
|
``purelib`` and ``platlib`` categories (installed into site-packages)
|
|
|
|
|
"editable". It does not make special provision for the other categories such as
|
|
|
|
|
``headers``, ``data`` and ``scripts``. Package authors are encouraged to use
|
|
|
|
|
``console_scripts``, make their ``scripts`` tiny wrappers around library
|
|
|
|
|
functionality, or manage these from the source checkout during development.
|
|
|
|
|
|
2021-06-22 14:32:18 -04:00
|
|
|
|
Prototypes
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
At the time of writing this PEP, several prototype implementations are
|
|
|
|
|
available in various frontends and backends. We provide links below to
|
|
|
|
|
illustrate possible approaches.
|
|
|
|
|
|
|
|
|
|
Frontends:
|
|
|
|
|
|
|
|
|
|
- pip (`pull request <https://github.com/pypa/pip/pull/8212>`__)
|
|
|
|
|
|
|
|
|
|
Build backends:
|
|
|
|
|
|
|
|
|
|
- enscons (`pull request 1 <https://github.com/dholth/enscons/pull/9>`__,
|
|
|
|
|
`pull request 2 <https://github.com/dholth/enscons/pull/21>`__)
|
|
|
|
|
- flit (`pull request <https://github.com/takluyver/flit/pull/400>`__)
|
|
|
|
|
- hatchling (`sdist <https://pypi.org/project/hatchling/#files>`__)
|
|
|
|
|
- pdm (`pull request <https://github.com/pdm-project/pdm-pep517/pull/36>`__)
|
|
|
|
|
- setuptools (`setuptools_pep660 repository <https://github.com/dholth/setuptools_pep660>`_)
|
|
|
|
|
|
2021-05-20 14:58:22 -04:00
|
|
|
|
Rejected ideas
|
|
|
|
|
==============
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
``editable`` local version identifier
|
|
|
|
|
-------------------------------------
|
|
|
|
|
|
2021-05-20 14:58:22 -04:00
|
|
|
|
The ideas of having build backends append or modify the local version
|
|
|
|
|
identifier to include the ``editable`` string has been rejected because it
|
|
|
|
|
would not satisfy ``==`` version speicifier that include the local version
|
2021-06-21 05:57:03 -04:00
|
|
|
|
identifier. In other words ``pkg==1.0+local`` is not satisfied by version
|
2021-05-20 14:58:22 -04:00
|
|
|
|
``1.0+local.editable``.
|
|
|
|
|
|
2021-06-21 05:57:03 -04:00
|
|
|
|
Virtual wheel
|
|
|
|
|
-------------
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
Another approach was proposed in :pep:`662`, where
|
2021-06-21 05:57:03 -04:00
|
|
|
|
the build backend returns a mapping from source files and directories to the
|
|
|
|
|
installed layout. It is then up to the installer frontend to realize the
|
|
|
|
|
editable installation by whatever means it deems adequate for its users.
|
|
|
|
|
|
|
|
|
|
In terms of capabilities, both proposals provide the core "editable" feature.
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
The key difference is that :pep:`662` leaves it to the frontend to decide how the
|
2021-06-21 05:57:03 -04:00
|
|
|
|
editable installation will be realized, while with this PEP, the choice must be
|
|
|
|
|
made by the backend. Both approaches can in principle provide several editable
|
|
|
|
|
installation methods for a given project, and let the developer choose one at
|
|
|
|
|
install time.
|
|
|
|
|
|
|
|
|
|
At the time of writing this PEP, it is clear that the community has a wide
|
|
|
|
|
range of theoretical and practical expectations about editable installs. The
|
|
|
|
|
reality is that the only one there is wide experience with is path insertion
|
|
|
|
|
via .pth (i.e. what setup.py develop does).
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
We believe that :pep:`660` better addresses these "unknown unknowns" today in the
|
2021-06-21 05:57:03 -04:00
|
|
|
|
most reliable way, by letting project authors select the backend or implement
|
|
|
|
|
the method that provides the editable mechanism that best suit their
|
|
|
|
|
requirements, and test it works correctly. Since the frontend has no latitude
|
|
|
|
|
in *how* to install the "editable" wheel, in case of issue, there is only one
|
|
|
|
|
place to investigate: the build backend.
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
With :pep:`662`, issues need to be investigated in the frontend,
|
2021-06-21 05:57:03 -04:00
|
|
|
|
the backend and possiblty the specification. There is also a high probability
|
|
|
|
|
that different frontends, implementing the specification in different ways,
|
|
|
|
|
will produce installations that behave differently than project authors
|
|
|
|
|
intended, creating confusion, or worse, projects that only work with specific
|
|
|
|
|
frontends or IDEs.
|
|
|
|
|
|
|
|
|
|
Unpacked wheel
|
|
|
|
|
--------------
|
|
|
|
|
|
|
|
|
|
A `prototype <https://github.com/pypa/pip/pull/8154/files>`_ was made that
|
|
|
|
|
created an unpacked wheel in a temporary directory, to be copied to the target
|
|
|
|
|
environment by the frontend. This approach was not pursued because a wheel
|
|
|
|
|
archive is easy to create for the backend, and using a wheel as communication
|
2022-01-21 06:03:51 -05:00
|
|
|
|
mechanism is a better fit with the :pep:`517` philosophy, and therefore keeps
|
2021-06-21 05:57:03 -04:00
|
|
|
|
things simpler for the frontend.
|
|
|
|
|
|
2021-05-20 14:58:22 -04:00
|
|
|
|
References
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
.. _`editables`: https://pypi.org/project/editables/
|
|
|
|
|
|
|
|
|
|
Copyright
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
This document is placed in the public domain or under the
|
|
|
|
|
CC0-1.0-Universal license, whichever is more permissive.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
..
|
|
|
|
|
Local Variables:
|
|
|
|
|
mode: indented-text
|
|
|
|
|
indent-tabs-mode: nil
|
|
|
|
|
sentence-end-double-space: t
|
|
|
|
|
fill-column: 70
|
|
|
|
|
coding: utf-8
|
|
|
|
|
End:
|