474 lines
19 KiB
Plaintext
474 lines
19 KiB
Plaintext
PEP: 516
|
|
Title: Build system abstraction for pip/conda etc
|
|
Author: Robert Collins <rbtcollins@hp.com>,
|
|
Nathaniel J. Smith <njs@pobox.com>
|
|
BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com>
|
|
Discussions-To: distutils-sig@python.org
|
|
Status: Rejected
|
|
Type: Standards Track
|
|
Topic: Packaging
|
|
Content-Type: text/x-rst
|
|
Created: 26-Oct-2015
|
|
Resolution: https://mail.python.org/pipermail/distutils-sig/2017-May/030517.html
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP specifies a programmatic interface for pip [#pip]_ and other
|
|
distribution or installation tools to use when working with Python
|
|
source trees (both the developer tree - e.g. the git tree - and source
|
|
distributions).
|
|
|
|
The programmatic interface allows decoupling of pip from its current
|
|
hard dependency on setuptools [#setuptools]_ able for two
|
|
key reasons:
|
|
|
|
1. It enables new build systems that may be much easier to use without
|
|
requiring them to even appear to be setuptools.
|
|
|
|
2. It facilitates setuptools itself changing its user interface without
|
|
breaking pip, giving looser coupling.
|
|
|
|
The interface needed to permit pip to install build systems also enables pip to
|
|
install build time requirements for packages which is an important step in
|
|
getting pip to full feature parity with the installation components of
|
|
easy-install.
|
|
|
|
As :pep:`426` is draft, we cannot utilise the metadata format it
|
|
defined. However :pep:`427` wheels are in wide use and fairly well specified, so
|
|
we have adopted the METADATA format from that for specifying distribution
|
|
dependencies and general project metadata. :pep:`508` provides a
|
|
self-contained language for describing a dependency, which we encapsulate in a
|
|
thin JSON schema to describe bootstrap dependencies.
|
|
|
|
Since Python sdists specified in :pep:`314` are also source trees, this
|
|
PEP is updating the definition of sdists.
|
|
|
|
|
|
PEP Rejection
|
|
=============
|
|
|
|
The CLI based approach proposed in this PEP has been rejected in favour of the
|
|
Python API based approach proposed in :pep:`517`. The specific CLI used to
|
|
communicate with build backends running as isolated subprocesses will be
|
|
considered an implementation detail of front-end developer tool implementations.
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
There is significant pent-up frustration in the Python packaging ecosystem
|
|
around the current lock-in between build system and pip. Breaking that lock-in
|
|
is better for pip, for setuptools, and for other build systems like flit
|
|
[#flit]_.
|
|
|
|
Specification
|
|
=============
|
|
|
|
Overview
|
|
--------
|
|
|
|
Build tools will be located by reading a file ``pypa.json`` from the root
|
|
directory of the source tree. That file describes how to get the build tool
|
|
and the name of the command to run to invoke the tool.
|
|
|
|
All tools will be expected to conform to a single command line interface
|
|
modelled on pip's existing use of the setuptools setup.py interface.
|
|
|
|
pypa.json
|
|
---------
|
|
|
|
The file ``pypa.json`` acts as a neutral configuration file for pip and other
|
|
tools that want to build source trees to consult for configuration. The
|
|
absence of a ``pypa.json`` file in a Python source tree implies a setuptools
|
|
or setuptools compatible build system.
|
|
|
|
The JSON has the following schema. Extra keys are ignored, which permits the
|
|
use of ``pypa.json`` as a configuration file for other related tools. If doing
|
|
that the chosen keys must be namespaced under ``tools``::
|
|
|
|
{"tools": {"flit": ["Flits content here"]}}
|
|
|
|
schema
|
|
The version of the schema. This PEP defines version "1". Defaults to "1"
|
|
when absent. All tools reading the file must error on an unrecognised
|
|
schema version.
|
|
|
|
bootstrap_requires
|
|
Optional list of :pep:`508` dependency specifications that must be
|
|
installed before running the build tool. For instance, if using flit, then
|
|
the requirements might be::
|
|
|
|
bootstrap_requires: ["flit"]
|
|
|
|
build_command
|
|
A mandatory key, this is a list of Python format strings [#strformat]_
|
|
describing the command to run. For instance, if using flit then the build
|
|
command might be::
|
|
|
|
build_command: ["flit"]
|
|
|
|
If using a command which is a runnable module fred::
|
|
|
|
build_command: ["{PYTHON}", "-m", "fred"]
|
|
|
|
Process interface
|
|
-----------------
|
|
|
|
The command to run is defined by a simple Python format string [#strformat]_.
|
|
|
|
This permits build systems with dedicated scripts and those that are invoked
|
|
using "python -m somemodule".
|
|
|
|
Processes will be run with the current working directory set to the root of
|
|
the source tree.
|
|
|
|
When run, processes should not read from stdin - while pip currently runs
|
|
build systems with stdin connected to its own stdin, stdout and stderr are
|
|
redirected and no communication with the user is possible.
|
|
|
|
As usual with processes, a non-zero exit status indicates an error.
|
|
|
|
Available format variables
|
|
--------------------------
|
|
|
|
PYTHON
|
|
The Python interpreter in use. This is important to enable calling things
|
|
which are just Python entry points.
|
|
|
|
{PYTHON} -m foo
|
|
|
|
Available environment variables
|
|
-------------------------------
|
|
|
|
These variables are set by the caller of the build system and will always be
|
|
available.
|
|
|
|
PATH
|
|
The standard system path.
|
|
|
|
PYTHON
|
|
As for format variables.
|
|
|
|
PYTHONPATH
|
|
Used to control sys.path per the normal Python mechanisms.
|
|
|
|
Subcommands
|
|
-----------
|
|
|
|
There are a number of separate subcommands that build systems must support.
|
|
The examples below use a build_command of ``flit`` for illustrative purposes.
|
|
|
|
build_requires
|
|
Query build requirements. Build requirements are returned as a UTF-8
|
|
encoded JSON document with one key ``build_requires`` consisting of a list
|
|
of :pep:`508` dependency specifications. Additional keys must be
|
|
ignored. The build_requires command is the only command run without
|
|
setting up a build environment.
|
|
|
|
Example command::
|
|
|
|
flit build_requires
|
|
|
|
metadata
|
|
Query project metadata. The metadata and only the metadata should
|
|
be output on stdout in UTF-8 encoding. pip would run metadata just once to
|
|
determine what other packages need to be downloaded and installed. The
|
|
metadata is output as a wheel METADATA file per :pep:`427`.
|
|
|
|
Note that the metadata generated by the metadata command, and the metadata
|
|
present in a generated wheel must be identical.
|
|
|
|
Example command::
|
|
|
|
flit metadata
|
|
|
|
wheel -d OUTPUT_DIR
|
|
Command to run to build a wheel of the project. OUTPUT_DIR will point to
|
|
an existing directory where the wheel should be output. Stdout and stderr
|
|
have no semantic meaning. Only one file should be output - if more are
|
|
output then pip would pick an arbitrary one to consume.
|
|
|
|
Example command::
|
|
|
|
flit wheel -d /tmp/pip-build_1234
|
|
|
|
develop [--prefix PREFIX]
|
|
Command to do an in-place 'development' installation of the project.
|
|
Stdout and stderr have no semantic meaning.
|
|
|
|
Not all build systems will be able to perform develop installs. If a build
|
|
system cannot do develop installs, then it should error when run. Note
|
|
that doing so will cause use operations like ``pip install -e foo`` to
|
|
fail.
|
|
|
|
The prefix option is used for defining an alternative prefix for the
|
|
installation. While setuptools has ``--root`` and ``--user`` options,
|
|
they can be done equivalently using ``--prefix``, and pip or other
|
|
tools that accept ``--root`` or ``--user`` options should translate
|
|
appropriately.
|
|
|
|
The root option is used to define an alternative root within which the
|
|
command should operate.
|
|
|
|
For instance::
|
|
|
|
flit develop --root /tmp/ --prefix /usr/local
|
|
|
|
Should install scripts within `/tmp/usr/local/bin`, even if the Python
|
|
environment in use reports that the sys.prefix is `/usr/` which would lead
|
|
to using `/tmp/usr/bin/`. Similar logic applies for package files etc.
|
|
|
|
The build environment
|
|
---------------------
|
|
|
|
Except for the build_requires command, all commands are run within a build
|
|
environment. No specific implementation is required, but a build environment
|
|
must achieve the following requirements.
|
|
|
|
1. All dependencies specified by the project's build_requires must be
|
|
available for import from within ``$PYTHON``.
|
|
|
|
1. All command-line scripts provided by the build-required packages must be
|
|
present in ``$PATH``.
|
|
|
|
A corollary of this is that build systems cannot assume access to any Python
|
|
package that is not declared as a build_requires or in the Python standard
|
|
library.
|
|
|
|
Hermetic builds
|
|
---------------
|
|
|
|
This specification does not prescribe whether builds should be hermetic or not.
|
|
Existing build tools like setuptools will use installed versions of build time
|
|
requirements (e.g. setuptools_scm) and only install other versions on version
|
|
conflicts or missing dependencies. However its likely that better consistency
|
|
can be created by always isolation builds and using only the specified dependencies.
|
|
|
|
However, there are nuanced problems there - such as how can users force the
|
|
avoidance of a bad version of a build requirement which meets some packages
|
|
dependencies. Future PEPs may tackle this problem, but it is not currently in
|
|
scope - it does not affect the metadata required to coordinate between build
|
|
systems and things that need to do builds, and thus is not PEP material.
|
|
|
|
Upgrades
|
|
--------
|
|
|
|
'pypa.json' is versioned to permit future changes without requiring
|
|
compatibility.
|
|
|
|
The sequence for upgrading either of schemas in a new PEP will be:
|
|
|
|
1. Issue new PEP defining an updated schema. If the schema is not entirely
|
|
backward compatible then a new version number must be defined.
|
|
2. Consumers (e.g. pip) implement support for the new schema version.
|
|
3. Package authors opt into the new schema when they are happy to introduce a
|
|
dependency on the version of 'pip' (and potentially other consumers) that
|
|
introduced support for the new schema version.
|
|
|
|
The *same* process will take place for the initial deployment of this PEP:-
|
|
the propagation of the capability to use this PEP without a `setuptools shim`_
|
|
will be largely gated by the adoption rate of the first version of pip that
|
|
supports it.
|
|
|
|
Static metadata in sdists
|
|
-------------------------
|
|
|
|
This PEP does not tackle the current inability to trust static metadata in
|
|
sdists. That is a separate problem to identifying and consuming the build
|
|
system that is in use in a source tree, whether it came from an sdist or not.
|
|
|
|
Handling of compiler options
|
|
----------------------------
|
|
|
|
Handling of different compiler options is out of scope for this specification.
|
|
|
|
pip currently handles compiler options by appending user supplied strings to
|
|
the command line it runs when running setuptools. This approach is sufficient
|
|
to work with the build system interface defined in this PEP, with the
|
|
exception that globally specified options will stop working globally as
|
|
different build systems evolve. That problem can be solved in pip (or conda or
|
|
other installers) without affecting interoperability.
|
|
|
|
In the long term, wheels should be able to express the difference between
|
|
wheels built with one compiler or options vs another, and that is PEP
|
|
material.
|
|
|
|
Examples
|
|
========
|
|
|
|
An example 'pypa.json' for using flit::
|
|
|
|
{"bootstrap_requires": ["flit"],
|
|
"build_command": "flit"}
|
|
|
|
When 'pip' reads this it would prepare an environment with flit in it before
|
|
trying to use flit.
|
|
|
|
Because flit doesn't have setup-requires support today,
|
|
`flit build_requires` would just output a constant string::
|
|
|
|
{"build_requires": []}
|
|
|
|
`flit metadata` would interrogate `flit.ini` and marshal the metadata into
|
|
a wheel METADATA file and output that on stdout.
|
|
|
|
`flit wheel` would need to accept a `-d` parameter that tells it where to output the
|
|
wheel (pip needs this).
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
Older pips will remain unable to handle alternative build systems.
|
|
This is no worse than the status quo - and individual build system
|
|
projects can decide whether to include a shim ``setup.py`` or not.
|
|
|
|
All existing build systems that can product wheels and do develop installs
|
|
should be able to run under this abstraction and will only need a specific
|
|
adapter for them constructed and published on PyPI.
|
|
|
|
In the absence of a ``pypa.json`` file, tools like pip should assume a
|
|
setuptools build system and use setuptools commands directly.
|
|
|
|
Network effects
|
|
---------------
|
|
|
|
Projects that adopt build systems that are not setuptools compatible - that
|
|
is that they have no setup.py, or the setup.py doesn't accept commands that
|
|
existing tools try to use - will not be installable by those existing tools.
|
|
|
|
Where those projects are used by other projects, this effect will cascade.
|
|
|
|
In particular, because pip does not handle setup-requires today, any project
|
|
(A) that adopts a setuptools-incompatible build system and is consumed as a
|
|
setup-requirement by a second project (B) which has not itself transitioned to
|
|
having a pypa.json will make B uninstallable by any version of pip. This is
|
|
because setup.py in B will trigger easy-install when 'setup.py egg_info' is
|
|
run by pip, and that will try and fail to install A.
|
|
|
|
As such we recommend that tools which are currently used as setup-requires
|
|
either ensure that they keep a `setuptools shim`_ or find their consumers and
|
|
get them all to upgrade to the use of a `pypa.json` in advance of moving
|
|
themselves. Pragmatically that is impossible, so the advice is to keep a
|
|
setuptools shim indefinitely - both for projects like pbr, setuptools_scm and
|
|
also projects like numpy.
|
|
|
|
setuptools shim
|
|
---------------
|
|
|
|
It would be possible to write a generic setuptools shim that looks like
|
|
``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This
|
|
is not needed for pip to use the system, but would allow package authors to
|
|
use the new features while still retaining compatibility with older pip
|
|
versions.
|
|
|
|
Rationale
|
|
=========
|
|
|
|
This PEP started with a long mailing list thread on distutils-sig [#thread]_.
|
|
Subsequent to that an online meeting was held to debug all the positions folk
|
|
had. Minutes from that were posted to the list [#minutes]_.
|
|
|
|
This specification is a translation of the consensus reached there into PEP
|
|
form, along with some arbitrary choices on the minor remaining questions.
|
|
|
|
The basic heuristic for the design has been to focus on introducing an
|
|
abstraction without requiring development not strictly tied to the
|
|
abstraction. Where the gap is small to improvements, or the cost of using the
|
|
existing interface is very high, then we've taken on having the improvement as
|
|
a dependency, but otherwise deferred such to future iterations.
|
|
|
|
We chose wheel METADATA files rather than defining a new specification,
|
|
because pip can already handle wheel .dist-info directories which encode all
|
|
the necessary data in a METADATA file. :pep:`426` can't be used as it's still
|
|
draft, and defining a new metadata format, while we should do that, is a
|
|
separate problem. Using a directory on disk would not add any value to the
|
|
interface (pip has to do that today due to limitations in the setuptools
|
|
CLI).
|
|
|
|
The use of 'develop' as a command is because there is no PEP specifying the
|
|
interoperability of things that do what 'setuptools develop' does - so we'll
|
|
need to define that before pip can take on the responsibility for doing the
|
|
'develop' step. Once that's done we can issue a successor PEP to this one.
|
|
|
|
The use of a command line API rather than a Python API is a little
|
|
contentious. Fundamentally anything can be made to work, and the pip
|
|
maintainers have spoken strongly in favour of retaining a process based
|
|
interface - something that is mature and robust in pip today.
|
|
|
|
The choice of JSON as a file format is a compromise between several
|
|
constraints. Firstly there is no stdlib YAML interpreter, nor one for any of
|
|
the other low-friction structured file formats. Secondly, INIParser is a poor
|
|
format for a number of reasons, primarily that it has very minimal structure -
|
|
but pip's maintainers are not fond of it. JSON is in the stdlib, has
|
|
sufficient structure to permit embedding anything we want in future without
|
|
requiring embedded DSL's.
|
|
|
|
Donald suggested using ``setup.cfg`` and the existing setuptools command line
|
|
rather than inventing something new. While that would permit interoperability
|
|
with less visible changes, it requires nearly as much engineering on the pip
|
|
side - looking for the new key in setup.cfg, implementing the non-installed
|
|
environments to run the build in. And the desire from other build system
|
|
authors not to confuse their users by delivering something that looks like but
|
|
behaves quite differently to setuptools seems like a bigger issue than pip
|
|
learning how to invoke a custom build tool.
|
|
|
|
The metadata and wheel commands are required to have consistent metadata to
|
|
avoid a race condition that could otherwise happen where pip reads the
|
|
metadata, acts on it, and then the resulting wheel has incompatible
|
|
requirements. That race is exploited today by packages using :pep:`426`
|
|
environment markers, to work with older pip versions that do not support
|
|
environment markers. That exploit is not needed with this PEP, because either
|
|
the setuptools shim is in use (with older pip versions), or an environment
|
|
marker ready pip is in use. The setuptools shim can take care of exploiting
|
|
the difference older pip versions require.
|
|
|
|
We discussed having an sdist verb. The main driver for this was to make sure
|
|
that build systems were able to produce sdists that pip can build - but this is
|
|
circular: the whole point of this PEP is to let pip consume such sdists or VCS
|
|
source trees reliably and without requiring an implementation of setuptools.
|
|
Being able to create new sdists from existing source trees isn't a thing pip
|
|
does today, and while there is a PR to do that as part of building from
|
|
source, it is contentious and lacks consensus. Rather than impose a
|
|
requirement on all build systems, we are treating it as a YAGNI, and will add
|
|
such a verb in a future version of the interface if required. The existing
|
|
:pep:`314` requirements for sdists still apply, and distutils or setuptools
|
|
users can use ``setup.py sdist`` to create an sdist. Other tools should create
|
|
sdists compatible with :pep:`314`. Note that pip itself does not require
|
|
:pep:`314` compatibility - it does not use any of the metadata from sdists - they
|
|
are treated like source trees from disk or version control.
|
|
|
|
References
|
|
==========
|
|
|
|
.. [#pip] pip, the recommended installer for Python packages
|
|
(http://pip.readthedocs.org/en/stable/)
|
|
|
|
.. [#setuptools] setuptools, the de facto Python package build system
|
|
(https://pythonhosted.org/setuptools/)
|
|
|
|
.. [#flit] flit, a simple way to put packages in PyPI
|
|
(http://flit.readthedocs.org/en/latest/)
|
|
|
|
.. [#pypi] PyPI, the Python Package Index
|
|
(https://pypi.python.org/)
|
|
|
|
.. [#shellvars] Shellvars, an implementation of shell variable rules for Python.
|
|
(https://github.com/testing-cabal/shellvars)
|
|
|
|
.. [#thread] The kick-off thread.
|
|
(https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)
|
|
|
|
.. [#minutes] The minutes.
|
|
(https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)
|
|
|
|
.. [#strformat] The Python string formatting syntax.
|
|
(https://docs.python.org/3.1/library/string.html#format-string-syntax)
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|