PEP 517: Add support for self-hosting backends and in-tree hooks (#901)

pep-0517.txt

@@ -166,6 +166,43 @@ Where the ``build-backend`` key exists, it takes precedence over
 Projects may still wish to include a ``setup.py`` for compatibility
 with tools that do not use this spec.
 
+This PEP also defines a ``backend-path`` key for use in ``pyproject.toml``, see
+the "In-Tree Build Backends" section below. This key would be used as follows::
+
+    [build-system]
+    # Defined by PEP 518:
+    requires = ["flit"]
+    # Defined by this PEP:
+    build-backend = "local_backend"
+    backend-path = ["backend"]
+
+
+Build Requirements
+==================
+
+This PEP places a number of additional requirements on the "build requirements"
+section of ``pyproject.toml``. These are intended to ensure that projects do
+not create impossible to satisfy conditions with their build requirements.
+
+- Project build requirements will define a directed graph of requirements
+  (project A needs B to build, B needs C and D, etc.) This graph MUST NOT
+  contain cycles.  If (due to lack of co-ordination between projects, for
+  example) a cycle is present, front ends MAY refuse to build the project.
+- Where build requirements are available as wheels, front ends SHOULD use these
+  where practical, to avoid deeply nested builds.  However front ends MAY have
+  modes where they do not consider wheels when locating build requirements, and
+  so projects MUST NOT assume that publishing wheels is sufficient to break a
+  requirement cycle.
+- Front ends SHOULD check explicitly for requirement cycles, and terminate
+  the build with an informative message if one is found.
+
+Note in particular that the requirement for no requirement cycles means that
+backends wishing to self-host (i.e., building a wheel for a backend uses that
+backend for the build) need to make special provision to avoid causing cycles.
+Typically this will involve specifying themselves as an in-tree backend, and
+avoiding external build dependencies (usually by vendoring them).
+
+
 =========================
  Build backend interface
 =========================
@@ -503,6 +540,48 @@ package *authors*, while still allowing *end-users* to open up the
 hood and apply duct tape when necessary.
 
 
+In-Tree Build Backends
+======================
+
+In certain circumstances, projects may wish to include the source code for the
+build backend directly in the source tree, rather than referencing the backend
+via the ``requires`` key. Two specific situations where this would be expected
+are:
+
+- Backends themselves, which want to use their own features for building
+  themselves ("self-hosting backends")
+- Project-specific backends, typically consisting of a custom wrapper around a
+  standard backend, where the wrapper is too project-specific to be worth
+  distributing independently ("in-tree backends")
+
+Projects can specify that their backend code is hosted in-tree by including the
+``backend-path`` key in ``pyproject.toml``. This key contains a list of
+directories, which the frontend will add to the start of ``sys.path`` when
+loading the backend, and running the backend hooks.
+
+There are two restrictions on the content of the ``backend-path`` key:
+
+- Directories in ``backend-path`` are interpreted as relative to the project
+  root, and MUST refer to a location within the source tree (after relative
+  paths and symbolic links have been resolved).
+- The backend code MUST be loaded from one of the directories specified in
+  ``backend-path`` (i.e., it is not permitted to specify ``backend-path`` and
+  *not* have in-tree backend code).
+
+The first restriction is to ensure that source trees remain self-contained,
+and cannot refer to locations outside of the source tree. Frontends SHOULD
+check this condition (typically by resolving the location to an absolute path
+and resolving symbolic links, and then checking it against the project root),
+and fail with an error message if it is violated.
+
+The ``backend-path`` feature is intended to support the implementation of
+in-tree backends, and not to allow configuration of existing backends. The
+second restriction above is specifically to ensure that this is how the feature
+is used. Front ends MAY enforce this check, but are not required to. Doing so
+would typically involve checking the backend's ``__file__`` attribute against
+the locations in ``backend-path``.
+
+
 ======================
  Source distributions
 ======================
@@ -599,9 +678,25 @@ automatically upgrade packages to the new format:
   reason, because we are quite tired of it.
 * Allowing the backend to be imported from files in the source tree would be
   more consistent with the way Python imports often work. However, not allowing
-  this prevents confusing errors from clashing module names. Projects which
-  need to provide their own backend could use a special proxy backend which
-  loads the hooks from the source tree, so the flexibility is not lost.
+  this prevents confusing errors from clashing module names. The initial
+  version of this PEP did not provide a means to allow backends to be
+  imported from files within the source tree, but the ``backend-path`` key
+  was added in the next revision to allow projects to opt into this behaviour
+  if needed.
+
+
+===============================
+ Summary of Changes to PEP 517
+===============================
+
+The following changes were made to this PEP after the initial reference
+implementation was released in pip 19.0.
+
+* Cycles in build requirements were explicitly prohibited.
+* Support for in-tree backends and self-hosting of backends was added by
+  the introduction of the ``backend-path`` key in the ``[build-system]``
+  table.
+
 
 ===================================
  Appendix A: Comparison to PEP 516