PEP 735: allow core metadata to include groups (#3943)

PEP 735 is expanded here to allow `project.dependencies` and
`project.optional-dependencies` to use Dependency Group Includes. This
allows for includes which help satisfy the use-cases described by the
`--only-deps` pip use-cases.

The changes for this are various:
- support for these includes is added to the rationale section
- a new use case is added, aligned with the `--only-deps` use case
- the spec section is updated to list changes to the `project` table
- backwards compatibility is extended significantly to address
  repackaging concerns as well as dependency analyzers[^1]
- "how to teach this" is also extended to note compatibility and
  repackaging issues
- rejected ideas has added an item on `[build-system.requires]`

[^1]: It may be overstatement to say these concerns are addressed.
They are considered and not dismissed, but it is not clear that any
choices can be made in the spec to address these issues other than to
inform users.

Additionally, one unrelated change is included: the spec now notes
explicitly that extra names and dependency group names can overlap
(this was already possible, implicitly) and instructs tools to decide
on their own how to handle that if they present dependency groups and
extras via a uniform interface.

Co-authored-by: Brett Cannon <brett@python.org>

peps/pep-0735.rst

@@ -145,7 +145,7 @@ another.
 Dependency Groups have two additional features which are similar to
 ``requirements.txt`` files:
 
-* they are not published as part of any built distribution
+* they are not published as distinct metadata in any built distribution
 
 * installation of a dependency group does not imply installation of a package's
   dependencies or the package itself
@@ -163,6 +163,24 @@ defined in greater detail in the :ref:`Use Cases Appendix <use_cases>`.
   be used as a location for locked dependency data)
 * Input data to an environment manager, such as tox, Nox, or Hatch
 * Configurable IDE discovery of test and linter requirements
+* Exposure of package dependencies for install, without the package itself
+
+Support for Inclusion from ``project.dependencies``
+---------------------------------------------------
+
+The ``project.dependencies`` and ``project.optional-dependencies`` tables are
+allowed to include Dependency Groups, requiring an update to the specification
+of these tables.
+
+The drivers for such changes are that some usages are well solved by the
+addition of such support, and that failing to include support in the initial
+Dependency Group PEP, but adding such support later in a subsequent PEP,
+would make the support landscape significantly more difficult for tool
+maintainers.
+
+Inclusion of a Dependency Group in ``project.dependencies`` or
+``project.optional-dependencies`` takes the form of a Dependency Group Include,
+defined in the specification section below.
 
 Regarding Poetry and PDM Dependency Groups
 ------------------------------------------
@@ -190,7 +208,7 @@ Dependency Groups are not Hidden Extras
 ---------------------------------------
 
 Dependency Groups are very similar to extras which go unpublished.
-However, there are two major features which distinguish them from extras
+However, there are three major features which distinguish them from extras
 further:
 
 * they support non-package projects
@@ -198,6 +216,8 @@ further:
 * installation of a Dependency Group does not imply installation of a package's
   dependencies (or the package itself)
 
+* a package's requirements (and extras) may depend upon Dependency Groups
+
 Future Compatibility & Invalid Data
 -----------------------------------
 
@@ -238,6 +258,10 @@ as defined in :pep:`508`.
 Tables in requirement lists must be valid Dependency Object Specifiers,
 defined below.
 
+The ``project`` table in ``pyproject.toml`` is modified such that
+``project.dependencies`` and the values of ``project.optional-dependencies``
+may contain Dependency Object Specifiers.
+
 Dependency Object Specifiers
 ----------------------------
 
@@ -286,6 +310,21 @@ Includes, in which case those includes should be expanded as well. Dependency
 Group Includes MUST NOT include cycles, and tools SHOULD report an error if
 they detect a cycle.
 
+``project`` Table Changes
+-------------------------
+
+The ``[project]`` table, originally defined in :pep:`621` is extended in two ways.
+
+``dependencies``
+
+In addition to :pep:`508` strings, the array may contain Dependency Object
+Specifiers.
+
+``optional-dependencies``
+
+In addition to :pep:`508` strings, the array values in this table may contain
+Dependency Object Specifiers.
+
 Example Dependency Groups Table
 -------------------------------
 
@@ -358,6 +397,19 @@ would be:
 Note that this is only an example. This PEP does not declare any requirements
 for how tools support the installation of Dependency Groups.
 
+Overlapping Install UX with Extras
+''''''''''''''''''''''''''''''''''
+
+Tools MAY choose to provide the same interfaces for installing Dependency
+Groups as they do for installing extras.
+
+Note that this specification does not forbid having an extra whose name matches
+a Dependency Group. In such cases, tools must define their own semantics for
+precedence order or disambiguation.
+
+Users are advised to avoid creating Dependency Groups whose names match extras.
+Tools SHOULD NOT treat such matching as an error.
+
 Validation and Compatibility
 ----------------------------
 
@@ -382,6 +434,13 @@ used:
     foo = ["pyparsing"]
     bar = [{set-phasers-to = "stun"}]
 
+Linters and Validators may be stricter
+''''''''''''''''''''''''''''''''''''''
+
+Eager validation is discouraged for tools which primarily install or resolve
+Dependency Groups.
+Linters and validation tools may have good cause to ignore this recommendation.
+
 Reference Implementation
 ========================
 
@@ -481,7 +540,53 @@ Backwards Compatibility
 At time of writing, the ``dependency-groups`` namespace within a
 ``pyproject.toml`` file is unused. Since the top-level namespace is
 reserved for use only by standards specified at packaging.python.org,
-there should be no direct backwards compatibility concerns.
+there are no direct backwards compatibility concerns.
+
+However, the introduction of the feature as a potential component of
+``project`` data has implications for a number of ecosystem tools.
+
+Audit and Update Tools
+----------------------
+
+A wide range of tools understand Python dependency data as expressed in
+``project.dependencies`` and ``project.optional-dependencies`` and may
+additionally support ``setup.cfg``, ``requirements.txt``, and even
+``setup.py``. (e.g., Dependabot, Tidelift, etc)
+
+Such tools inspect dependency data and, in some cases, offer tool-assisted or
+fully automated updates.
+It is our expectation that no such tools would support the new Dependency
+Groups at first, and broad ecosystem support could take many months or even some
+number of years to arrive.
+
+As a result, users of Dependency Groups would experience a degradation in their
+workflows and tool support at the time that they start using Dependency Groups.
+This is true of any new standard for where and how dependency data are encoded.
+
+Repackaging
+-----------
+
+Repackaging, and in particular tool-assisted repackaging such as Grayskull or
+PyInstaller, will need to contend with the change to package metadata
+definitions as well.
+
+Because repackagers for alternate ecosystems such as conda and linux distros
+are often distinct persons vs the package publishers, this compatibility
+concern is more difficult to address than cases in which the package
+maintainers experience the impact. Package maintainers may be unaware of the
+impact of beginning to use Dependency Groups, and may unknowingly make changes
+which harm downstream repackaging workflows.
+
+There are two primary ways in which this issue can be addressed:
+
+* via education and "How to Teach This" -- users whose packages are repackaged
+  should be made aware that using new standards may cause issues for downstream
+  package consumers
+
+* by driving more behavior through build-backends -- so long as dependency
+  metadata are gathered via :pep:`517` interfaces, downstream repackagers can
+  remain ignorant of which build system is being used and whether or not it
+  supports Dependency Groups
 
 Security Implications
 =====================
@@ -521,6 +626,22 @@ an include allows one Dependency Group to extend another. Similar configuration
 interfaces and the Python ``list.extend`` method may be used to explain the
 idea by analogy.
 
+Notes for Packages which are Repackaged
+---------------------------------------
+
+A special note should be given to package maintainers whose packages are
+repackaged by linux distros, homebrew, conda, etc.
+
+Use of Dependency Group Includes in core project metadata,
+``project.dependencies`` and ``project.optional-dependencies``, may break these
+consumers' uses of your package. Because they may be consuming and directly
+interact with the source for your repository, their toolchains may not support
+Dependency Groups at the same time that the package maintainers' tools are
+updated.
+
+Ensure that repackaging consumers can contact you if there is an issue, and
+make sure to note transitions to use Dependency Groups in your changelogs.
+
 Rejected Ideas
 ==============
 
@@ -627,28 +748,18 @@ has impacts on high-level tools like IDEs and Dependabot, which cannot support
 deep integration with these Dependency Groups. (For example, at time of writing
 Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.)
 
-Open Issues
-===========
+Why not support Dependency Group Includes in ``[build-system.requires]``?
+-------------------------------------------------------------------------
 
-Should it be possible for a Dependency Group to include ``[project.dependencies]`` or vice-versa?
--------------------------------------------------------------------------------------------------
+Although it may be interesting to allow this in the future, incorporating the
+proposal into the build-system table reduces the ability of users to rely on
+this to bootstrap support. :pep:`517` frontends would be required to support
+Dependency Groups in order to achieve this, and the versions of these frontends
+are not easily controlled by packages.
 
-A topic of debate is how -- or if -- Dependency Groups should interact with
-``[project.dependencies]`` and ``[project.optional-dependencies]``.
-
-An additional Dependency Object Specifier could be added for including
-``[project.dependencies]`` or ``[project.optional-dependencies]`` data to a
-Dependency Group. However, it is a goal of this spec that
-Dependency Groups should always be resolvable to a list of packages
-without the use of a build backend. Therefore, an inclusion of
-``[project.dependencies]`` or ``[project.optional-dependencies]`` would need to
-be defined carefully with respect to dynamic dependencies.
-
-The inclusion running in the opposite direction -- a ``[project.dependencies]``
-list containing a Dependency Group reference, possibly re-using Dependency
-Group Include objects as the mechanism -- is also possible but presents
-different challenges. Such an addition would introduce new syntax into the
-``[project]`` table, which not all tools would support at first.
+By only defining build backend support, it is possible for packages to start
+leveraging the new syntax and capabilities without being concerned about
+controlling the environment in which the package is built and installed.
 
 .. _prior_art:
 
@@ -1269,6 +1380,42 @@ for various purposes.
 This declaration allows the project author's knowledge of the appropriate tools
 for the project to be shared with all editors of that project.
 
+Exposure of package dependencies without the package itself
+-----------------------------------------------------------
+
+There are a variety of use-cases in which package developers would like to
+install only the dependencies of a package, without the package itself.
+
+For example:
+
+* Specify different environment variables or options when building dependencies
+  vs when building the package itself
+
+* Creating layered container images in which the dependencies are isolated from
+  the package being installed
+
+* Providing the dependencies to analysis environments (e.g., type checking)
+  without having to build and install the package itself
+
+For an example of the last case, consider the following sample
+``pyproject.toml``:
+
+.. code-block:: toml
+
+    [project]
+    dependencies = [{include = "runtime"}]
+    [optional-dependencies]
+    foo = [{include = "foo"}]
+    [dependency-groups]
+    runtime = ["a", "b"]
+    foo = ["c", "d"]
+    typing = ["mypy", {include = "runtime"}, {include = "foo"}]
+
+In this case, a ``typing`` group can be defined, with all of the package's
+runtime dependencies, but without the package itself. This allows uses of the
+``typing`` Dependency Group to skip installation of the package -- not only is
+this more efficient, but it may reduce the requirements for testing systems.
+
 Copyright
 =========