diff --git a/peps/pep-0735.rst b/peps/pep-0735.rst index d19f6925d..4a133943e 100644 --- a/peps/pep-0735.rst +++ b/peps/pep-0735.rst @@ -145,7 +145,7 @@ another. Dependency Groups have two additional features which are similar to ``requirements.txt`` files: -* they are not published as part of any built distribution +* they are not published as distinct metadata in any built distribution * installation of a dependency group does not imply installation of a package's dependencies or the package itself @@ -163,6 +163,24 @@ defined in greater detail in the :ref:`Use Cases Appendix `. be used as a location for locked dependency data) * Input data to an environment manager, such as tox, Nox, or Hatch * Configurable IDE discovery of test and linter requirements +* Exposure of package dependencies for install, without the package itself + +Support for Inclusion from ``project.dependencies`` +--------------------------------------------------- + +The ``project.dependencies`` and ``project.optional-dependencies`` tables are +allowed to include Dependency Groups, requiring an update to the specification +of these tables. + +The drivers for such changes are that some usages are well solved by the +addition of such support, and that failing to include support in the initial +Dependency Group PEP, but adding such support later in a subsequent PEP, +would make the support landscape significantly more difficult for tool +maintainers. + +Inclusion of a Dependency Group in ``project.dependencies`` or +``project.optional-dependencies`` takes the form of a Dependency Group Include, +defined in the specification section below. Regarding Poetry and PDM Dependency Groups ------------------------------------------ @@ -190,7 +208,7 @@ Dependency Groups are not Hidden Extras --------------------------------------- Dependency Groups are very similar to extras which go unpublished. -However, there are two major features which distinguish them from extras +However, there are three major features which distinguish them from extras further: * they support non-package projects @@ -198,6 +216,8 @@ further: * installation of a Dependency Group does not imply installation of a package's dependencies (or the package itself) +* a package's requirements (and extras) may depend upon Dependency Groups + Future Compatibility & Invalid Data ----------------------------------- @@ -238,6 +258,10 @@ as defined in :pep:`508`. Tables in requirement lists must be valid Dependency Object Specifiers, defined below. +The ``project`` table in ``pyproject.toml`` is modified such that +``project.dependencies`` and the values of ``project.optional-dependencies`` +may contain Dependency Object Specifiers. + Dependency Object Specifiers ---------------------------- @@ -286,6 +310,21 @@ Includes, in which case those includes should be expanded as well. Dependency Group Includes MUST NOT include cycles, and tools SHOULD report an error if they detect a cycle. +``project`` Table Changes +------------------------- + +The ``[project]`` table, originally defined in :pep:`621` is extended in two ways. + +``dependencies`` + +In addition to :pep:`508` strings, the array may contain Dependency Object +Specifiers. + +``optional-dependencies`` + +In addition to :pep:`508` strings, the array values in this table may contain +Dependency Object Specifiers. + Example Dependency Groups Table ------------------------------- @@ -358,6 +397,19 @@ would be: Note that this is only an example. This PEP does not declare any requirements for how tools support the installation of Dependency Groups. +Overlapping Install UX with Extras +'''''''''''''''''''''''''''''''''' + +Tools MAY choose to provide the same interfaces for installing Dependency +Groups as they do for installing extras. + +Note that this specification does not forbid having an extra whose name matches +a Dependency Group. In such cases, tools must define their own semantics for +precedence order or disambiguation. + +Users are advised to avoid creating Dependency Groups whose names match extras. +Tools SHOULD NOT treat such matching as an error. + Validation and Compatibility ---------------------------- @@ -382,6 +434,13 @@ used: foo = ["pyparsing"] bar = [{set-phasers-to = "stun"}] +Linters and Validators may be stricter +'''''''''''''''''''''''''''''''''''''' + +Eager validation is discouraged for tools which primarily install or resolve +Dependency Groups. +Linters and validation tools may have good cause to ignore this recommendation. + Reference Implementation ======================== @@ -481,7 +540,53 @@ Backwards Compatibility At time of writing, the ``dependency-groups`` namespace within a ``pyproject.toml`` file is unused. Since the top-level namespace is reserved for use only by standards specified at packaging.python.org, -there should be no direct backwards compatibility concerns. +there are no direct backwards compatibility concerns. + +However, the introduction of the feature as a potential component of +``project`` data has implications for a number of ecosystem tools. + +Audit and Update Tools +---------------------- + +A wide range of tools understand Python dependency data as expressed in +``project.dependencies`` and ``project.optional-dependencies`` and may +additionally support ``setup.cfg``, ``requirements.txt``, and even +``setup.py``. (e.g., Dependabot, Tidelift, etc) + +Such tools inspect dependency data and, in some cases, offer tool-assisted or +fully automated updates. +It is our expectation that no such tools would support the new Dependency +Groups at first, and broad ecosystem support could take many months or even some +number of years to arrive. + +As a result, users of Dependency Groups would experience a degradation in their +workflows and tool support at the time that they start using Dependency Groups. +This is true of any new standard for where and how dependency data are encoded. + +Repackaging +----------- + +Repackaging, and in particular tool-assisted repackaging such as Grayskull or +PyInstaller, will need to contend with the change to package metadata +definitions as well. + +Because repackagers for alternate ecosystems such as conda and linux distros +are often distinct persons vs the package publishers, this compatibility +concern is more difficult to address than cases in which the package +maintainers experience the impact. Package maintainers may be unaware of the +impact of beginning to use Dependency Groups, and may unknowingly make changes +which harm downstream repackaging workflows. + +There are two primary ways in which this issue can be addressed: + +* via education and "How to Teach This" -- users whose packages are repackaged + should be made aware that using new standards may cause issues for downstream + package consumers + +* by driving more behavior through build-backends -- so long as dependency + metadata are gathered via :pep:`517` interfaces, downstream repackagers can + remain ignorant of which build system is being used and whether or not it + supports Dependency Groups Security Implications ===================== @@ -521,6 +626,22 @@ an include allows one Dependency Group to extend another. Similar configuration interfaces and the Python ``list.extend`` method may be used to explain the idea by analogy. +Notes for Packages which are Repackaged +--------------------------------------- + +A special note should be given to package maintainers whose packages are +repackaged by linux distros, homebrew, conda, etc. + +Use of Dependency Group Includes in core project metadata, +``project.dependencies`` and ``project.optional-dependencies``, may break these +consumers' uses of your package. Because they may be consuming and directly +interact with the source for your repository, their toolchains may not support +Dependency Groups at the same time that the package maintainers' tools are +updated. + +Ensure that repackaging consumers can contact you if there is an issue, and +make sure to note transitions to use Dependency Groups in your changelogs. + Rejected Ideas ============== @@ -627,28 +748,18 @@ has impacts on high-level tools like IDEs and Dependabot, which cannot support deep integration with these Dependency Groups. (For example, at time of writing Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.) -Open Issues -=========== +Why not support Dependency Group Includes in ``[build-system.requires]``? +------------------------------------------------------------------------- -Should it be possible for a Dependency Group to include ``[project.dependencies]`` or vice-versa? -------------------------------------------------------------------------------------------------- +Although it may be interesting to allow this in the future, incorporating the +proposal into the build-system table reduces the ability of users to rely on +this to bootstrap support. :pep:`517` frontends would be required to support +Dependency Groups in order to achieve this, and the versions of these frontends +are not easily controlled by packages. -A topic of debate is how -- or if -- Dependency Groups should interact with -``[project.dependencies]`` and ``[project.optional-dependencies]``. - -An additional Dependency Object Specifier could be added for including -``[project.dependencies]`` or ``[project.optional-dependencies]`` data to a -Dependency Group. However, it is a goal of this spec that -Dependency Groups should always be resolvable to a list of packages -without the use of a build backend. Therefore, an inclusion of -``[project.dependencies]`` or ``[project.optional-dependencies]`` would need to -be defined carefully with respect to dynamic dependencies. - -The inclusion running in the opposite direction -- a ``[project.dependencies]`` -list containing a Dependency Group reference, possibly re-using Dependency -Group Include objects as the mechanism -- is also possible but presents -different challenges. Such an addition would introduce new syntax into the -``[project]`` table, which not all tools would support at first. +By only defining build backend support, it is possible for packages to start +leveraging the new syntax and capabilities without being concerned about +controlling the environment in which the package is built and installed. .. _prior_art: @@ -1269,6 +1380,42 @@ for various purposes. This declaration allows the project author's knowledge of the appropriate tools for the project to be shared with all editors of that project. +Exposure of package dependencies without the package itself +----------------------------------------------------------- + +There are a variety of use-cases in which package developers would like to +install only the dependencies of a package, without the package itself. + +For example: + +* Specify different environment variables or options when building dependencies + vs when building the package itself + +* Creating layered container images in which the dependencies are isolated from + the package being installed + +* Providing the dependencies to analysis environments (e.g., type checking) + without having to build and install the package itself + +For an example of the last case, consider the following sample +``pyproject.toml``: + +.. code-block:: toml + + [project] + dependencies = [{include = "runtime"}] + [optional-dependencies] + foo = [{include = "foo"}] + [dependency-groups] + runtime = ["a", "b"] + foo = ["c", "d"] + typing = ["mypy", {include = "runtime"}, {include = "foo"}] + +In this case, a ``typing`` group can be defined, with all of the package's +runtime dependencies, but without the package itself. This allows uses of the +``typing`` Dependency Group to skip installation of the package -- not only is +this more efficient, but it may reduce the requirements for testing systems. + Copyright =========