diff --git a/peps/pep-0735.rst b/peps/pep-0735.rst index 31c4fba63..f1efa0608 100644 --- a/peps/pep-0735.rst +++ b/peps/pep-0735.rst @@ -34,23 +34,31 @@ standardized answer: * How should dependencies be defined for projects which do not build distributions (non-package projects)? -In the absence of any standard, two known workflow tools, PDM and Poetry, have -defined their own solutions for Dependency Groups to address the first of these -two needs. Their definitions are very similar to one another, although PDM -structures them much more similarly to -`extras `__ -than Poetry does. -Neither of them addresses the needs of non-package projects, and neither can be -referenced natively from other tools like ``tox``, ``nox``, or ``pip``. +In support of these two needs, there are two common solutions which are similar +to this proposal: -There are two pre-existing solutions which are similar to this proposal: the -use of ``requirements.txt`` files and package extras. +* ``requirements.txt`` files -Regarding ``requirements.txt``, many projects may define one or more of these files, +* package `extras `__ + +Both ``requirements.txt`` files and ``extras`` have limitations which this +standard seeks to overcome. + +Note that the two use cases above describe two different types of projects +which this PEP seeks to support: + +* Python packages, such as libraries + +* non-package projects, such as data science projects + +Limitations of ``requirements.txt`` files +----------------------------------------- + +Many projects may define one or more ``requirements.txt`` iles, and may arrange them either at the project root (e.g. ``requirements.txt`` and ``test-requirements.txt``) or else in a directory (e.g. ``requirements/base.txt`` and ``requirements/test.txt``). However, there are -two major issues with the use of requirements files in this way: +major issues with the use of requirements files in this way: * There is no standardized naming convention such that tools can discover or use these files by name. @@ -58,67 +66,50 @@ two major issues with the use of requirements files in this way: * ``requirements.txt`` files are *not standardized*, but instead provide options to ``pip``. -Therefore, their use is not portable to any new -installer or tool which wishes to process them without relying upon ``pip``. -Additionally, ``requirements.txt`` files may be viewed as a heavyweight -solution for very small dependency sets of only one or two items, and a terser -declaration will be beneficial to projects with a number of small groups of +As a result, it is difficult to define tool behaviors based on +``requirements.txt`` files. They are not trivial to discover or identify by +name, and their contents may contain a mix of package specifiers and additional +``pip`` options. + +The lack of a standard for ``requirements.txt`` contents also means they are +not portable to any alternative tools which wish to process them other than +``pip``. + +Additionally, ``requirements.txt`` files require a file per dependency list. +For some use-cases, this makes the marginal cost of dependency groupings high, +relative to their benefit. +A terser declaration is beneficial to projects with a number of small groups of dependencies. -Regarding extras, the use of extras to define development dependencies is a -widespread practice, but it has two major downsides: +In contrast with this, Dependency Groups are defined at a well known location +in ``pyproject.toml`` with fully standardized contents. Not only will they have +immediate utility, but they will also serve as a starting point for future +standards. -* An extra is defined as an optional part of a package which may specify - additional dependencies. - This means that it cannot be installed without installing all of the package - dependencies, and the project must be defined as a package. +Limitations of ``extras`` +------------------------- -* Many developers view their extras as part of the public interface for their - package. Because these are published data, package developers often are +``extras`` are additional package metadata declared in the +``[project.optional-dependencies]`` table. They provide names for lists of +package specifiers which are published as part of a package's metadata, and +which a user can request under that name, as in ``pip install 'foo[bar]'`` to +install ``foo`` with the ``bar`` extra. + +Because ``extras`` are package metadata, they are not usable when a project +does not build a distribution (i.e., is not a package). + +For projects which are packages, ``extras`` are a common solution for defining +development dependencies, but even under these circumstances they have +downsides: + +* Because an ``extra`` defines optional *additional* dependencies, it is not + possible to install an ``extra`` without installing the current package and + its dependencies. + +* Because they are user-installable, ``extras`` are part of the public interface + for packages. Because ``extras`` are published, package developers often are concerned about ensuring that their development extras are not confused with - user-facing extras. Therefore, their development needs are not appropriate to - publish in the manner of extras. - -Providing a standardized place to store dependency data which matches the -typical use cases of ``requirements.txt`` files will allow for better -cross-compatibility between tools and will onboard beginning users into -``pyproject.toml`` data more quickly (before they learn about the -``[build-system]`` table, for example). It will also resolve the long-standing -ambiguity and tension within the Python community about whether or not extras -should be used to declare development dependencies. - -Supported Project Types ------------------------ - -This PEP aims to serve the needs of two distinct project types: - -* libraries and other packages which want to define development dependencies - -* projects which do not build distributions which want a standardized way to - declare their dependency data - -The Python packaging toolchain is oriented towards the production of wheel and -sdist files, containing project source code for installation. However, not all -projects are designed or intended for this mode of distribution. Primary -examples include webapps (which often run from source), and data science -projects (which may start as collections of scripts and only later evolve -package structures). This PEP seeks to offer these projects a way to specify -their package dependencies in a way which is not constrained by the -requirements of distribution building -- for example, a data science project -does not need to declare a version or authors in order for these data to be -valid, nor does it need to define an installable package source tree. - -Simultaneously, this PEP should benefit projects which *are* intended to build -distributions. -Because Dependency Groups, as specified in this PEP, are explicitly defined to -not be included in any built distribution, two common use cases are addressed -by this addition: - -* the use of ``extras``, as discussed above - -* the use of ``requirements.txt`` files for distributed libraries and - applications, which takes a similar form to their use in - non-distribution-building projects + user-facing extras. Rationale ========= @@ -130,139 +121,111 @@ This name was chosen to match the canonical name of the feature This format should be as simple and learnable as possible, having a format very similar to existing ``requirements.txt`` files for many cases. Each list -in ``[dependency-groups]`` is defined as a list of package specifiers. However, -there are a number of use cases which require data which cannot be expressed in -PEP 508 package specifiers. Therefore, this PEP also defines an object-format -for package specification, capable of defining these additional data. +in ``[dependency-groups]`` is defined as a list of package specifiers. For +example: -The format does not support many forms of non-package data from -``requirements.txt`` files, such as ``pip`` options for package servers or -hashes. Including them in the initial stage complicates this proposal and it is -not clear that the majority of use-cases require them. Future expansions to -this specification may extend the object format for dependencies in -``[dependency-groups]`` values. Because this specification provides for an -object format for the data, it is possible for these to be added in future -specifications. +.. code-block:: toml -The following use cases are considered important targets for this PEP: + [dependency-groups] + test = ["pytest>7", "coverage"] -* A web application (e.g. a Django or Flask app) which does not build a - distribution, but bundles and ships its source to a deployment toolchain. The - app has runtime dependencies, testing dependencies, and development - environment dependencies. All of these dependency sets should be declared - independently, and none of these runtime contexts should require that the - project be built into a wheel. -* A library which builds a distribution, but has a variety of testing and - development environments it wishes to declare. Some of these rely on the - library's own dependencies (``test`` and ``typing`` environments) while - others do not (``docs`` and ``lint`` environments). -* A data science project which consists of multiple scripts that depend on the same suite - of core libraries (e.g. ``numpy``, ``pandas``, ``matplotlib``), and which - also has additional libraries, sometimes conflicting, for specific scripts - (e.g. ``scikit-learn==1.3.2`` for one script and ``scikit-learn==0.24.2`` for - another). -* *Input data* to a lockfile generator. Because there is no standardized - lockfile format, it is still the prerogative of tools like ``poetry`` and - ``pip-compile`` to describe their own formats. However, it should be possible - for the data from this PEP to be used as an input to these tools. - Furthermore, tools may define their own structures and conventions such that - the generated lockfiles can be referenced by the names of their originating - Dependency Groups in ``pyproject.toml``. -* *Input data* to a tox, Nox, or Hatch environment, as can - currently be achieved, for example, with ``deps = -r requirements.txt`` in - ``tox.ini``. These tools will need to add additional options for processing - Dependency Groups. -* Embeddable data for ``pyproject.toml`` within a script, as in :pep:`723`, - which defines embedded ``pyproject.toml`` data within scripts. This - PEP does not define exactly how :pep:`723` should be modified, but being - consumable by that interface is a stated goal. -* IDE discovery of requirements data. For example, VS Code could look for a dependency - group named ``test`` to use when running tests. +There are a number of use cases for ``requirements.txt`` files which require +data which cannot be expressed in :pep:`508` dependency specifiers. Such +fields are not valid in Dependency Groups. Including many of the data and +fields which ``pip`` supports, such as index servers, hashes, and path +dependencies, requires new standards. This standard leaves room for new +standards and developments, but does not attempt to support all valid +``requirements.txt`` contents. -Note that this PEP does not reserve any names for specific use cases. It is -considered a problem for downstream standards and conventions to define -well-known names for certain needs, such as ``test`` or ``docs``. +The only exception to this is the ``-r`` flag which ``requirements.txt`` files +use to include one file in another. Dependency Groups support an "include" +mechanism which is similar in meaning, allowing one dependency group to extend +another. + +Dependency Groups have two additional features which are similar to +``requirements.txt`` files: + +* they are not published as part of any built distribution + +* installation of a dependency group does not imply installation of a package's + dependencies or the package itself + +Use Cases +--------- + +The following use cases are considered important targets for this PEP. They are +defined in greater detail in the `Use Cases Appendix `_. + +* Web Applications deployed via a non-python-packaging build process +* Libraries with unpublished dev dependency groups +* Data science projects with groups of dependencies but no core package +* *Input data* to lockfile generation (Dependency Groups should generally not + be used as a location for locked dependency data) +* Input data to an environment manager, such as tox, Nox, or Hatch +* Embedded ``pyproject.toml`` data in scripts, as proposed in :pep:`723` +* Configurable IDE discovery of test and linter requirements Regarding Poetry and PDM Dependency Groups ------------------------------------------ -Poetry and PDM already offer a feature which each calls "Dependency Groups", -but using non-standard data belonging to the ``poetry`` and ``pdm`` tools. +The existing Poetry and PDM tools already offer a feature which each calls +"Dependency Groups", but using non-standard data belonging to the ``poetry`` +and ``pdm`` tools. (PDM also uses extras for some Dependency Groups, and overlaps the notion heavily with extras.) -This PEP is not guaranteed to be a perfectly substitutable solution for the -same problem space for each tool. However, the ideas are extremely similar, and -it should be possible for Poetry and PDM to support at least some -PEP-735-standardized Dependency Group configurations using their own Dependency -Group nomenclature. +This PEP does not support all of the features of Poetry and PDM, which, like +``requirements.txt`` files for ``pip``, support several non-standard extensions +to common dependency specifiers. -A level of interoperability with Poetry and PDM is a goal of this PEP, but -certain features and behaviors defined here may not be supported by Poetry and -PDM. Matching the existing Poetry and PDM *semantics* for Dependency Groups is -a non-goal. +It should be possible for such tools to use standardized Dependency Groups as +extensions of their own Dependency Group mechanisms. +However, defining a new data format which replaces the existing Poetry and PDM +solutions is a non-goal, as it would require standardizing their various +non-standard features. Dependency Groups are not Hidden Extras --------------------------------------- -One could be forgiven for thinking that Dependency Groups are just extras which -go unpublished. - -However, there are two major features which distinguish them from -extras: +Dependency Groups are very similar to extras which go unpublished. +However, there are two major features which distinguish them from extras +further: * they support non-package projects * installation of a Dependency Group does not imply installation of a package's dependencies (or the package itself) -Object Specification Does Not Allow for "PEP 508 Decomposition" ---------------------------------------------------------------- - -Poetry and PDM both allow for their object formats for dependency data to be -used to decompose a PEP 508 specification into parts. For example, a dependency -on ``requests==2.21.0`` could be written under these tools in a form like -``{name = "requests", version = "==2.21.0"}``. - -This spec does not allow for such a structure. There is only one way to write -down a PEP 508 dependency, as a string. This has two positive effects: - -* There are fewer ways of writing identical data - -* It is not possible, by design, to combine a PEP 508 specifier with other - key-value pairs in a dependency specifier - Specification ============= This PEP defines a new section (table) in ``pyproject.toml`` files named ``dependency-groups``. The ``dependency-groups`` table contains an arbitrary number of user-defined keys, each of which has, as its value, a list of -requirements specifiers (defined below). These keys must match the following +requirements (defined below). These keys must match the following regular expression: ``[a-z0-9][a-z0-9-]*[a-z0-9]``. Meaning that they must be all lower-case alphanumerics, with ``-`` allowed only in the middle, and at least two characters long. These requirements are chosen so that the normalization rules used for PyPI package names are unnecessary as the names are already normalized. -Requirements specifiers will use a definition based on standardized -`Dependency Specifiers `__ -introduced in :pep:`508`. +Requirement lists under ``dependency-groups`` may contain strings, tables +("dicts" in Python), or a mix of strings and tables. -This PEP also proposes an object format to define a dependency, called a -"Dependency Object Specifier" (defined below). The elements in -``[dependency-groups]`` lists must either be strings, in which case they must -be valid Dependency Specifiers (PEP 508) or else Dependency Object Specifiers. +Strings in requirement lists must be valid +`Dependency Specifiers `__, +as defined in :pep:`508`. + +Tables in requirement lists must be valid Dependency Object Specifiers, +defined below. Dependency Object Specifiers ---------------------------- -Dependency Object Specifiers are objects (tables in TOML) which define a set of -dependencies. They do not necessarily refer to a single package, as will become -clear from the definitions below. +Dependency Object Specifiers are tables which define zero or more dependencies. -There are two forms of Dependency Object Specifiers: "Dependency Group Includes" -and "Path Dependencies". +This PEP standardizes only one type of Dependency Object Specifier, a +"Dependency Group Include". Other types may be added in future standards. Dependency Group Include '''''''''''''''''''''''' @@ -270,115 +233,12 @@ Dependency Group Include A Dependency Group Include includes the dependencies of another Dependency Group in the current Dependency Group. -An include is defined as an object with exactly one key, ``"include"``, whose +An include is defined as a table with exactly one key, ``"include"``, whose value is a string, the name of another Dependency Group. For example, ``{include = "test"}`` is an include which expands to the contents of the ``test`` Dependency Group. -Path Dependency -''''''''''''''' - -A Path Dependency is a dependency on a package found via a local filesystem -path OR on dependencies of a package found via a local filesystem path. - -It contains the following keys, with associated types: - -* ``path``: a string, required -* ``extras``: a list of strings, optional -* ``editable``: a boolean, optional -* ``only-deps``: a boolean, optional - -For a simple example, ``{path = ".", editable = true, extras = ["mysql"]}`` is -a Path Dependency on the current project including its ``mysql`` extra. In -``requirements.txt`` files, a similar idea can be expressed as ``-e '.[mysql]'``. - -``path`` -~~~~~~~~ - -The ``path`` must refer to the path to a built distribution (wheel, sdist, or -any future file format) or a directory containing python package source code. - -If the ``path`` is relative, it is relative to the directory containing -``pyproject.toml``. - -If the path refers to a built distribution, that package should be installed -when the Dependency Group is installed. - -If the path refers to a directory, the directory SHOULD be a valid Python -package. Implementations MAY refuse to process ``path`` directives which are -not packages, even when ``only-deps`` is specified (see below for how -``only-deps`` would potentially make it possible to process non-package paths). - -``extras`` -~~~~~~~~~~ - -The ``extras`` key is a list of strings, each of which is the name of an extra -which should be included in the package installation. - -``editable`` -~~~~~~~~~~~~ - -If ``editable`` is ``true``, implementations installing the Dependency Group -SHOULD install the dependency in editable mode. If it is ``false``, they SHOULD -install it in non-editable mode. If ``editable`` is absent, no default behavior -is specified and implementations may choose their preferred default. - -Implementations MAY provide options to users to configure or override this behavior. -For example, a tool may have an option ``--never-editable`` which always treats -``editable`` as ``false``. -However, implementations SHOULD prefer to use the ``editable`` value if it is -present. - -If ``editable`` is specified on a Path Dependency which refers to a built -distribution, implementations MUST treat this as an error. - -``only-deps`` -~~~~~~~~~~~~~ - -If ``only-deps`` is ``true``, implementations MUST NOT install the package -at the specified ``path``. Instead, they should only install the dependencies -of that package. This may still require building a package from a source tree -in order to discover ``dynamic`` dependency data. - -If ``only-deps`` is ``false`` or absent, implementations should install the -package at the specified ``path``. - -It is possible to specify ``only-deps`` on a Path Dependency which does not -refer to a valid python package and for tools to, at least in theory, process -such a dependency successfully. For example, a ``pyproject.toml`` file -containing only ``[project.dependencies]`` and none of the other required keys -in the ``[project]`` table could be supported. Implementations SHOULD NOT -support such structures and SHOULD fail if a Path Dependency refers to a python -project which is not a package. - -If ``only-deps`` is ``true`` and ``extras`` are specified, implementations -should install the ``extras`` as well as all of the non-optional dependencies -of the package. - -Handling of Multiple Path Dependencies Referring to the Same Path -''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' - -It is possible for resolution of a Dependency Group to refer to the same -path multiple times (for example, via two combined includes) using distinct -Path Dependencies. - -If the ``path`` values are distinct, but refer to the same concrete file or -directory on the filesystem, handling behavior is unspecified. Implementations -MAY normalize these paths to a single value. - -If the ``path`` values are identical, implementations MUST treat the result as -a singular Path Dependency following the rules below: - -- The ``extras`` keys are concatenated into a single list -- If ``editable`` is always ``true`` or always ``false``, it is treated as - such. Otherwise, the ``editable`` key is treated as though it were absent. -- If the ``only-deps`` key is present, a ``false`` or absent value takes - priority over any ``true`` values. In other words, implementations respect - ``only-deps`` if it is ``true`` for all instances of the same ``path``. - Otherwise, they treat it as ``false``. - - Example Dependency Groups Table ------------------------------- @@ -389,32 +249,60 @@ define four Dependency Groups: ``test``, ``docs``, ``typing``, and .. code:: toml [dependency-groups] - test = ["pytest", "coverage", {path = ".", editable = true}] + test = ["pytest", "coverage"] docs = ["sphinx", "sphinx-rtd-theme"] - typing = ["mypy", "types-requests", {path = ".", extras = ["types"], only-deps = true}] + typing = ["mypy", "types-requests"] typing-test = [{include = "typing"}, {include = "test"}, "useful-types"] - [project.optional-dependencies] - types = ["typing-extensions"] +Note that none of these Dependency Group declarations implicitly install the +current package, its dependencies, or any optional dependencies. +Use of a Dependency Group like ``test`` to test a package requires that the +user's configuration or toolchain also installs ``.``. For example, -Note how ``test`` and ``typing`` refer to the current package while ``docs`` -does not. This reflects the ability of Dependency Groups to be used in the same -manner as extras, adding to dependencies, or completely independently. +.. code-block:: shell -``typing-test`` is defined as a union of two existing groups, plus an -additional package. ``typing`` includes an extra, ``types``, and this is -included by extension under ``typing-test``. -Under ``typing-test`` implementations may choose whether or not to use an -editable installation of the current package or not, but they MUST treat -``only-deps`` as ``false``. + $TOOL install-dependency-group test + pip install -e . + +could be used (supposing ``$TOOL`` is a tool which supports installing +Dependency Groups) to build a testing environment. + +This also allows for the ``docs`` dependency group to be used without +installing the project as a package: + +.. code-block:: shell + + $TOOL install-dependency-group docs Package Building ---------------- -Build backends MUST NOT include Dependency Group data in built distributions. +Build backends MUST NOT include Dependency Group data in built distributions as +package metadata. -Use of Dependency Groups ------------------------- +It is valid to use Dependency Groups in the evaluation of dynamic metadata. +For example, a build backend may define ``dependencies`` as dynamic and use +dependency groups to compute the value of ``dependencies``. + +For example, a build backend could define the following data to evaluate +equivalently to ``dependencies=["aiohttp", "sqlalchemy"]``: + +.. code:: toml + + [project] + dynamic = ["dependencies"] + + [dependency-groups] + http = ["aiohttp"] + db = ["sqlalchemy"] + + [tool.some-build-tool.dynamic] + dependencies = { dependency-groups = ["http", "db"] } + +Build backends may use Dependency Groups in this way. + +Installing Dependency Groups +---------------------------- Tools which support Dependency Groups are expected to provide new options and interfaces to allow users to install from Dependency Groups. @@ -441,7 +329,72 @@ for how tools support the installation of Dependency Groups. Reference Implementation ======================== -There is currently no reference implementation/consumer of this specification. +The following Reference Implementation prints the contents of a Dependency +Group to stdout, newline delimited. +The output is therefore valid ``requirements.txt`` data. + +Although this PEP does not specify that cyclic includes are forbidden, the +Reference Implementation raises errors if they are encountered. + +.. code-block:: python + + import sys + import tomllib + + from packaging.requirements import Requirement + + + def _resolve_dependency_group( + dependency_groups: dict, group: str, past_groups: tuple[str] = () + ) -> list[str]: + if group in past_groups: + raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}") + + if group not in dependency_groups: + raise LookupError(f"Dependency group '{group}' not found") + + raw_group = dependency_groups[group] + if not isinstance(raw_group, list): + raise ValueError(f"Dependency group '{group}' is not a list") + + realized_group = [] + for item in raw_group: + if isinstance(item, str): + # packaging.requirements.Requirement parsing ensures that this is a valid + # PEP 508 Dependency Specifier + # raises InvalidRequirement on failure + Requirement(item) + realized_group.append(item) + elif isinstance(item, dict): + if tuple(item.keys()) != ("include",): + raise ValueError(f"Invalid dependency group item: {item}") + + include_group = next(iter(item.values())) + realized_group.extend( + _resolve_dependency_group( + dependency_groups, include_group, past_groups + (group,) + ) + ) + else: + raise ValueError(f"Invalid dependency group item: {item}") + + return realized_group + + + def resolve(dependency_groups: dict, group: str) -> list[str]: + if not isinstance(dependency_groups, dict): + raise TypeError("Dependency Groups table is not a dict") + if not isinstance(group, str): + raise TypeError("Dependency group name is not a str") + return _resolve_dependency_group(dependency_groups, group) + + + if __name__ == "__main__": + with open("pyproject.toml", "rb") as fp: + pyproject = tomllib.load(fp) + + dependency_groups = pyproject["dependency-groups"] + print("\n".join(resolve(pyproject["dependency-groups"], sys.argv[1]))) Backwards Compatibility ======================= @@ -482,15 +435,12 @@ without needing to learn about package building. A ``pyproject.toml`` file with only ``[dependency-groups]`` and no other tables is valid. -For both new and experienced users, the object style used in Dependency Object -Specifiers will need to be explained. Support for -``{path = ".", editable = true}`` and -``{path = ".", editable = true, extras = ["extra"]}`` should be taught -similarly to teaching ``pip install -e .`` and ``pip install -e '.[extra]'`` --- it intentionally mirrors the effects of those commands. - -Support for inclusion of one Dependency Group in another can be -taught as a homologue for one requirements file including another using ``-r``. +For both new and experienced users, the Dependency Group Includes will need to +be explained. For users with experience using ``requirements.txt``, this can be +described as an analogue for ``-r``. For new users, they should be taught that +an include allows one Dependency Group to extend another. Similar configuration +interfaces and the Python ``list.extend`` method may be used to explain the +idea by analogy. Rejected Ideas ============== @@ -519,24 +469,32 @@ However, there were three major issues with this approach: * the resulting strings would always need to be disambiguated from PEP 508 specifiers, complicating implementations -* support for the matrix of Path Dependency requirements (``editable``, ``only-deps``, - ``extras``) would require a complex syntax whose design was unclear +Why not allow for more non-PEP 508 dependency specifiers? +--------------------------------------------------------- -Why not restrict dependencies to PEP 508 only? ----------------------------------------------- +Several use cases surfaced during discussion which need more expressive +specifiers than are possible with :pep:`508`. -There are known use cases for: +"Path Dependencies", referring to local paths, and references to +``[project.dependencies]`` were of particular interest. -* including one Dependency Group in another -* including the current package (if the project is a package) -* including the current package with extras (if the project is a package) -* installing ``project.dependencies`` from the current project but not - installing the current project (as can be done with - ``{path = ".", only-deps = true}``) -* specifying that an installation be done in editable mode +However, there are no existing standards for these features (excepting the +de-facto standard of ``pip``'s implementation details). -These are not satisfiable without some expansion of syntax beyond what is -possible with existing Dependency Specifiers (:pep:`508`). +As a result, attempting to include these features in this PEP results in a +significant growth in scope, to attempt to standardize these various features +and ``pip`` behaviors. + +Special attention was devoted to attempting to standardize the expression of +editable installations, as expressed by ``pip install -e`` and :pep:`660`. +However, although the creation of editable installs is standardized for build +backends, the behavior of editables is not standardized for installers. +Inclusion of editables in this PEP requires that any supporting tool allows for +the installation of editables. + +Therefore, although Poetry and PDM provide syntaxes for some of these features, +they are considered insufficiently standardized at present for inclusion in +Dependency Groups. Why is the table not named ``[run]``, ``[project.dependency-groups]``, ...? --------------------------------------------------------------------------- @@ -553,13 +511,15 @@ outlines multiple groups of dependencies, which makes ``[run]`` a less appropriate fit -- this is not just dependency data for a specific runtime context, but for multiple contexts. -``[project.dependency-groups]`` would be ideal, but has major downsides for -non-package projects. ``[project]`` requires several keys to be defined, such -as ``name`` and ``version``. Using this name would either require redefining -the ``[project]`` table to allow for these keys to be absent, or else would -impose a requirement on non-package projects to define and use these keys. By -extension, it would effectively require any non-package project allow itself to -be treated as a package. +``[project.dependency-groups]`` would offer a nice parallel with +``[project.dependencies]`` and ``[project.optional-dependencies]``, but has +major downsides for non-package projects. +``[project]`` requires several keys to be defined, such as ``name`` and +``version``. Using this name would either require redefining the ``[project]`` +table to allow for these keys to be absent, or else would impose a requirement +on non-package projects to define and use these keys. By extension, it would +effectively require any non-package project allow itself to be treated as a +package. Why is pip's planned implementation of ``--only-deps`` not sufficient? ---------------------------------------------------------------------- @@ -572,9 +532,6 @@ without installing the current package. It does not address the needs of non-package projects, nor does it allow for the installation of an extra without the package dependencies. -Therefore, while it may be a useful feature for pip to pursue, it does not -address the same use-cases addressed here. - Why isn't a solution? ------------------------------------------- @@ -594,41 +551,13 @@ Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.) Open Issues =========== -Documenting Distinctions from Poetry and PDM Object Formats ------------------------------------------------------------ - -The object format here is similar to the ones used by Poetry and PDM, but not -identical. -The differences should be captured in Appendix B, as a part of documenting Poetry -and PDM behaviors and data formats. - -Editable Installation May or May Not Be In Scope ------------------------------------------------- - -The ``editable`` field formalizes a behavior which is supported by setuptools -and pip, but which has three major issues: - -- it combines information about the existence of a dependency with information - about *how* that dependency should be installed - -- editable installs, as implemented today, do not provide for seamless updates - to package metadata - -- there is no specification for the behavior of an editable install -- this is - an implementation-specific feature - -Therefore, ``editable`` may need to be removed. -Such a removal must account for how existing tools and workflows which rely on -editable installs will be impacted. - Should ``include`` accept a list? --------------------------------- This would enable more compact includes of multiple other Dependency Groups, at the cost of a minor complication to the specification. -Footnotes -========= +.. _prior_art: Appendix A: Prior Art in Non-Python Languages ============================================= @@ -636,6 +565,8 @@ Appendix A: Prior Art in Non-Python Languages This section is primarily informational and serves to document how other language ecosystems solve similar problems. +.. _javascript_prior_art: + JavaScript and ``package.json`` ------------------------------- @@ -781,6 +712,8 @@ behavior more precisely. Furthermore, these values can be declared in ``.npmrc`` files, allowing per-user and per-project configurations to control installation behaviors. +.. _ruby_prior_art: + Ruby & Ruby Gems ---------------- @@ -948,15 +881,337 @@ following ``Gemfile`` content: gem 'kramdown' end +.. _python_prior_art: + Appendix B: Prior Art in Python =============================== -TODO +In the absence of any prior standard for Dependency Groups, two known workflow +tools, PDM and Poetry, have defined their own solutions. + +This section will primarily focus on these two tools as cases of prior art +regarding the definition and use of Dependency Groups in Python. + +Projects are Packages +--------------------- + +Both PDM and Poetry treat the projects they support as packages. +This allows them to use and interact with standard ``pyproject.toml`` metadata +for some of their needs, and allows them to support installation of the +"current project" by doing a build and install using their build backends. + +Effectively, this means that neither Poetry nor PDM supports non-package projects. + +Non-Standard Dependency Specifiers +---------------------------------- + +PDM and Poetry extend :pep:`508` dependency specifiers with additional features +which are not part of any shared standard. +The two tools use slightly different approaches to these problems, however. + +PDM supports specifying local paths, and editable installs, via a syntax which +looks like a set of arguments to ``pip install``. For example, the following +dependency group includes a local package in editable mode: + +.. code-block:: toml + + [tool.pdm.dev-dependencies] + mygroup = ["-e file:///${PROJECT_ROOT}/foo"] + +This declares a dependency group ``mygroup`` which includes a local editable +install from the ``foo`` directory. + +Poetry describes dependency groups as tables, mapping package names to +specifiers. For example, the same configuration as the above ``mygroup`` +example might appear as follows under Poetry: + +.. code-block:: toml + + [tool.poetry.group.mygroup] + foo = { path = "foo", editable = true } + +PDM restricts itself to a string syntax, and Poetry introduces tables which +describe dependencies. + +Installing and Referring to Dependency Groups +--------------------------------------------- + +Both PDM and Poetry have tool-specific support for installing dependency +groups. Because both projects support their own lockfile formats, they also +both have the capability to transparently use a dependency group name to refer +to the *locked* dependency data for that group. + +However, neither tool's dependency groups can be referenced natively from other +tools like ``tox``, ``nox``, or ``pip``. +Attempting to install a dependency group under ``tox``, for example, requires +an explicit call to PDM or Poetry to parse their dependency data and do the +relevant installation step. + +.. _use_cases: Appendix C: Use Cases ===================== -TODO +Web Applications +---------------- + +A web application (e.g. a Django or Flask app) often does not need to build a +distribution, but bundles and ships its source to a deployment toolchain. + +For example, a source code repository may define python packaging metadata as +well as containerization or other build pipeline metadata (``Dockerfile``, +etc). +The python application is built by copying the entire repository into a +build context, installing dependencies, and bundling the result as a machine +image or container. + +Such applications have dependency groups for the build, but also for linting, +testing, etc. In practice, today, these applications often define themselves as +packages to be able to use packaging tools and mechanisms like ``extras`` to +manage their dependency groups. However, they are not conceptually packages, +meant for distribution in sdist or wheel format. + +Dependency Groups allow these applications to define their various dependencies +without relying on packaging metadata, and without trying to express their +needs in packaging terms. + +Libraries +''''''''' + +Libraries are python packages which build distributions (sdist and wheel) and +publish them to PyPI. + +For libraries, Dependency Groups represent an alternative to ``extras`` for +defining groups of development dependencies, with the important advantages +noted above. + +A library may define groups for ``test`` and ``typing`` which allow testing and +type-checking, and therefore rely on the library's own dependencies (as +specified in ``[project.dependencies]``). + +Other development needs may not require installation of the package at all. For +example, a ``lint`` Dependency Group may be valid and faster to install without +the library, as it only installs tools like ``black``, ``ruff``, or ``flake8``. + +``lint`` and ``test`` environments may also be valuable locations to hook in +IDE or editor support. See the case below for a fuller description of such +usage. + +Here's an example Dependency Groups table which might be suitable for a +library: + +.. code-block:: toml + + [dependency-groups] + test = ["pytest<8", "coverage"] + typing = ["mypy==1.7.1", "types-requests"] + lint = ["black", "flake8"] + typing-test = [{include = "typing"}, "pytest<8"] + +Note that none of these implicitly install the library itself. +It is therefore the responsibility of any environment management toolchain to +install the appropriate Dependency Groups along with the library when needed, +as in the case of ``test``. + +Data Science Projects +''''''''''''''''''''' + +Data Science Projects typically take the form of a logical collection of +scripts and utilities for processing and analyzing data, using a common +toolchain. Components may be defined in the Jupyter Notebook format (ipynb), +but rely on the same common core set of utlities. + +In such a project, there is no package to build or install. Therefore, +``pyproject.toml`` currently does not offer any solution for dependency +management or declaration. + +It is valuable for such a project to be able to define at least one major +grouping of dependencies. For example: + +.. code-block:: toml + + [dependency-groups] + main = ["numpy", "pandas", "matplotlib"] + +However, it may also be necessary for various scripts to have additional +supporting tools. Projects may even have conflicting or incompatible tools or +tool versions for different components, as they evolve over time. + +Consider the following more elaborate configuration: + +.. code-block:: toml + + [dependency-groups] + main = ["numpy", "pandas", "matplotlib"] + scikit = [{include = "main"}, "scikit-learn==1.3.2"] + scikit-old = [{include = "main"}, "scikit-learn==0.24.2"] + +This defines ``scikit`` and ``scikit-old`` as two similar variants of the +common suite of dependencies, pulling in different versions of ``scikit-learn`` +to suit different scripts. + +This PEP only defines these data. It does not formalize any mechanism for a +Data Science Project (or any other type of project) to install the dependencies +into known environments or associate those environments with the various +scripts. Such combinations of data are left as a problem for tool authors to +solve, and perhaps eventually standardize. + +Lockfile Generation +''''''''''''''''''' + +There are a number of tools which generate lockfiles in the Python ecosystem +today. PDM and Poetry each use their own lockfile formats, and pip-tools +generates ``requirements.txt`` files with version pins and hashes. + +Dependency Groups are not an appropriate place to store lockfiles, as they lack +many of the necessary features. Most notably, they cannot store hashes, which +most lockfile users consider essential. + +However, Dependency Groups are a valid input to tools which generate lockfiles. +Furthermore, PDM and Poetry both allow a Dependency Group name (under their +notions of Dependency Groups) to be used to refer to its locked variant. + +Therefore, consider a tool which produces lockfiles, here called ``$TOOL``. +It might be used as follows: + +.. code:: shell + + $TOOL lock --dependency-group=test + $TOOL install --dependency-group=test --use-locked + +All that such a tool needs to do is to ensure that its lockfile data records +the name ``test`` in order to support such usage. + +The mutual compatibility of Dependency Groups is not guaranteed. For example, +the Data Science example above shows conflicting versions of ``scikit-learn``. +Therefore, installing multiple locked dependency groups in tandem may require +that tools apply additional constraints or generate additional lockfile data. +These problems are considered out of scope for this PEP. + +As two examples of how combinations might be locked: + +* A tool might require that lockfile data be explicitly generated for any + combination to be considered valid + +* Poetry implements the requirement that all Dependency Groups be mutually + compatible, and generates only one locked version. (Meaning it finds a single + solution, rather than a set or matrix of solutions.) + +Environment Manager Inputs +'''''''''''''''''''''''''' + +A common usage in tox, Nox, and Hatch is to install a set of dependencies into +a testing environment. + +For example, under ``tox.ini``, type checking dependencies may be defined +inline: + +.. code-block:: ini + + [testenv:typing] + deps = + pyright + useful-types + commands = pyright src/ + +This combination provides a desirable developer experience within a limited +context. Under the relevant environment manager, the dependencies which are +needed for the test environment are declared alongside the commands which need +those dependencies. They are not published in package metadata, as ``extras`` +would be, and they are discoverable for the tool which needs them to build the +relevant environment. + +Dependency Groups apply to such usages by effectively "lifting" these +requirements data from a tool-specific location into a more broadly available +one. In the example above, only ``tox`` has access to the declared list of +dependencies. Under an implementation supporting dependency groups, the same +data might be available in a Dependency Group: + +.. code-block:: toml + + [dependency-groups] + typing = ["pyright", "useful-types"] + +The data can then be used under multiple tools. For example, ``tox`` might +implement support as ``dependency_groups = typing``, replacing the ``deps`` +usage above. + +In order for Dependency Groups to be a viable alternative for users of +environment managers, the environment managers will need to support processing +Dependency Groups similarly to how they support inline dependency declaration. + +Embedded ``pyproject.toml`` in Scripts +'''''''''''''''''''''''''''''''''''''' + +:pep:`723`, defines embedded ``pyproject.toml`` data within scripts. For this +use case, it is necessary to declare the dependencies of a script in a data +format which is also valid ``pyproject.toml`` content. However, +``[project.dependencies]`` is considered inappropriate because a script is not +a package -- and the ``[project]`` table is defined under constraints which +reflect valid metadata for packages. + +:pep:`723` provisionally uses a ``[run.dependencies]`` table for this purpose, +but Dependency Groups offer a more general solution to the problem of +dependency declaration covering a broader set of use cases than the (informal) +``[run]`` proposal. + +Rather than a singular group of dependencies, and a singular runtime context, +Dependency Groups support multiple named groups for different purposes and +environments. + +Because Dependency Groups are multiple, unlike ``[run.dependencies]``, it is +necessary for any standard which wants to use Dependency Groups to define how +it will leverage them. +This PEP does not assign special meanings to any names for Dependency Groups, +but it is valid for standards consuming Dependency Groups to define +conventional names. + +To use Dependency Groups within :pep:`723`, there are two primary options: + +* declare, as part of the specification of :pep:`723`, that the ``run`` + Dependency Group is conventionally the one which will be used + +* declare a mechanism for naming a Dependency Group to use + +For example, the following two ``pyproject.toml`` contents would be valid ways +of declaring dependencies for a script: + +.. code-block:: toml + + [dependency-groups] + run = ["numpy"] + +or + +.. code-block:: toml + + [dependency-groups] + mygroupname = ["numpy"] + [run] + use-group = "mygroupname" + +This PEP declares no preference for how other standards consume this +information, but aims to make such consumption feasible. + +IDE and Editor Use of Requirements Data +''''''''''''''''''''''''''''''''''''''' + +Similar to the :pep:`723` case above, IDE and Editor integrations may benefit +from conventional name definitions or configurable ones. + +However, there are at least two known scenarios in which it is valuable for an +editor or IDE to be capable of discovering the non-published dependencies of a +project: + +* testing: IDEs such as VS Code support GUI interfaces for running particular + tests + +* linting: editors and IDEs often support linting and autoformatting + integrations which highlight or autocorrect errors + +These cases could be handled by defining conventional group names like +``test``, ``lint``, and ``fix``, or by defining configuration mechanisms which +allow the selection of Dependency Groups. Copyright =========