python-peps/peps/pep-0735.rst

PEP: 735
Title: Dependency Groups in pyproject.toml
Author: Stephen Rosen <sirosen0@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/39233
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 20-Nov-2023
Post-History: `14-Nov-2023 <https://discuss.python.org/t/29684>`__, `20-Nov-2023 <https://discuss.python.org/t/39233>`__

Abstract
========

This PEP specifies a mechanism for storing package requirements in
``pyproject.toml`` files such that they are not included in any built distribution of
the project.

This is suitable for creating named groups of dependencies, similar to
``requirements.txt`` files, which launchers, IDEs, and other tools can find and
identify by name.

The feature defined here is referred to as "Dependency Groups".

Motivation
==========

There are two major use cases for which the Python community has no
standardized answer:

* How should development dependencies be defined for packages?

* How should dependencies be defined for projects which do not build
  distributions (non-package projects)?

In support of these two needs, there are two common solutions which are similar
to this proposal:

* ``requirements.txt`` files

* package `extras <https://packaging.python.org/en/latest/specifications/dependency-specifiers/#extras>`__

Both ``requirements.txt`` files and ``extras`` have limitations which this
standard seeks to overcome.

Note that the two use cases above describe two different types of projects
which this PEP seeks to support:

* Python packages, such as libraries

* non-package projects, such as data science projects

Limitations of ``requirements.txt`` files
-----------------------------------------

Many projects may define one or more ``requirements.txt`` iles,
and may arrange them either at the project root (e.g. ``requirements.txt`` and
``test-requirements.txt``) or else in a directory (e.g.
``requirements/base.txt`` and ``requirements/test.txt``). However, there are
major issues with the use of requirements files in this way:

* There is no standardized naming convention such that tools can discover or
  use these files by name.

* ``requirements.txt`` files are *not standardized*, but instead provide
  options to ``pip``.

As a result, it is difficult to define tool behaviors based on
``requirements.txt`` files. They are not trivial to discover or identify by
name, and their contents may contain a mix of package specifiers and additional
``pip`` options.

The lack of a standard for ``requirements.txt`` contents also means they are
not portable to any alternative tools which wish to process them other than
``pip``.

Additionally, ``requirements.txt`` files require a file per dependency list.
For some use-cases, this makes the marginal cost of dependency groupings high,
relative to their benefit.
A terser declaration is beneficial to projects with a number of small groups of
dependencies.

In contrast with this, Dependency Groups are defined at a well known location
in ``pyproject.toml`` with fully standardized contents. Not only will they have
immediate utility, but they will also serve as a starting point for future
standards.

Limitations of ``extras``
-------------------------

``extras`` are additional package metadata declared in the
``[project.optional-dependencies]`` table. They provide names for lists of
package specifiers which are published as part of a package's metadata, and
which a user can request under that name, as in ``pip install 'foo[bar]'`` to
install ``foo`` with the ``bar`` extra.

Because ``extras`` are package metadata, they are not usable when a project
does not build a distribution (i.e., is not a package).

For projects which are packages, ``extras`` are a common solution for defining
development dependencies, but even under these circumstances they have
downsides:

* Because an ``extra`` defines optional *additional* dependencies, it is not
  possible to install an ``extra`` without installing the current package and
  its dependencies.

* Because they are user-installable, ``extras`` are part of the public interface
  for packages. Because ``extras`` are published, package developers often are
  concerned about ensuring that their development extras are not confused with
  user-facing extras.

Rationale
=========

This PEP defines the storage of requirements data in lists within a
``[dependency-groups]`` table.
This name was chosen to match the canonical name of the feature
("Dependency Groups").

This format should be as simple and learnable as possible, having a format
very similar to existing ``requirements.txt`` files for many cases. Each list
in ``[dependency-groups]`` is defined as a list of package specifiers. For
example:

.. code-block:: toml

    [dependency-groups]
    test = ["pytest>7", "coverage"]

There are a number of use cases for ``requirements.txt`` files which require
data which cannot be expressed in :pep:`508` dependency specifiers. Such 
fields are not valid in Dependency Groups. Including many of the data and
fields which ``pip`` supports, such as index servers, hashes, and path
dependencies, requires new standards. This standard leaves room for new
standards and developments, but does not attempt to support all valid
``requirements.txt`` contents.

The only exception to this is the ``-r`` flag which ``requirements.txt`` files
use to include one file in another. Dependency Groups support an "include"
mechanism which is similar in meaning, allowing one dependency group to extend
another.

Dependency Groups have two additional features which are similar to
``requirements.txt`` files:

* they are not published as part of any built distribution

* installation of a dependency group does not imply installation of a package's
  dependencies or the package itself

Use Cases
---------

The following use cases are considered important targets for this PEP. They are
defined in greater detail in the `Use Cases Appendix <use_cases>`_.

* Web Applications deployed via a non-python-packaging build process
* Libraries with unpublished dev dependency groups
* Data science projects with groups of dependencies but no core package
* *Input data* to lockfile generation (Dependency Groups should generally not
  be used as a location for locked dependency data)
* Input data to an environment manager, such as tox, Nox, or Hatch
* Embedded ``pyproject.toml`` data in scripts, as proposed in :pep:`723`
* Configurable IDE discovery of test and linter requirements

Regarding Poetry and PDM Dependency Groups
------------------------------------------

The existing Poetry and PDM tools already offer a feature which each calls
"Dependency Groups", but using non-standard data belonging to the ``poetry``
and ``pdm`` tools.
(PDM also uses extras for some Dependency Groups, and overlaps the notion
heavily with extras.)

This PEP does not support all of the features of Poetry and PDM, which, like
``requirements.txt`` files for ``pip``, support several non-standard extensions
to common dependency specifiers.

It should be possible for such tools to use standardized Dependency Groups as
extensions of their own Dependency Group mechanisms.
However, defining a new data format which replaces the existing Poetry and PDM
solutions is a non-goal, as it would require standardizing their various
non-standard features.

Dependency Groups are not Hidden Extras
---------------------------------------

Dependency Groups are very similar to extras which go unpublished.
However, there are two major features which distinguish them from extras
further:

* they support non-package projects

* installation of a Dependency Group does not imply installation of a package's
  dependencies (or the package itself)

Specification
=============

This PEP defines a new section (table) in ``pyproject.toml`` files named
``dependency-groups``. The ``dependency-groups`` table contains an arbitrary
number of user-defined keys, each of which has, as its value, a list of
requirements (defined below).  These keys must match the following
regular expression: ``[a-z0-9][a-z0-9-]*[a-z0-9]``. Meaning that they must be
all lower-case alphanumerics, with ``-`` allowed only in the middle, and at
least two characters long. These requirements are chosen so that the
normalization rules used for PyPI package names are unnecessary as the names
are already normalized.

Requirement lists under ``dependency-groups`` may contain strings, tables
("dicts" in Python), or a mix of strings and tables.

Strings in requirement lists must be valid
`Dependency Specifiers <https://packaging.python.org/en/latest/specifications/dependency-specifiers/>`__,
as defined in :pep:`508`.

Tables in requirement lists must be valid Dependency Object Specifiers,
defined below.

Dependency Object Specifiers
----------------------------

Dependency Object Specifiers are tables which define zero or more dependencies.

This PEP standardizes only one type of Dependency Object Specifier, a
"Dependency Group Include". Other types may be added in future standards.

Dependency Group Include
''''''''''''''''''''''''

A Dependency Group Include includes the dependencies of another Dependency
Group in the current Dependency Group.

An include is defined as a table with exactly one key, ``"include"``, whose
value is a string, the name of another Dependency Group.

For example, ``{include = "test"}`` is an include which expands to the
contents of the ``test`` Dependency Group.

Example Dependency Groups Table
-------------------------------

The following is an example of a partial ``pyproject.toml`` which uses this to
define four Dependency Groups: ``test``, ``docs``, ``typing``, and
``typing-test``:

.. code:: toml

    [dependency-groups]
    test = ["pytest", "coverage"]
    docs = ["sphinx", "sphinx-rtd-theme"]
    typing = ["mypy", "types-requests"]
    typing-test = [{include = "typing"}, {include = "test"}, "useful-types"]

Note that none of these Dependency Group declarations implicitly install the
current package, its dependencies, or any optional dependencies.
Use of a Dependency Group like ``test`` to test a package requires that the
user's configuration or toolchain also installs ``.``. For example,

.. code-block:: shell

    $TOOL install-dependency-group test
    pip install -e .

could be used (supposing ``$TOOL`` is a tool which supports installing
Dependency Groups) to build a testing environment.

This also allows for the ``docs`` dependency group to be used without
installing the project as a package:

.. code-block:: shell

    $TOOL install-dependency-group docs

Package Building
----------------

Build backends MUST NOT include Dependency Group data in built distributions as
package metadata.

It is valid to use Dependency Groups in the evaluation of dynamic metadata.
For example, a build backend may define ``dependencies`` as dynamic and use
dependency groups to compute the value of ``dependencies``.

For example, a build backend could define the following data to evaluate
equivalently to ``dependencies=["aiohttp", "sqlalchemy"]``:

.. code:: toml

    [project]
    dynamic = ["dependencies"]

    [dependency-groups]
    http = ["aiohttp"]
    db = ["sqlalchemy"]

    [tool.some-build-tool.dynamic]
    dependencies = { dependency-groups = ["http", "db"] }

Build backends may use Dependency Groups in this way.

Installing Dependency Groups
----------------------------

Tools which support Dependency Groups are expected to provide new options and
interfaces to allow users to install from Dependency Groups.

No syntax is defined for expressing the Dependency Group of a package, for two
reasons:

* it would not be valid to refer to the Dependency Groups of a third-party
  package from PyPI (because the data is defined to be unpublished)

* there is not guaranteed to be a current package for Dependency Groups -- part
  of their purpose is to support non-package projects

For example, a possible pip interface for installing Dependency Groups
would be:

.. code:: shell

    pip install --dependency-groups=test,typing

Note that this is only an example. This PEP does not declare any requirements
for how tools support the installation of Dependency Groups.

Reference Implementation
========================

The following Reference Implementation prints the contents of a Dependency
Group to stdout, newline delimited.
The output is therefore valid ``requirements.txt`` data.

Although this PEP does not specify that cyclic includes are forbidden, the
Reference Implementation raises errors if they are encountered.

.. code-block:: python

    import sys
    import tomllib

    from packaging.requirements import Requirement


    def _resolve_dependency_group(
        dependency_groups: dict, group: str, past_groups: tuple[str] = ()
    ) -> list[str]:
        if group in past_groups:
            raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}")

        if group not in dependency_groups:
            raise LookupError(f"Dependency group '{group}' not found")

        raw_group = dependency_groups[group]
        if not isinstance(raw_group, list):
            raise ValueError(f"Dependency group '{group}' is not a list")

        realized_group = []
        for item in raw_group:
            if isinstance(item, str):
                # packaging.requirements.Requirement parsing ensures that this is a valid
                # PEP 508 Dependency Specifier
                # raises InvalidRequirement on failure
                Requirement(item)
                realized_group.append(item)
            elif isinstance(item, dict):
                if tuple(item.keys()) != ("include",):
                    raise ValueError(f"Invalid dependency group item: {item}")

                include_group = next(iter(item.values()))
                realized_group.extend(
                    _resolve_dependency_group(
                        dependency_groups, include_group, past_groups + (group,)
                    )
                )
            else:
                raise ValueError(f"Invalid dependency group item: {item}")

        return realized_group


    def resolve(dependency_groups: dict, group: str) -> list[str]:
        if not isinstance(dependency_groups, dict):
            raise TypeError("Dependency Groups table is not a dict")
        if not isinstance(group, str):
            raise TypeError("Dependency group name is not a str")
        return _resolve_dependency_group(dependency_groups, group)


    if __name__ == "__main__":
        with open("pyproject.toml", "rb") as fp:
            pyproject = tomllib.load(fp)

        dependency_groups = pyproject["dependency-groups"]
        print("\n".join(resolve(pyproject["dependency-groups"], sys.argv[1])))

Backwards Compatibility
=======================

At time of writing, the ``dependency-groups`` namespace within a
``pyproject.toml`` file is unused. Since the top-level namespace is
reserved for use only by standards specified at packaging.python.org,
there should be no direct backwards compatibility concerns.

Security Implications
=====================

This PEP introduces new syntaxes and data formats for specifying dependency
information in projects. However, it does not introduce newly specified
mechanisms for handling or resolving dependencies.

It therefore does not carry security concerns other than those inherent in any
tools which may already be used to install dependencies -- i.e. malicious
dependencies may be specified here, just as they may be specified in
``requirements.txt`` files.

How to Teach This
=================

This feature should be referred to by its canonical name, "Dependency Groups".

The basic form of usage should be taught as a variant on typical
``requirements.txt`` data. Standard dependency specifiers (:pep:`508`) can be
added to a named list. Rather than asking pip to install from a
``requirements.txt`` file, either pip or a relevant workflow tool will install
from a named Dependency Group.

For new Python users, they may be taught directly to create a section in
``pyproject.toml`` containing their Dependency Groups, similarly to how they
are currently taught to use ``requirements.txt`` files.
This also allows new Python users to learn about ``pyproject.toml`` files
without needing to learn about package building.
A ``pyproject.toml`` file with only ``[dependency-groups]`` and no other tables
is valid.

For both new and experienced users, the Dependency Group Includes will need to
be explained. For users with experience using ``requirements.txt``, this can be
described as an analogue for ``-r``. For new users, they should be taught that
an include allows one Dependency Group to extend another. Similar configuration
interfaces and the Python ``list.extend`` method may be used to explain the
idea by analogy.

Rejected Ideas
==============

Why not define each Dependency Group as a table?
------------------------------------------------

If our goal is to allow for future expansion, then defining each Dependency
Group as a subtable, thus enabling us to attach future keys to each group,
allows for the greatest future flexibility.

However, it also makes the structure nested more deeply, and therefore harder
to teach and learn. One of the goals of this PEP is to be an easy replacement
for many ``requirements.txt`` use-cases.

Why not define a special string syntax to extend Dependency Specifiers?
-----------------------------------------------------------------------

Earlier drafts of this specification defined syntactic forms for Dependency
Group Includes and Path Dependencies.

However, there were three major issues with this approach:

* it complicates the string syntax which must be taught, beyond PEP 508

* the resulting strings would always need to be disambiguated from PEP 508
  specifiers, complicating implementations

Why not allow for more non-PEP 508 dependency specifiers?
---------------------------------------------------------

Several use cases surfaced during discussion which need more expressive
specifiers than are possible with :pep:`508`.

"Path Dependencies", referring to local paths, and references to
``[project.dependencies]`` were of particular interest.

However, there are no existing standards for these features (excepting the
de-facto standard of ``pip``'s implementation details).

As a result, attempting to include these features in this PEP results in a
significant growth in scope, to attempt to standardize these various features
and ``pip`` behaviors.

Special attention was devoted to attempting to standardize the expression of
editable installations, as expressed by ``pip install -e`` and :pep:`660`.
However, although the creation of editable installs is standardized for build
backends, the behavior of editables is not standardized for installers.
Inclusion of editables in this PEP requires that any supporting tool allows for
the installation of editables.

Therefore, although Poetry and PDM provide syntaxes for some of these features,
they are considered insufficiently standardized at present for inclusion in
Dependency Groups.

Why is the table not named ``[run]``, ``[project.dependency-groups]``, ...?
---------------------------------------------------------------------------

There are many possible names for this concept.
It will have to live alongside the already existing ``[project.dependencies]``
and ``[project.optional-dependencies]`` tables, and possibly a new
``[external]`` dependency table as well (at time of writing, :pep:`725`, which
defines the ``[external]`` table, is in progress).

``[run]`` was a leading proposal in earlier discussions, but its proposed usage
centered around a single set of runtime dependencies. This PEP explicitly
outlines multiple groups of dependencies, which makes ``[run]`` a less
appropriate fit -- this is not just dependency data for a specific runtime
context, but for multiple contexts.

``[project.dependency-groups]`` would offer a nice parallel with
``[project.dependencies]`` and ``[project.optional-dependencies]``, but has
major downsides for non-package projects.
``[project]`` requires several keys to be defined, such as ``name`` and
``version``. Using this name would either require redefining the ``[project]``
table to allow for these keys to be absent, or else would impose a requirement
on non-package projects to define and use these keys. By extension, it would
effectively require any non-package project allow itself to be treated as a
package.

Why is pip's planned implementation of ``--only-deps`` not sufficient?
----------------------------------------------------------------------

pip currently has a feature on the roadmap to add an
`--only-deps flag <https://github.com/pypa/pip/issues/11440>`_.
This flag is intended to allow users to install package dependencies and extras
without installing the current package.

It does not address the needs of non-package projects, nor does it allow for
the installation of an extra without the package dependencies.

Why isn't <environment manager> a solution?
-------------------------------------------

Existing environment managers like tox, Nox, and Hatch already have
the ability to list inlined dependencies as part of their configuration data.
This meets many development dependency needs, and clearly associates dependency
groups with relevant tasks which can be run.
These mechanisms are *good* but they are not *sufficient*.

First, they do not address the needs of non-package projects.

Second, there is no standard for other tools to use to access these data. This
has impacts on high-level tools like IDEs and Dependabot, which cannot support
deep integration with these Dependency Groups. (For example, at time of writing
Dependabot will not flag dependencies which are pinned in ``tox.ini`` files.)

Open Issues
===========

Should ``include`` accept a list?
---------------------------------

This would enable more compact includes of multiple other Dependency Groups, at
the cost of a minor complication to the specification.

.. _prior_art:

Appendix A: Prior Art in Non-Python Languages
=============================================

This section is primarily informational and serves to document how other
language ecosystems solve similar problems.

.. _javascript_prior_art:

JavaScript and ``package.json``
-------------------------------

In the JavaScript community, packages contain a canonical configuration and
data file, similar in scope to ``pyproject.toml``, at ``package.json``.

Two keys in ``package.json`` control dependency data: ``"dependencies"`` and
``"devDependencies"``. The role of ``"dependencies"`` is effectively the same
as that of ``[project.dependencies]`` in ``pyproject.toml``, declaring the
direct dependencies of a package.

``"dependencies"`` data
'''''''''''''''''''''''

Dependency data is declared in ``package.json`` as a mapping from package names
to version specifiers.

Version specifiers support a small grammar of possible versions, ranges, and
other values, similar to Python's :pep:`440` version specifiers.

For example, here is a partial ``package.json`` file declaring a few
dependencies:

.. code-block:: json

    {
        "dependencies": {
            "@angular/compiler": "^17.0.2",
            "camelcase": "8.0.0",
            "diff": ">=5.1.0 <6.0.0"
        }
    }

The use of the ``@`` symbol is a `scope
<https://docs.npmjs.com/cli/v10/using-npm/scope>`__ which declares the package
owner, for organizationally owned packages.
``"@angular/compiler"`` therefore declares a package named ``compiler`` grouped
under ``angular`` ownership.

Dependencies Referencing URLs and Local Paths
'''''''''''''''''''''''''''''''''''''''''''''

Dependency specifiers support a syntax for URLs and Git repositories, similar
to the provisions in Python packaging.

URLs may be used in lieu of version numbers.
When used, they implicitly refer to tarballs of package source code.

Git repositories may be similarly used, including support for committish
specifiers.

Unlike :pep:`440`, NPM allows for the use of local paths to package source code
directories for dependencies. When these data are added to ``package.json`` via
the standard ``npm install --save`` command, the path is normalized to a
relative path, from the directory containing ``package.json``, and prefixed
with ``file:``. For example, the following partial ``package.json`` contains a
reference to a sibling of the current directory:

.. code-block:: json

    {
        "dependencies": {
            "my-package": "file:../foo"
        }
    }

The `official NPM documentation
<https://docs.npmjs.com/cli/v8/configuring-npm/package-json#local-paths>`__
states that local path dependencies "should not" be published to public package
repositories, but makes no statement about the inherent validity or invalidity
of such dependency data in published packages.

``"devDependencies"`` data
''''''''''''''''''''''''''

``package.json`` is permitted to contain a second section named
``"devDependencies"``, in the same format as ``"dependencies"``.
The dependencies declared in ``"devDependencies"`` are not installed by default
when a package is installed from the package repository (e.g. as part of a
dependency being resolved) but are installed when ``npm install`` is run in the
source tree containing ``package.json``.

Just as ``"dependencies"`` supports URLs and local paths, so does
``"devDependencies"``.

``"peerDependencies"`` and ``"optionalDependencies"``
'''''''''''''''''''''''''''''''''''''''''''''''''''''

There are two additional, related sections in ``package.json`` which have
relevance.

``"peerDependencies"`` declares a list of dependencies in the same format as
``"dependencies"``, but with the meaning that these are a compatibility
declaration.
For example, the following data declares compatibility with package ``foo``
version 2:

.. code-block:: json

    {
        "peerDependencies": {
            "foo": "2.x"
        }
    }

``"optionalDependencies"`` declares a list of dependencies which should be
installed if possible, but which should not be treated as failures if they are
unavailable. It also uses the same mapping format as ``"dependencies"``.

``"peerDependenciesMeta"``
~~~~~~~~~~~~~~~~~~~~~~~~~~

``"peerDependenciesMeta"`` is a section which allows for additional control
over how ``"peerDependencies"`` are treated.

Warnings about missing dependencies can be disabled by setting packages to
``optional`` in this section, as in the following sample:

.. code-block:: json

    {
        "peerDependencies": {
            "foo": "2.x"
        },
        "peerDependenciesMeta": {
            "foo": {
                "optional": true
            }
        }
    }

``--omit`` and ``--include``
''''''''''''''''''''''''''''

The ``npm install`` command supports two options, ``--omit`` and ``--include``,
which can control whether "prod", "dev", "optional", or "peer" dependencies are installed.

The "prod" name refers to dependencies listed under ``"dependencies"``.

By default, all four groups are installed when ``npm install`` is executed
against a source tree, but these options can be used to control installation
behavior more precisely.
Furthermore, these values can be declared in ``.npmrc`` files, allowing
per-user and per-project configurations to control installation behaviors.

.. _ruby_prior_art:

Ruby & Ruby Gems
----------------

Ruby projects may or may not be intended to produce packages ("gems") in the
Ruby ecosystem. In fact, the expectation is that most users of the language do
not want to produce gems and have no interest in producing their own packages.
Many tutorials do not touch on how to produce packages, and the toolchain never
requires user code to be packaged for supported use-cases.

Ruby splits requirement specification into two separate files.

- ``Gemfile``: a dedicated file which only supports requirement data in the form
  of dependency groups
- ``<package>.gemspec``: a dedicated file for declaring package (gem) metadata

The ``bundler`` tool, providing the ``bundle`` command, is the primary interface
for using ``Gemfile`` data.

The ``gem`` tool is responsible for building gems from ``.gemspec`` data, via the
``gem build`` command.

Gemfiles & bundle
'''''''''''''''''

A `Gemfile <https://bundler.io/v1.12/man/gemfile.5.html>`__ is a Ruby file
containing ``gem`` directives enclosed in any number of ``group`` declarations.
``gem`` directives may also be used outside of the ``group`` declaration, in which
case they form an implicitly unnamed group of dependencies.

For example, the following ``Gemfile`` lists ``rails`` as a project dependency.
All other dependencies are listed under groups:

.. code-block:: ruby

    source 'https://rubygems.org'

    gem 'rails'

    group :test do
      gem 'rspec'
    end

    group :lint do
      gem 'rubocop'
    end

    group :docs do
      gem 'kramdown'
      gem 'nokogiri'
    end

If a user executes ``bundle install`` with these data, all groups are
installed. Users can deselect groups by creating or modifying a bundler config
in ``.bundle/config``, either manually or via the CLI. For example, ``bundle
config set --local without 'lint:docs'``.

It is not possible, with the above data, to exclude the top-level use of the
``'rails'`` gem or to refer to that implicit grouping by name.

gemspec and packaged dependency data
''''''''''''''''''''''''''''''''''''

A `gemspec file <https://guides.rubygems.org/specification-reference/>`__ is a
ruby file containing a `Gem::Specification
<https://ruby-doc.org/stdlib-3.0.1/libdoc/rubygems/rdoc/Gem/Specification.html>`__
instance declaration.

Only two fields in a ``Gem::Specification`` pertain to package dependency data.
These are ``add_development_dependency`` and ``add_runtime_dependency``.
A ``Gem::Specification`` object also provides methods for adding dependencies
dynamically, including ``add_dependency`` (which adds a runtime dependency).

Here is a variant of the ``rails.gemspec`` file, with many fields removed or
shortened to simplify:

.. code-block:: ruby

    version = '7.1.2'

    Gem::Specification.new do |s|
      s.platform    = Gem::Platform::RUBY
      s.name        = "rails"
      s.version     = version
      s.summary     = "Full-stack web application framework."

      s.license = "MIT"
      s.author   = "David Heinemeier Hansson"

      s.files = ["README.md", "MIT-LICENSE"]

      # shortened from the real 'rails' project
      s.add_dependency "activesupport", version
      s.add_dependency "activerecord",  version
      s.add_dependency "actionmailer",  version
      s.add_dependency "activestorage", version
      s.add_dependency "railties",      version
    end

Note that there is no use of ``add_development_dependency``.
Some other mainstream, major packages (e.g. ``rubocop``) do not use development
dependencies in their gems.

Other projects *do* use this feature. For example, ``kramdown`` makes use of
development dependencies, containing the following specification in its
``Rakefile``:

.. code-block:: ruby

    s.add_dependency "rexml"
    s.add_development_dependency 'minitest', '~> 5.0'
    s.add_development_dependency 'rouge', '~> 3.0', '>= 3.26.0'
    s.add_development_dependency 'stringex', '~> 1.5.1'

The purpose of development dependencies is only to declare an implicit group,
as part of the ``.gemspec``, which can then be used by ``bundler``.

For full details, see the ``gemspec`` directive in ``bundler``\'s
`documentation on Gemfiles
<https://bundler.io/v1.12/man/gemfile.5.html#GEMSPEC-gemspec->`__.
However, the integration between ``.gemspec`` development dependencies and
``Gemfile``/``bundle`` usage is best understood via an example.

gemspec development dependency example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Consider the following simple project in the form of a ``Gemfile`` and ``.gemspec``.
The ``cool-gem.gemspec`` file:

.. code-block:: ruby

    Gem::Specification.new do |s|
      s.author = 'Stephen Rosen'
      s.name = 'cool-gem'
      s.version = '0.0.1'
      s.summary = 'A very cool gem that does cool stuff'
      s.license = 'MIT'

      s.files = []

      s.add_dependency 'rails'
      s.add_development_dependency 'kramdown'
    end

and the ``Gemfile``:

.. code-block:: ruby

    source 'https://rubygems.org'

    gemspec

The ``gemspec`` directive in ``Gemfile`` declares a dependency on the local
package, ``cool-gem``, defined in the locally available ``cool-gem.gemspec``
file. It *also* implicitly adds all development dependencies to a dependency
group named ``development``.

Therefore, in this case, the ``gemspec`` directive is equivalent to the
following ``Gemfile`` content:

.. code-block:: ruby

    gem 'cool-gem', :path => '.'

    group :development do
      gem 'kramdown'
    end

.. _python_prior_art:

Appendix B: Prior Art in Python
===============================

In the absence of any prior standard for Dependency Groups, two known workflow
tools, PDM and Poetry, have defined their own solutions.

This section will primarily focus on these two tools as cases of prior art
regarding the definition and use of Dependency Groups in Python.

Projects are Packages
---------------------

Both PDM and Poetry treat the projects they support as packages.
This allows them to use and interact with standard ``pyproject.toml`` metadata
for some of their needs, and allows them to support installation of the
"current project" by doing a build and install using their build backends.

Effectively, this means that neither Poetry nor PDM supports non-package projects.

Non-Standard Dependency Specifiers
----------------------------------

PDM and Poetry extend :pep:`508` dependency specifiers with additional features
which are not part of any shared standard.
The two tools use slightly different approaches to these problems, however.

PDM supports specifying local paths, and editable installs, via a syntax which
looks like a set of arguments to ``pip install``. For example, the following
dependency group includes a local package in editable mode:

.. code-block:: toml

    [tool.pdm.dev-dependencies]
    mygroup = ["-e file:///${PROJECT_ROOT}/foo"]

This declares a dependency group ``mygroup`` which includes a local editable
install from the ``foo`` directory.

Poetry describes dependency groups as tables, mapping package names to
specifiers. For example, the same configuration as the above ``mygroup``
example might appear as follows under Poetry:

.. code-block:: toml

    [tool.poetry.group.mygroup]
    foo = { path = "foo", editable = true }

PDM restricts itself to a string syntax, and Poetry introduces tables which
describe dependencies.

Installing and Referring to Dependency Groups
---------------------------------------------

Both PDM and Poetry have tool-specific support for installing dependency
groups. Because both projects support their own lockfile formats, they also
both have the capability to transparently use a dependency group name to refer
to the *locked* dependency data for that group.

However, neither tool's dependency groups can be referenced natively from other
tools like ``tox``, ``nox``, or ``pip``.
Attempting to install a dependency group under ``tox``, for example, requires
an explicit call to PDM or Poetry to parse their dependency data and do the
relevant installation step.

.. _use_cases:

Appendix C: Use Cases
=====================

Web Applications
----------------

A web application (e.g. a Django or Flask app) often does not need to build a
distribution, but bundles and ships its source to a deployment toolchain.

For example, a source code repository may define python packaging metadata as
well as containerization or other build pipeline metadata (``Dockerfile``,
etc).
The python application is built by copying the entire repository into a
build context, installing dependencies, and bundling the result as a machine
image or container.

Such applications have dependency groups for the build, but also for linting,
testing, etc. In practice, today, these applications often define themselves as
packages to be able to use packaging tools and mechanisms like ``extras`` to
manage their dependency groups. However, they are not conceptually packages,
meant for distribution in sdist or wheel format.

Dependency Groups allow these applications to define their various dependencies
without relying on packaging metadata, and without trying to express their
needs in packaging terms.

Libraries
'''''''''

Libraries are python packages which build distributions (sdist and wheel) and
publish them to PyPI.

For libraries, Dependency Groups represent an alternative to ``extras`` for
defining groups of development dependencies, with the important advantages
noted above.

A library may define groups for ``test`` and ``typing`` which allow testing and
type-checking, and therefore rely on the library's own dependencies (as
specified in ``[project.dependencies]``).

Other development needs may not require installation of the package at all. For
example, a ``lint`` Dependency Group may be valid and faster to install without
the library, as it only installs tools like ``black``, ``ruff``, or ``flake8``.

``lint`` and ``test`` environments may also be valuable locations to hook in
IDE or editor support. See the case below for a fuller description of such
usage.

Here's an example Dependency Groups table which might be suitable for a
library:

.. code-block:: toml

    [dependency-groups]
    test = ["pytest<8", "coverage"]
    typing = ["mypy==1.7.1", "types-requests"]
    lint = ["black", "flake8"]
    typing-test = [{include = "typing"}, "pytest<8"]

Note that none of these implicitly install the library itself.
It is therefore the responsibility of any environment management toolchain to
install the appropriate Dependency Groups along with the library when needed,
as in the case of ``test``.

Data Science Projects
'''''''''''''''''''''

Data Science Projects typically take the form of a logical collection of
scripts and utilities for processing and analyzing data, using a common
toolchain. Components may be defined in the Jupyter Notebook format (ipynb),
but rely on the same common core set of utlities.

In such a project, there is no package to build or install. Therefore,
``pyproject.toml`` currently does not offer any solution for dependency
management or declaration.

It is valuable for such a project to be able to define at least one major
grouping of dependencies. For example:

.. code-block:: toml

    [dependency-groups]
    main = ["numpy", "pandas", "matplotlib"]

However, it may also be necessary for various scripts to have additional
supporting tools. Projects may even have conflicting or incompatible tools or
tool versions for different components, as they evolve over time.

Consider the following more elaborate configuration:

.. code-block:: toml

    [dependency-groups]
    main = ["numpy", "pandas", "matplotlib"]
    scikit = [{include = "main"}, "scikit-learn==1.3.2"]
    scikit-old = [{include = "main"}, "scikit-learn==0.24.2"]

This defines ``scikit`` and ``scikit-old`` as two similar variants of the
common suite of dependencies, pulling in different versions of ``scikit-learn``
to suit different scripts.

This PEP only defines these data. It does not formalize any mechanism for a
Data Science Project (or any other type of project) to install the dependencies
into known environments or associate those environments with the various
scripts. Such combinations of data are left as a problem for tool authors to
solve, and perhaps eventually standardize.

Lockfile Generation
'''''''''''''''''''

There are a number of tools which generate lockfiles in the Python ecosystem
today. PDM and Poetry each use their own lockfile formats, and pip-tools
generates ``requirements.txt`` files with version pins and hashes.

Dependency Groups are not an appropriate place to store lockfiles, as they lack
many of the necessary features. Most notably, they cannot store hashes, which
most lockfile users consider essential.

However, Dependency Groups are a valid input to tools which generate lockfiles.
Furthermore, PDM and Poetry both allow a Dependency Group name (under their
notions of Dependency Groups) to be used to refer to its locked variant.

Therefore, consider a tool which produces lockfiles, here called ``$TOOL``.
It might be used as follows:

.. code:: shell

    $TOOL lock --dependency-group=test
    $TOOL install --dependency-group=test --use-locked

All that such a tool needs to do is to ensure that its lockfile data records
the name ``test`` in order to support such usage.

The mutual compatibility of Dependency Groups is not guaranteed. For example,
the Data Science example above shows conflicting versions of ``scikit-learn``.
Therefore, installing multiple locked dependency groups in tandem may require
that tools apply additional constraints or generate additional lockfile data.
These problems are considered out of scope for this PEP.

As two examples of how combinations might be locked:

* A tool might require that lockfile data be explicitly generated for any
  combination to be considered valid

* Poetry implements the requirement that all Dependency Groups be mutually
  compatible, and generates only one locked version. (Meaning it finds a single
  solution, rather than a set or matrix of solutions.)

Environment Manager Inputs
''''''''''''''''''''''''''

A common usage in tox, Nox, and Hatch is to install a set of dependencies into
a testing environment.

For example, under ``tox.ini``, type checking dependencies may be defined
inline:

.. code-block:: ini

    [testenv:typing]
    deps =
        pyright
        useful-types
    commands = pyright src/

This combination provides a desirable developer experience within a limited
context. Under the relevant environment manager, the dependencies which are
needed for the test environment are declared alongside the commands which need
those dependencies. They are not published in package metadata, as ``extras``
would be, and they are discoverable for the tool which needs them to build the
relevant environment.

Dependency Groups apply to such usages by effectively "lifting" these
requirements data from a tool-specific location into a more broadly available
one. In the example above, only ``tox`` has access to the declared list of
dependencies. Under an implementation supporting dependency groups, the same
data might be available in a Dependency Group:

.. code-block:: toml

    [dependency-groups]
    typing = ["pyright", "useful-types"]

The data can then be used under multiple tools. For example, ``tox`` might
implement support as ``dependency_groups = typing``, replacing the ``deps``
usage above.

In order for Dependency Groups to be a viable alternative for users of
environment managers, the environment managers will need to support processing
Dependency Groups similarly to how they support inline dependency declaration.

Embedded ``pyproject.toml`` in Scripts
''''''''''''''''''''''''''''''''''''''

:pep:`723`, defines embedded ``pyproject.toml`` data within scripts. For this
use case, it is necessary to declare the dependencies of a script in a data
format which is also valid ``pyproject.toml`` content. However,
``[project.dependencies]`` is considered inappropriate because a script is not
a package -- and the ``[project]`` table is defined under constraints which
reflect valid metadata for packages.

:pep:`723` provisionally uses a ``[run.dependencies]`` table for this purpose,
but Dependency Groups offer a more general solution to the problem of
dependency declaration covering a broader set of use cases than the (informal)
``[run]`` proposal.

Rather than a singular group of dependencies, and a singular runtime context,
Dependency Groups support multiple named groups for different purposes and
environments.

Because Dependency Groups are multiple, unlike ``[run.dependencies]``, it is
necessary for any standard which wants to use Dependency Groups to define how
it will leverage them.
This PEP does not assign special meanings to any names for Dependency Groups,
but it is valid for standards consuming Dependency Groups to define
conventional names.

To use Dependency Groups within :pep:`723`, there are two primary options:

* declare, as part of the specification of :pep:`723`, that the ``run``
  Dependency Group is conventionally the one which will be used

* declare a mechanism for naming a Dependency Group to use

For example, the following two ``pyproject.toml`` contents would be valid ways
of declaring dependencies for a script:

.. code-block:: toml

    [dependency-groups]
    run = ["numpy"]

or

.. code-block:: toml

    [dependency-groups]
    mygroupname = ["numpy"]
    [run]
    use-group = "mygroupname"

This PEP declares no preference for how other standards consume this
information, but aims to make such consumption feasible.

IDE and Editor Use of Requirements Data
'''''''''''''''''''''''''''''''''''''''

Similar to the :pep:`723` case above, IDE and Editor integrations may benefit
from conventional name definitions or configurable ones.

However, there are at least two known scenarios in which it is valuable for an
editor or IDE to be capable of discovering the non-published dependencies of a
project:

* testing: IDEs such as VS Code support GUI interfaces for running particular
  tests

* linting: editors and IDEs often support linting and autoformatting
  integrations which highlight or autocorrect errors

These cases could be handled by defining conventional group names like
``test``, ``lint``, and ``fix``, or by defining configuration mechanisms which
allow the selection of Dependency Groups.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.