python-peps/pep-0633.rst

PEP: 633
Title: Dependency specification in pyproject.toml using an exploded TOML table
Author: Laurie Opperman <laurie_opperman@hotmail.com>,
        Arun Babu Neelicattu <arun.neelicattu@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/dependency-specification-in-pyproject-toml-using-an-exploded-toml-table/5123/
Status: Rejected
Type: Standards Track
Topic: Packaging
Content-Type: text/x-rst
Created: 02-Sep-2020
Post-History: 02-Sep-2020
Resolution: https://discuss.python.org/t/how-to-specify-dependencies-pep-508-strings-or-a-table-in-toml/5243/38


Rejection Notice
================

This PEP has been rejected in favour of :pep:`631` due to its popularity,
consistency with the existing usage of :pep:`508` strings, and compatibility
with existing packaging tool suites.


Abstract
========

This PEP specifies how to write a project's dependencies in a
``pyproject.toml`` file for packaging-related tools to consume using the fields
defined in :pep:`621`, as an alternative to the :pep:`508`-based approach
defined in :pep:`631`.


Motivation
==========

There are multiple benefits to using TOML tables and other data-types to
represent requirements rather than :pep:`508` strings:

- Easy initial validation via the TOML syntax.

- Easy secondary validation using a schema, for example a `JSON Schema`_.

- Potential for users to guess the keys of given features, rather than
  memorising a syntax.

- Users of multiple other popular languages may already be familiar with the
  TOML syntax.

- TOML directly represents the same data structures as in JSON, and therefore a
  sub-set of Python literals, so users can understand the hierarchy and type of
  value

.. _JSON Schema: https://json-schema.org/


Rationale
=========

Most of this is taken from discussions in the `PEP 621 dependencies topic`_.
This has elements from `Pipfile`_, `Poetry`_, `Dart's dependencies`_ and
`Rust's Cargo`_. A `comparison document`_ shows advantages and disadvantages
between this format and :pep:`508`-style specifiers.

In the specification of multiple requirements with the same distribution name
(where environment markers choose the appropriate dependency), the chosen
solution is similar to `Poetry`_'s, where an array of requirements is allowed.

The direct-reference keys closely align with and utilise :pep:`610` and
:pep:`440` as to reduce differences in the packaging ecosystem and rely on
previous work in specification.

.. _PEP 621 dependencies topic: https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599
.. _Pipfile: https://github.com/pypa/pipfile
.. _Poetry: https://python-poetry.org/docs/dependency-specification/
.. _Dart's dependencies: https://dart.dev/tools/pub/dependencies
.. _Rust's Cargo: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html
.. _comparison document: https://github.com/uranusjr/packaging-metadata-comparisons/blob/master/topics/dependency-entries.md


Specification
=============

As in :pep:`621`, if metadata is improperly specified then tools MUST raise an
error. The metadata MUST conform to the `TOML`_ specification.

To reduce confusion with this document being a specification for specifying
dependencies, the word "requirement" is used to mean a :pep:`508` dependency
specification.

The following tables are added to the ``project`` table specified in
:pep:`621`.

.. _TOML: https://toml.io/

``dependencies``
----------------

Format: table

The keys inside this table are the names of the required distribution. The
values can have one of the following types:

- string: the requirement is defined only by a version requirement, with same
  specification as ``version`` in the requirement table, except allowing the
  empty string ``""`` to place no restriction on the version.

- table: a requirement table.

- array: an array of requirement tables. It is an error to specify an empty
  array ``[]`` as a value.

Requirement table
^^^^^^^^^^^^^^^^^

The keys of the requirement table are as follows (all are optional):

- ``version`` (string): a :pep:`440` version specifier, which is a
  comma-delimited list of version specifier clauses. The string MUST be
  non-empty.

- ``extras`` (array of strings): a list of :pep:`508` extras declarations for
  the distribution. The list MUST be non-empty.

- ``markers`` (string): a :pep:`508` environment marker expression. The string
  MUST be non-empty.

- ``url`` (string): the URL of the artifact to install and satisfy the
  requirement. Note that ``file://`` is the prefix used for packages to be
  retrieved from the local filesystem.

- ``git``, ``hg``, ``bzr`` or ``svn`` (string): the URL of a VCS repository
  (as specified in :pep:`440`)
  to clone, whose tree will be installed to satisfy the requirement. Further
  VCS keys will be added via amendments to :pep:`610`, however tools MAY opt to
  support other VCS's using their command-line command prior to the acceptance
  of the amendment.

- ``revision`` (string): the identifier for a specific revision of the
  specified VCS repository to check-out before installation. Users MUST only
  provide this when one of ``git``, ``hg``, ``bzr``, ``svn``, or another VCS
  key is used to identify the distribution to install. Revision identifiers are
  suggested in :pep:`610`.

At most one of the following keys can be specified simultaneously, as they
logically conflict with each other in the requirement: ``version``, ``url``,
``git``, ``hg``, ``bzr``, ``svn``, and any other VCS key.

An empty requirement table ``{}`` places no restriction on the requirement, in
addition to the empty string ``""``.

Any keys provided which are not specified in this document MUST cause an error
in parsing.

``optional-dependencies``
--------------------------

Format: table

The keys inside this table are the names of an extra's required distribution.
The values can have one of the following types:

- table: a requirement table.

- array: an array of requirement tables.

These requirement tables have
`the same specification as above <requirement table_>`_, with the addition of
the following required key:

- ``for-extra`` (string): the name of the :pep:`508` extra that this
  requirement is required for.


Reference implementation
========================

Tools will need to convert this format to :pep:`508` requirement strings. Below
is an example implementation of that conversion (assuming validation is already
performed):

.. code-block::

    def convert_requirement_to_pep508(name, requirement):
        if isinstance(requirement, str):
            requirement = {"version": requirement}
        pep508 = name
        if "extras" in requirement:
            pep508 += " [" + ", ".join(requirement["extras"]) + "]"
        if "version" in requirement:
            pep508 += " " + requirement["version"]
        if "url" in requirement:
            pep508 += " @ " + requirement["url"]
        for vcs in ("git", "hg", "bzr", "svn"):
            if vcs in requirement:
                pep508 += " @ " + vcs + "+" + requirement[vcs]
                if "revision" in requirement:
                    pep508 += "@" + requirement["revision"]
        extra = None
        if "for-extra" in requirement:
            extra = requirement["for-extra"]
        if "markers" in requirement:
            markers = requirement["markers"]
            if extra:
                markers = "extra = '" + extra + "' and (" + markers + ")"
            pep508 += "; " + markers
        return pep508, extra


    def convert_requirements_to_pep508(dependencies):
        pep508s = []
        extras = set()
        for name, req in dependencies.items():
            if isinstance(req, list):
                for sub_req in req:
                    pep508, extra = convert_requirement_to_pep508(name, sub_req)
                    pep508s.append(pep508)
                    if extra:
                        extras.add(extra)
            else:
                pep508, extra = convert_requirement_to_pep508(name, req)
                pep508s.append(pep508)
                if extra:
                    extras.add(extra)
        return pep508s, extras


    def convert_project_requirements_to_pep508(project):
        reqs, _ = convert_requirements_to_pep508(project.get("dependencies", {}))
        optional_reqs, extras = convert_requirements_to_pep508(
            project.get("optional-dependencies", {})
        )
        reqs += optional_reqs
        return reqs, extras

JSON schema
-----------

For initial validation, a JSON-schema can be used. Not only does this help
tools have a consistent validation, but it allows code editors to highlight
validation errors as users are building the dependencies list.

.. code-block::

    {
        "$id": "spam",
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "Project metadata",
        "type": "object",
        "definitions": {
            "requirementTable": {
                "title": "Full project dependency specification",
                "type": "object",
                "properties": {
                    "extras": {
                        "title": "Dependency extras",
                        "type": "array",
                        "items": {
                            "title": "Dependency extra",
                            "type": "string"
                        }
                    },
                    "markers": {
                        "title": "Dependency environment markers",
                        "type": "string"
                    }
                },
                "propertyNames": {
                    "enum": [
                        "extras",
                        "markers",
                        "version",
                        "url",
                        "git",
                        "hg",
                        "bzr",
                        "svn",
                        "for-extra"
                    ]
                },
                "oneOf": [
                    {
                        "title": "Version requirement",
                        "properties": {
                            "version": {
                                "title": "Version",
                                "type": "string"
                            }
                        }
                    },
                    {
                        "title": "URL requirement",
                        "properties": {
                            "url": {
                                "title": "URL",
                                "type": "string",
                                "format": "uri"
                            }
                        },
                        "required": [
                            "url"
                        ]
                    },
                    {
                        "title": "VCS requirement",
                        "properties": {
                            "revision": {
                                "title": "VCS repository revision",
                                "type": "string"
                            }
                        },
                        "oneOf": [
                            {
                                "title": "Git repository",
                                "properties": {
                                    "git": {
                                        "title": "Git URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "git"
                                ]
                            },
                            {
                                "title": "Mercurial repository",
                                "properties": {
                                    "hg": {
                                        "title": "Mercurial URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "hg"
                                ]
                            },
                            {
                                "title": "Bazaar repository",
                                "properties": {
                                    "bzr": {
                                        "title": "Bazaar URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "bzr"
                                ]
                            },
                            {
                                "title": "Subversion repository",
                                "properties": {
                                    "svn": {
                                        "title": "Subversion URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "svn"
                                ]
                            }
                        ]
                    }
                ]
            },
            "requirementVersion": {
                "title": "Version project dependency specification",
                "type": "string"
            },
            "requirement": {
                "title": "Project dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/requirementVersion"
                    },
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/requirementTable"
                        },
                        "minLength": 1
                    }
                ]
            },
            "optionalRequirementTable": {
                "title": "Project optional dependency specification table",
                "allOf": [
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "properties": {
                            "for-extra": {
                                "title": "Dependency's extra",
                                "type": "string"
                            }
                        },
                        "required": [
                            "for-extra"
                        ]
                    }
                ]
            },
            "optionalRequirement": {
                "title": "Project optional dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/optionalRequirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/optionalRequirementTable"
                        },
                        "minLength": 1
                    }
                ]
            }
        },
        "properties": {
            "dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/requirement"
                }
            },
            "optional-dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/optionalRequirement"
                }
            }
        }
    }


Examples
========

Full artificial example:

.. code-block::

    [project.dependencies]
    flask = { }
    django = { }
    requests = { version = ">= 2.8.1, == 2.8.*", extras = ["security", "tests"], markers = "python_version < '2.7'" }
    pip = { url = "https://github.com/pypa/pip/archive/1.3.1.zip" }
    sphinx = { git = "ssh://git@github.com/sphinx-doc/sphinx.git" }
    numpy = "~=1.18"
    pytest = [
        { version = "<6", markers = "python_version < '3.5'" },
        { version = ">=6", markers = "python_version >= '3.5'" },
    ]

    [project.optional-dependencies]
    pytest-timout = { for-extra = "dev" }
    pytest-mock = [
        { version = "<6", markers = "python_version < '3.5'", for-extra = "dev" },
        { version = ">=6", markers = "python_version >= '3.5'", for-extra = "dev" },
    ]

In homage to :pep:`631`, the following is an equivalent dependencies
specification for `docker-compose`_:

.. code-block::

    [project.dependencies]
    cached-property = ">= 1.2.0, < 2"
    distro = ">= 1.2.0, < 2"
    docker = { extras = ["ssh"], version = ">= 4.2.2, < 5" }
    docopt = ">= 0.6.1, < 1"
    jsonschema = ">= 2.5.1, < 4"
    PyYAML = ">= 3.10, < 6"
    python-dotenv = ">= 0.13.0, < 1"
    requests = ">= 2.20.0, < 3"
    texttable = ">= 0.9.0, < 2"
    websocket-client = ">= 0.32.0, < 1"

    # Conditional
    "backports.shutil_get_terminal_size" = { version = "== 1.0.0", markers = "python_version < '3.3'" }
    "backports.ssl_match_hostname" = { version = ">= 3.5, < 4", markers = "python_version < '3.5'" }
    colorama = { version = ">= 0.4, < 1", markers = "sys_platform == 'win32'" }
    enum34 = { version = ">= 1.0.4, < 2", markers = "python_version < '3.4'" }
    ipaddress = { version = ">= 1.0.16, < 2", markers = "python_version < '3.3'" }
    subprocess32 = { version = ">= 3.5.4, < 4", markers = "python_version < '3.2'" }

    [project.optional-dependencies]
    PySocks = { version = ">= 1.5.6, != 1.5.7, < 2", for-extra = "socks" }
    ddt = { version = ">= 1.2.2, < 2", for-extra = "tests" }
    pytest = { version = "< 6", for-extra = "tests" }
    mock = { version = ">= 1.0.1, < 4", markers = "python_version < '3.4'", for-extra = "tests" }

.. _docker-compose: https://github.com/docker/compose/blob/789bfb0e8b2e61f15f423d371508b698c64b057f/setup.py#L28-L61


Compatibility Examples
======================

The authors of this PEP recognise that various tools need to both read
from and write to this format for dependency specification. This section
aims to provide direct comparison with and examples for translating to/from
the currently used standard, :pep:`508`.

.. note::

        For simplicity and clarity, various ways in which TOML allows you to specify each
        specification is not represented. These examples use the standard inline representation.

        For example, while following are considered equivalent in TOML, we choose the
        second form for the examples in this section.

        .. code-block::

            aiohttp.version = "== 3.6.2"
            aiohttp = { version = "== 3.6.2" }


Version Constrained Dependencies
--------------------------------

**No Version Constraint**

.. code-block::

        aiohttp


.. code-block::

        aiohttp = {}

**Simple Version Constraint**

.. code-block::

        aiohttp >= 3.6.2, < 4.0.0


.. code-block::

        aiohttp = { version = ">= 3.6.2, < 4.0.0" }


.. note::

        This can, for conciseness, be also represented as a string.

        .. code-block::

            aiohttp = ">= 3.6.2, < 4.0.0"



Direct Reference Dependencies
-----------------------------

**URL Dependency**

.. code-block::

        aiohttp @ https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl


.. code-block::

        aiohttp = { url = "https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl" }

**VCS Dependency**

.. code-block::

        aiohttp @ git+ssh://git@github.com/aio-libs/aiohttp.git@master


.. code-block::

        aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master" }


Environment Markers
-------------------

.. code-block::

        aiohttp >= 3.6.1; python_version >= '3.8'


.. code-block::

        aiohttp = { version = ">= 3.6.1", markers = "python_version >= '3.8'" }


A slightly extended example of the above, where a particular version of ``aiohttp`` is required based on the interpreter version.

.. code-block::

        aiohttp >= 3.6.1; python_version >= '3.8'
        aiohttp >= 3.0.0, < 3.6.1; python_version < '3.8'


.. code-block::

        aiohttp = [
            { version = ">= 3.6.1", markers = "python_version >= '3.8'" },
            { version = ">= 3.0.0, < 3.6.1", markers = "python_version < '3.8'" }
        ]


Package Extras
--------------

**Specifying dependency for a package extra**

.. code-block::

        aiohttp >= 3.6.2; extra == 'http'


.. code-block::

        aiohttp = { version = ">= 3.6.2", for-extra = "http" }

**Using extras from a dependency**

.. code-block::

        aiohttp [speedups] >= 3.6.2


.. code-block::

        aiohttp = { version = ">= 3.6.2", extras = ["speedups"] }


Complex Examples
----------------

**Version Constraint**

.. code-block::

        aiohttp [speedups] >= 3.6.2; python_version >= '3.8' and extra == 'http'


.. code-block::

        aiohttp = { version = ">= 3.6.2", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }


**Direct Reference (VCS)**

.. code-block::

        aiohttp [speedups] @ git+ssh://git@github.com/aio-libs/aiohttp.git@master ; python_version >= '3.8' and extra == 'http'


.. code-block::

        aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }


Rejected Ideas
==============

Switch to an array for ``dependencies``
---------------------------------------

Use an array instead of a table in order to have each element only be a table
(with a ``name`` key) and no arrays of requirement tables. This was very
verbose and restrictive in the TOML format, and having multiple requirements
for a given distribution isn't very common.

Replace ``optional-dependencies`` with ``extras``
-------------------------------------------------

Remove the ``optional-dependencies`` table in favour of both including an
``optional`` key in the requirement and an ``extras`` table which specifies
which (optional) requirements are needed for a project's extra. This reduces
the number of table with the same specification (to 1) and allows for
requirements to be specified once but used in multiple extras, but distances
some of the requirement's properties (which extra(s) it belongs to), groups
required and optional dependencies together (possibly mixed), and there may not
be a simple way to choose a requirement when a distribution has multiple
requirements. This was rejected as ``optional-dependencies`` has already been
used in the :pep:`621` draft.

``direct`` table in requirement
-------------------------------

Include the direct-reference keys in a ``direct`` table, have the VCS specified
as the value of a ``vcs`` key. This was more explicit and easier to include in
a JSON-schema validation, but was decided to be too verbose and not as
readable.

Include hash
------------

Include hash in direct-reference requirements. This was only for package
lock-files, and didn't really have a place in the project's metadata.

Dependency tables for each extra
--------------------------------

Have the ``optional-dependencies`` be a table of dependency tables for each
extra, with the table name being the extra's name. This made
``optional-dependencies`` a different type (table of tables of requirements)
from ``dependencies`` (table of requirements), which could be jarring for users
and harder to parse.

Environment marker keys
-----------------------

Make each :pep:`508` environment marker as a key (or child-table key) in
the requirement. This arguably increases readability and ease of parsing.
The ``markers`` key would still be allowed for more advanced specification,
with which the key-specified environment markers are ``and``'d with the
result of. This was deferred as more design needs to be undertaken.

Multiple extras which one requirement can satisfy
-------------------------------------------------

Replace the ``for-extra`` key with ``for-extras``, with the value being an
array of extras which the requirement satisfies. This reduces some
duplication, but in this case that duplication makes explicit which extras
have which dependencies.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.