diff --git a/pep-0426.txt b/pep-0426.txt index 745c8757c..2b4745350 100644 --- a/pep-0426.txt +++ b/pep-0426.txt @@ -5,12 +5,12 @@ Last-Modified: $Date$ Author: Nick Coghlan , Daniel Holth , Donald Stufft -BDFL-Delegate: Nick Coghlan +BDFL-Delegate: Donald Stufft Discussions-To: Distutils SIG -Status: Deferred -Type: Standards Track +Status: Draft +Type: Informational Content-Type: text/x-rst -Requires: 440 +Requires: 440, 508, 518 Created: 30 Aug 2012 Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013, 27 May 2013, 20 Jun 2013, 23 Jun 2013, 14 Jul 2013, @@ -26,74 +26,53 @@ related to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 2.0 of the metadata format. + Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. -Version 2.0 of the metadata format migrates from a custom key-value format -to a JSON-compatible in-memory representation. +Version 2.0 of the metadata format migrates from directly defining a +custom key-value file format to instead defining a JSON-compatible in-memory +representation that may be used to define metadata representation in other +contexts (such as API and archive format definitions). -This version also adds fields designed to make third-party packaging of -Python software easier, defines a formal extension mechanism, and adds -support for optional dependencies. Finally, this version addresses -several issues with the previous iteration of the standard version -identification scheme. +This version also defines a formal extension mechanism, allowing new +fields to be added for particular purposes without requiring updates to +the core metadata format. -PEP Deferral -============ -Migrating to a new metadata format is current deferred indefinitely, with -distutils-sig instead focusing on a more incremental approach to allowing -parts of the ecosystem to be independently upgraded without breaking the world. +Note on PEP Deferral +==================== -Some of those incremental improvements include: +This PEP was deferred for an extended period, from December 2013 through to +March 2017, as distutils-sig worked through a number of other changes. These +changes included: -* PEP 425, defining a compatibility tagging mechanism for binary files -* PEP 427, defining the pre-built "wheel" format -* PEP 440, covering the versioning identification and selection scheme -* PEP 503, ``/simple/`` package repository API -* PEP 508, covering dependency declarations and environment markers -* PEP 513, defining the "manylinux1" compatibility tag -* PEP 518, defining a static config file for sdist build system dependencies +* defining a binary compatibility tagging format in PEP 425 +* defining a binary archive format (``wheel``) in PEP 427 +* explicitly defining versioning and version comparison in PEP 440 +* explicitly defining the PyPI "simple" API in PEP 503 +* explicitly defining dependency specifiers and the extras system in PEP 508 +* declaring static build system dependencies (``pyproject.toml``) in PEP 518 +* migrating PyPI hosting to Rackspace, and placing it behind the Fastly CDN +* shipping ``pip`` with CPython by default in PEP 453, and backporting that + addition to Python 2.7 in PEP 477 +* establishing `packaging.python.org`_ as the common access point for Python + packaging ecosystem documentation +* migrating to using the `specifications`_ section of packaging.python.org + as the central location for tracking packaging related PEPs -Other planned improvements in various stages of development include: +The time spent pursuing these changes provided additional perspective on which +metadata format changes were genuinely desirable, and which could be omitted +from the revised specification as merely being "change for change's sake". -* Supporting build systems that don't provide the `setup.py` command -* Supporting additional semantic categories for distribution contents (based on - GNU autotools) -* Support for declaring dependencies on shared system libraries +It is allowed a number of features that aren't critical to the core activity +of publishing and distributing software to be moved out to PEP 459, a separate +proposal for a number of standard metadata extensions that provide additional +optional information about a release. -.. note:: - - "I" in this doc refers to Nick Coghlan. Daniel and Donald either wrote or - contributed to earlier versions, and provided feedback as the JSON-based - rewrite took shape. Daniel and Donald were also vetting the proposal as we - went to ensure it remained practical to implement for both clients and - index servers. - - Metadata 2.0 represents a major upgrade to the Python packaging ecosystem, - and attempts to incorporate experience gained over the 15 years(!) since - distutils was first added to the standard library. Some of that is just - incorporating existing practices from setuptools/pip/etc, some of it is - copying from other distribution systems (like Linux distros or other - development language communities) and some of it is attempting to solve - problems which haven't yet been well solved by anyone (like supporting - clean conversion of Python source packages to distro policy compliant - source packages for at least Debian and Fedora, and perhaps other - platform specific distribution systems). - - It's going to take a while to work through all of these and make them - a reality. The main change from our last attempt at this is that we're - trying to design the different pieces so we can implement them - independently of each other, without requiring users to switch to - a whole new tool chain (although they may have to upgrade their existing - ones to start enjoying the benefits in their own work). - - Many of the inline notes in this version of the PEP are there to aid - reviewers that are familiar with the old metadata standards. Before this - version is finalised, most of that content will be moved down to the - "rationale" section at the end of the document, as it would otherwise be - an irrelevant distraction for future readers. +.. _packaging.python.org: https://packaging.python.org/ +.. _specifications: https://packaging.python.org/specifications/ Purpose @@ -107,10 +86,10 @@ Python code by those doing the analysis. Another aim is to encourage good software distribution practices by default, while continuing to support the current practices of almost all existing users of the Python Package Index (both publishers and integrators). Finally, the aim is to support an upgrade -path from the existing setuptools defined dependency and entry point -metadata formats that is transparent to end users. +path from the currently in use metadata formats that is transparent to end +users. -The design draws on the Python community's 15 years of experience with +The design draws on the Python community's nearly 20 years of experience with distutils based software distribution, and incorporates ideas and concepts from other distribution systems, including Python's setuptools, pip and other projects, Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer @@ -121,32 +100,6 @@ of the ideas may also be useful in the future evolution of other dependency management ecosystems. -A Note on Time Frames -===================== - -There's a lot of work going on in the Python packaging space at the moment. -In the near term (up until the release of Python 3.4), those efforts are -focused on the existing metadata standards, both those defined in Python -Enhancement Proposals, and the de facto standards defined by the setuptools -project. - -This PEP is about setting out a longer term goal for the ecosystem that -captures those existing capabilities in a format that is easier to work -with. There are still a number of key open questions (mostly related to -source based distribution), and those won't be able to receive proper -attention from the development community until the other near term -concerns have been resolved. - -At this point in time, the PEP is quite possibly still overengineered, as -we're still trying to make sure we have all the use cases covered. The -"transparent upgrade path from setuptools" goal brings in a lot of required -functionality though, and then the aim of supporting automated creation of -policy compliant downstream packages for Linux distributions adds more. -However, we've at least reached the point where we're taking a critical -look at the core metadata, and are pushing as much functionality out to -standard metadata extensions as we can. - - Development, Distribution and Deployment of Python Software =========================================================== @@ -177,14 +130,14 @@ the following phases: The publication and integration phases are collectively referred to as the distribution phase, and the individual software components distributed -in that phase are formally referred to as "distributions", but are more -colloquially known as "packages" (relying on context to disambiguate them +in that phase are formally referred to as "distribution packages", but are more +colloquially known as just "packages" (relying on context to disambiguate them from the "module with submodules" kind of Python package). The exact details of these phases will vary greatly for particular use cases. Deploying a web application to a public Platform-as-a-Service provider, publishing a new release of a web framework or scientific library, -creating an integrated Linux distribution or upgrading a custom application +creating an integrated Linux distribution, or upgrading a custom application running in a secure enclave are all situations this metadata design should be able to handle. @@ -208,7 +161,7 @@ the `Python Package Index`_. "Releases" are uniquely identified snapshots of a project. -"Distributions" are the packaged files which are used to publish +"Distribution packages" are the packaged files which are used to publish and distribute a release. Depending on context, "package" may refer to either a distribution, or @@ -272,10 +225,8 @@ along with the supporting metadata file formats defined by the ``setuptools`` project. "Distro" is used as the preferred term for Linux distributions, to help -avoid confusion with the Python-specific meaning of the term "distribution". - -"Dist" is the preferred abbreviation for "distributions" in the sense defined -in this PEP. +avoid confusion with the Python-specific use of the term "distribution +package". "Qualified name" is a dotted Python identifier. For imported modules and packages, the qualified name is available as the ``__name__`` attribute, @@ -323,8 +274,8 @@ is that it minimizes the dependencies that need to be installed on deployment targets (as the build dependencies will be needed only on the build systems). -The published metadata for distributions SHOULD allow integrators, with the -aid of build and integration tools, to: +The published metadata for distribution packages SHOULD allow integrators, with +the aid of build and integration tools, to: * obtain the original source code that was used to create a distribution * identify and retrieve the dependencies (if any) required to use a @@ -333,10 +284,6 @@ aid of build and integration tools, to: distribution from source * identify and retrieve the dependencies (if any) required to run a distribution's test suite -* find resources on using and contributing to the project -* access sufficiently rich metadata to support contacting distribution - publishers through appropriate channels, as well as finding distributions - that are relevant to particular problems Development and publication of distributions @@ -362,47 +309,6 @@ and publishers, with the aid of build and publication tools, to: publish the distribution -Standard build system ---------------------- - -.. note:: - - The standard build system currently described in the PEP is a draft based - on existing practices for projects using distutils or setuptools as their - build system (or other projects, like ``d2to1``, that expose a setup.py - file for backwards compatibility with existing tools) - - The specification doesn't currently cover expected argument support for - the commands, which is a limitation that needs to be addressed before the - PEP can be considered ready for acceptance. - - It is also possible that the "meta build system" will be separated out - into a distinct PEP in the coming months (similar to the separation of - the versioning and requirement specification standard out to PEP 440). - - If a `suitable API can be worked out `__, then it may - even be possible to switch to a more declarative API for build system - specification. - -Both development and integration of distributions relies on the ability to -build extension modules and perform other operations in a distribution -independent manner. - -The current iteration of the metadata relies on the -``distutils``/``setuptools`` commands system to support these necessary -development and integration activities: - -* ``python setup.py dist_info``: generate distribution metadata in place - given a source archive or VCS checkout -* ``python setup.py sdist``: create an sdist from a source archive - or VCS checkout -* ``python setup.py build_ext --inplace``: build extension modules in place - given an sdist, source archive or VCS checkout -* ``python setup.py test``: run the distribution's test suite in place - given an sdist, source archive or VCS checkout -* ``python setup.py bdist_wheel``: create a binary archive from an sdist, - source archive or VCS checkout - Metadata format =============== @@ -419,9 +325,9 @@ access APIs. The individual field descriptions show examples of the key name and value as they would be serialised as part of a JSON mapping. -The fields identified as core metadata are required. Automated tools MUST -NOT accept distributions with missing core metadata as valid Python -distributions. +Unless otherwise indicated, the fields identified as core metadata are required. +Automated tools MUST NOT accept distributions with missing core metadata as +valid Python distributions. All other fields are optional. Automated tools MUST operate correctly if a distribution does not provide them, except for those operations @@ -453,7 +359,7 @@ described in this PEP. When serialising metadata, automated tools SHOULD lexically sort any keys and list elements in order to simplify reviews of any changes. -There are three standard locations for these metadata files: +There are expected to be three standard locations for these metadata files: * as a ``{distribution}-{version}.dist-info/pydist.json`` file in an ``sdist`` source distribution archive @@ -469,20 +375,27 @@ There are three standard locations for these metadata files: also be a wheel 1.1 format update after this PEP is approved that mandates provision of 2.0+ metadata. -Note that these metadata files SHOULD NOT be processed if the version of the +Note that these metadata files MAY be processed even if the version of the containing location is too low to indicate that they are valid. Specifically, unversioned ``sdist`` archives, unversioned installation database directories -and version 1.0 of the ``wheel`` specification do not cover ``pydist.json`` +and version 1.0 of the ``wheel`` specification may still provide ``pydist.json`` files. +.. note:: + + Until this specification is formally marked as Active, it is recommended + that tools following the draft format use an alternative filename like + ``metadata.json`` or ``pep426-20131213.json`` to avoid colliding with + the eventually standardised files. + Other tools involved in Python distribution MAY also use this format. As JSON files are generally awkward to edit by hand, it is RECOMMENDED that these metadata files be generated by build tools based on other -input formats (such as ``setup.py``) rather than being used directly as -a data input format. Generating the metadata as part of the publication -process also helps to deal with version specific fields (including the -source URL and the version field itself). +input formats (such as ``setup.py`` and ``pyproject.toml``) rather than being +used directly as a data input format. Generating the metadata as part of the +publication process also helps to deal with version specific fields (including +the source URL and the version field itself). For backwards compatibility with older installation tools, metadata 2.0 files MAY be distributed alongside legacy metadata. @@ -512,7 +425,9 @@ with RFC 3986. The current version of the schema file covers the previous draft of the PEP, and has not yet been updated for the split into the essential - dependency resolution metadata and multiple standard extensions. + dependency resolution metadata and multiple standard extensions, and nor + has it been updated for the replacement of the various "semantic dependency" + fields with particular conventional "extras" declarations. Core metadata @@ -558,15 +473,16 @@ Generator Name (and optional version) of the program that generated the file, if any. A manually produced file would omit this field. -Example:: +Examples:: - "generator": "setuptools (0.9)" + "generator": "flit" + "generator": "setuptools (34.3.1)" Name ---- -The name of the distribution. +The name of the distribution, as defined in PEP 508. As distribution names are used as part of URLs, filenames, command line parameters and must also interoperate with other packaging systems, the @@ -580,7 +496,10 @@ permitted characters are constrained to: Distribution names MUST start and end with an ASCII letter or digit. -Automated tools MUST reject non-compliant names. +Automated tools MUST reject non-compliant names. A regular expression to +enforce these constraints (when run with ``re.IGNORECASE``) is:: + + ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent. @@ -681,6 +600,11 @@ characters: Source labels MUST start and end with an ASCII letter or digit. +A regular expression to rnforce these constraints (when run with +``re.IGNORECASE``) is:: + + ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-+]*[A-Z0-9])$ + A source label for a project MUST NOT match any defined version for that project. This restriction ensures that there is no ambiguity between version identifiers and source labels. @@ -720,16 +644,14 @@ automated tools SHOULD at least emit a warning and MAY refuse to rely on the URL. If such a source URL also uses an insecure transport, automated tools SHOULD NOT rely on the URL. -It is RECOMMENDED that only hashes which are unconditionally provided by -the latest version of the standard library's ``hashlib`` module be used -for source archive hashes. At time of writing, that list consists of -``'md5'``, ``'sha1'``, ``'sha224'``, ``'sha256'``, ``'sha384'``, and -``'sha512'``. - For source archive references, an expected hash value may be specified by including a ``=`` entry as part of the URL fragment. +As of 2017, it is RECOMMENDED that ``'sha256'`` hashes be used for source +URLs, as this hash is not yet known to be vulnerable to generation of +malicious collisions, while also being widely available on client systems. + For version control references, the ``VCS+protocol`` scheme SHOULD be used to identify both the version control system and the secure transport, and a version control system with hash based commit identifiers SHOULD be @@ -744,7 +666,7 @@ notation. .. note:: This isn't *quite* the same as the existing VCS reference notation - supported by pip. Firstly, the distribution name is moved in front rather + supported by pip. Firstly, the distribution name is a separate field rather than embedded as part of the URL. Secondly, the commit hash is included even when retrieving based on a tag, in order to meet the requirement above that *every* link should include a hash to make things harder to @@ -753,7 +675,7 @@ notation. Example:: - "source_url": "https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686" + "source_url": "https://github.com/pypa/pip/archive/1.3.1.zip#sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8 "source_url": "git+https://github.com/pypa/pip.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d" "source_url": "git+https://github.com/pypa/pip.git@7921be1537eac1e97bc40179a57f0349c2aee67d" @@ -761,113 +683,52 @@ Example:: Semantic dependencies ===================== -Dependency metadata allows distributions to make use of functionality -provided by other distributions, without needing to bundle copies of those -distributions. +Dependency metadata allows published projects to make use of functionality +provided by other published projects, without needing to bundle copies of +particular releases of those projects. Semantic dependencies allow publishers to indicate not only which other -distributions are needed, but also *why* they're needed. This additional +projects are needed, but also *why* they're needed. This additional information allows integrators to install just the dependencies they need for specific activities, making it easier to minimise installation footprints in constrained environments (regardless of the reasons for those constraints). -Distributions may declare five different kinds of dependency: +By default, dependency declarations are assumed to be for +"runtime dependencies": other releases that are needed to actually use the +published release. -* Runtime dependencies: other distributions that are needed to actually use - this distribution (but are not considered subdistributions). -* "Meta" dependencies: subdistributions that are grouped together into a - single larger metadistribution for ease of reference and installation. -* Test dependencies: other distributions that are needed to run the - automated test suite for this distribution (but are not needed just to - use it). -* Build dependencies: other distributions that are needed to build this - distribution. -* Development dependencies: other distributions that are needed when - working on this distribution (but do not fit into one of the other - dependency categories). +There are also four different kinds of optional dependency that releases may +declare: -Within each of these categories, distributions may also declare "Extras". -Extras are dependencies that may be needed for some optional functionality, -or which are otherwise complementary to the distribution. +* ``test`` dependencies: other releases that are needed to run the + automated test suite for this release, but are not needed just to + use it (e.g. ``nose2`` or ``pytest``) +* ``build`` dependencies: other releases that are needed to build this + a deployable binary version of this release from source + (e.g. ``flit`` or ``setuptools``) +* ``doc`` dependencies: other releases that are needed to build the + documentation for this distribution (e.g. the ``sphinx`` build tool) +* ``dev`` dependencies: other releases that are needed when working on this + distribution, but do not fit into exactly one of the other optional + dependency categories (e.g. ``pylint``, ``flake8``). ``dev`` dependencies + are also effectively considered as combined ``test``, ``build``, and ``doc`` + dependencies, without needing to be listed three times + +These optional categories are known as +`Extras `_. In addition to the four +standard categories, projects may also declare their own custom categories +in the `Extras`_ field. Dependency management is heavily dependent on the version identification -and specification scheme defined in PEP 440. +and specification scheme defined in PEP 440 and the dependency specification, +extra, and environment marker schemes defined in PEP 508. All of these fields are optional. Automated tools MUST operate correctly if a distribution does not provide them, by assuming that a missing field indicates "Not applicable for this distribution". -Dependency specifiers ---------------------- - -While many dependencies will be needed to use a distribution at all, others -are needed only on particular platforms or only when particular optional -features of the distribution are needed. To handle this, dependency -specifiers are mappings with the following subfields: - -* ``requires``: a list of `requirement specifiers - `__ needed to satisfy the dependency -* ``extra``: the name of a set of optional dependencies that are requested - and installed together. See `Extras (optional dependencies)`_ for details. -* ``environment``: an environment marker defining the environment that - needs these dependencies. See `Environment markers`_ for details. - -``requires`` is the only required subfield. When it is the only subfield, the -dependencies are said to be *unconditional*. If ``extra`` or ``environment`` -is specified, then the dependencies are *conditional*. - -All three fields may be supplied, indicating that the dependencies are -needed only when the named extra is requested in a particular environment. - -Automated tools MUST combine related dependency specifiers (those with -common values for ``extra`` and ``environment``) into a single specifier -listing multiple requirements when serialising metadata or -passing it to an install hook. - -Despite this required normalisation, the same extra name or environment -marker MAY appear in multiple conditional dependencies. This may happen, -for example, if an extra itself only needs some of its dependencies in -specific environments. It is only the combination of extras and environment -markers that is required to be unique in a list of dependency specifiers. - -Any extras referenced from a dependency specifier MUST be named in the -`Extras`_ field for this distribution. This helps avoid typographical -errors and also makes it straightforward to identify the available extras -without scanning the full set of dependencies. - - -Requirement specifiers ----------------------- - -Individual requirements are defined as strings containing a distribution -name (as found in the ``name`` field). The distribution name -may be followed by an extras specifier (enclosed in square -brackets) and by a version specifier or direct reference. - -Whitespace is permitted between the distribution name and an opening -square bracket or parenthesis. Whitespace is also permitted between a -closing square bracket and the version specifier. - -See `Extras (optional dependencies)`_ for details on extras and PEP 440 -for details on version specifiers and direct references. - -The distribution names should correspond to names as found on the `Python -Package Index`_; while these names are often the same as the module names -as accessed with ``import x``, this is not always the case (especially -for distributions that provide multiple top level modules or packages). - -Example requirement specifiers:: - - "Flask" - "Django" - "Pyramid" - "SciPy ~= 0.12" - "ComfyChair[warmup]" - "ComfyChair[warmup] > 0.1" - - Mapping dependencies to development and distribution activities --------------------------------------------------------------- @@ -875,29 +736,24 @@ The different categories of dependency are based on the various distribution and development activities identified above, and govern which dependencies should be installed for the specified activities: -* Implied runtime dependencies: +* Required runtime dependencies: - * ``run_requires`` - * ``meta_requires`` + * unconditional dependencies -* Implied build dependencies: +* Required build dependencies: - * ``build_requires`` + * the ``build`` extra + * the ``dev`` extra * If running the distribution's test suite as part of the build process, - request the ``:run:``, ``:meta:``, and ``:test:`` extras to also - install: + also install the unconditional dependencies and ``test`` extra - * ``run_requires`` - * ``meta_requires`` - * ``test_requires`` +* Required development and publication dependencies: -* Implied development and publication dependencies: - - * ``run_requires`` - * ``meta_requires`` - * ``build_requires`` - * ``test_requires`` - * ``dev_requires`` + * unconditional dependencies + * the ``test`` extra + * the ``build`` extra + * the ``doc`` extra + * the ``dev`` extra The notation described in `Extras (optional dependencies)`_ SHOULD be used to determine exactly what gets installed for various operations. @@ -920,23 +776,35 @@ conditional dependencies in dependency fields. See The names of extras MUST abide by the same restrictions as those for distribution names. +The following extra names are available by default and SHOULD NOT be +declared explicitly: + +* ``test`` +* ``build`` +* ``doc`` +* ``dev`` + +The following extra names are reserved, and MUST NOT be declared: + +* ``self`` +* ``runtime`` + Example:: "extras": ["warmup"] -Run requires +Dependencies ------------ -A list of other distributions needed to actually run this distribution. +A list of release requirements needed to actually run this release. -Automated tools MUST NOT allow strict version matching clauses or direct -references in this field - if permitted at all, such clauses should appear -in ``meta_requires`` instead. +Public index servers MAY prohibit strict version matching clauses or direct +references in this field. Example:: - "run_requires": + "dependencies": { "requires": ["SciPy", "PasteDeploy", "zope.interface > 3.5.0"] }, @@ -950,133 +818,46 @@ Example:: } ] +While many dependencies will be needed to use a project release at all, others +are needed only on particular platforms or only when particular optional +features of the release are needed. -Meta requires -------------- +To handle this, release dependency specifiers are mappings with the following +subfields: -An abbreviation of "metadistribution requires". This is a list of -subdistributions that can easily be installed and used together by -depending on this metadistribution. +* ``requires``: a list of requirements needed to satisfy the dependency +* ``extra``: the name of a set of optional dependencies that are requested + and installed together. See `Extras (optional dependencies)`_ for details +* ``environment``: an environment marker defining the environment that + needs these dependencies. The syntax and capabilities of environment + markers are defined in PEP 508 -In this field, automated tools: +Individual entries in the ``requires`` lists are strings using the dependency +declaration format defined in PEP 508, with the exception that environment +markers MUST NOT be included in the individual dependency declarations, and +are instead supplied in the separate ``environment`` field. -* MUST allow strict version matching -* MUST NOT allow more permissive version specifiers. -* MAY allow direct references +``requires`` is the only required subfield. When it is the only subfield, the +dependencies are said to be *unconditional*. If ``extra`` or ``environment`` +is specified, then the dependencies are *conditional*. -Public index servers SHOULD NOT allow the use of direct references in -uploaded distributions. Direct references are intended primarily as a -tool for software integrators rather than publishers. +All three fields may be supplied, indicating that the dependencies are +needed only when the named extra is requested in a particular environment. -Distributions that rely on direct references to platform specific binary -archives SHOULD define appropriate constraints in their -``supports_environments`` field. +Automated tools MUST combine related dependency specifiers (those with +common values for ``extra`` and ``environment``) into a single specifier +listing multiple requirements when serialising metadata. -Example:: +Despite this required normalisation, the same extra name or environment +marker MAY appear in multiple conditional dependencies. This may happen, +for example, if an extra itself only needs some of its dependencies in +specific environments. It is only the combination of extras and environment +markers that is required to be unique in a list of dependency specifiers. - "meta_requires": - { - "requires": ["ComfyUpholstery == 1.0a2", - "ComfySeatCushion == 1.0a2"] - }, - { - "requires": ["CupOfTeaAtEleven == 1.0a2"], - "environment": "'linux' in sys_platform" - } - ] - - -Test requires -------------- - -A list of other distributions needed in order to run the automated tests -for this distribution.. - -Automated tools MAY disallow strict version matching clauses and direct -references in this field and SHOULD at least emit a warning for such clauses. - -Public index servers SHOULD NOT allow strict version matching clauses or -direct references in this field. - -Example:: - - "test_requires": - { - "requires": ["unittest2"] - }, - { - "requires": ["pywin32 > 1.0"], - "environment": "sys_platform == 'win32'" - }, - { - "requires": ["CompressPadding"], - "extra": "warmup" - } - ] - - -Build requires --------------- - -A list of other distributions needed when this distribution is being built -(creating a binary archive from an sdist, source archive or VCS checkout). - -Note that while these are build dependencies for the distribution being -built, the installation is a *deployment* scenario for the dependencies. - -Automated tools MAY disallow strict version matching clauses and direct -references in this field and SHOULD at least emit a warning for such clauses. - -Public index servers SHOULD NOT allow strict version matching clauses or -direct references in this field. - -Example:: - - "build_requires": - { - "requires": ["setuptools >= 0.7"] - }, - { - "requires": ["pywin32 > 1.0"], - "environment": "sys_platform == 'win32'" - }, - { - "requires": ["cython"], - "extra": "c-accelerators" - } - ] - - -Dev requires ------------- - -A list of any additional distributions needed during development of this -distribution that aren't already covered by the deployment and build -dependencies. - -Additional dependencies that may be listed in this field include: - -* tools needed to create an sdist from a source archive or VCS checkout -* tools needed to generate project documentation that is published online - rather than distributed along with the rest of the software - -Automated tools MAY disallow strict version matching clauses and direct -references in this field and SHOULD at least emit a warning for such clauses. - -Public index servers SHOULD NOT allow strict version matching clauses or -direct references in this field. - -Example:: - - "dev_requires": - { - "requires": ["hgtools", "sphinx >= 1.0"] - }, - { - "requires": ["pywin32 > 1.0"], - "environment": "sys_platform == 'win32'" - } - ] +Aside from the four standard extra categories, any extras referenced from a +dependency specifier MUST be named in the `Extras`_ field for this distribution. +This helps avoid typographical errors and also makes it straightforward to +identify the available extras without scanning the full set of dependencies. Provides @@ -1094,10 +875,6 @@ For example, this may be used to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. -For instance, with distribute merged back into setuptools, the merged -project is able to include a ``"provides": ["distribute"]`` entry to -satisfy any projects that require the now obsolete distribution's name. - To avoid malicious hijacking of names, when interpreting metadata retrieved from a public index server, automated tools MUST NOT pay any attention to ``"provides"`` entries that do not correspond to a published distribution. @@ -1149,10 +926,6 @@ Automated tools SHOULD report a warning when installing an obsolete project. Possible uses for this field include handling project name changes and project mergers. -For instance, with distribute merging back into setuptools, a new version -of distribute may be released that depends on the new version of setuptools, -and also explicitly indicates that distribute itself is now obsolete. - Note that without a corresponding ``provides``, there is no expectation that the replacement project will be a "drop-in" replacement for the obsolete project - at the very least, upgrading to the new distribution @@ -1163,9 +936,6 @@ Examples:: "name": "BadName", "obsoleted_by": "AcceptableName" - "name": "distribute", - "obsoleted_by": "setuptools >= 0.7" - Metadata Extensions =================== @@ -1259,86 +1029,64 @@ back on attempting to install from source rather than failing entirely. Extras (optional dependencies) ============================== -.. note:: +As defined in PEP 508, extras are additional dependencies that enable an +optional aspect of a project release, often corresponding to a ``try: import +optional_dependency ...`` block in the code. They are also used to indicate +semantic dependencies for activities other than normal runtime using (such as +testing, building, or working on the component). - The general definition of extras has been moved out to the - specification for dependency declarations: PEP 508. - -Extras are additional dependencies that enable an optional aspect -of the distribution, often corresponding to a ``try: import -optional_dependency ...`` block in the code. To support the use of the -distribution with or without the optional dependencies they are listed -separately from the distribution's core dependencies and must be requested -explicitly, either in the dependency specifications of another distribution, -or else when issuing a command to an installation tool. - -Note that installation of extras is not tracked directly by installation -tools: extras are merely a convenient way to indicate a set of dependencies -that is needed to provide some optional functionality of the distribution. -If selective *installation* of components is desired, then multiple -distributions must be defined rather than relying on the extras system. - -The names of extras MUST abide by the same restrictions as those for -distribution names. +To support the use of the release with or without the optional dependencies, +they are listed separately from the release's core runtime dependencies +and must be requested explicitly, either in the dependency specifications of +another project, or else when issuing a command to an installation tool. Example of a distribution with optional dependencies:: "name": "ComfyChair", - "extras": ["warmup", "c-accelerators"] - "run_requires": [ + "extras": ["warmup"] + "dependencies": [ { "requires": ["SoftCushions"], "extra": "warmup" - } - ] - "build_requires": [ + }, { "requires": ["cython"], - "extra": "c-accelerators" + "extra": "build" } ] Other distributions require the additional dependencies by placing the relevant extra names inside square brackets after the distribution name when -specifying the dependency. +specifying the dependency. Multiple extras from a dependency can be requested +by placing to -Extra specifications MUST allow the following additional syntax: +In addition to the required syntax defined in PEP 508, automated tools +MAY offer the following additional enhancements to control precisely +which dependencies are installed: -* Multiple extras can be requested by separating them with a comma within - the brackets. +* The special extra ``*`` to request that all optional dependencies be + installed (unless excluded based on environment markers) -* The following special extras request processing of the corresponding - lists of dependencies: +* The special extra ``-self`` to indicate that the project itself should not + be installed, only its dependencies - * ``:meta:``: ``meta_requires`` - * ``:run:``: ``run_requires`` - * ``:test:``: ``test_requires`` - * ``:build:``: ``build_requires`` - * ``:dev:``: ``dev_requires`` - * ``:*:``: process *all* dependency lists +* The special extra ``-runtime`` to indicate that the otherwise unconditional + runtime dependencies should not be installed, only the dependencies for the + requested extras -* The ``*`` character as an extra is a wild card that enables all of the - entries defined in the distribution's ``extras`` field. - -* Extras may be explicitly excluded by prefixing their name with a ``-`` +* Other extras may be explicitly excluded by prefixing their name with a ``-`` character (this is useful in conjunction with ``*`` to exclude only particular extras that are definitely not wanted, while enabling all - others). - -* The ``-`` character as an extra specification indicates that the - distribution itself should NOT be installed, and also disables the - normally implied processing of ``:meta:`` and ``:run:`` dependencies - (those may still be requested explicitly using the appropriate extra - specifications). - -Command line based installation tools SHOULD support this same syntax to -allow extras to be requested explicitly. + others) The full set of dependency requirements is then based on the top level dependencies, along with those of any requested extras. Dependency examples (showing just the ``requires`` subfield):: + "requires": ["ComfyChair"] + -> requires ``ComfyChair`` only + "requires": ["ComfyChair[warmup]"] -> requires ``ComfyChair`` and ``SoftCushions`` @@ -1349,113 +1097,17 @@ Dependency examples (showing just the ``requires`` subfield):: Command line examples:: pip install ComfyChair - -> installs ComfyChair with applicable :meta: and :run: dependencies + -> installs ComfyChair with applicable runtime dependencies - pip install ComfyChair[*] - -> as above, but also installs all extra dependencies + pip install ComfyChair[-build,-dev,-doc,-test,*] + -> as above, but also installs all non-standard extra dependencies - pip install ComfyChair[-,:build:,*] - -> installs just the build dependencies with all extras + pip install ComfyChair[-self,-runtime,build,dev] + -> installs just the build and development dependencies - pip install ComfyChair[-,:build:,:run:,:meta:,:test:,*] - -> as above, but also installs dependencies needed to run the tests - - pip install ComfyChair[-,:*:,*] - -> installs the full set of development dependencies, but avoids - installing ComfyChair itself - - -Environment markers -=================== - -.. note:: - - The definition of environment markers has been moved out to the - specification for dependency declarations: PEP 508. - -An **environment marker** describes a condition about the current execution -environment. They are used to indicate when certain dependencies are only -required in particular environments, and to indicate supported platforms -for distributions with additional constraints beyond the availability of a -Python runtime. - -Here are some examples of such markers:: - - "sys_platform == 'win32'" - "platform_machine == 'i386'" - "python_version == '2.4' or python_version == '2.5'" - "'linux' in sys_platform" - -And here's an example of some conditional metadata for a distribution that -requires PyWin32 both at runtime and buildtime when using Windows:: - - "name": "ComfyChair", - "run_requires": [ - { - "requires": ["pywin32 > 1.0"], - "environment": "sys.platform == 'win32'" - } - ] - "build_requires": [ - { - "requires": ["pywin32 > 1.0"], - "environment": "sys.platform == 'win32'" - } - ] - -The micro-language behind this is a simple subset of Python: it compares -only strings, with the ``==`` and ``in`` operators (and their opposites), -and with the ability to combine expressions. Parentheses are supported -for grouping. - -The pseudo-grammar is :: - - MARKER: EXPR [(and|or) EXPR]* - EXPR: ("(" MARKER ")") | (SUBEXPR [CMPOP SUBEXPR]) - CMPOP: (==|!=|<|>|<=|>=|in|not in) - -where ``SUBEXPR`` is either a Python string (such as ``'2.4'``, or -``'win32'``) or one of the following marker variables: - -* ``python_version``: ``'{0.major}.{0.minor}'.format(sys.version_info)`` -* ``python_full_version``: see definition below -* ``os_name``: ``os.name`` -* ``sys_platform``: ``sys.platform`` -* ``platform_release``: ``platform.release()`` -* ``platform_version``: ``platform.version()`` -* ``platform_machine``: ``platform.machine()`` -* ``platform_python_implementation``: ``platform.python_implementation()`` -* ``implementation_name``: ``sys.implementation.name`` -* ``implementation_version``: see definition below - -If a particular value is not available (such as the ``sys.implementation`` -subattributes in versions of Python prior to 3.3), the corresponding marker -variable MUST be considered equivalent to the empty string. - -Note that all subexpressions are restricted to strings or one of the -marker variable names (which refer to string values), meaning that it is -not possible to use other sequences like tuples or lists on the right -side of the ``in`` and ``not in`` operators. - -Chaining of comparison operations is permitted using the normal Python -semantics of an implied ``and``. - -The ``python_full_version`` and ``implementation_version`` marker variables -are derived from ``sys.version_info()`` and ``sys.implementation.version`` -respectively, in accordance with the following algorithm:: - - def format_full_version(info): - version = '{0.major}.{0.minor}.{0.micro}'.format(info) - kind = info.releaselevel - if kind != 'final': - version += kind[0] + str(info.serial) - return version - - python_full_version = format_full_version(sys.version_info) - implementation_version = format_full_version(sys.implementation.version) - -``python_full_version`` will typically correspond to the leading segment -of ``sys.version()``. + pip install ComfyChair[-self,*] + -> installs the full set of runtime and development dependencies, but + avoids installing ComfyChair itself Updating the metadata specification @@ -1507,21 +1159,22 @@ As an example of mapping this PEP to Linux distro packages, assume an example project without any extras defined is split into 2 RPMs in a SPEC file: ``example`` and ``example-devel``. -The ``meta_requires`` and ``run_requires`` dependencies would be mapped -to the Requires dependencies for the "example" RPM (a mapping from -environment markers relevant to Linux to SPEC file conditions would -also allow those to be handled correctly) +The unconditional dependencies would be mapped to the Requires dependencies +for the "example" RPM (a mapping from environment markers relevant to Linux +to SPEC file conditions would also allow those to be handled correctly). -The ``build_requires`` dependencies would be mapped to the BuildRequires -dependencies for the "example" RPM. +The ``build`` and ``dev`` extra dependencies would be mapped to the +BuildRequires dependencies for the "example" RPM. Depending on how the +``%check`` section in the RPM was defined, the ``test`` extra may also be +mapped to the BuildRequires declaration for the RPM. -All defined dependencies relevant to Linux, including those in -``dev_requires`` and ``test_requires`` would become Requires dependencies -for the "example-devel" RPM. +All defined dependencies relevant to Linux in the ``dev``, ``test``, ``build``, +and ``doc`` extras would become Requires dependencies for the "example-devel" +RPM. -A documentation toolchain dependency like Sphinx would either go in -``build_requires`` (for example, if man pages were included in the -built distribution) or in ``dev_requires`` (for example, if the +A documentation toolchain dependency like Sphinx would either go in the +``build`` extra (for example, if man pages were included in the +built distribution) or in the ``doc`` extra (for example, if the documentation is published solely through ReadTheDocs or the project website). This would be enough to allow an automated converter to map it to an appropriate dependency in the spec file. @@ -1529,16 +1182,12 @@ to map it to an appropriate dependency in the spec file. If the project did define any extras, those could be mapped to additional virtual RPMs with appropriate BuildRequires and Requires entries based on the details of the dependency specifications. Alternatively, they could -be mapped to other system package manager features (such as package lists -in ``yum``). +be mapped to other system package manager features (such as weak dependencies). -Other system package managers may have other options for dealing with -extras (Debian packagers, for example, would have the option to map them -to "Recommended" or "Suggested" package entries). - -The metadata extension format should also allow distribution specific hints -to be included in the upstream project metadata without needing to manually -duplicate any of the upstream metadata in a distribution specific format. +The metadata extension format should also provide a way for distribution +specific hints to be included in the upstream project metadata without needing +to manually duplicate any of the upstream metadata in a distribution specific +format. Appendix C: Summary of differences from \PEP 345 @@ -1565,34 +1214,16 @@ Appendix C: Summary of differences from \PEP 345 * Added the source label mechanism as described in PEP 440 -* Support for different kinds of dependencies +* Formally defined dependency declarations, extras, and environment markers + in PEP 508 -* The "Extras" optional dependency mechanism - -* A well-defined metadata extension mechanism, and migration of any fields - not needed for dependency resolution to standard extensions. - -* Clarify and simplify various aspects of environment markers: - - * allow use of parentheses for grouping in the pseudo-grammar - * consistently use underscores instead of periods in the variable names - * allow ordered string comparisons and chained comparisons - -* New constraint mechanism to define supported environments and ensure - compatibility between independently built binary components at - installation time +* Support for different kinds of dependencies through additional reserved + extra names * Updated obsolescence mechanism -* More flexible system for defining contact points and contributors - -* Defined a recommended set of project URLs - -* Identification of supporting documents in the ``dist-info`` directory: - - * Allows markup formats to be indicated through file extensions - * Standardises the common practice of taking the description from README - * Also supports inclusion of license files and changelogs +* A well-defined metadata extension mechanism, and migration of any fields + not needed for dependency resolution to standard extensions * With all due respect to Charles Schulz and Peanuts, many of the examples have been updated to be more `thematically appropriate`_ for Python ;) @@ -1641,8 +1272,8 @@ eventually even the option to embed arbitrary JSON inside particular subfields. The old serialisation format also wasn't amenable to easy conversion to -standard Python data structures for use in the new install hook APIs, or -in future extensions to the importer APIs to allow them to provide +standard Python data structures for use in any new install hook APIs, or +in future extensions to the runtime importer APIs to allow them to provide information for inclusion in the installation database. Accordingly, we've taken the step of switching to a JSON-compatible metadata @@ -1681,33 +1312,6 @@ version, like a hash or tag name that allows the release to be reconstructed from the project version control system. -Support for different kinds of dependencies -------------------------------------------- - -The separation of the five different kinds of dependency allows a -distribution to indicate whether a dependency is needed specifically to -develop, build, test or use the distribution. - -To allow for metadistributions like PyObjC, while still actively -discouraging overly strict dependency specifications, the separate -``meta`` dependency fields are used to separate out those dependencies -where exact version specifications are appropriate. - -The advantage of having these distinctions supported in the upstream Python -specific metadata is that even if a project doesn't care about these -distinction themselves, they may be more amenable to patches from -downstream redistributors that separate the fields appropriately. Over time, -this should allow much greater control over where and when particular -dependencies end up being installed. - -The names for the dependency fields have been deliberately chosen to avoid -conflicting with the existing terminology in setuptools and previous -versions of the metadata standard. Specifically, the names ``requires``, -``install_requires`` and ``setup_requires`` are not used, which will -hopefully reduce confusion when converting legacy metadata to the new -standard. - - Support for optional dependencies for distributions --------------------------------------------------- @@ -1724,103 +1328,47 @@ in particular those associated with build systems (with optional support for running the test suite) and development systems. +Support for different kinds of semantic dependencies +---------------------------------------------------- + +The separation of the five different kinds of dependency through the Extras +system allows a project to optionally indicate whether a dependency is needed +specifically to develop, build, test or use the distribution. + +The advantage of having these distinctions supported in the upstream Python +specific metadata is that even if a project doesn't care about these +distinction themselves, they may be more amenable to patches from +downstream redistributors that separate the fields appropriately. Over time, +this should allow much greater control over where and when particular +dependencies end up being installed. + + Support for metadata extensions ------------------------------- The new extension effectively allows sections of the metadata -namespace to be delegated to other distributions, while preserving a +namespace to be delegated to other projects, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. -It also works well in combination with the new ``build_requires`` field +It also works well in combination with the new ``build`` extra to allow a distribution to depend on tools which *do* know how to handle -the chosen extension, and the new extras mechanism, allowing support for -particular extensions to be provided as optional features. +the chosen extension, and the new extras mechanism in general, allowing +support for particular extensions to be provided as optional features. Possible future uses for extensions include declaration of plugins for -other distributions and hints for automatic conversion to Linux system +other projects and hints for automatic conversion to Linux system packages. The ability to declare an extension as required is included primarily to allow the definition of the metadata hooks extension to be deferred until some time after the initial adoption of the metadata 2.0 specification. If -a distribution needs a ``postinstall`` hook to run in order to complete +a release needs a ``postinstall`` hook to run in order to complete the installation successfully, then earlier versions of tools should fall back to installing from source rather than installing from a wheel file and then failing to run the expected postinstall hook. -Changes to environment markers ------------------------------- - -There are three substantive changes to environment markers in this version: - -* ``platform_release`` was added, as it provides more useful information - than ``platform_version`` on at least Linux and Mac OS X (specifically, - it provides details of the running kernel version) -* ordered comparison of strings is allowed, as this is more useful for - setting minimum and maximum versions where conditional dependencies - are needed or where a platform is supported -* comparison chaining is explicitly allowed, as this becomes useful in the - presence of ordered comparisons - -The other changes to environment markers are just clarifications and -simplifications to make them easier to use. - -The arbitrariness of the choice of ``.`` and ``_`` in the different -variables was addressed by standardising on ``_`` (as these are all -predefined variables rather than live references into the Python module -namespace) - -The use of parentheses for grouping was explicitly noted to address some -underspecified behaviour in the previous version of the specification. - - -Updated contact information ---------------------------- - -This feature is provided by the ``python.project`` and -``python.integrator`` extensions in :pep:`459`. - -The switch to JSON made it possible to provide a more flexible -system for defining multiple contact points for a project, as well as -listing other contributors. - -The ``type`` concept allows for preservation of the distinction between -the original author of a project, and a lead maintainer that takes over -at a later date. - - -Changes to project URLs ------------------------ - -This feature is provided by the ``python.project`` and -``python.integrator`` extensions in :pep:`459`. - -In addition to allow arbitrary strings as project URL labels, the new -metadata standard also defines a recommend set of four URL labels for -a distribution's home page, documentation, source control and issue tracker. - - -Changes to platform support ---------------------------- - -This feature is provided by the ``python.constraints`` extension in -:pep:`459`. - -The new environment marker system makes it possible to define supported -platforms in a way that is actually amenable to automated processing. This -has been used to replace several older fields with poorly defined semantics. - -The constraints mechanism also allows additional information to be -conveyed through metadata extensions and then checked for consistency at -install time. - -For the moment, the old ``Requires-External`` field has been removed -entirely. The metadata extension mechanism will hopefully prove to be a more -useful replacement. - - Updated obsolescence mechanism ------------------------------ @@ -1837,62 +1385,15 @@ is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. -Included text documents ------------------------ - -This feature is provided by the ``python.details`` extension in :pep:`459`. - -Currently, PyPI attempts to determine the description's markup format by -rendering it as reStructuredText, and if that fails, treating it as plain -text. - -Furthermore, many projects simply read their long description in from an -existing README file in ``setup.py``. The popularity of this practice is -only expected to increase, as many online version control systems -(including both GitHub and BitBucket) automatically display such files -on the landing page for the project. - -Standardising on the inclusion of the long description as a separate -file in the ``dist-info`` directory allows this to be simplified: - -* An existing file can just be copied into the ``dist-info`` directory as - part of creating the sdist -* The expected markup format can be determined by inspecting the file - extension of the specified path - -Allowing the intended format to be stated explicitly in the path allows -the format guessing to be removed and more informative error reports to be -provided to users when a rendering error occurs. - -This is especially helpful since PyPI applies additional restrictions to -the rendering process for security reasons, thus a description that renders -correctly on a developer's system may still fail to render on the server. - -The document naming system used to achieve this then makes it relatively -straightforward to allow declaration of alternative markup formats like -HTML, Markdown and AsciiDoc through the use of appropriate file -extensions, as well as to define similar included documents for the -project's license and changelog. - -Grouping the included document names into a single top level field gives -automated tools the option of treating them as arbitrary documents without -worrying about their contents. - -Requiring that the included documents be added to the ``dist-info`` metadata -directory means that the complete metadata for the distribution can be -extracted from an sdist or binary archive simply by extracting that -directory, without needing to check for references to other files in the -sdist. - - Appendix D: Deferred features ============================= Several potentially useful features have been deliberately deferred in order to better prioritise our efforts in migrating to the new metadata standard. These all reflect information that may be nice to have in the -new metadata, but which can be readily added in metadata 2.1 without -breaking any use cases already supported by metadata 2.0. +new metadata, but which can be readily added through metadata extensions or +in metadata 2.1 without breaking any use cases already supported by metadata +2.0. Once the ``pypi``, ``setuptools``, ``pip``, ``wheel`` and ``distlib`` projects support creation and consumption of metadata 2.0, then we may @@ -1900,15 +1401,24 @@ revisit the creation of metadata 2.1 with some or all of these additional features. +Standard extensions +------------------- + +Some of the information provided by the legacy metadata system has been +moved out to standard extensions defined in PEP 459. + +This allows publication of the dependency metadata in a more readily +consumable format to proceed even before the full details of those extensions +have been resolved. + + MIME type registration ---------------------- At some point after acceptance of the PEP, we may submit the -following MIME type registration requests to IANA: +following MIME type registration request to IANA: -* Full metadata: ``application/vnd.python.pydist+json`` -* Essential dependency resolution metadata: - ``application/vnd.python.pydist-dependencies+json`` +* ``application/vnd.python.pydist+json`` It's even possible we may be able to just register the ``vnd.python`` namespace under the banner of the PSF rather than having to register @@ -1927,152 +1437,6 @@ and the fact that 64-bit Windows still shows up as ``win32`` is more than a little strange. -Support for metadata hooks ---------------------------- - -While a draft proposal for a `metadata hook system -`__ -has been created, that proposal is not part of the initial set of standard -metadata extensions in PEP 459. - -A metadata hook system would allow the wheel format to fully replace direct -installation on deployment targets, by allowing projects to explicitly -define code that should be executed following installation from a wheel file. - -This may be something relatively simple, like the `two line -refresh `__ -of the Twisted plugin caches that the Twisted developers recommend for -any project that provides Twisted plugins, to more complex platform -dependent behaviour, potentially in conjunction with appropriate -metadata extensions and ``supports_environments`` entries. - -For example, upstream declaration of external dependencies for various -Linux distributions in a distribution neutral format may be supported by -defining an appropriate metadata extension that is read by a postinstall -hook and converted into an appropriate invocation of the system package -manager. Other operations (such as registering COM DLLs on Windows, -registering services for automatic startup on any platform, or altering -firewall settings) may need to be undertaken with elevated privileges, -meaning they cannot be deferred to implicit execution on first use of the -distribution. - -For the time being, any such system is being left to the realm of tool -specific metadata extensions. This does mean that affected projects may -choose not to publish wheel files, instead continuing to rely on source -distributions until the relevant extension is well defined and widely -supported. - - -Metabuild system ----------------- - -This version of the metadata specification continues to use ``setup.py`` -and the distutils command syntax to invoke build and test related -operations on a source archive or VCS checkout. - -It may be desirable to replace these in the future with tool independent -entry points that support: - -* Generating the metadata file on a development system -* Generating an sdist on a development system -* Generating a binary archive on a build system -* Running the test suite on a built (but not installed) distribution - -Metadata 2.0 deliberately focuses on wheel based installation, leaving -sdist, source archive, and VCS checkout based installation to use the -existing ``setup.py`` based ``distutils`` command interface. - -In the meantime, the above operations will be handled through the -``distutils``/``setuptools`` command system: - -* ``python setup.py dist_info`` -* ``python setup.py sdist`` -* ``python setup.py build_ext --inplace`` -* ``python setup.py test`` -* ``python setup.py bdist_wheel`` - -The following metabuild hooks may be defined in metadata 2.1 to -cover these operations without relying on ``setup.py``: - -* ``make_dist_info``: generate the sdist's dist_info directory -* ``make_sdist``: create the contents of an sdist -* ``build_dist``: create the contents of a binary wheel archive from an - unpacked sdist -* ``test_built_dist``: run the test suite for a built distribution - -Tentative signatures have been designed for those hooks, but in order to -better focus initial development efforts on the integration and installation -use cases, they will not be pursued further until metadata 2.1:: - - def make_dist_info(source_dir, info_dir): - """Generate the contents of dist_info for an sdist archive - - *source_dir* points to a source checkout or unpacked tarball - *info_dir* is the destination where the sdist metadata files should - be written - - Returns the distribution metadata as a dictionary. - """ - - def make_sdist(source_dir, contents_dir, info_dir): - """Generate the contents of an sdist archive - - *source_dir* points to a source checkout or unpacked tarball - *contents_dir* is the destination where the sdist contents should be - written (note that archiving the contents is the responsibility of - the metabuild tool rather than the hook function) - *info_dir* is the destination where the sdist metadata files should - be written - - Returns the distribution metadata as a dictionary. - """ - - def build_dist(sdist_dir, built_dir, info_dir, compatibility=None): - """Generate the contents of a binary wheel archive - - *sdist_dir* points to an unpacked sdist - *built_dir* is the destination where the wheel contents should be - written (note that archiving the contents is the responsibility of - the metabuild tool rather than the hook function) - *info_dir* is the destination where the wheel metadata files should - be written - *compatibility* is an optional PEP 425 compatibility tag indicating - the desired target compatibility for the build. If the tag cannot - be satisfied, the hook should throw ``ValueError``. - - Returns the actual compatibility tag for the build - """ - - def test_built_dist(sdist_dir, built_dir, info_dir): - """Check a built (but not installed) distribution works as expected - - *sdist_dir* points to an unpacked sdist - *built_dir* points to a platform appropriate unpacked wheel archive - (which may be missing the wheel metadata directory) - *info_dir* points to the appropriate wheel metadata directory - - Requires that the distribution's test dependencies be installed - (indicated by the ``:test:`` extra). - - Returns ``True`` if the check passes, ``False`` otherwise. - """ - -As with the existing install hooks, checking for extras would be done -using the same import based checks as are used for runtime extras. That -way it doesn't matter if the additional dependencies were requested -explicitly or just happen to be available on the system. - -There are still a number of open questions with this design, such as whether -a single build hook is sufficient to cover both "build for testing" and -"prep for deployment", as well as various complexities like support for -cross-compilation of binaries, specification of target platforms and -Python versions when creating wheel files, etc. - -Opting to retain the status quo for now allows us to make progress on -improved metadata publication and binary installation support, rather than -having to delay that awaiting the creation of a viable metabuild framework. - - Appendix E: Rejected features ============================= @@ -2090,6 +1454,28 @@ automated tools and removing it also made the PEP and metadata schema substantially shorter, suggesting it was actually harder to explain as well. +Separate lists for semantic dependencies +---------------------------------------- + +Earlier versions of this PEP used separate fields rather than the extras +system for test, build, documentation, and development dependencies. This +turned out to be annoying to handle in automated tools and removing it also +made the PEP and metadata schema substantially shorter, suggesting it was +actually harder to explain as well. + + +Introducing friction for overly precise dependency declarations +--------------------------------------------------------------- + +Earlier versions of this PEP attempted to introduce friction into the +inappropriate use of overly strict dependency declarations in published +releases. Discussion on distutils-sig came to the conclusion that wasn't +a serious enough problem to tackle directly at the interoperability +specification layer, and if it does become a problem in the future, +it would be better tackled at the point where projects are uploaded to +the public Python Package Index. + + Disallowing underscores in distribution names --------------------------------------------- @@ -2098,7 +1484,7 @@ unduly restrictive for this spec given the common practice of using valid Python identifiers as Python distribution names. A Debian side policy of converting underscores to hyphens seems easy enough to implement (and the requirement to consider hyphens and underscores as -equivalent ensures that doing so won't introduce any conflicts). +equivalent ensures that doing so won't introduce any name conflicts). Allowing the use of Unicode in distribution names @@ -2119,19 +1505,6 @@ challenging enough without adding more general Unicode identifier support into the mix. -Single list for conditional and unconditional dependencies ----------------------------------------------------------- - -It's technically possible to store the conditional and unconditional -dependencies of each kind in a single list and switch the handling based on -the entry type (string or mapping). - -However, the current ``*requires`` vs ``*may-require`` two list design seems -easier to understand and work with, since it's only the conditional -dependencies that need to be checked against the requested extras list and -the target installation environment. - - Depending on source labels -------------------------- @@ -2145,7 +1518,7 @@ Alternative dependencies ------------------------ An earlier draft of this PEP considered allowing lists in place of the -usual strings in dependency specifications to indicate that there aren +usual strings in dependency specifications to indicate that there are multiple ways to satisfy a dependency. If at least one of the individual dependencies was already available, then @@ -2162,8 +1535,7 @@ since Pillow/PIL style forks aren't common, and the database driver use case would arguably be better served by an SQL Alchemy defined "supported database driver" metadata extension where a project depends on SQL Alchemy, and then declares in the extension which database drivers are checked for -compatibility by the upstream project (similar to the advisory -``supports_environments`` field in the main metadata). +compatibility by the upstream project. We're also getting better support for "virtual provides" in this version of the metadata standard, so this may end up being an installer and index