From a75b50475eb2fd156fe7e5b58c6dd67fd8719462 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Wed, 2 Jul 2014 22:55:34 -0700 Subject: [PATCH] Update PEP 426 for PEP 440 changes & add python.constraints extension --- pep-0426.txt | 229 ++++++++++++++++++--------------------------------- pep-0459.txt | 194 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 254 insertions(+), 169 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt index 9c9fe1ee3..789c1a520 100644 --- a/pep-0426.txt +++ b/pep-0426.txt @@ -602,15 +602,18 @@ Example:: Version ------- -The distribution's public version identifier, as defined in PEP 440. Public -versions are designed for consumption by automated tools and support a -variety of flexible version specification mechanisms (see PEP 440 for -details). +The distribution's public or local version identifier, as defined in PEP 440. +Version identifiers are designed for consumption by automated tools and +support a variety of flexible version specification mechanisms (see PEP 440 +for details). Version identifiers MUST comply with the format defined in PEP 440. Version identifiers MUST be unique within each project. +Index servers MAY place restrictions on the use of local version identifiers +as described in PEP 440. + Example:: "version": "1.0a2" @@ -646,17 +649,31 @@ a distribution does not provide them, including failing cleanly when an operation depending on one of these fields is requested. -Source label ------------- +Source labels +------------- -A constrained identifying text string, as defined in PEP 440. Source labels -cannot be used in version specifiers - they are included for information -purposes only. +Source labels are text strings with minimal defined semantics. They are +intended to allow the original source code to be unambiguously identified, +even if an integrator has applied additional local modifications to a +particular distribution. -Source labels MUST meet the character restrictions defined in PEP 440. +To ensure source labels can be readily incorporated as part of file names +and URLs, and to avoid formatting inconsistencies in hexadecimal hash +representations they MUST be limited to the following set of permitted +characters: -Source labels MUST be unique within each project and MUST NOT match any -defined version for the project. +* Lowercase ASCII letters (``[a-z]``) +* ASCII digits (``[0-9]``) +* underscores (``_``) +* hyphens (``-``) +* periods (``.``) +* plus signs (``+``) + +Source labels MUST start and end with an ASCII letter or digit. + +A source label for a project MUST NOT match any defined version for that +project. This restriction ensures that there is no ambiguity between version +identifiers and source labels. Examples:: @@ -817,12 +834,11 @@ Requirement specifiers Individual requirements are defined as strings containing a distribution name (as found in the ``name`` field). The distribution name may be followed by an extras specifier (enclosed in square -brackets) and by a version specifier or direct reference (within -parentheses). +brackets) and by a version specifier or direct reference. Whitespace is permitted between the distribution name and an opening square bracket or parenthesis. Whitespace is also permitted between a -closing square bracket and an opening parenthesis. +closing square bracket and the version specifier. See `Extras (optional dependencies)`_ for details on extras and PEP 440 for details on version specifiers and direct references. @@ -837,9 +853,9 @@ Example requirement specifiers:: "Flask" "Django" "Pyramid" - "SciPy (0.12)" + "SciPy ~= 0.12" "ComfyChair[warmup]" - "ComfyChair[warmup] (> 0.1)" + "ComfyChair[warmup] > 0.1" Mapping dependencies to development and distribution activities @@ -912,10 +928,10 @@ Example:: "run_requires": { - "requires": ["SciPy", "PasteDeploy", "zope.interface (>3.5.0)"] + "requires": ["SciPy", "PasteDeploy", "zope.interface > 3.5.0"] }, { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys_platform == 'win32'" }, { @@ -950,11 +966,11 @@ Example:: "meta_requires": { - "requires": ["ComfyUpholstery (== 1.0a2)", - "ComfySeatCushion (== 1.0a2)"] + "requires": ["ComfyUpholstery == 1.0a2", + "ComfySeatCushion == 1.0a2"] }, { - "requires": ["CupOfTeaAtEleven (== 1.0a2)"], + "requires": ["CupOfTeaAtEleven == 1.0a2"], "environment": "'linux' in sys_platform" } ] @@ -979,7 +995,7 @@ Example:: "requires": ["unittest2"] }, { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys_platform == 'win32'" }, { @@ -1008,10 +1024,10 @@ Example:: "build_requires": { - "requires": ["setuptools (>= 0.7)"] + "requires": ["setuptools >= 0.7"] }, { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys_platform == 'win32'" }, { @@ -1044,10 +1060,10 @@ Example:: "dev_requires": { - "requires": ["hgtools", "sphinx (>= 1.0)"] + "requires": ["hgtools", "sphinx >= 1.0"] }, { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys_platform == 'win32'" } ] @@ -1138,59 +1154,7 @@ Examples:: "obsoleted_by": "AcceptableName" "name": "distribute", - "obsoleted_by": "setuptools (>= 0.7)" - - -Supports Environments ---------------------- - -A list of strings specifying the environments that the distribution -explicitly supports. An environment is considered supported if it -matches at least one of the environment markers given. - -If this field is not given in the metadata, it is assumed that the -distribution supports any platform supported by Python. - -Individual entries are environment markers, as described in -`Environment markers`_. - -Installation tools SHOULD report an error if supported environments are -specified by the distribution and the current platform fails to match -any of them, MUST at least emit a warning, and MAY allow the user to -force the installation to proceed regardless. - -The two main uses of this field are to declare which versions of Python -and which underlying operating systems are supported. - -Examples indicating supported Python versions:: - - # Supports Python 2.6+ - "supports_environments": ["python_version >= '2.6'"] - - # Supports Python 2.6+ (for 2.x) or 3.3+ (for 3.x) - "supports_environments": ["python_version >= '3.3'", - "'3.0' > python_version >= '2.6'"] - -Examples indicating supported operating systems:: - - # Windows only - "supports_environments": ["sys_platform == 'win32'"] - - # Anything except Windows - "supports_environments": ["sys_platform != 'win32'"] - - # Linux or BSD only - "supports_environments": ["'linux' in sys_platform", - "'bsd' in sys_platform"] - -Example where the supported Python version varies by platform:: - - # The standard library's os module has long supported atomic renaming - # on POSIX systems, but only gained atomic renaming on Windows in Python - # 3.3. A distribution that needs atomic renaming support for reliable - # operation might declare the following supported environments. - "supports_environments": ["python_version >= '2.6' and sys_platform != 'win32'", - "python_version >= '3.3' and sys_platform == 'win32'"] + "obsoleted_by": "setuptools >= 0.7" Metadata Extensions @@ -1204,7 +1168,7 @@ Two key names are reserved and MUST NOT be used by extensions, except as described below: * ``extension_version`` -* ``required_extension`` +* ``installer_must_handle`` The following example shows the ``python.details`` and ``python.commands`` standard extensions from :pep:`459`:: @@ -1263,16 +1227,16 @@ extension metadata using the lowest metadata version that includes all of the needed fields. -Required extensions -------------------- +Required extension handling +--------------------------- A project may consider correct handling of some extensions to be essential to correct installation of the software. This is indicated by setting the -``required_extension`` field to ``true``. Setting it to ``false`` or +``installer_must_handle`` field to ``true``. Setting it to ``false`` or omitting it altogether indicates that processing the extension when installing the distribution is not considered mandatory by the developers. -Installation tools MUST fail if ``required_extension`` is set to ``true`` +Installation tools MUST fail if ``installer_must_handle`` is set to ``true`` for an extension and the tool does not have any ability to process that particular extension (whether directly or through a tool-specific plugin system). @@ -1408,13 +1372,13 @@ requires PyWin32 both at runtime and buildtime when using Windows:: "name": "ComfyChair", "run_requires": [ { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys.platform == 'win32'" } ] "build_requires": [ { - "requires": ["pywin32 (>1.0)"], + "requires": ["pywin32 > 1.0"], "environment": "sys.platform == 'win32'" } ] @@ -1594,7 +1558,9 @@ Appendix C: Summary of differences from \PEP 345 * consistently use underscores instead of periods in the variable names * allow ordered string comparisons and chained comparisons -* New system for defining supported environments +* New constraint mechanism to define supported environments and ensure + compatibility between independently built binary components at + installation time * Updated obsolescence mechanism @@ -1665,6 +1631,13 @@ generate correctly. Changing the name of the metadata file also makes it easy to distribute 1.x and 2.x metadata in parallel, greatly simplifying several aspects of the migration to the new metadata format. +The specific choice of ``pydist.json`` as the preferred file name relates +to the fact that the metadata described in these files applies to the +distribution as a whole, rather than to any particular build. Additional +metadata formats may be defined in the future to hold information that can +only be determined after building a binary distribution for a particular +target environment. + Changing the version scheme --------------------------- @@ -1676,7 +1649,16 @@ versioning scheme. Source labels ------------- -See PEP 440 for the rationale behind the addition of this field. +The new source label support is intended to make it clearer that the +constraints on public version identifiers are there primarily to aid in +the creation of reliable automated dependency analysis tools. Projects +are free to use whatever versioning scheme they like internally, so long +as they are able to translate it to something the dependency analysis tools +will understand. + +Source labels also make it straightforward to record specific details of a +version, like a hash or tag name that allows the release to be reconstructed +from the project version control system. Support for different kinds of dependencies @@ -1803,10 +1785,17 @@ a distribution's home page, documentation, source control and issue tracker. Changes to platform support --------------------------- +This feature is provided by the ``python.constraints`` extension in +:pep:`459`. + The new environment marker system makes it possible to define supported platforms in a way that is actually amenable to automated processing. This has been used to replace several older fields with poorly defined semantics. +The constraints mechanism also allows additional information to be +conveyed through metadata extensions and then checked for consistency at +install time. + For the moment, the old ``Requires-External`` field has been removed entirely. The metadata extension mechanism will hopefully prove to be a more useful replacement. @@ -1894,7 +1883,7 @@ features. MIME type registration ---------------------- -At some point after acceptance of the PEP, I will likely submit the +At some point after acceptance of the PEP, we may submit the following MIME type registration requests to IANA: * Full metadata: ``application/vnd.python.pydist+json`` @@ -2146,7 +2135,7 @@ entry would be added to the dependency set. Alternative dependency specification example:: ["Pillow", "PIL"] - ["mysql", "psycopg2 (>= 4)", "sqlite3"] + ["mysql", "psycopg2 >= 4", "sqlite3"] However, neither of the given examples is particularly compelling, since Pillow/PIL style forks aren't common, and the database driver use @@ -2187,64 +2176,6 @@ is rejected unless someone can present a compelling use case (and even then the idea won't be reconsidered until metadata 2.1 at the earliest). -A hook to run tests against installed distributions ---------------------------------------------------- - -Earlier drafts of this PEP defined a hook for running automated -tests against an *installed* distribution. This isn't actually what you -generally want - you want the ability to test a *built* distribution, -potentially relying on files which won't be included in the binary archives. - -RPM's "check" step also runs between the build step and the install step, -rather than after the install step. - -Accordingly, the ``test_installed_dist`` hook has been removed, and the -``test_built_dist`` metabuild hook has been tentatively defined. However, -along with the rest of the metabuild hooks, further consideration has been -deferred until metadata 2.1 at the earliest. - - -Extensible signatures for the install hooks -------------------------------------------- - -The install hooks have been deliberately designed to NOT accept arbitary -keyword arguments that the hook implementation is then expected to ignore. - -The argument in favour of that API design technique is to allow the addition -of new optional arguments in the future, without requiring the definition -of a new install hook, or migration to version 3.0 of the metadata -specification. It is a technique very commonly seen in function wrappers -which merely pass arguments along to the inner function rather than -processing them directly. - -However, the install hooks are already designed to have access to the full -metadata for the distribution (including all metadata extensions and -the previous/next version when appropriate), as well as to the full target -deployment environment. - -This means there are two candidates for additional information that -could be passed as arbitrary keyword arguments: - -* installer dependent settings -* user provided installation options - -The first of those runs explicitly counter to one of the core goals of the -metadata 2.0 specification: decoupling the software developer's choice of -development and publication tools from the software integrator's choice of -integration and deployment tools. - -The second is a complex problem that has a readily available workaround in -the form of operating system level environment variables (this is also -one way to interoperate with platform specific installation tools). - -Alternatively, installer developers may either implicitly inject an -additional metadata extension when invoking the install hook, or else -define an alternate hook signature as a distinct metadata extension to be -provided by the distribution. Either of these approaches makes the -reliance on installer-dependent behaviour suitably explicit in either -the install hook implementation or the distribution metadata. - - References ========== diff --git a/pep-0459.txt b/pep-0459.txt index eae584ca7..414409814 100644 --- a/pep-0459.txt +++ b/pep-0459.txt @@ -45,21 +45,22 @@ The currently defined standard extensions are: * ``python.details`` * ``python.project`` * ``python.integrator`` -* ``python.commands`` * ``python.exports`` +* ``python.commands`` +* ``python.constraints`` All standard extensions are currently at version ``1.0``, and thus the ``extension_metadata`` field may be omitted without losing access to any functionality. -The ``details`` extension -========================= +The ``python.details`` extension +================================ -The ``details`` extension allows for more information to be provided +The ``python.details`` extension allows for more information to be provided regarding the software distribution. -The ``details`` extension contains three subfields: +The ``python.details`` extension contains four custom subfields: * ``license``: the copyright license for the distribution * ``keywords``: package index keywords for the distribution @@ -169,13 +170,13 @@ Example:: } -The ``project`` extension -========================= +The ``python.project`` extension +================================ -The ``project`` extension allows for more information to be provided +The ``python.project`` extension allows for more information to be provided regarding the creation and maintenance of the distribution. -The ``project`` extension contains three subfields: +The ``python.project`` extension contains three custom subfields: * ``contacts``: key contact points for the distribution * ``contributors``: other contributors to the distribution @@ -290,10 +291,11 @@ Example:: } -The ``integrator`` extension -============================ +The ``python.integrator`` extension +=================================== -Structurally, this extension is identical to the ``project`` extension. +Structurally, this extension is largely identical to the ``python.project`` +extension (the extension name is the only difference). However, where the ``project`` metadata refers to the upstream creators of the software, the ``integrator`` metadata refers to the downstream @@ -303,20 +305,20 @@ If the software is being redistributed unmodified, then typically this extension will not be used. However, if the software has been patched (for example, backporting compatible fixes from a later version, or addressing a platform compatibility issue), then this extension SHOULD be used, and -an integrator suffix added to the package version number. +a local version label added to the distribution's version identifier. If there are multiple redistributors in the chain, each one just overwrites this extension with their particular metadata. -The ``exports`` extension -========================= +The ``python.exports`` extension +================================ Most Python distributions expose packages and modules for import through the Python module namespace. Distributions may also expose other interfaces when installed. -The ``exports`` extension contains three subfields: +The ``python.exports`` extension contains three custom subfields: * ``modules``: modules exported by the distribution * ``namespaces``: namespace packages that the distribution contributes to @@ -464,10 +466,10 @@ Example:: } -The ``commands`` extension -========================== +The ``python.commands`` extension +================================= -The ``commands`` extension contains three subfields: +The ``python.commands`` extension contains three custom subfields: * ``wrap_console``: console wrapper scripts to be generated by the installer * ``wrap_gui``: GUI wrapper scripts to be generated by the installer @@ -495,6 +497,8 @@ a wheel file or following installation. They are provided for informational purpose only - installing them is handled through the normal processes for files created when building a distribution. +Build tools SHOULD mark this extension as requiring handling by installers. + Index servers SHOULD allow multiple distributions to publish the same commands, but MAY notify distribution authors of potential conflicts. @@ -504,13 +508,163 @@ previously installed, distribution. Example:: - "commands": { + "python.commands": { + "installer_must_handle": true, "wrap_console": [{"chair": "chair:run_cli"}], "wrap_gui": [{"chair-gui": "chair:run_gui"}], "prebuilt": ["reduniforms"] } +The ``python.constraints`` extension +==================================== + +The ``python.constraints`` extension contains two custom subfields: + +* ``environments``: supported installation environments +* ``extension_metadata``: required exact matches in extension metadata + fields published by other installed components + +Build tools SHOULD mark this extension as requiring handling by installers. + +Index servers SHOULD allow distributions to be uploaded with constraints +that cannot be satisfied using that index, but MAY notify distribution +authors of any such potential compatibility issues. + +Installation tools SHOULD report an error if constraints are specified by +the distribution and the target installation environment fails to satisfy +them, MUST at least emit a warning, and MAY allow the user to +force the installation to proceed regardless. + +Example:: + + "python.constraints": { + "installer_must_handle": true, + "environments": ["python_version >= 2.6"], + "extension_metadata": { + "fortranlib": { + "fortranlib.compatibility": { + "fortran_abi": "openblas-g77" + } + } + } + } + + +Supported Environments +---------------------- + +The ``environments`` subfield is a list of strings specifying the +environments that the distribution explicitly supports. An environment is +considered supported if it matches at least one of the environment markers +given. + +If this field is not given in the metadata, it is assumed that the +distribution supports any platform supported by Python. + +Individual entries are environment markers, as described in :pep:`426`. + +The two main uses of this field are to declare which versions of Python +and which underlying operating systems are supported. + +Examples indicating supported Python versions:: + + # Supports Python 2.6+ + "environments": ["python_version >= '2.6'"] + + # Supports Python 2.6+ (for 2.x) or 3.3+ (for 3.x) + "environments": ["python_version >= '3.3'", + "'3.0' > python_version >= '2.6'"] + +Examples indicating supported operating systems:: + + # Windows only + "environments": ["sys_platform == 'win32'"] + + # Anything except Windows + "environments": ["sys_platform != 'win32'"] + + # Linux or BSD only + "environments": ["'linux' in sys_platform", + "'bsd' in sys_platform"] + +Example where the supported Python version varies by platform:: + + # The standard library's os module has long supported atomic renaming + # on POSIX systems, but only gained atomic renaming on Windows in Python + # 3.3. A distribution that needs atomic renaming support for reliable + # operation might declare the following supported environments. + "environment": ["python_version >= '2.6' and sys_platform != 'win32'", + "python_version >= '3.3' and sys_platform == 'win32'"] + + +Extension metadata constraints +------------------------------ + +The ``extension_metadata`` subfield is a mapping from distribution names +to extension metadata snippets that are expected to exactly match the +metadata of the named distribution in the target installation environment. + +Each submapping then consists of a mapping from metadata extension names to +the exact expected values of a subset of fields. + +For example, a distribution called ``fortranlib`` may publish a different +FORTRAN ABI depending on how it is built, and any related projects that are +installed into the same runtime environment should use matching build +options. This can be handled by having the base distribution publish a +custom extension that indicates the build option that was used to create +the binary extensions:: + + "extensions": { + "fortranlib.compatibility": { + "fortran_abi": "openblas-g77" + } + } + +Other distributions that contain binary extensions that need to be compatible +with the base distribution would then define a suitable constraint in their +own metadata:: + + "python.constraints": { + "installer_must_handle": true, + "extension_metadata": { + "fortranlib": { + "fortranlib.compatibility": { + "fortran_abi": "openblas-g77" + } + } + } + } + +This constraint specifies that: + +* ``fortranlib`` must be installed (this should also be expressed as a + normal dependency so that installers ensure it is satisfied) +* The installed version of ``fortranlib`` must include the custom + ``fortranlib.compatibility`` extension in its published metadata +* The ``fortan_abi`` subfield of that extension must have the *exact* + value ``openblas-g77``. + +If all of these conditions are met (the distribution is installed, the +specified extension is included in the metadata, the specified subfields +have the exact specified value), then the constraint is considered to be +satisfied. + +.. note:: + + The primary intended use case here is allowing C extensions with additional + ABI compatibility requirements to declare those in a way that any + installation tool can enforce without needing to understand the details. + In particular, many NumPy based scientific libraries need to be built + using a consistent set of FORTRAN libraries, hence the "fortranlib" + example. + + This is the reason there's no support for pattern matching or boolean + logic: even the "simple" version of this extension is relatively + complex, and there's currently no compelling rationale for making it + more complicated than it already is. + + Copyright =========