PEP 643: Further updates (#1713)

pep-0643.rst

@@ -7,7 +7,7 @@ Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 24-Oct-2020
-Post-History: 24-Oct-2020, 01-Nov-2020, 02-Nov-2020
+Post-History: 24-Oct-2020, 01-Nov-2020, 02-Nov-2020, 14-Nov-2020
 
 
 Abstract
@@ -17,7 +17,7 @@ Python package metadata is stored in the distribution file in a standard
 format, defined in the `Core Metadata Specification`_. However, for
 source distributions, while the format of the data is defined, there has
 traditionally been a lot of inconsistency in what data is recorded in
-the sdist. See `here
+the source distribution. See `here
 <https://discuss.python.org/t/why-isnt-source-distribution-metadata-trustworthy-can-we-make-it-so/2620>`_
 for a discussion of this issue.
 
@@ -28,9 +28,7 @@ mechanisms to extract medatata.
 This PEP defines a standard that allows build backends to reliably store
 package metadata in the source distribution, while still retaining the
 necessary flexibility to handle metadata fields that have to be calculated
-at build time. It further defines a set of metadata values that must be
-fixed when the sdist is created, ensuring that consumers have a minimum
-"core" of metadata they can be sure is available.
+at build time.
 
 
 Motivation
@@ -51,84 +49,87 @@ stored in source distributions:
 
 This PEP proposes an update to the metadata specification to allow
 recording of fields which are expected to be "filled in later", and
-updates the sdist specification to clarify that backends should record
-sdist metadata using that version of the spec (or later). It restricts
-which fields can be "filled in later", so that a core set of metadata is
-available, and reliable, when read from a sdist.
+updates the source distribution specification to clarify that backends
+should record sdist metadata using that version of the spec (or later).
 
 
 Rationale
 =========
 
-:pep:`621` proposes a mechanism for users to specify metadata in
-``pyproject.toml``. As part of that mechanism, a way was needed to say
-that a particular field is defined dynamically by the backend. During
-discussions on the PEP, it became clear that the same type of mechanism
-would address the issue of distinguishing between "not known yet" and
-"definitely has no value" in sdists. This PEP defines the ``Dynamic``
-metadata field by analogy with the ``dynamic`` field in :pep:`621`.
+This PEP allows projects to define source distribution metadata values
+as being "dynamic". In this context, saying that a field is "dynamic"
+means that the value has not been fixed at the time that the source
+distribution was generated. Dynamic values will be supplied by the build
+backend at the time when the wheel is generated, and could depend on
+details of the build environment.
+
+:pep:`621` has a similar concept, of "dynamic" values that will be
+"filled in later", and so we choose to use the same term here by
+analogy.
 
 
 Specification
 =============
 
-This PEP defines the relationship between metadata values specified in
-a sdist, and the corresponding values in wheels built from that sdist.
-It requires build backends to clearly mark any fields which will *not*
-simply be copied unchanged from the sdist to the wheel.
+This PEP defines the relationship between metadata values specified in a
+source distribution, and the corresponding values in wheels built from
+it. It requires build backends to clearly mark any fields which will
+*not* simply be copied unchanged from the sdist to the wheel.
 
 In addition, this PEP makes the `PyPA Specifications`_ document the
-canonical location for the specification of the sdist format (collecting
-the information in :pep:`517` and in this PEP).
+canonical location for the specification of the source distribution
+format (collecting the information in :pep:`517` and in this PEP).
 
 A new field, ``Dynamic``, will be added to the `Core Metadata Specification`_.
 This field will be multiple use, and will be allowed to contain the name
-of another core metadata field. The ``Dynamic`` metadata item is only
-allowed in source distribution metadata.
+of another core metadata field.
 
-If tools that read metadata encounter the ``Dynamic`` field anywhere except
-in a sdist, they MAY fail with an error. If they do not fail, they MUST
-ignore the field, and may report a warning.
+When found in the metadata of a source distribution, the following
+rules apply:
 
-If a field is marked as ``Dynamic``, there is no restriction placed on
-its value in a wheel built from the sdist. A field which is marked as
-``Dynamic``, MUST NOT have an explicit value in the sdist.
+1. If a field is *not* marked as ``Dynamic``, then the value of the field
+   in any wheel built from the sdist MUST match the value in the sdist.
+   If the field is not in the sdist, and not marked as ``Dynamic``, then
+   it MUST NOT be present in the wheel.
+2. If a field is marked as ``Dynamic``, it may contain any valid value in
+   a wheel built from the sdist (including not being present at all).
+3. Backends MUST NOT mark a field as ``Dynamic`` if they can determine that
+   it was generated from data that will not change at build time. 
 
-If a field is *not* marked as ``Dynamic``, then the value of the field
-in any wheel built from the sdist MUST match the value in the sdist.
-If the field is not in the sdist, and not marked as ``Dynamic``, then it
-MUST NOT be present in the wheel.
+Backends MAY record the value they calculated for a field they mark as
+``Dynamic`` in a source distribution. Consumers, however, MUST NOT treat
+this value as canonical, but MAY use it as an hint about what the final
+value in a wheel could be.
 
-Build backends MUST ensure that these rules are followed, and MUST
-report an error if they are unable to do so.
+In any context other than a source distribution, if a field is marked as
+``Dynamic``, that indicates that the value was generated at wheel build
+time and may not match the value in the sdist (or in other builds of the
+project). Backends are not required to record this information, though,
+and consumers MUST NOT assume that the lack of a ``Dynamic`` marking has
+any significance, except in a source distribution.
 
-The following fields are the only ones allowed to be marked as ``Dynamic``:
-
-* ``Platform``
-* ``Supported-Platform``
-* ``Requires-Dist``
-* ``Requires-External``
-* ``Provides-Extra``
-* ``Provides-Dist``
-* ``Obsoletes-Dist``
+The fields ``Name`` and ``Version`` MUST NOT be marked as ``Dynamic``.
 
 As it adds a new metadata field, this PEP updates the core metadata
 format to version 2.2.
 
-Source distributions MUST use the latest version of the core metadata
-specification (which will be version 2.2 or later).
+Source distributions SHOULD use the latest version of the core metadata
+specification that was available when they were created.
 
-Build backends SHOULD encourage projects to avoid using ``Dynamic``,
-preferring to use environment markers on static values to adapt to
+Build backends are strongly encouraged to only mark fields as
+``Dynamic`` when absolutely necessary, and to encourage projects to
+avoid backend features that require the use of ``Dynamic``. Projects
+should prefer to use environment markers on static values to adapt to
 details of the install location.
 
+
 Backwards Compatibility
 =======================
 
 As this proposal increments the core metadata version, it is compatible
-with existing sdists, which will use an older metadata version. Tools
-can determine whether a sdist conforms to this PEP by checking the
-metadata version.
+with existing source distributions, which will use an older metadata
+version. Tools can determine whether a source distribution conforms to
+this PEP by checking the metadata version.
 
 
 Security Implications
@@ -159,6 +160,10 @@ Rejected Ideas
    so the PEP chooses to make dynamic fields the exception, and require
    backends to "opt in" to making a field dynamic.
 
+   In addition, if dynamic is the default, then in future, as more
+   and more metadata becomes static, metadata files will include an
+   increasing number of ``Static`` declarations.
+
 2. Rather than having a ``Dynamic`` field, add a special value that
    indicates that a field is "not yet defined".
 
@@ -169,26 +174,29 @@ Rejected Ideas
    does not seem to be enough benefit to this approach to make it worth
    using instead of the proposed mechanism.
 
-3. Allow ``Requires-Python`` to be ``Dynamic``, as it cannot include environment
-   markers to tailor the requirement to the target environment.
+3. Allow ``Requires-Python`` to be ``Dynamic``, as it cannot include
+   environment markers to tailor the requirement to the target
+   environment.
 
-   Currently, no projects on PyPI have a ``Requires-Python`` value that varies
-   between different wheels for the same version, so there is no practical
-   need for this flexibility at present. If a genuine use case is identified
-   later, the specification can be changed to allow ``Rquires-Python`` to be
-   dynamic at that time.
+   Currently, no projects on PyPI have a ``Requires-Python`` value that
+   varies between different wheels for the same version, so there is no
+   practical need for this flexibility at present. If a genuine use case
+   is identified later, the specification can be changed to allow
+   ``Requires-Python`` to be dynamic at that time.
 
-4. Allow ``Dynamic`` to be used in wheels and/or installed distributions.
+   In fact, this became irrelevant during subsequent discussions, when
+   the explicit whitelist of fields allowed to be ``Dynamic`` was
+   removed.
 
-   There is no obvious value to allowing this, and it seems like it is simply
-   adding complexity for no real reason. Allowing this could be done in a
-   follow-up proposal if there turned out to be sufficient benefit.
+4. Restrict the use of ``Dynamic`` to a minimal "white list" of
+   permitted fields.
 
-5. Allow a field to be marked as ``Dynamic``, but *also* have a value in the
-   sdist metadata.
+   This approach was likely to prove extremely difficult for setuptools
+   to implement in a backward compatible way, due to the dynamic nature
+   of the setuptools interface. Instead, the proposal now allows most
+   fields to be dynamic, but encourages backends to avoid dynamic values
+   unless essential.
 
-   There appears to be no use case for allowing this. If a use case is
-   identified in the future, the specification can be updated at that time.
 
 Open Issues
 ===========