PEP 740: Index support for digital attestations (#3618)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Facundo Tuesca <facundo.tuesca@trailofbits.com>
This commit is contained in:
William Woodruff
GitHub
parent
9d82dd1cd8
commit
a85fa9110d
|
|
@ -616,6 +616,7 @@ peps/pep-0735.rst @brettcannon
|
||||||
peps/pep-0736.rst @gvanrossum @Rosuav
|
peps/pep-0736.rst @gvanrossum @Rosuav
|
||||||
peps/pep-0737.rst @vstinner
|
peps/pep-0737.rst @vstinner
|
||||||
peps/pep-0738.rst @encukou
|
peps/pep-0738.rst @encukou
|
||||||
|
peps/pep-0740.rst @dstufft
|
||||||
# ...
|
# ...
|
||||||
# peps/pep-0754.rst
|
# peps/pep-0754.rst
|
||||||
# ...
|
# ...
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,370 @@
|
||||||
|
PEP: 740
|
||||||
|
Title: Index support for digital attestations
|
||||||
|
Author: William Woodruff <william@yossarian.net>,
|
||||||
|
Facundo Tuesca <facundo.tuesca@trailofbits.com>
|
||||||
|
Sponsor: Donald Stufft <donald@stufft.io>
|
||||||
|
PEP-Delegate: Donald Stufft <donald@stufft.io>
|
||||||
|
Discussions-To: https://discuss.python.org/t/pre-pep-exposing-trusted-publisher-provenance-on-pypi/42337
|
||||||
|
Status: Draft
|
||||||
|
Type: Informational
|
||||||
|
Topic: Packaging
|
||||||
|
Created: 08-Jan-2024
|
||||||
|
|
||||||
|
Abstract
|
||||||
|
========
|
||||||
|
|
||||||
|
This PEP proposes a collection of changes related to the upload and distribution
|
||||||
|
of digitally signed attestations and metadata used to verify them on a Python
|
||||||
|
package repository, such as PyPI.
|
||||||
|
|
||||||
|
These changes have two subcomponents:
|
||||||
|
|
||||||
|
* Changes to the currently unstandardized PyPI upload API, allowing clients
|
||||||
|
to upload digital attestations;
|
||||||
|
* Changes to the :pep:`503` and :pep:`691` "simple" APIs, allowing clients
|
||||||
|
to retrieve both digital attestations and
|
||||||
|
`Trusted Publishing <https://docs.pypi.org/trusted-publishers/>`_ metadata
|
||||||
|
for individual release files.
|
||||||
|
|
||||||
|
This PEP does not recommend a specific digital attestation format, nor does
|
||||||
|
it make a policy recommendation around mandatory digital attestations on
|
||||||
|
release uploads or their subsequent verification by installing clients like
|
||||||
|
``pip``.
|
||||||
|
|
||||||
|
Rationale
|
||||||
|
=========
|
||||||
|
|
||||||
|
Desire for digital signatures on Python packages has been repeatedly
|
||||||
|
expressed by both package maintainers and downstream users:
|
||||||
|
|
||||||
|
* Maintainers wish to demonstrate the integrity and authenticity of their
|
||||||
|
package uploads;
|
||||||
|
* Individual downstream users wish to verify package integrity and authenticity
|
||||||
|
without placing additional trust in their index's honesty;
|
||||||
|
* "Bulk" downstream users (such as Operating System distributions) wish to
|
||||||
|
perform similar verifications and potentially re-expose or countersign
|
||||||
|
for their own downstream packaging ecosystems.
|
||||||
|
|
||||||
|
This proposal seeks to accommodate each of the above use cases.
|
||||||
|
|
||||||
|
While this PEP does not recommend a specific digital attestation format,
|
||||||
|
it does recognize the utility of Trusted Publishing as a pre-existing,
|
||||||
|
"zero-configuration" source of strong provenance for Python packages.
|
||||||
|
Consequently this PEP includes a proposed scheme for exposing each release
|
||||||
|
file's Trusted Publisher metadata, with the expectation that a future digital
|
||||||
|
attestation format will likely make use of it.
|
||||||
|
|
||||||
|
Design Considerations
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
This PEP identifies the following design considerations when evaluating
|
||||||
|
both its own proposed changes and previous work in the same or adjacent
|
||||||
|
areas of Python packaging:
|
||||||
|
|
||||||
|
1. Index accessibility: digital attestations for Python packages
|
||||||
|
are ideally retrievable directly from the index itself, as "detached"
|
||||||
|
resources.
|
||||||
|
|
||||||
|
This both simplifies some compatibility concerns (by avoiding
|
||||||
|
the need to modify the distribution formats themselves) and also simplifies
|
||||||
|
the behavior of potential installing clients (by allowing them to
|
||||||
|
retrieve each attestation before its corresponding package without needing
|
||||||
|
to do streaming decompression).
|
||||||
|
|
||||||
|
2. Verification by the index itself: in addition to enabling verification
|
||||||
|
by installing clients, each digital attestation is *ideally* verifiable
|
||||||
|
in some form by the index itself.
|
||||||
|
|
||||||
|
This both increases the overall quality
|
||||||
|
of attestations uploaded to the index (preventing, for example, users
|
||||||
|
from accidentally uploading incorrect or invalid attestations) and also
|
||||||
|
enables UI and UX refinements on the index itself (such as a "provenance"
|
||||||
|
view for each uploaded package).
|
||||||
|
|
||||||
|
3. General applicability: digital attestations should be applicable to
|
||||||
|
*any and every* package uploaded to the index, regardless of its format
|
||||||
|
(sdist or wheel) or interior contents.
|
||||||
|
|
||||||
|
4. Metadata support: this PEP refers to "digital attestations" rather than
|
||||||
|
just "digital signatures" to emphasize the ideal presence of additional
|
||||||
|
metadata within the cryptographic envelope.
|
||||||
|
|
||||||
|
For example, to prevent domain separation between a distribution's name and
|
||||||
|
its contents, the digital attestation could be performed over
|
||||||
|
``HASH(name || HASH(contents))`` rather than just ``HASH(contents)``.
|
||||||
|
|
||||||
|
5. Consistent release attestations: if a file belonging to a release has a
|
||||||
|
set of digital attestations, then all of the other files belonging to that
|
||||||
|
release should also have the same types of attestations.
|
||||||
|
|
||||||
|
This simplifies the downstream use story for digital attestations, and
|
||||||
|
prevents potentially vulnerable "swiss cheese" release patterns (where
|
||||||
|
a verifier checks for a valid attestation on ``HolyGrail-1.0.tar.gz``
|
||||||
|
but their installing client actually resolves an attacker-controlled,
|
||||||
|
platform-specific ``.whl`` instead).
|
||||||
|
|
||||||
|
|
||||||
|
Previous Work
|
||||||
|
-------------
|
||||||
|
|
||||||
|
PGP signatures
|
||||||
|
^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
PyPI and other indices have historically supported PGP signatures on uploaded
|
||||||
|
distributions. These could be supplied during upload, and could be retrieved
|
||||||
|
by installing clients via the ``data-gpg-sig`` attribute in the :pep:`503`
|
||||||
|
API, the ``gpg-sig`` key on the :pep:`691` API, or via an adjacent
|
||||||
|
``.asc``-suffixed URL.
|
||||||
|
|
||||||
|
PGP signature uploads have been disabled on PyPI since
|
||||||
|
`May 2023 <https://blog.pypi.org/posts/2023-05-23-removing-pgp/>`_, after
|
||||||
|
`an investigation <https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI-worse-than-useless>`_
|
||||||
|
determined that the majority of signatures (which, themselves, constituted a
|
||||||
|
tiny percentage of overall uploads) could not be associated with a public key or
|
||||||
|
otherwise meaningfully verified.
|
||||||
|
|
||||||
|
In their previously supported form on PyPI, PGP signatures satisfied
|
||||||
|
considerations (1) and (3) above but not (2) (owing to the need for external
|
||||||
|
keyservers and key distribution) or (4) (due to PGP signatures typically being
|
||||||
|
constructed over just an input file, without any associated signed metadata).
|
||||||
|
Similarly, PyPI's historical implementation of PGP did not satisfy consideration
|
||||||
|
(5), due to a lack of consistency checks between different release files
|
||||||
|
(and an inability to perform those checks due to no access to the signer's
|
||||||
|
public key).
|
||||||
|
|
||||||
|
Wheel signatures
|
||||||
|
^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
:pep:`427` (and its :ref:`living PyPA counterpart <packaging:binary-distribution-format>`)
|
||||||
|
specify the :term:`wheel format <packaging:Wheel>`.
|
||||||
|
|
||||||
|
This format includes accommodations for digital signatures embedded directly
|
||||||
|
into the wheel, in either JWS or S/MIME format. These signatures are specified
|
||||||
|
over a :pep:`376` RECORD, which is modified to include a cryptographic digest
|
||||||
|
for each recorded file in the wheel.
|
||||||
|
|
||||||
|
While wheel signatures are fully specified, they do not appear to be broadly
|
||||||
|
used; the official `wheel tooling <https://github.com/pypa/wheel>`_ deprecated
|
||||||
|
signature generation and verification support
|
||||||
|
`in 0.32.0 <https://wheel.readthedocs.io/en/stable/news.html>`_, which was
|
||||||
|
released in 2018.
|
||||||
|
|
||||||
|
Additionally, wheel signatures do not satisfy any of
|
||||||
|
the above considerations (due to the "attached" nature of the signatures,
|
||||||
|
non-verifiability on the index itself, and support for wheels only).
|
||||||
|
|
||||||
|
Specification
|
||||||
|
=============
|
||||||
|
|
||||||
|
.. _upload-endpoint:
|
||||||
|
|
||||||
|
Upload endpoint changes
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The current upload API is not standardized. However, we propose the following
|
||||||
|
changes to it:
|
||||||
|
|
||||||
|
* In addition to the current top-level ``content`` and ``gpg_signature`` fields,
|
||||||
|
the index **SHALL** accept ``attestations`` as an additional multipart form
|
||||||
|
field.
|
||||||
|
* The new ``attestations`` field **SHALL** be a JSON object.
|
||||||
|
* The JSON object **SHALL** have one or more keys, each identifying an
|
||||||
|
attestation format known to the index. If any key does not identify an
|
||||||
|
attestation format known to the index, the index **MUST** reject the upload.
|
||||||
|
* The value associated with each well-known key **SHALL** be a JSON object.
|
||||||
|
* Each attestation value **MUST** be verifiable by the index. If the index fails
|
||||||
|
to verify any attestation in ``attestations``, it **MUST** reject the upload.
|
||||||
|
|
||||||
|
In addition to the above, the index **SHALL** enforce a consistency
|
||||||
|
policy for release attestations via the following:
|
||||||
|
|
||||||
|
* If the first file under a new release is supplied with ``attestations``,
|
||||||
|
then all subsequently uploaded files under the same release **MUST** also
|
||||||
|
have ``attestations``. Conversely, if the first file under a new release
|
||||||
|
does not have any ``attestations``, then all subsequent uploads under the
|
||||||
|
same release **MUST NOT** have ``attestations``.
|
||||||
|
* All files under the same release **MUST** have the same set of well-known
|
||||||
|
attestation format keys.
|
||||||
|
|
||||||
|
The index **MUST** reject any file upload that does not satisfy these
|
||||||
|
consistency properties.
|
||||||
|
|
||||||
|
Index changes
|
||||||
|
-------------
|
||||||
|
|
||||||
|
.. _provenance-object:
|
||||||
|
|
||||||
|
Provenance objects
|
||||||
|
^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
The index will serve uploaded attestations along with metadata that can assist
|
||||||
|
in verifying them in the form of JSON serialized objects.
|
||||||
|
|
||||||
|
These "provenance objects" will be available via both the :pep:`503` Simple Index
|
||||||
|
and :pep:`691` JSON-based Simple API as described below, and will have the
|
||||||
|
following structure:
|
||||||
|
|
||||||
|
.. code-block:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"publisher": {
|
||||||
|
"type": "important-ci-service",
|
||||||
|
"claims": {},
|
||||||
|
"vendor-property": "foo",
|
||||||
|
"another-property": 123
|
||||||
|
},
|
||||||
|
"attestations": {
|
||||||
|
"some-attestation": {/* ... */},
|
||||||
|
"another-attestation": {/* ... */}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* ``publisher`` is an **optional** JSON object, containing a
|
||||||
|
representation of the file's Trusted Publisher configuration at the time
|
||||||
|
the file was uploaded to the package index. The keys within the ``publisher``
|
||||||
|
object are specific to each Trusted Publisher but include, at minimum:
|
||||||
|
|
||||||
|
* A ``type`` key, which **MUST** be a JSON string that uniquely identifies the
|
||||||
|
kind of Trusted Publisher.
|
||||||
|
* A ``claims`` key, which **MUST** be a JSON object containing any context-specific
|
||||||
|
claims retained by the index during Trusted Publisher authentication.
|
||||||
|
|
||||||
|
All other keys in the ``publisher`` object are publisher-specific. A full
|
||||||
|
illustrative example of a ``publisher`` object is provided in :ref:`appendix-2`.
|
||||||
|
* ``attestations`` is a **required** JSON object, containing one or
|
||||||
|
more attestation objects as identified by their keys. This object is
|
||||||
|
a superset of ``attestations`` object supplied by the uploader through the
|
||||||
|
``attestations`` field, as described in :ref:`upload-endpoint`.
|
||||||
|
|
||||||
|
Because ``attestations`` is a superset of the file's original uploaded attestations,
|
||||||
|
the index **MAY** chose to embed additional attestations of its own.
|
||||||
|
|
||||||
|
Simple Index
|
||||||
|
^^^^^^^^^^^^
|
||||||
|
|
||||||
|
* When an uploaded file has one or more attestations, the index **MAY** include a
|
||||||
|
``data-provenance`` attribute on its file link, with a value of either
|
||||||
|
``true`` or ``false``.
|
||||||
|
* When ``data-provenance`` is ``true``, the index **MUST** serve a
|
||||||
|
:ref:`provenance object <provenance-object>` at the same URL, but with
|
||||||
|
``.provenance`` appended to it. For example, if ``HolyGrail-1.0.tar.gz``
|
||||||
|
exists and has associated attestations, those attestations would be located
|
||||||
|
within the provenance object hosted at ``HolyGrail-1.0.tar.gz.provenance``.
|
||||||
|
|
||||||
|
JSON-based Simple API
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
* When an uploaded file has one or more attestations, the index **MAY** include a
|
||||||
|
``provenance`` object in the ``file`` dictionary for that file.
|
||||||
|
* ``provenance``, when present, **MUST** be a :ref:`provenance object <provenance-object>`.
|
||||||
|
|
||||||
|
Security Implications
|
||||||
|
=====================
|
||||||
|
|
||||||
|
This PEP is "mechanical" in nature; it provides only the plumbing for future
|
||||||
|
digital attestations on package indices, without specifying their concrete
|
||||||
|
cryptographic details.
|
||||||
|
|
||||||
|
As such, we do not identify any positive or negative security implications
|
||||||
|
for this PEP.
|
||||||
|
|
||||||
|
Index trust
|
||||||
|
-----------
|
||||||
|
|
||||||
|
This PEP does **not** increase (or decrease) trust in the index itself:
|
||||||
|
the index is still effectively trusted to honestly deliver unmodified package
|
||||||
|
distributions, since a dishonest index capable of modifying package
|
||||||
|
contents could also dishonestly modify or omit package attestations.
|
||||||
|
As a result, this PEP's presumption of index trust is equivalent to the
|
||||||
|
unstated presumption with earlier mechanisms, like PGP and Wheel signatures.
|
||||||
|
|
||||||
|
This PEP does not preclude or exclude future index trust mechanisms, such
|
||||||
|
as :pep:`458` and/or :pep:`480`.
|
||||||
|
|
||||||
|
Recommendations
|
||||||
|
===============
|
||||||
|
|
||||||
|
This PEP does not recommend specific attestation formats. It does,
|
||||||
|
however, make the following recommendations to package indices seeking
|
||||||
|
to create new or implement pre-existing attestation formats:
|
||||||
|
|
||||||
|
1. Consult the :ref:`living PyPA specifications <packaging:packaging-specifications>`
|
||||||
|
first, and determine if any currently defined attestation formats suit
|
||||||
|
your purpose.
|
||||||
|
2. If no suitable attestation format is defined under the PyPA specifications,
|
||||||
|
consider submitting it to the PyPA specifications for longevity and reuse
|
||||||
|
purposes.
|
||||||
|
|
||||||
|
When designing a new attestation format, we make the following recommendations:
|
||||||
|
|
||||||
|
1. Pick a short, but unique name for your attestation format; this name will
|
||||||
|
serve as the attestation's identifier in the upload and index APIs.
|
||||||
|
|
||||||
|
When appropriate for an attestation format, we recommend using ``:`` as a
|
||||||
|
domain separator. For example, an attestation format that provides publish
|
||||||
|
provenance using `Sigstore <https://www.sigstore.dev/>`_ might have the
|
||||||
|
name ``sigstore:publish``.
|
||||||
|
2. Prefer parsimony in your format: avoid optional fields and functionality,
|
||||||
|
avoid unnecessary cryptographic agility and message malleability, and ensure
|
||||||
|
that verifying the attestation communicates something meaningful beyond a
|
||||||
|
basic integrity check (since the index itself already supplies cryptographic
|
||||||
|
digests for this purpose).
|
||||||
|
|
||||||
|
.. _appendix-1:
|
||||||
|
|
||||||
|
Appendix 1: Example Uploaded Attestations
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
This appendix provides a fictional example of the ``attestations`` field
|
||||||
|
submitted on file upload, with two fictional attestations (``publish`` and
|
||||||
|
``timestamp``):
|
||||||
|
|
||||||
|
.. code-block:: json
|
||||||
|
|
||||||
|
{
|
||||||
|
"publish": {
|
||||||
|
"mediaType": "application/vnd.dev.sigstore.bundle+json;version=0.2",
|
||||||
|
"verificationMaterial": { /* omitted for brevity */ },
|
||||||
|
"messageSignature": {
|
||||||
|
"messageDigest": {
|
||||||
|
"algorithm": "some-hash-algo",
|
||||||
|
"digest": "digest-here"
|
||||||
|
},
|
||||||
|
"signature": "signature-here"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"timestamp": {
|
||||||
|
"cms": "some-long-blob-here"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
The payloads of these fictional attestations are purely illustrative.
|
||||||
|
|
||||||
|
.. _appendix-2:
|
||||||
|
|
||||||
|
Appendix 2: Example Trusted Publisher Representation
|
||||||
|
====================================================
|
||||||
|
|
||||||
|
This appendix provides a fictional example of a ``publisher`` key within
|
||||||
|
a :pep:`691` ``project.files[].provenance`` listing:
|
||||||
|
|
||||||
|
.. code-block:: json
|
||||||
|
|
||||||
|
"publisher": {
|
||||||
|
"type": "GitHub",
|
||||||
|
"claims": {
|
||||||
|
"ref": "refs/tags/v1.0.0",
|
||||||
|
"sha": "da39a3ee5e6b4b0d3255bfef95601890afd80709"
|
||||||
|
},
|
||||||
|
"repository_name": "HolyGrail",
|
||||||
|
"repository_owner": "octocat",
|
||||||
|
"repository_owner_id": "1",
|
||||||
|
"workflow_filename": "publish.yml",
|
||||||
|
"environment": null
|
||||||
|
}
|
||||||
|
|
||||||
|
Copyright
|
||||||
|
=========
|
||||||
|
|
||||||
|
This document is placed in the public domain or under the
|
||||||
|
CC0-1.0-Universal license, whichever is more permissive.
|
||||||
Loading…
Reference in New Issue