python-peps/pep-0700.rst

PEP: 700
Title: Additional Fields for the Simple API for Package Indexes
Author: Paul Moore <p.f.moore@gmail.com>
PEP-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177
Status: Draft
Type: Standards Track
Topic: Packaging
Content-Type: text/x-rst
Created: 21-Oct-2022
Post-History: `21-Oct-2022 <https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177>`__


Abstract
========

:pep:`691` defined a JSON form for the "Simple Repository API". This allowed
clients to more easily query the data that was previously only available in
HTML, as defined in :pep:`503`.

This proposal adds three fields to the JSON form, which allow it to be used in
place of PyPI's `JSON API <https://warehouse.pypa.io/api-reference/json.html>`__
in a number of situations.

- A field to allow retrieval of a list of all the published versions of a project.
- Fields containing the size and upload time for a project file.

The new fields are all part of the data returned from the "project details" URL.


Rationale
=========

With the introduction of the JSON form of the simple API in :pep:`691`, the
simple API offers functionality that is almost as complete as the PyPI JSON API.
This PEP adds a number of fields which were previously only available through
the JSON API, in order to allow more clients which were previously Warehouse
specific to support arbitrary standards-compliant indexes.


Specification
=============

This specification defines version 1.1 of the simple repository API. For the
HTML version of the API, there is no change from version 1.0. For the JSON
version of the API, the following changes are made:

- The ``api-version`` must specify version 1.1 or later.
- A new ``versions`` key is added at the top level.
- Two new "file information" keys, ``size`` and ``upload-time``, are added to
  the ``files`` data.
- Keys (at any level) with a leading underscore are reserved as private for
  index server use. No future standard will assign a meaning to any such key.

The ``versions`` and ``size`` keys are mandatory. The ``upload-time`` key is
optional.

Versions
--------

An additional key, ``versions`` MUST be present at the top level, in addition to
the keys ``name``, ``files`` and ``meta`` defined in :pep:`691`. This key MUST
contain a list of version strings specifying all of the project versions uploaded
for this project. The value is logically a set, and as such may not contain
duplicates, and the order of the values is not significant.

All of the files listed in the ``files`` key MUST be associated with one of the
versions in the ``versions`` key. The ``versions`` key MAY contain versions with
no associated files (to represent versions with no files uploaded, if the server
has such a concept).

Note that because servers may hold "legacy" data from before the adoption of
:pep:`440`, version strings currently cannot be required to be valid :pep:`440`
versions, and therefore cannot be assumed to be orderable using the :pep:`440`
rules. However, servers SHOULD use normalised :pep:`440` versions where
possible.


Additional file information
---------------------------

Two new keys are added to the ``files`` key.

- ``size``: This field is mandatory. It MUST contain an integer which is the
  file size in bytes.
- ``upload-time``: This field is optional. If present, it MUST contain a valid
  ISO 8601 date/time string, in the format ``yyyy-mm-ddThh:mm:ss.ffffffZ``,
  which represents the time the file was uploaded to the index. As indicated by
  the ``Z`` suffix, the upload time MUST use the UTC timezone. The fractional
  seconds part of the timestamp (the ``.ffffff`` part) is optional, and if
  present may contain up to 6 digits of precision. If a server does not record
  upload time information for a file, it MAY omit the ``upload-time`` key.


FAQ
===

Why not add this data to the HTML API as well?
----------------------------------------------

It would be possible to add the data to the HTML API, but the vast majority of
consumers for this data are likely to be currently getting it from the PyPI JSON
API, and so will already be expecting to parse JSON. Traditional consumers of
the HTML API have never needed this data previously.

Does this imply that the HTML API is obsolete?
----------------------------------------------

No. The FAQ of :pep:`691` was clear that the HTML API is not being deprecated,
and this PEP does not change that position. However, clients wishing to access
the new data introduced by this PEP will need to use the JSON API to get it. And
indexes wanting to provide it will need to serve the JSON format.

Is the simple API replacing the Warehouse JSON and XML-RPC APIs?
----------------------------------------------------------------

Where possible, clients should prefer the simple API over the JSON or XML-RPC
APIs, as the former is standardised and can be assumed to be available from any
index, whereas the latter are exclusive to the Warehouse project.

However, while this PEP brings the simple API closer to being able to replace
the JSON API, there is no formal policy that the simple API will replicate all
of the functionality covered by the existing Warehouse APIs. Proposed additions
to the simple API will still be considered on their individual merits, and the
requirement that the API should be simple and fast for the primary use case of
locating files for a project will remain the overriding consideration.

Why not allow other date formats?
---------------------------------

The ISO 8601 standard is complex, and there seems little value in requiring
clients to deal with that. The standard library ``datetime`` module provides
methods to parse ISO 8601 strings, but it is possible that users may want to
access index data *without* using Python (for example, piping the output of
``curl`` into ``jq``). Having a single, well-defined format makes this easier,
and doesn't have any significant disadvantages.

What if file sizes are too big for a JSON number?
-------------------------------------------------

The JSON standard does not specify how numbers are to be interpreted. Python can
read and write arbitrary-length integers in a JSON file, so this should not be
an issue for code written in Python. Non-Python implementations may need to take
care to handle large integers correctly, but this is not expected to be a
significant problem.

Why not require PEP 440 versions?
---------------------------------

At the time this PEP was written, PyPI still contains (and serves) projects and
files with "legacy" versions. Requiring :pep:`440` versions would make it
impossible for PyPI to follow this specification while still serving the
existing content.

Ideally, at some future time, the simple index API will be updated to require
:pep:`440` versions, at which time this specification should be updated to
reflect that. However, that change will need to be co-ordinated with existing
index providers including PyPI, to desupport and remove non-conforming projects
and/or files.

Why not provide a "latest version" value?
-----------------------------------------

For :pep:`440` versions, this is easy enough for the client to do (using the
``packaging`` library, ``latest = max(Version(s) for s in proj["versions"])``).
For non-standard versions, there is no well-defined ordering, and clients will
need to decide on what rule is appropriate for their needs. Requiring the server
to supply a latest version value takes the choice away from the client.

Servers with an explicit concept of which version is the "latest", which cannot
be calculated from data available to the client, can provide a non-standard,
underscore-prefixed key to convey that information to the client if they wish.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.