From 00a02152858da38fc85d4ceabafff671fc3782f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul?= Date: Thu, 16 Jan 2020 00:29:20 +0100 Subject: [PATCH] PEP 610: 2nd draft (#1276) --- pep-0610.rst | 159 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 100 insertions(+), 59 deletions(-) diff --git a/pep-0610.rst b/pep-0610.rst index bac5e3bf8..7eb1e89ba 100644 --- a/pep-0610.rst +++ b/pep-0610.rst @@ -1,6 +1,7 @@ PEP: 610 Title: Recording the Direct URL Origin of installed distributions -Author: Stéphane Bidoul , Chris Jerdonek +Author: Stéphane Bidoul , Chris Jerdonek +Sponsor: Nick Coghlan Discussions-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535 Status: Draft Type: Standards Track @@ -72,7 +73,8 @@ goal of this command is to help users generating a list of requirements that will later allow the re-installation the same environment with the highest possible fidelity. -The ``pip freeze`` command outputs a ``name==version`` line for each installed +As of pip version 19.3, the ``pip freeze`` command outputs a ``name==version`` +line for each installed distribution (except for editable installs). To achieve the goal of reinstalling the same environment, this requires the (name, version) tuple to refer to an immutable version of the @@ -115,7 +117,7 @@ For VCS URLs, there is additional origin information available only at install time useful for introspection and certain workflows. For example, when installing a revision from a VCS URL, a tool can determine if the revision corresponds to a branch, tag or (in the case of Git) a ref. This -information can be used when introspecting the database of installed packages +information can be used when introspecting the Database of Installed Distributions to communicate to users more information about what version was installed (e.g. whether a branch or tag was installed and, if so, the name of the branch or tag). This also permits one to know whether a PEP 440 direct @@ -147,8 +149,9 @@ about the purpose of the editable mode, and works only when the distribution can be installed with setuptools (i.e. it is not usable with other PEP 517 build backends). -For the sake of clarity, it is important to note that this PEP is otherwise -unrelated to editable installs. +When this PEP is implemented, it will not be necessary anymore to use +editable installs for the purpose of making pip freeze work correctly with +VCS references. Rationale ========= @@ -164,7 +167,7 @@ installed version that is already available for editable installs by virtue of a VCS checkout being present. Since at least three different ways already exist to encode this type of -information, this PEP uses a key-value format, so as not to make any +information, this PEP uses a dictionary format, so as not to make any assumption on how a direct URL reference must ultimately be encoded in a requirement or lockfile. See also the `Alternatives`_ section below for more discussion about this choice. @@ -182,17 +185,20 @@ This PEP specifies a ``direct_url.json`` file in the ``.dist-info`` directory of an installed distribution, to record the Direct URL Origin of the distribution. This file MUST be created by installers when installing a distribution -from a requirement specifying a direct URL reference (including a VCS URL) -in *non*-editable mode. +from a requirement specifying a direct URL reference (including a VCS URL). This file MUST NOT be created when installing a distribution from an other -type of requirement (i.e. name plus version specifier, or URL in editable mode). +type of requirement (i.e. name plus version specifier). -This JSON MUST be a flat dictionary where all keys and values are of string type. -For the sake of forward compatibility, tools SHOULD ignore values which are -not of string type. +This JSON file MUST be a dictionary, compliant with `RFC 8259`_ and UTF-8 +encoded. -If present, it MUST contain at least one field with name ``url``. +If present, it MUST contain at least two fields. The first one is ``url``, with +type ``string``. Depending on what ``url`` refers to, the second field MUST be +one of ``vcs_info`` (if ``url`` is a VCS reference), ``archive_info`` (if +``url`` is a source archives or a wheel), or ``dir_info`` (if ``url`` is a +local directory). These info fields have a (possibly empty) subdictionary as +value, with the possible keys defined below. ``url`` MUST be stripped of any sensitive authentication information, for security reasons. The user:password section of the URL MAY however @@ -201,46 +207,61 @@ expression:: \$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})? -When ``url`` refers to a VCS repository: +When ``url`` refers to a VCS repository, the ``vcs_info`` key MUST be present +as a dictionary with the following keys: -- A ``vcs`` field MUST be present, containing the name of the VCS +- A ``vcs`` key (type ``string``) MUST be present, containing the name of the VCS (i.e. one of ``git``, ``hg``, ``bzr``, ``svn``). Other VCS's SHOULD be registered by - amending this PEP. -- The ``url`` value MUST be compatible with the corresponding VCS, + amending this PEP. + The ``url`` value MUST be compatible with the corresponding VCS, so an installer can hand it off without transformation to a checkout/download command of the VCS. -- A ``requested_revision`` field MAY be present naming a +- A ``requested_revision`` key (type ``string``) MAY be present naming a branch/tag/ref/commit/revision/etc (in a format compatible with the VCS) to install. -- A ``commit_id`` field MUST be present, containing the +- A ``commit_id`` key (type ``string``) MUST be present, containing the exact commit/revision number that was installed. If the VCS supports commit-hash based revision identifiers, such commit-hash MUST be used as ``commit_id`` in order to reference the immutable version of the source code that was installed. -- A ``tag`` field naming a tag MAY be present to indicate that a particular - tag was installed. -- A ``branch`` field naming a branch MAY be present to indicate that a - particular branch was installed. If ``branch`` is present, ``tag`` MUST not - be present. +- If the installer could discover additional information about + the requested revision, it MAY add a ``resolved_revision`` and/or + ``resolved_revision_type`` field. If no revision was provided in + the requested URL, ``requested_revision`` MAY contain the default branch + that was installed, and ``requested_revision_type`` will be ``branch``. + If the installer determines that ``requested_revision`` was a tag, it MAY + add ``requested_revision_type`` with value ``tag``. -When ``url`` refers to a source archive, a wheel, or a local directory: +When ``url`` refers to a source archive or a wheel, the ``archive_info`` key +MUST be present as a dictionary with the following key: -- A ``hash`` field SHOULD be present, with value +- A ``hash`` key (type ``string``) SHOULD be present, with value ``=``. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used for source archive hashes. At time of writing, that list consists of 'md5', 'sha1', 'sha224', 'sha256', 'sha384', and 'sha512'. +When ``url`` refers to a local directory, the ``dir_info`` key MUST be +present as a dictionary with the following key: + +- ``editable`` (type: ``boolean``): `true` if the distribution was installed + in editable mode, `false` otherwise. If absent, default to `false`. + +When ``url`` refers to a local directory, it MUST have the ``file`` sheme +and be compliant with `RFC 8089`_. In particular, the path component must +be absolute. Symbolic links SHOULD be preserved when make relative +paths absolute. + .. note:: When the requested URL points to a local directory that happens to contain a VCS checkout, installers MUST NOT attempt to infer any VCS information and - therefore MUST NOT output any vcs related information (such as the ``vcs`` field) + therefore MUST NOT output any VCS related information (such as ``vcs_info``) in ``direct_url.json``. -A ``subdirectory`` field MAY be present containing a directory path, +A top-level ``subdirectory`` field MAY be present containing a directory path, relative to the root of the VCS repository, source archive or local directory, to specify where ``pyproject.toml`` or ``setup.py`` is located. @@ -257,7 +278,8 @@ Registered VCS -------------- This section lists the registered VCS's; expanded, VCS-specific information -on how to use the ``vcs``, ``requested_revision``, and other fields; and in +on how to use the ``vcs``, ``requested_revision``, and other fields of +``vcs_info``; and in some cases additional VCS-specific fields. Tools MAY support other VCS's although it is RECOMMENDED to register them by amending this PEP. The ``vcs`` field SHOULD be the command name @@ -275,31 +297,19 @@ vcs command git -vcs field +``vcs`` field git -requested_revision field +``requested_revision`` field A tag name, branch name, Git ref, commit hash, shortened commit hash, or other commit-ish. -commit_id field +``commit_id`` field A commit hash (40 hexadecimal characters sha1). -branch field - - If no ``requested_revision`` is provided and the remote repository has - a default branch, this field can be used to record the default branch that - was installed. - -VCS-specific fields: - -- A ``git_ref`` field naming a Git ref (string beginning with ``refs/``) MAY - be present to indicate that a particular ref was installed (e.g. - ``refs/pull/123/head``). - .. note:: Installers can use the ``git show-ref`` and ``git symbolic-ref`` commands @@ -319,15 +329,15 @@ vcs command hg -vcs field +``vcs`` field hg -requested_revision field +``requested_revision`` field A tag name, branch name, changeset ID, shortened changeset ID. -commit_id field +``commit_id`` field A changeset ID (40 hexadecimal characters). @@ -342,15 +352,15 @@ vcs command bzr -vcs field +``vcs`` field bzr -requested_revision field +``requested_revision`` field A tag name, branch name, revision id. -commit_id field +``commit_id`` field A revision id. @@ -365,16 +375,16 @@ vcs command svn -vcs field +``vcs`` field svn -requested_revision field +``requested_revision`` field ``requested_revision`` must be compatible with ``svn checkout`` ``--revision`` option. In Subversion, branch or tag is part of ``url``. -commit_id +``commit_id`` field Since Subversion does not support globally unique identifiers, this field is the Subversion revision number in the corresponding @@ -392,7 +402,9 @@ Source archive: { "url": "https://github.com/pypa/pip/archive/1.3.1.zip", - "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8" + "archive_info": { + "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8" + } } Git URL with tag and commit-hash: @@ -401,11 +413,35 @@ Git URL with tag and commit-hash: { "url": "https://github.com/pypa/pip.git", - "vcs": "git", - "requested_revision": "1.3.1", - "commit_id": "7921be1537eac1e97bc40179a57f0349c2aee67d" + "vcs_info": { + "vcs": "git", + "requested_revision": "1.3.1", + "resolved_revision_type": "tag", + "commit_id": "7921be1537eac1e97bc40179a57f0349c2aee67d" + } } +Local directory: + +.. code:: + + { + "url": "file:///home/user/project", + "dir_info": {} + } + +Local directory installed in editable mode: + +.. code:: + + { + "url": "file:///home/user/project", + "dir_info": { + "editable": true + } + } + + Example pip commands and their effect on direct_url.json -------------------------------------------------------- @@ -416,13 +452,16 @@ Commands that generate a ``direct_url.json``: * pip install "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" * pip install ./app * pip install file:///home/user/app +* pip install --editable "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" + (in which case, ``url`` will be the local directory where the git repository has been + cloned to, and ``dir_info`` will be present with ``"editable": true`` and no + ``vcs_info`` will be set) +* pip install -e ./app Commands that *do not* generate a ``direct_url.json`` * pip install app * pip install app --no-index --find-links https://example.com/ -* pip install --editable "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" -* pip install -e ./app Use cases ========= @@ -484,6 +523,8 @@ References .. _`Pipfile and Pipfile.lock`: https://github.com/pypa/pipfile .. _distlib: https://distlib.readthedocs.io .. _`environment variables in the user:password part of URLs`: https://pip.pypa.io/en/stable/reference/pip_install/#id10 +.. _`RFC 8259`: https://tools.ietf.org/html/rfc8259 +.. _`RFC 8089`: https://tools.ietf.org/html/rfc8089 Acknowledgements ================