242 lines
8.4 KiB
ReStructuredText
242 lines
8.4 KiB
ReStructuredText
PEP: 627
|
|
Title: Recording installed projects
|
|
Author: Petr Viktorin <encukou@gmail.com>
|
|
Discussions-To: https://discuss.python.org/t/pep-627/4126
|
|
Status: Draft
|
|
Type: Informational
|
|
Content-Type: text/x-rst
|
|
Created: 15-Jul-2020
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP clarifies and updates :pep:`376` (Database of Installed Python
|
|
Distributions), rewriting it as an interoperability standard.
|
|
It moves the canonical location of the standard to the Python
|
|
Packaging Authority (PyPA) standards repository, and sets up guidelines
|
|
for changing it.
|
|
|
|
Two files in installed ``.dist-info`` directories are made optional:
|
|
``RECORD`` (which PEP 376 lists as mandatory, but suggests it can be left out
|
|
for "system packages"), and ``INSTALLER``.
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
Python packaging is moving from relying on specific tools (Setuptools and pip)
|
|
toward a ecosystem of tools and tool-agnostic interoperability standards.
|
|
|
|
PEP 376 is not written as an interoperability standard.
|
|
It describes implementation details of specific tools and libraries,
|
|
and is underspecified, leaving much room for implementation-defined behavior.
|
|
|
|
This is a proposal to “distill” the standard from PEP 376, clarify it,
|
|
and rewrite it to be tool-agnostic.
|
|
|
|
The aim of this PEP is to have a better standard, not necessarily a perfect one.
|
|
Some issues are left to later clarification.
|
|
|
|
|
|
Rationale Change
|
|
================
|
|
|
|
PEP 376's rationale focuses on two problems:
|
|
|
|
* There are too many ways to install projects and this makes interoperation difficult.
|
|
* There is no API to get information on installed distributions.
|
|
|
|
The new document focuses only the on-disk format of information about
|
|
installed projects.
|
|
Providing API to install, uninstall or query this information is left to
|
|
be implemented by tools.
|
|
|
|
|
|
Standard and Changes Process
|
|
============================
|
|
|
|
The canonical standard for *Recording installed projects* (previously known as
|
|
*Database of Installed Python Distributions*) is the `documentation`_ at
|
|
``packaging.python.org``.
|
|
Any changes to the document (except trivial language or typography fixes) must
|
|
be made through the PEP process.
|
|
|
|
The document is normative (with examples to aid understanding).
|
|
PEPs that change it, such as this one, contain additional information that is
|
|
expected to get out of date, such as rationales and compatibility
|
|
considerations.
|
|
|
|
The proposed standard is submitted together with this PEP as a pull request to
|
|
``packaging.python.org``.
|
|
|
|
.. _documentation: https://packaging.python.org/specifications/recording-installed-packages/
|
|
|
|
|
|
Changes and their Rationale
|
|
===========================
|
|
|
|
Renaming to "Recording installed projects"
|
|
------------------------------------------
|
|
|
|
The standard is renamed from *Database of Installed Python Distributions*
|
|
to *Recording installed projects*.
|
|
|
|
While putting files in known locations on disk may be thought of as
|
|
a “database”, it's not what most people think about when they hear the term.
|
|
The PyPA links to PEP 376 under the heading *Recording installed distributions*.
|
|
|
|
The PyPA glossary defines “Distribution” (or, “Distribution Package” to prevent
|
|
confusion with e.g. Linux distributions) as “A versioned archive file […]”.
|
|
Since there may be other ways to install Python code than from archive files,
|
|
the document uses “installed project” rather than “installed distribution”.
|
|
|
|
|
|
Removal of Implementation Details
|
|
---------------------------------
|
|
|
|
All tool- and library-specific details are removed.
|
|
The mechanisms of how a project is installed are also left out: the document
|
|
focuses on the end state.
|
|
One exception is a sketch of an uninstallation algorithm,
|
|
which is given to better explain the purpose of the ``RECORD`` file.
|
|
|
|
References to ``.egg-info`` and ``.egg``,
|
|
formats specific to ``setuptools`` and ``distutils``,
|
|
are left out.
|
|
|
|
|
|
Explicitly Allowing Additional Files
|
|
------------------------------------
|
|
|
|
The ``.dist-info`` directory is allowed to contain files not specified in
|
|
the spec.
|
|
The current tools already do this.
|
|
|
|
A note in the specification mentions files in the ``.dist-info`` directory of *wheels*.
|
|
Current tools copy these files to the installed ``.dist-info``—something
|
|
to keep in mind for further standardization efforts.
|
|
|
|
|
|
Clarifications in the ``RECORD`` File
|
|
-------------------------------------
|
|
|
|
The CSV dialect is specified to be the default of Python's ``csv`` module.
|
|
This resolves edge cases around handling double-quotes and line terminators
|
|
in file names.
|
|
|
|
The “base” of relative paths in ``RECORD`` is specified relative to the
|
|
``.dist-info`` directory, rather than tool-specific ``--install-lib`` and
|
|
``--prefix`` options.
|
|
|
|
Both *hash* and *size* fields are now optional (for any file, not just
|
|
``.pyc``, ``.pyo`` and ``RECORD``).
|
|
This simplifies the spec, and makes it possible for tools to modify files
|
|
(for example, rewrite shebangs) after ``RECORD`` is generated.
|
|
|
|
The new spec explicitly says that the ``RECORD`` file must now include *all*
|
|
files of the installed project (the exception for ``.pyc`` files remains).
|
|
Since tools use ``RECORD`` for uninstallation, incomplete file lists could
|
|
introduce orphaned files to users' environments.
|
|
On the other hand, this means that there is no way to record hashes of some
|
|
any files if the full list of files is unknown.
|
|
|
|
A sketch of an uninstallation algorithm is included to clarify the file's
|
|
primary purpose and contents.
|
|
|
|
On Windows, files in ``RECORD`` may be separated by either ``/`` or ``\``.
|
|
PEP 376 was unclear on this: it mandates forward slashes in one place, but
|
|
shows backslackes in a Windows-specific example.
|
|
|
|
|
|
|
|
Optional ``RECORD`` File
|
|
------------------------
|
|
|
|
The ``RECORD`` file is made optional.
|
|
Not all tools can easily generate a list of installed files in a
|
|
Python-specific format.
|
|
|
|
Specifically, the ``RECORD`` file is unnecessary when projects are installed
|
|
by a Linux system packaging system, which has its own ways to keep track of
|
|
files, uninstall them or check their integrity.
|
|
Having to keep a ``RECORD`` file in sync with the disk and the system package
|
|
database would be unreasonably fragile, and no ``RECORD`` file is better
|
|
than one that does not correspond to reality.
|
|
|
|
(Full disclosure: The author of this PEP is an RPM packager active in the Fedora Linux distro.)
|
|
|
|
|
|
Optional ``INSTALLER`` File
|
|
---------------------------
|
|
|
|
The ``INSTALLER`` file is also made optional, and specified to be used for
|
|
informational purposes only.
|
|
|
|
This file was added to distinguish projects installed by the Python installer
|
|
(``pip``) from ones installed by other package managers (e.g. ``dnf``).
|
|
There were attempts to use this file to prevent ``pip`` from updating or
|
|
uninstalling packages it didn't install.
|
|
|
|
Our goal is supporting interoperating tools, and basing any action on
|
|
which tool happened to install a package runs counter to that goal.
|
|
|
|
Instead of relying on the installer name, tools should use feature detection.
|
|
The current document offers a crude way of making a project untouchable by
|
|
Python tooling: omitting ``RECORD`` file.
|
|
|
|
On the other hand, the installer name may be useful in hints to the user.
|
|
|
|
To align with this new purpose of the file, the new specification allows
|
|
any ASCII string in ``INSTALLER``, rather than a lowercase identifier.
|
|
It also suggests using the command-line command, if available.
|
|
|
|
|
|
The ``REQUESTED`` File
|
|
----------------------
|
|
|
|
The ``REQUESTED`` file is kept in the specification, with only tool-specific
|
|
notes removed.
|
|
|
|
Note that tools that do not create this file effectively mark all installed
|
|
projects as “removable unless needed by another project”.
|
|
This was was the case with ``pip`` before version 20.2.
|
|
Before ``REQUESTED`` becomes commonplace, automatically uninstalling “orphaned”
|
|
projects should be done with care.
|
|
|
|
|
|
Clarifications
|
|
--------------
|
|
|
|
When possible, terms (such as ``name`` and ``version``) are qualified by
|
|
references to existing specs.
|
|
|
|
|
|
Deferred Ideas
|
|
==============
|
|
|
|
To limit the scope of this PEP, some improvements are explicitly left to
|
|
future PEPs:
|
|
|
|
* Encoding of the ``RECORD`` file
|
|
* Limiting or namespacing files that can appear in ``.dist-info``
|
|
* Marking the difference between projects installed by a tool that does not
|
|
support the ``REQUESTED`` file and projects installed as dependencies
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|