406 lines
14 KiB
Plaintext
406 lines
14 KiB
Plaintext
PEP: 301
|
|
Title: Package Index and Metadata for Distutils
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Richard Jones <rjones@ekit-inc.com>
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 24-Oct-2002
|
|
Python-Version: 2.3
|
|
Post-History: 8-Nov-2002
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes several extensions to the Distutils packaging system
|
|
[1]_. These enhancements include a central package index server,
|
|
tools for submitting package information to the index and extensions
|
|
to the package metadata to include Trove [2]_ information.
|
|
|
|
This PEP does not address issues of package dependency. It also does
|
|
not address storage and download of packages as described in PEP 243
|
|
[6]_. Nor is it proposing a local database of packages as described
|
|
in PEP 262 [7]_.
|
|
|
|
Existing package repositories such as the Vaults of Parnassus [3]_,
|
|
CPAN [4]_ and PAUSE [5]_ will be investigated as prior art in this
|
|
field.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Python programmers have long needed a simple method of discovering
|
|
existing modules and systems available for their use. It is arguable
|
|
that the existence of these systems for other languages have been a
|
|
significant contribution to their popularity. The existence of the
|
|
Catalog-SIG, and the many discussions there indicate that there is a
|
|
large population of users who recognise this need.
|
|
|
|
The introduction of the Distutils packaging system to Python
|
|
simplified the process of distributing shareable code, and included
|
|
mechanisms for the capture of package metadata, but did little with
|
|
the metadata save ship it with the package.
|
|
|
|
An interface to the index should be hosted in the python.org domain,
|
|
giving it an air of legitimacy that existing catalog efforts do not
|
|
have.
|
|
|
|
The interface for submitting information to the catalog should be as
|
|
simple as possible - hopefully just a one-line command for most users.
|
|
|
|
Issues of package dependency are not addressed due to the complexity
|
|
of such a system. The original proposal for such a system, PEP 262,
|
|
was dropped as the author realised that platform packaging systems
|
|
(RPM, apt, etc) already handle dependencies, installation and removal.
|
|
|
|
Issues of package dissemination (storage on a central server) are
|
|
not addressed because they require assumptions about availability of
|
|
storage and bandwidth that I am not in a position to make. PEP 243,
|
|
which is still being developed, is tackling these issues and many
|
|
more. This proposal is considered compatible with, and adjunct to
|
|
the proposal in PEP 243.
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
The specification takes three parts, the `web interface`_, the
|
|
`Distutils register command`_ and the `Distutils Trove
|
|
classification`_.
|
|
|
|
|
|
Web Interface
|
|
-------------
|
|
|
|
A web interface is implemented over a simple store. The interface is
|
|
available through the python.org domain, either directly or as
|
|
packages.python.org.
|
|
|
|
The store has columns for all metadata fields. The (name, version)
|
|
double is used as a uniqueness key. Additional submissions for an
|
|
existing (name, version) will result in an *update* operation.
|
|
|
|
The web interface implements the following commands/interfaces:
|
|
|
|
**index**
|
|
Lists known packages, optionally filtered. An additional HTML page,
|
|
**search**, presents a form to the user which is used to customise
|
|
the index view. The index will include a browsing interface like
|
|
that presented in the Trove interface design section 4.3. The
|
|
results will be paginated, sorted alphabetically and only showing
|
|
the most recent version. The most recent version information will
|
|
be determined using the Distutils LooseVersion class.
|
|
|
|
**display**
|
|
Displays information about the package. All fields are displayed as
|
|
plain text. The "url" (or "home_page") field is hyperlinked.
|
|
|
|
**submit**
|
|
Accepts a POST submission of metadata about a package. The
|
|
"name" and "version" fields are mandatory, as they uniquely identify
|
|
an entry in the index. **Submit** will automatically determine
|
|
whether to create a new entry or update an existing entry. The
|
|
metadata is checked for correctness where appropriate - specifically
|
|
the Trove discriminators are compared with the allowed set. An
|
|
update will update all information about the package based on the
|
|
new submitted information.
|
|
|
|
There will also be a submit/edit form that will allow manual
|
|
submission and updating for those who do not use Distutils.
|
|
|
|
**submit_pkg_info**
|
|
Accepts a POST submission of a PKG-INFO file and performs the same
|
|
function as the **submit** interface.
|
|
|
|
**user**
|
|
Registers a new user with the index. Requires username, password
|
|
and email address. Passwords will be stored in the index database
|
|
as SHA hashes. If the username already exists in the database:
|
|
|
|
1. If valid HTTP Basic authentication is provided, the password and
|
|
email address are updated with the submission information, or
|
|
2. If no valid authentication is provided, the user is informed that
|
|
the login is already taken.
|
|
|
|
Registration will be a three-step process, involving:
|
|
|
|
1. User submission of details via the Distutils *register* command
|
|
or through the web,
|
|
2. Index server sending email to the user's email address with a URL
|
|
to visit to confirm registration with a random one-time key, and
|
|
3. User visits URL with the key and confirms registration.
|
|
|
|
**roles**
|
|
An interface for changing user Role assignments.
|
|
|
|
**password_reset**
|
|
Using a supplied email address as the key, this resets a user's
|
|
password and sends an email with the new password to the user.
|
|
|
|
The **submit** command will require HTTP Basic authentication,
|
|
preferably over an HTTPS connection.
|
|
|
|
The server interface will indicate success or failure of the commands
|
|
through a subset of the standard HTTP response codes:
|
|
|
|
===== ============== ================================================
|
|
Code Meaning Register command implications
|
|
===== ============== ================================================
|
|
200 OK Everything worked just fine
|
|
400 Bad request Data provided for submission was malformed
|
|
401 Unauthorised The username or password supplied were incorrect
|
|
403 Forbidden User does not have permission to update the
|
|
package information (not Owner or Maintainer)
|
|
===== ============== ================================================
|
|
|
|
User Roles
|
|
----------
|
|
|
|
Three user Roles will be assignable to users:
|
|
|
|
Owner
|
|
Owns a package name, may assign Maintainer Role for that name. The
|
|
first user to register information about a package is deemed Owner
|
|
of the package name. The Admin user may change this if necessary.
|
|
May submit updates for the package name.
|
|
|
|
Maintainer
|
|
Can submit and update info for a particular package name.
|
|
|
|
Admin
|
|
Can assign Owner Role and edit user details. Not specific to a
|
|
package name.
|
|
|
|
|
|
Index Storage (Schema)
|
|
----------------------
|
|
|
|
The index is stored in a set of relational database tables:
|
|
|
|
**packages**
|
|
Lists package names and holds package-level metadata (currently
|
|
just the stable release version)
|
|
|
|
**releases**
|
|
Each package has an entry in **releases** for each version of the
|
|
package that is released. A row holds the bulk of the information
|
|
given in the package's PKG-INFO file. There is one row for each
|
|
package (*name*, *version*).
|
|
|
|
**trove_discriminators**
|
|
Lists the Trove discriminator text and assigns each one a unique
|
|
ID.
|
|
|
|
**release_discriminators**
|
|
Each entry maps a package (*name*, *version*) to a
|
|
*discriminator_id*. We map to releases instead of packages because
|
|
the set of discriminators may change between releases.
|
|
|
|
**journals**
|
|
Holds information about changes to package information in the
|
|
index. Changes to the **packages**, **releases**, **roles**,
|
|
and **release_discriminators** tables are listed here by
|
|
package *name* and *version* if the change is release-specific.
|
|
|
|
**users**
|
|
Holds our user database - user name, email address and password.
|
|
|
|
**roles**
|
|
Maps *user_name* and *role_name* to a *package_name*.
|
|
|
|
An additional table, **rego_otk** holds the One Time Keys generated
|
|
during registration and is not interesting in the scope of the index
|
|
itself.
|
|
|
|
|
|
Distutils *register* Command
|
|
----------------------------
|
|
|
|
An additional Distutils command, ``register``, is implemented which
|
|
posts the package metadata to the central index. The *register*
|
|
command automatically handles user registration; the user is presented
|
|
with three options:
|
|
|
|
1. login and submit package information
|
|
2. register as a new packager
|
|
3. send password reminder email
|
|
|
|
On systems where the ``$HOME`` environment variable is set, the user
|
|
will be prompted at exit to save their username/password to a file
|
|
in their ``$HOME`` directory in the file ``.pypirc``.
|
|
|
|
Notification of changes to a package entry will be sent to all users
|
|
who have submitted information about the package. That is, the
|
|
original submitter and any subsequent updaters.
|
|
|
|
The *register* command will include a ``--verify`` option which
|
|
performs a test submission to the index without actually committing
|
|
the data. The index will perform its submission verification checks
|
|
as usual and report any errors it would have reported during a normal
|
|
submission. This is useful for verifying correctness of Trove
|
|
discriminators.
|
|
|
|
|
|
Distutils Trove Classification
|
|
------------------------------
|
|
|
|
The Trove concept of *discrimination* will be added to the metadata
|
|
set available to package authors through the new attribute
|
|
"classifiers". The list of classifiers will be available through the
|
|
web, and added to the package like so::
|
|
|
|
setup(
|
|
name = "roundup",
|
|
version = __version__,
|
|
classifiers = [
|
|
'Development Status :: 4 - Beta',
|
|
'Environment :: Console',
|
|
'Environment :: Web Environment',
|
|
'Intended Audience :: End Users/Desktop',
|
|
'Intended Audience :: Developers',
|
|
'Intended Audience :: System Administrators',
|
|
'License :: OSI Approved :: Python Software Foundation License',
|
|
'Operating System :: MacOS :: MacOS X',
|
|
'Operating System :: Microsoft :: Windows',
|
|
'Operating System :: POSIX',
|
|
'Programming Language :: Python',
|
|
'Topic :: Communications :: Email',
|
|
'Topic :: Office/Business',
|
|
'Topic :: Software Development :: Bug Tracking',
|
|
],
|
|
url = 'http://sourceforge.net/projects/roundup/',
|
|
...
|
|
)
|
|
|
|
It was decided that strings would be used for the classification
|
|
entries due to the deep nesting that would be involved in a more
|
|
formal Python structure.
|
|
|
|
The original Trove specification that classification namespaces be
|
|
separated by slashes ("/") unfortunately collides with many of the
|
|
names having slashes in them (e.g. "OS/2"). The double-colon solution
|
|
(" :: ") implemented by SourceForge and FreshMeat gets around this
|
|
limitation.
|
|
|
|
The list of classification values on the module index has been merged
|
|
from FreshMeat and SourceForge (with their permission). This list
|
|
will be made available both through the web interface and through the
|
|
*register* command's ``--list-classifiers`` option as a text list
|
|
which may then be copied to the ``setup.py`` file. The *register*
|
|
command's ``--verify`` option will check classifiers values against
|
|
the server's list.
|
|
|
|
Unfortunately, the addition of the "classifiers" property is not
|
|
backwards-compatible. A setup.py file using it will not work under
|
|
Python 2.1.3. It is hoped that a bug-fix release of Python 2.2 (most
|
|
likely 2.2.3) will relax the argument checking of the setup() command
|
|
to allow new keywords, even if they're not actually used. It is
|
|
preferable that a warning be produced, rather than a show-stopping
|
|
error. The use of the new keyword should be discouraged in situations
|
|
where the package is advertised as being compatible with python
|
|
versions earlier than 2.2.3 or 2.3.
|
|
|
|
In the PKG-INFO, the classifiers list items will appear as individual
|
|
``Classifier:`` entries::
|
|
|
|
Name: roundup
|
|
Version: 0.5.2
|
|
Classifier: Development Status :: 4 - Beta
|
|
Classifier: Environment :: Console (Text Based)
|
|
.
|
|
.
|
|
Classifier: Topic :: Software Development :: Bug Tracking
|
|
Url: http://sourceforge.net/projects/roundup/
|
|
|
|
|
|
Implementation
|
|
==============
|
|
|
|
The server is be available at:
|
|
|
|
http://www.python.org/pypi
|
|
|
|
The code is available from the SourceForge project:
|
|
|
|
http://sourceforge.net/projects/pypi/
|
|
|
|
The *register* command has been integrated into Python 2.3.
|
|
|
|
|
|
Rejected Proposals
|
|
==================
|
|
|
|
Originally, the index server was to return custom headers (inspired by
|
|
PEP 243):
|
|
|
|
**X-Pypi-Status**
|
|
Either "success" or "fail".
|
|
|
|
**X-Pypi-Reason**
|
|
A description of the reason for failure, or additional information
|
|
in the case of a success.
|
|
|
|
However, it has been pointed out [8]_ that this is a bad scheme to
|
|
use.
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [1] Distutils packaging system
|
|
(http://www.python.org/doc/current/lib/module-distutils.html)
|
|
|
|
.. [2] Trove
|
|
(http://tuxedo.org/~esr/trove/)
|
|
|
|
.. [3] Vaults of Parnassus
|
|
(http://www.vex.net/parnassus/)
|
|
|
|
.. [4] CPAN
|
|
(http://www.cpan.org/)
|
|
|
|
.. [5] PAUSE
|
|
(http://pause.cpan.org/)
|
|
|
|
.. [6] PEP 243, Module Repository Upload Mechanism
|
|
(http://www.python.org/peps/pep-0243.html)
|
|
|
|
.. [7] PEP 262, A Database of Installed Python Packages
|
|
(http://www.python.org/peps/pep-0262.html)
|
|
|
|
.. [8] [PEP243] upload status is bogus
|
|
(http://mail.python.org/pipermail/distutils-sig/2001-March/002262.html)
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
|
|
Acknowledgements
|
|
================
|
|
|
|
Anthony Baxter, Martin v. Loewis and David Goodger for encouragement
|
|
and feedback during initial drafting.
|
|
|
|
A.M. Kuchling for support including hosting the second prototype.
|
|
|
|
Greg Stein for recommending that the register command interpret the
|
|
HTTP response codes rather than custom X-PyPI-* headers.
|
|
|
|
The many participants of the Distutils and Catalog SIGs for their
|
|
ideas over the years.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
End:
|