python-peps/pep-0301.txt

PEP: 301
Title: Package Index and Metadata for Distutils
Version: $Revision$
Last-Modified: $Date$
Author: Richard Jones <rjones@ekit-inc.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Oct-2002
Python-Version: 2.3
Post-History: 8-Nov-2002


Abstract
========

This PEP proposes several extensions to the Distutils packaging system
[1]_.  These enhancements include a central package index server,
tools for submitting package information to the index and extensions
to the package metadata to include Trove [2]_ information.

This PEP does not address issues of package dependency.  It also does
not address storage and download of packages as described in PEP 243
[6]_.  Nor is it proposing a local database of packages as described
in PEP 262 [7]_.

Existing package repositories such as the Vaults of Parnassus [3]_,
CPAN [4]_ and PAUSE [5]_ will be investigated as prior art in this
field.


Rationale
=========

Python programmers have long needed a simple method of discovering
existing modules and systems available for their use.  It is arguable
that the existence of these systems for other languages have been a
significant contribution to their popularity.  The existence of the
Catalog-SIG, and the many discussions there indicate that there is a
large population of users who recognise this need.

The introduction of the Distutils packaging system to Python
simplified the process of distributing shareable code, and included
mechanisms for the capture of package metadata, but did little with
the metadata save ship it with the package.

An interface to the index should be hosted in the python.org domain,
giving it an air of legitimacy that existing catalog efforts do not
have.

The interface for submitting information to the catalog should be as
simple as possible - hopefully just a one-line command for most users.

Issues of package dependency are not addressed due to the complexity
of such a system.  PEP 262 proposes such a system, but as of this
writing the PEP is still unfinished.

Issues of package dissemination (storage on a central server) are
not addressed because they require assumptions about availability of
storage and bandwidth that I am not in a position to make.  PEP 243,
which is still being developed, is tackling these issues and many
more.  This proposal is considered compatible with, and adjunct to
the proposal in PEP 243.


Specification
=============

The specification takes three parts, the `web interface`_,  the
`Distutils register command`_ and the `Distutils Trove
classification`_.


Web Interface
-------------

A web interface is implemented over a simple store.  The interface is
available through the python.org domain, either directly or as
packages.python.org.

The store has columns for all metadata fields.  The (name, version)
double is used as a uniqueness key.  Additional submissions for an
existing (name, version) will result in an *update* operation.

The web interface implements the following commands/interfaces:

**index**
  Lists known packages, optionally filtered.  An additional HTML page,
  **search**, presents a form to the user which is used to customise
  the index view.  The index will include a browsing interface like
  that presented in the Trove interface design section 4.3.  The
  results will be paginated, sorted alphabetically and only showing
  the most recent version.  The most recent version information will
  be determined using the Distutils LooseVersion class.

**display**
  Displays information about the package.  All fields are displayed as
  plain text.  The "url" (or "home_page") field is hyperlinked.

**submit**
  Accepts a POST submission of metadata about a package.  The
  "name" and "version" fields are mandatory, as they uniquely identify
  an entry in the index.  **Submit** will automatically determine
  whether to create a new entry or update an existing entry.  The
  metadata is checked for correctness where appropriate - specifically
  the Trove discriminators are compared with the allowed set.  An
  update will update all information about the package based on the
  new submitted information.

  There will also be a submit/edit form that will allow manual
  submission and updating for those who do not use Distutils.

**submit_pkg_info**
  Accepts a POST submission of a PKG-INFO file and performs the same
  function as the **submit** interface.

**user**
  Registers a new user with the index.  Requires username, password
  and email address.  Passwords will be stored in the index database
  as SHA hashes.  If the username already exists in the database:

  1. If valid HTTP Basic authentication is provided, the password and
     email address are updated with the submission information, or
  2. If no valid authentication is provided, the user is informed that
     the login is already taken.

  Registration will be a three-step process, involving:

  1. User submission of details via the Distutils *register* command
     or through the web,
  2. Index server sending email to the user's email address with a URL
     to visit to confirm registration with a random one-time key, and
  3. User visits URL with the key and confirms registration.

**roles**
  An interface for changing user Role assignments.

**password_reset**
  Using a supplied email address as the key, this resets a user's
  password and sends an email with the new password to the user.

The **submit** command will require HTTP Basic authentication,
preferably over an HTTPS connection.

The server interface will indicate success or failure of the commands
through a subset of the standard HTTP response codes:

===== ============== ================================================
Code  Meaning        Register command implications
===== ============== ================================================
200   OK             Everything worked just fine
400   Bad request    Data provided for submission was malformed
401   Unauthorised   The username or password supplied were incorrect
403   Forbidden      User does not have permission to update the
                     package information (not Owner or Maintainer)
===== ============== ================================================

User Roles
----------

Three user Roles will be assignable to users:

Owner
  Owns a package name, may assign Maintainer Role for that name.  The
  first user to register information about a package is deemed Owner
  of the package name.  The Admin user may change this if necessary.
  May submit updates for the package name.

Maintainer
  Can submit and update info for a particular package name.

Admin
  Can assign Owner Role and edit user details. Not specific to a
  package name.


Index Storage (Schema)
----------------------

The index is stored in a set of relational database tables:

**packages**
  Lists package names and holds package-level metadata (currently
  just the stable release version)

**releases**
  Each package has an entry in **releases** for each version of the
  package that is released. A row holds the bulk of the information
  given in the package's PKG-INFO file. There is one row for each
  package (*name*, *version*).

**trove_discriminators**
  Lists the Trove discriminator text and assigns each one a unique
  ID.

**release_discriminators**
  Each entry maps a package (*name*, *version*) to a
  *discriminator_id*.  We map to releases instead of packages because
  the set of discriminators may change between releases.

**journals**
  Holds information about changes to package information in the
  index. Changes to the **packages**, **releases**, **roles**,
  and **release_discriminators** tables are listed here by
  package *name* and *version* if the change is release-specific.

**users**
  Holds our user database - user name, email address and password.

**roles**
  Maps *user_name* and *role_name* to a *package_name*.

An additional table, **rego_otk** holds the One Time Keys generated
during registration and is not interesting in the scope of the index
itself.


Distutils *register* Command
----------------------------

An additional Distutils command, ``register``, is implemented which
posts the package metadata to the central index.  The *register*
command automatically handles user registration; the user is presented
with three options:

1. login and submit package information
2. register as a new packager
3. send password reminder email

On systems where the ``$HOME`` environment variable is set, the user
will be prompted at exit to save their username/password to a file
in their ``$HOME`` directory in the file ``.pypirc``.

Notification of changes to a package entry will be sent to all users
who have submitted information about the package.  That is, the
original submitter and any subsequent updaters.

The *register* command will include a ``--verify`` option which
performs a test submission to the index without actually committing
the data.  The index will perform its submission verification checks
as usual and report any errors it would have reported during a normal
submission.  This is useful for verifying correctness of Trove
discriminators.


Distutils Trove Classification
------------------------------

The Trove concept of *discrimination* will be added to the metadata
set available to package authors through the new attribute
"classifiers".  The list of classifiers will be available through the
web, and added to the package like so::

    setup(
        name = "roundup", 
        version = __version__,
        classifiers = [
            'Development Status :: 4 - Beta',
            'Environment :: Console',
            'Environment :: Web Environment',
            'Intended Audience :: End Users/Desktop',
            'Intended Audience :: Developers',
            'Intended Audience :: System Administrators',
            'License :: OSI Approved :: Python Software Foundation License',
            'Operating System :: MacOS :: MacOS X',
            'Operating System :: Microsoft :: Windows',
            'Operating System :: POSIX',
            'Programming Language :: Python',
            'Topic :: Communications :: Email',
            'Topic :: Office/Business',
            'Topic :: Software Development :: Bug Tracking',
        ],
        url = 'http://sourceforge.net/projects/roundup/',
        ...
    )

It was decided that strings would be used for the classification
entries due to the deep nesting that would be involved in a more
formal Python structure.

The original Trove specification that classification namespaces be
separated by slashes ("/") unfortunately collides with many of the
names having slashes in them (e.g. "OS/2").  The double-colon solution
(" :: ") implemented by SourceForge and FreshMeat gets around this
limitation.

The list of classification values on the module index has been merged
from FreshMeat and SourceForge (with their permission).  This list
will be made available both through the web interface and through the
*register* command's ``--list-classifiers`` option as a text list
which may then be copied to the ``setup.py`` file.  The *register*
command's ``--verify`` option will check classifiers values against
the server's list.

Unfortunately, the addition of the "classifiers" property is not
backwards-compatible.  A setup.py file using it will not work under
Python 2.1.3.  It is hoped that a bug-fix release of Python 2.2 (most
likely 2.2.3) will relax the argument checking of the setup() command
to allow new keywords, even if they're not actually used.  It is
preferable that a warning be produced, rather than a show-stopping
error. The use of the new keyword should be discouraged in situations
where the package is advertised as being compatible with python
versions earlier than 2.2.3 or 2.3.

In the PKG-INFO, the classifiers list items will appear as individual
``Classifier:`` entries::

        Name: roundup 
        Version: 0.5.2
        Classifier: Development Status :: 4 - Beta
        Classifier: Environment :: Console (Text Based)
                    .
                    .
        Classifier: Topic :: Software Development :: Bug Tracking
        Url: http://sourceforge.net/projects/roundup/


Implementation
==============

The server is available at:

  http://www.python.org/pypi

The code is available from the SourceForge project:

  http://sourceforge.net/projects/pypi/

The *register* command has been integrated into Python 2.3.


Rejected Proposals
==================

Originally, the index server was to return custom headers (inspired by
PEP 243):

**X-Pypi-Status**
  Either "success" or "fail".

**X-Pypi-Reason**
  A description of the reason for failure, or additional information
  in the case of a success.

However, it has been pointed out [8]_ that this is a bad scheme to
use.


References
==========

.. [1] Distutils packaging system
   (http://www.python.org/doc/current/lib/module-distutils.html)

.. [2] Trove
   (http://tuxedo.org/~esr/trove/)

.. [3] Vaults of Parnassus
   (http://www.vex.net/parnassus/)

.. [4] CPAN
   (http://www.cpan.org/)

.. [5] PAUSE
   (http://pause.cpan.org/)

.. [6] PEP 243, Module Repository Upload Mechanism
   (http://www.python.org/peps/pep-0243.html)

.. [7] PEP 262, A Database of Installed Python Packages
   (http://www.python.org/peps/pep-0262.html)

.. [8] [PEP243] upload status is bogus
   (http://mail.python.org/pipermail/distutils-sig/2001-March/002262.html)


Copyright
=========

This document has been placed in the public domain.


Acknowledgements
================

Anthony Baxter, Martin v. Loewis and David Goodger for encouragement
and feedback during initial drafting.

A.M. Kuchling for support including hosting the second prototype.

Greg Stein for recommending that the register command interpret the
HTTP response codes rather than custom X-PyPI-* headers.

The many participants of the Distutils and Catalog SIGs for their
ideas over the years.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: