python-peps/pep-0376.txt

PEP: 376
Title: Changing the .egg-info structure
Version: $Revision$
Last-Modified: $Date$
Author: Tarek Ziadé <tarek@ziade.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 22-Feb-2009
Python-Version: 2.7, 3.1
Post-History:


Abstract
========

This PEP proposes various enhancements for Distutils:

- A new format for the .egg-info structure.
- Some APIs to read the meta-data of a project

Definitions
===========

A **project** is a Python application composed of one or many Python packages.
It is distributed using a `setup.py` script with Distutils and/or Setuptools.

Once installed, one or several **packages** are added in Python's site-packages.

Rationale
=========

There are three problems right now in the way projects are installed in 
Python:

- There are too many ways to install a project in Python.
- There is no API to get the metadata of installed packages.

How projects are installed
--------------------------

Right now, when a project is installed in Python, every package its contains 
is installed in the `site-packages` directory with the Distutils `install` 
command.

The `install_egg_info` subcommand is called during this process, in order to 
create an `.egg-info` file in the `site-packages` directory.

For example, if the `zlib` project is installed (which contains one package), 
two elements will be installed in `site-packages`::

 - zlib
 - zlib-2.5.2-py2.4.egg-info

Where `zlib` is the package, and `zlib-2.5.2-py2.4.egg-info` is
a file containing the package metadata as described in PEP 314.

This file corresponds to the file called `PKG-INFO`, built by 
the `sdist` command.

The problem is that many people use `easy_install` (setuptools) or `pip` 
to install their packages, and these third-party tools do not install 
packages in the same way that Distutils does:

- `easy_install` creates an `EGG-INFO` directory inside an `.egg` directory, 
  and adds a `PKG-INFO` file inside this directory. The `.egg` directory 
  contains in that case the packages of the project.

- `pip` creates an `.egg-info` directory inside the site-packages directory
  and adds a `PKG-INFO` file inside it. Packages are installed in 
  site-packages directory in a regular way.

They both add other files in the `EGG-INFO` or `.egg-info` directory, and
create or modify `.pth` files.

Uninstall information
---------------------

Distutils doesn't provide any `uninstall` command. If you want to uninstall
a project, you have to be a power user and remove the various package 
directories from the right `site-packages` directory, then look over the right 
`pth` files. And this method differs, depending on the tools you are using.

The worst issue is that you depend on the way the packager created his package.
When you call `python setup.py install`, it will not be installed the same way
depending on the tool used by the packager (mainly Distutils or Setuptools).

But there's common behavior: files are copied in your installation.
And there's a way to keep track of theses file, so to remove them.

What this PEP proposes
----------------------

To address those issues, this PEP proposes a few changes:

- a new `.egg-info` structure using a directory;
- a list of elements this directory holds;
- new functions in `pkgutil` to be able to query the information
  of installed projects.

.egg-info becomes a directory
=============================

The first change would be to make `.egg-info` a directory and let it
hold the `PKG-INFO` file built by the `write_pkg_file` method.

This change will not impact Python itself, because this file is not
used anywhere yet in the standard library. So there's no need of
deprecation.

Although it will impact the `setuptools` and `pip` projects, but given 
the fact that they already work with a directory that contains a 
`PKG-INFO` file, the change will be small.

For example, if the `zlib` package is installed, two elements
will be installed in `site-packages`::

    - zlib
    - zlib-2.5.2.egg-info/
        PKG-INFO

The Python version will also be removed from the .egg-info directory
name. To be able to implement this change, the impacted code in Distutils
is the `install_egg_info` command, and the various third-party projects.

Adding a RECORD in the .egg-info directory
==========================================

A `RECORD` file will be added inside the `.egg-info` directory at installation
time.

- the `RECORD` file will hold the list of installed files. These
  correspond to the files listed by the `record` option of the `install` 
  command, and will always be generated. This will allow uninstall, as
  explained later in this PEP.

The `install` command will record by default installed files in the
RECORD file, using these rules:

- if the installed file is located in a directory in `site-packages`,
  it will be a '/'-separated relative path, no matter what is the target 
  system. This makes this information cross-compatible and allows simple
  installation to be relocatable.

- if the installed file is located elswhere in the system, a 
  '/'-separated absolute path is used.

This will require changing the way the `install` command writes the record
file, so the old `record` behavior will be deprecated. 
XXX see how to handle old record (new option, or wait for 2 version?)

Listing the .egg-info elements in Distutils
===========================================

In Distutils, the `dist` module will introduce an `EGG_INFO_FILES` constant 
to list all files located in the `.egg-info` directory::

    from collections import namedtuple

    EggInfos = namedtuple('EggInfo', 'manifest record pkg_info')

    # files added in egg-info    
    EGG_INFO_FILES = EggInfos('RECORD', 'PKG-INFO')

Back to our `zlib` example, we will have::

    - zlib
    - zlib-2.5.2.egg-info/
        PKG-INFO
        RECORD

XXX See if we want to add Python version in the PKG-INFO

New functions in pkgutil
========================

To use the `.egg-info` directory content, we need to add in the standard 
library a set of APIs. The best place to put these APIs seems to be `pkgutil`.

The new functions added in the package are :

- get_egg_info(project_name) -> path or None

  Scans all site-packages directories and looks for all `project_name.egg-info` 
  directories. Returns the directory path that contains a PKG-INFO that matches
  `project_name` for the `name` metadata. Notice that there should be at most 
  one result. The first result founded will be returned.

  If the directory is not found, returns None.

- get_metadata(project_name) -> DistributionMetadata or None

  Uses `get_egg_info` to get the `PKG-INFO` file, and returns a 
  `DistributionMetadata` instance that contains the metadata.
  This will require a small change in `DistributionMetadata` (see #4908).

- get_egg_info_file(project_name, filename) -> file object or None

  Uses `get_egg_info` and gets any file inside the directory,
  pointed by filename.

  filename can be any value found in `distutils.sdist.EGG_INFO_FILES`.

Let's use it with our `zlib` example::

    >>> from pkgutil import get_egg_info, get_metadata, get_egg_info_file
    >>> get_egg_info('zlib')
    '/opt/local/lib/python2.6/site-packages/zlib-2.5.2.egg-info'
    >>> metadata = get_metadata('zlib')
    >>> metadata.version
    '2.5.2'
    >>> from distutils.dist import EGG_INFO_FILES
    >>> get_egg_info_file('zlib', EGG_INFO_FILES.manifest).read()
    some
    ...
    files


Adding an Uninstall API
=======================

Distutils provides a very basic way to install a project, which is running
the `install` command over the `setup.py` script of the distribution.

Distutils will provide a very basic `uninstall` command that will remove
all files listed in the `RECORD` file of a project, as long as they are not 
mentioned in another `RECORD` file.

This command will be added in the `util` module and will take the name
of the project to uninstall::

    >>> from distutils.util import uninstall
    >>> uninstall('zlib')

To make it a reference API for third-party projects that wish to provide
an `uninstall feature`. The `uninstall` API can also be invoked with a 
second callable argument, that will be invoked for each file to be removed.
If it returns `True`, the file will be removed.

Examples::

    >>> def _remove_and_log(path):
    ...     logging.info('Removing %s' % path)
    ...     return True
    >>> uninstall('zlib', _remove_and_log)
    >>> def _dry_run(path):
    ...     logging.info('Removing %s (dry run)' % path)
    ...     return False
    >>> uninstall('zlib', _dry_run)


Aknowledgments
==============

Jim Fulton, Ian Bicking, Phillip Eby, and many people at Pycon and Distutils-SIG.

Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: