diff --git a/pep-0376.txt b/pep-0376.txt index 09e321e9f..5a506517c 100644 --- a/pep-0376.txt +++ b/pep-0376.txt @@ -22,28 +22,41 @@ This PEP proposes various enhancements for Distutils: Definitions =========== -A **project** is a Python application composed of one or many Python packages. -It is distributed using a `setup.py` script with Distutils and/or Setuptools. +A **project** is a Python application composed of one or several files, which can +be Python modules, extensions or data. It is distributed using a `setup.py` script +with Distutils and/or Setuptools. The `setup.py` script indicates where each +elements should be installed. -Once installed, one or several **packages** are added in Python's site-packages. +Once installed, the elements are located in various places in the system, like: + +- in Python's site-packages (Python modules, Python modules organized into packages, + Extensions, etc.) +- in Python's `include` directory. +- in Python's `bin` or `Script` directory. +- etc. Rationale ========= There are two problems right now in the way projects are installed in Python: -- There are too many ways to install a project in Python. +- There are too many ways to do it. - There is no API to get the metadata of installed projects. How projects are installed -------------------------- -Right now, when a project is installed in Python, every package its contains -is installed in the `site-packages` directory with the Distutils `install` -command. +Right now, when a project is installed in Python, every elements its contains +is installed in various directories. + +The pure Python code for instance is installed in the `purelib` directory, +which is located in the Python installation in `lib\python2.6\site-packages` +for example under unix-like systems or Mac OS X, and in `Lib/site-packages` +under Windows. This is done with the Distutils `install` command, which calls +various subcommands. The `install_egg_info` subcommand is called during this process, in order to -create an `.egg-info` file in the `site-packages` directory. +create an `.egg-info` file in the `purelib` directory. For example, if the `zlib` project (which contains one package) is installed, two elements will be installed in `site-packages`:: @@ -51,8 +64,8 @@ two elements will be installed in `site-packages`:: - zlib - zlib-2.5.2-py2.4.egg-info -Where `zlib` is the package, and `zlib-2.5.2-py2.4.egg-info` is -a file containing the package metadata as described in PEP 314. +Where `zlib` is a Python package, and `zlib-2.5.2-py2.4.egg-info` is +a file containing the project metadata as described in PEP 314 [#pep314]_. This file corresponds to the file called `PKG-INFO`, built by the `sdist` command. @@ -63,11 +76,13 @@ packages in the same way that Distutils does: - `easy_install` creates an `EGG-INFO` directory inside an `.egg` directory, and adds a `PKG-INFO` file inside this directory. The `.egg` directory - contains in that case the packages of the project. + contains in that case all the elements of the project that are supposed to + be installed in `site-packages`, and is placed in the `site-packages` + directory. -- `pip` creates an `.egg-info` directory inside the site-packages directory - and adds a `PKG-INFO` file inside it. Packages are installed in - site-packages directory in a regular way. +- `pip` creates an `.egg-info` directory inside the `site-packages` directory + and adds a `PKG-INFO` file inside it. Elements of the project are then + installed in various places like Distutils does. They both add other files in the `EGG-INFO` or `.egg-info` directory, and create or modify `.pth` files. @@ -76,16 +91,19 @@ Uninstall information --------------------- Distutils doesn't provide any `uninstall` command. If you want to uninstall -a project, you have to be a power user and remove the various package -directories from the right `site-packages` directory, then look over the right -`pth` files. And this method differs, depending on the tools you are using. +a project, you have to be a power user and remove the various elements that +were installed. Then look over the `.pth` file to clean them if necessary. -The worst issue is that you depend on the way the packager created his package. -When you call `python setup.py install`, it will not be installed the same way -depending on the tool used by the packager (mainly Distutils or Setuptools). +And the process differs, depending on the tools you have used to install the +project, and if the project's `setup.py` uses Distutils or Setuptools. -But there's common behavior: files are copied in your installation. -And there's a way to keep track of theses file, so to remove them. +Under some circumstances, you might not be able to know for sure that you +have removed everything, or that you didn't break another project by +removing a file that was shared among the two projects. + +But there's common behavior: when you install a project, files are copied +in your system. And there's a way to keep track of theses files, so to remove +them. What this PEP proposes ---------------------- @@ -101,51 +119,58 @@ To address those issues, this PEP proposes a few changes: ============================= The first change would be to make `.egg-info` a directory and let it -hold the `PKG-INFO` file built by the `write_pkg_file` method. +hold the `PKG-INFO` file built by the `write_pkg_file` method of +the `Distribution` class in Distutils. -This change will not impact Python itself, because this file is not -used anywhere yet in the standard library. So there's no need of -deprecation. +This change will not impact Python itself, because `egg-info` files are not +used anywhere yet in the standard library besides Distutils. Although it will impact the `setuptools` and `pip` projects, but given -the fact that they already work with a directory that contains a -`PKG-INFO` file, the change will be small. +the fact that they already work with a directory that contains a `PKG-INFO` +file, the change will have no deep consequences. -For example, if the `zlib` package is installed, two elements -will be installed in `site-packages`:: +For example, if the `zlib` package is installed, the elements that +will be installed in `site-packages` will become:: - zlib - zlib-2.5.2.egg-info/ PKG-INFO -The Python version will also be removed from the .egg-info directory -name. To be able to implement this change, the impacted code in Distutils -is the `install_egg_info` command, and the various third-party projects. +The Python version will also be removed from the `.egg-info` directory +name. Adding a RECORD in the .egg-info directory ========================================== A `RECORD` file will be added inside the `.egg-info` directory at installation -time. +time. The `RECORD` file will hold the list of installed files. These correspond +to the files listed by the `record` option of the `install` command, and will +always be generated. This will allow uninstallation, as explained later in this +PEP. This RECORD file is inspired from PEP 262 FILES [#pep262]_. -- the `RECORD` file will hold the list of installed files. These - correspond to the files listed by the `record` option of the `install` - command, and will always be generated. This will allow uninstallation, as - explained later in this PEP. +The RECORD format +----------------- -The `install` command will record by default installed files in the -RECORD file, using these rules: +The `RECORD` file is composed of records, one line per installed file. +Each record is composed of three elements separated by a `;` character: -- if the installed file is located in a directory in `site-packages`, - it will be a '/'-separated relative path, no matter what is the target - system. This makes this information cross-compatible and allows simple - installation to be relocatable. +- the file's full **path** -- if the installed file is located elsewhere in the system, a - '/'-separated absolute path is used. + - if the installed file is located in a directory in `site-packages`, + it will be a '/'-separated relative path, no matter what is the target + system. This makes this information cross-compatible and allows simple + installation to be relocatable. -This will require changing the way the `install` command writes the record -file, so the old `record` behavior will be deprecated. + - if the installed file is located elsewhere in the system, a + '/'-separated absolute path is used. + +- the **MD5** hash of the file, encoded in hex. Notice that `pyc` and `pyo` + generated files will not have a hash. + +- the file's size in bytes + +Example +------- Back to our `zlib` example, we will have:: @@ -154,6 +179,21 @@ Back to our `zlib` example, we will have:: PKG-INFO RECORD +And the RECORD file will contain:: + + zlib/include/zconf.h;b690274f621402dda63bf11ba5373bf2;9544 + zlib/include/zlib.h;9c4b84aff68aa55f2e9bf70481b94333;66188 + zlib/lib/libz.a;e6d43fb94292411909404b07d0692d46;91128 + zlib/share/man/man3/zlib.3;785dc03452f0508ff0678fba2457e0ba;4486 + zlib-2.5.2.egg-info/PKG-INFO;6fe57de576d749536082d8e205b77748;195 + zlib-2.5.2.egg-info/RECORD + +Notice that: + +- the `RECORD` file can't contain a hash of itself and is just mentioned here +- `zlib` and `zlib-2.5.2.egg-info` are located in `site-packages` so the file + paths are relative to it. + New functions in pkgutil ======================== @@ -178,16 +218,31 @@ The new functions added in the package are : Uses `get_egg_info` to get the `PKG-INFO` file, and returns a `DistributionMetadata` instance that contains the metadata. - This will require a small change in `DistributionMetadata` (see #4908). -- get_egg_info_file(project_name, filename) -> file object or None +- get_files(project_name) -> iterator of (path, hash, size, other_projects) - Uses `get_egg_info` and gets any file inside the directory, - pointed by filename. + Uses `get_egg_info` to get the `RECORD` file, and returns an iterator. + + Each returned element is a tuple `(path, hash, size, other_projects)` where + ``path`, ``hash``, ``size`` are the values found in the RECORD file. + + `other_projects` is a tuple containing the name of the projects that are + also referring to this file in their own RECORD file (same path). + + If `other_projects` is empty, it means that the file is only referred by the + current project. In other words, it can be removed if the project is removed. + +- get_egg_info_file(project_name, path) -> file object or None + + Uses `get_egg_info` and gets any element inside the directory, + pointed by its relative path. `get_egg_info_file` will perform + an `os.path.join` on `get_egg_info(project_name)` and `path` to build the + whole path. Let's use it with our `zlib` example:: - >>> from pkgutil import get_egg_info, get_metadata, get_egg_info_file + >>> from pkgutil import (get_egg_info, get_metadata, get_egg_info_file, + ... get_files) >>> get_egg_info('zlib') '/opt/local/lib/python2.6/site-packages/zlib-2.5.2.egg-info' >>> metadata = get_metadata('zlib') @@ -197,6 +252,16 @@ Let's use it with our `zlib` example:: some ... files + >>> for path, hash, size, other_projects in get_files('zlib'): + ... print '%s %s %d %s' % (path, hash, size, ','.join(other_projects)) + ... + zlib/include/zconf.h b690274f621402dda63bf11ba5373bf2 9544 + zlib/include/zlib.h 9c4b84aff68aa55f2e9bf70481b94333 66188 + zlib/lib/libz.a e6d43fb94292411909404b07d0692d46 91128 + zlib/share/man/man3/zlib.3 785dc03452f0508ff0678fba2457e0ba 4486 + zlib-2.5.2.egg-info/PKG-INFO 6fe57de576d749536082d8e205b77748 195 + zlib-2.5.2.egg-info/RECORD None None + Adding an Uninstall function ============================ @@ -204,14 +269,13 @@ Adding an Uninstall function Distutils provides a very basic way to install a project, which is running the `install` command over the `setup.py` script of the distribution. -Distutils will provide a very basic `uninstall` command that will remove -all files listed in the `RECORD` file of a project, as long as they are not -mentioned in another `RECORD` file and as long as the package is installed -using the standard described earlier. +Distutils will provide a very basic ``uninstall`` function, that will be added +in ``distutils.util`` and will take the name of the project to uninstall as +its argument. ``uninstall`` will use ``pkgutil.get_files`` and remove all +unique files, as long as their hash didn't change. Then it will remove +directories where it removed the last elements. -This command will be added in ``distutils.util`` and will take the name -of the project to uninstall as its argument. A call to uninstall will return a -list of uninstalled files:: +``uninstall`` will return a list of uninstalled files:: >>> from distutils.util import uninstall >>> uninstall('zlib') @@ -220,23 +284,28 @@ list of uninstalled files:: If the project is not found, a ``DistutilsUninstallError`` will be raised. -To make it a reference API for third-party projects that wish to provide -an `uninstall feature`. The ``uninstall`` function can also be invoked with a -second callable argument, that will be invoked for each file to be removed. -If this callable returns `True`, the file will be removed. +To make it a reference API for third-party projects that wish to control +how `uninstall` works, a second callable argument can be used. It will be +called for each file that is removed. If the callable returns `True`, the +file will be removed. If it returns False, it will be left alone. Examples:: >>> def _remove_and_log(path): ... logging.info('Removing %s' % path) ... return True + ... >>> uninstall('zlib', _remove_and_log) >>> def _dry_run(path): ... logging.info('Removing %s (dry run)' % path) ... return False + ... >>> uninstall('zlib', _dry_run) +Of course, a third-party tool can use ``pkgutil.get_files`` for a maximum +control, to implement their own uninstall feature. + Backward compatibility and roadmap ================================== @@ -244,10 +313,20 @@ These changes will not introduce any compatibility problems with the previous version of Distutils, and will also work with existing third-party tools. Although, a backport of the new Distutils for 2.5, 2.6, 3.0 and 3.1 will be -provided so people can benefit from the new features. +provided so people can benefit from these new features. The plan is to integrate them for Python 2.7 and Python 3.2 +References +========== + +.. [#pep262] + http://www.python.org/dev/peps/pep-0262 + +.. [#pep314] + http://www.python.org/dev/peps/pep-0314 + + Aknowledgments ==============