python-peps/pep-0347.txt

PEP: 347
Title: Migrating the Python CVS to Subversion
Version: $Revision$
Last-Modified: $Date$
Author: Martin v. L<>wis <martin@v.loewis.de>
Discussions-To: <python-dev@python.org>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 14-Jul-2004
Post-History: 14-Jul-2004


Abstract
========

The Python source code is currently managed in a CVS repository on
sourceforge.net.  This PEP proposes to move it to a Subversion
repository on svn.python.org.


Rationale
=========

This change has two aspects: moving from CVS to Subversion, and moving
from SourceForge to python.org.  For each, a rationale will be given.


Moving to Subversion
--------------------

CVS has a number of limitations that have been eliminated by
Subversion.  For the development of Python, the most notable
improvements are:

- the ability to rename files and directories, and to remove
  directories, while keeping the history of these files.

- support for change sets (sets of correlated changes to multiple
  files) through global revision numbers.  Change sets are
  transactional.

- atomic, fast tagging: a cvs tag might take many minutes; a
  Subversion tag (svn cp) will complete quickly, and atomically.
  Likewise, branches are very efficient.

- support for offline diffs, which is useful when creating patches.


Moving to python.org
--------------------

SourceForge has kindly provided an important infrastructure for the
past years.  Unfortunately, the attention that SF received has also
caused repeated overload situations in the past, to which the SF
operators could not always respond in a timely manner.  In particular,
for CVS, they had to reduce the load on the primary CVS server by
introducing a second, read-only CVS server for anonymous access.  This
server is regularly synchronized, but lags behind the the read-write
CVS repository between synchronizations.  As a result, users without
commit access can see recent changes to the repository only after a
delay.

On python.org, it would be possible to make the repository accessible
for anonymous access.


Migration Procedure
===================

To move the Python CVS repository, the following steps need to be
executed.  The steps are elaborated upon in the following sections.

1. Assign passwords for all current committers for use on
   svn.python.org.  User names on SF and svn.python.org should be
   identical, unless some committer requests a different user name.

2. At the beginning of the migration, announce that the repository on
   SourceForge closed.

3. 24 hours after the last commit, download the CVS repository.

4. Convert the CVS repository into a Subversion repository.

5. Publish the repository with write access for committers, and
   read-only anonymous access.

6. Disable CVS access on SF.


Assign Passwords
----------------

Currently, access to Subversion on svn.python.org uses WebDAV over
https, using basic authentication.  For this to work, authorized users
need to provide a password.  This mechanism should be used, at least
initially, for the Python CVS as well, since various committers also
have a username/password pair for the www SVN repository already.


Downloading the CVS Repository
------------------------------

The CVS repository can be downloaded from

    http://cvs.sourceforge.net/cvstarballs/python-cvsroot.tar.bz2

Since this tarball is generated only once a day, some time must pass
after the repository freeze before the tarball can be picked up.  It
should be verified that the last commit, as recorded on the
python-commits mailing list, is indeed included in the tarball.

After the conversion, the converted CVS tarball should be kept
forever on www.python.org/archive/python-cvsroot-<date>.tar.bz2


Converting the CVS Repository
-----------------------------

The Python CVS repository contains two modules: distutils and python.
The python module is further structured into dist and nondist,
where dist only contains src (the python code proper). nondist
contains various subdirectories.

These should be reorganized in the Subversion repository to get
shorter URLs, following the <project>/{trunk,tags,branches}
structure.  A project will be created for each nondist directory,
plus for src (called python), plus distutils.  Reorganizing the
repository is best done in the CVS tree, as shown below.

The fsfs backend should be used as the repository format (which
requires Subversion 1.1).  The fsfs has the advantage of being more
backup-friendly, as it allows incremental repository backups, without
requiring any dump commands to be run.

The conversion should be done using the cvs2svn utility, available
e.g. in the cvs2svn Debian package.  As cvs2svn does not currently
support the project/trunk structure, each project needs to be
converted separately.  To get each conversion result into a separate
directory in the target repository, svnadmin load must be used.
In summary, the conversion script is::

  #!/bin/sh
  rm cvs2svn-*
  rm -rf python py.new
  tar xjf python-cvsroot.tar.bz2
  rm -rf python/CVSROOT
  svnadmin create --fs-type fsfs py.new
  mv python/python python/orig
  mv python/orig/dist/src python/python
  mv python/orig/nondist/* python
  # nondist/nondist is empty
  rmdir python/nondist
  rm -rf python/orig
  for a in python/*
  do
    b=`basename $a`
    cvs2svn -q --dump-only --encoding=latin1 --force-branch=cnri-16-start \
    --force-branch=descr-branch --force-branch=release152p1-patches \
    --force-tag=r16b1 $a
    svn mkdir -m"Conversion to SVN" file:///`pwd`/py.new/$b
    svnadmin load -q --parent-dir $b py.new < cvs2svn-dump
    rm cvs2svn-dump
  done

Sample results of this conversion are available at

    http://www.dcl.hpi.uni-potsdam.de/pysvn/


Publish the Repository
------------------------

The repository should be published at https://svn.python.org/projects.
Read-write access should be granted through basic authentication to
all current SF committers; read-only anonymous access should also be
granted.

As an option, websvn (available e.g. from the Debian websvn package)
could be provided. Unfortunately, in the test installation, websvn
breaks because it runs out of memory.

The current SF project admins should get write access to the password
file, in order to create or delete new users.


Disable CVS
-----------

It appears that CVS cannot be disabled entirely.  Only the user
interface can be removed from the project page; the repository itself
remains available.  If desired, write access to the python and
distutils modules can be disabled through a CVS commitinfo entry.


Discussion
----------

Several alternatives had been suggested to the procedure above.
The rejected alternatives are shortly discussed here:

- create multiple repositories, one for python and one for
  distutils. This would have allowed even shorter URLs, but
  was rejected because a single repository supports moving code
  across projects.

- Nick Bastin has offered to host a Perforce repository, instead
  of using Subversion. Details of that offer are not clear yet.

- Several people suggested to create the project/trunk structure
  through standard cvs2svn, followed by renames. This would have
  the disadvantage that old revisions use different path names
  than recent revisions; the suggested approach through dump files
  works without renames.

- Several people also expressed concern about the administrative
  overhead that hosting the repository on python.org would cause
  to pydotorg admins.  As a specific alternative, BerliOS has been
  suggested.  The pydotorg admins themselves haven\'t objected
  to the additional workload; migrating the repository again if
  they get overworked is an option.

- People have expressed dislike of the basic auth (username/password)
  authentication.  Two alternatives have been suggested:

  * Subversion over SSH, using SSH key pairs for a single Unix
    account, restricted to the execution of svnserve.  The pydotorg
    admins have ruled out creation of one account per committer;
    whether a single account would be acceptable is not yet
    decided.

  * Subversion over WebDAV, using SSL client certificates. This would
    work, but would require us to administer a certificate authority.


Copyright
---------

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: