python-peps/pep-0507.txt

PEP: 507
Title: Migrate CPython to Git and GitLab
Version: $Revision$
Last-Modified: $Date$
Author: Barry Warsaw <barry@python.org>
Status: Rejected
Type: Process
Content-Type: text/x-rst
Created: 2015-09-30
Post-History:
Resolution: https://mail.python.org/pipermail/core-workflow/2016-January/000345.html


Abstract
========

This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git.  Further, it proposes adopting a
hosted GitLab instance as the primary way of handling merge requests,
code reviews, and code hosting.  It is similar in intent to PEP 481
but proposes an open source alternative to GitHub and omits the
proposal to run Phabricator.  As with PEP 481, this particular PEP is
offered as an alternative to PEP 474 and PEP 462.


Rationale
=========

CPython is an open source project which relies on a number of
volunteers donating their time.  As with any healthy, vibrant open
source project, it relies on attracting new volunteers as well as
retaining existing developers.  Given that volunteer time is the most
scarce resource, providing a process that maximizes the efficiency of
contributors and reduces the friction for contributions, is of vital
importance for the long-term health of the project.

The current tool chain of the CPython project is a custom and unique
combination of tools.  This has two critical implications:

* The unique nature of the tool chain means that contributors must
  remember or relearn, the process, workflow, and tools whenever they
  contribute to CPython, without the advantage of leveraging long-term
  memory and familiarity they retain by working with other projects in
  the FLOSS ecosystem.  The knowledge they gain in working with
  CPython is unlikely to be applicable to other projects.

* The burden on the Python/PSF infrastructure team is much greater in
  order to continue to maintain custom tools, improve them over time,
  fix bugs, address security issues, and more generally adapt to new
  standards in online software development with global collaboration.

These limitations act as a barrier to contribution both for highly
engaged contributors (e.g. core Python developers) and especially for
more casual "drive-by" contributors, who care more about getting their
bug fix than learning a new suite of tools and workflows.

By proposing the adoption of both a different version control system
and a modern, well-maintained hosting solution, this PEP addresses
these limitations.  It aims to enable a modern, well-understood
process that will carry CPython development for many years.


Version Control System
----------------------

Currently the CPython and supporting repositories use Mercurial.  As a
modern distributed version control system, it has served us well since
the migration from Subversion.  However, when evaluating the VCS we
must consider the capabilities of the VCS itself as well as the
network effect and mindshare of the community around that VCS.

There are really only two real options for this, Mercurial and Git.
The technical capabilities of the two systems are largely equivalent,
therefore this PEP instead focuses on their social aspects.

It is not possible to get exact numbers for the number of projects or
people which are using a particular VCS, however we can infer this by
looking at several sources of information for what VCS projects are
using.

The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that
37% of the repositories indexed by The Open Hub are using Git (second
only to Subversion which has 48%) while Mercurial has just 2%, beating
only Bazaar which has 1%.  This has Git being just over 18 times as
popular as Mercurial on The Open Hub.

Another source of information on VCS popularity is PyPI itself. This
source is more targeted at the Python community itself since it
represents projects developed for Python.  Unfortunately PyPI does not
have a standard location for representing this information, so this
requires manual processing.  If we limit our search to the top 100
projects on PyPI (ordered by download counts) we can see that 62% of
them use Git, while 22% of them use Mercurial, and 13% use something
else.  This has Git being just under 3 times as popular as Mercurial
for the top 100 projects on PyPI.

These numbers back up the anecdotal evidence for Git as the far more
popular DVCS for open source projects.  Choosing the more popular VCS
has a number of positive benefits.

For new contributors it increases the likelihood that they will have already
learned the basics of Git as part of working with another project or if they
are just now learning Git, that they'll be able to take that knowledge and
apply it to other projects.  Additionally a larger community means more people
writing how to guides, answering questions, and writing articles about Git
which makes it easier for a new user to find answers and information about the
tool they are trying to learn and use.  Given its popularity, there may also
be more auxiliary tooling written *around* Git.  This increases options for
everything from GUI clients, helper scripts, repository hosting, etc.

Further, the adoption of Git as the proposed back-end repository
format doesn't prohibit the use of Mercurial by fans of that VCS!
Mercurial users have the [#hg-git]_ plugin which allows them to push
and pull from a Git server using the Mercurial front-end.  It's a
well-maintained and highly functional plugin that seems to be
well-liked by Mercurial users.


Repository Hosting
------------------

Where and how the official repositories for CPython are hosted is in
someways determined by the choice of VCS.  With Git there are several
options.  In fact, once the repository is hosted in Git, branches can
be mirrored in many locations, within many free, open, and proprietary
code hosting sites.

It's still important for CPython to adopt a single, official
repository, with a web front-end that allows for many convenient and
common interactions entirely through the web, without always requiring
local VCS manipulations.  These interactions include as a minimum,
code review with inline comments, branch diffing, CI integration, and
auto-merging.

This PEP proposes to adopt a [#GitLab]_ instance, run within the
python.org domain, accessible to and with ultimate control from the
PSF and the Python infrastructure team, but donated, hosted, and
primarily maintained by GitLab, Inc.

Why GitLab?  Because it is a fully functional Git hosting system, that
sports modern web interactions, software workflows, and CI
integration.  GitLab's Community Edition (CE) is open source software,
and thus is closely aligned with the principles of the CPython
community.


Code Review
-----------

Currently CPython uses a custom fork of Rietveld modified to not run
on Google App Engine and which is currently only really maintained by
one person.  It is missing common features present in many modern code
review tools.

This PEP proposes to utilize GitLab's built-in merge requests and
online code review features to facilitate reviews of all proposed
changes.


GitLab merge requests
---------------------

The normal workflow for a GitLab hosted project is to submit a *merge request*
asking that a feature or bug fix branch be merged into a target branch,
usually one or more of the stable maintenance branches or the next-version
master branch for new features.  GitLab's merge requests are similar in form
and function to GitHub's pull requests, so anybody who is already familiar
with the latter should be able to immediately utilize the former.

Once submitted, a conversation about the change can be had between the
submitter and reviewer.  This includes both general comments, and inline
comments attached to a particular line of the diff between the source and
target branches.  Projects can also be configured to automatically run
continuous integration on the submitted branch, the results of which are
readily visible from the merge request page.  Thus both the reviewer and
submitter can immediately see the results of the tests, making it much easier
to only land branches with passing tests.  Each new push to the source branch
(e.g. to respond to a commenter's feedback or to fix a failing test) results
in a new run of the CI, so that the state of the request always reflects the
latest commit.

Merge requests have a fairly major advantage over the older "submit a patch to
a bug tracker" model.  They allow developers to work completely within the VCS
using standard VCS tooling, without requiring the creation of a patch file or
figuring out the right location to upload the patch to.  This lowers the
barrier for sending a change to be reviewed.

Merge requests are far easier to review.  For example, they provide nice
syntax highlighted diffs which can operate in either unified or side by side
views.  They allow commenting inline and on the merge request as a whole and
they present that in a nice unified way which will also hide comments which no
longer apply.  Comments can be hidden and revealed.

Actually merging a merge request is quite simple, if the source branch applies
cleanly to the target branch.  A core reviewer simply needs to press the
"Merge" button for GitLab to automatically perform the merge.  The source
branch can be optionally rebased, and once the merge is completed, the source
branch can be automatically deleted.

GitLab also has a good workflow for submitting pull requests to a project
completely through their web interface.  This would enable the Python
documentation to have "Edit on GitLab" buttons on every page and people who
discover things like typos, inaccuracies, or just want to make improvements to
the docs they are currently reading.  They can simply hit that button and get
an in browser editor that will let them make changes and submit a merge
request all from the comfort of their browser.


Criticism
=========

X is not written in Python
--------------------------

One feature that the current tooling (Mercurial, Rietveld) has is that the
primary language for all of the pieces are written in Python.  This PEP
focuses more on the *best* tools for the job and not necessarily on the *best*
tools that happen to be written in Python.  Volunteer time is the most
precious resource for any open source project and we can best respect and
utilize that time by focusing on the benefits and downsides of the tools
themselves rather than what language their authors happened to write them in.

One concern is the ability to modify tools to work for us, however one of the
Goals here is to *not* modify software to work for us and instead adapt
ourselves to a more standardized workflow.  This standardization pays off in
the ability to re-use tools out of the box freeing up developer time to
actually work on Python itself as well as enabling knowledge sharing between
projects.

However if we do need to modify the tooling, Git itself is largely written in
C the same as CPython itself.  It can also have commands written for it using
any language, including Python.  GitLab itself is largely written in Ruby and
since it is Open Source software, we would have the ability to submit merge
requests to the upstream Community Edition, albeit in language potentially
unfamiliar to most Python programmers.


Mercurial is better than Git
----------------------------

Whether Mercurial or Git is better on a technical level is a highly subjective
opinion.  This PEP does not state whether the mechanics of Git or Mercurial
are better, and instead focuses on the network effect that is available for
either option.  While this PEP proposes switching to Git, Mercurial users are
not left completely out of the loop.  By using the hg-git extension for
Mercurial, working with server-side Git repositories is fairly easy and
straightforward.


CPython Workflow is too Complicated
-----------------------------------

One sentiment that came out of previous discussions was that the multi-branch
model of CPython was too complicated for GitLab style merge requests.  This
PEP disagrees with that sentiment.

Currently any particular change requires manually creating a patch for 2.7 and
3.x which won't change at all in this regards.

If someone submits a fix for the current stable branch (e.g. 3.5) the merge
request workflow can be used to create a request to merge the current stable
branch into the master branch, assuming there is no merge conflicts.  As
always, merge conflicts must be manually and locally resolved.  Because
developers also have the *option* of performing the merge locally, this
provides an improvement over the current situation where the merge *must*
always happen locally.

For fixes in the current development branch that must also be applied to
stable release branches, it is possible in many situations to locally cherry
pick and apply the change to other branches, with merge requests submitted for
each stable branch.  It is also possible just cherry pick and complete the
merge locally.  These are all accomplished with standard Git commands and
techniques, with the advantage that all such changes can go through the review
and CI test workflows, even for merges to stable branches.  Minor changes may
be easily accomplished in the GitLab web editor.

No system can hide all the complexities involved in maintaining several long
lived branches.  The only thing that the tooling can do is make it as easy as
possible to submit and commit changes.


Open issues
===========

* What level of hosted support will GitLab offer?  The PEP author has been in
  contact with the GitLab CEO, with positive interest on their part.  The
  details of the hosting offer would have to be discussed.

* What happens to Roundup and do we switch to the GitLab issue tracker?
  Currently, this PEP is *not* suggesting we move from Roundup to GitLab
  issues.  We have way too much invested in Roundup right now and migrating
  the data would be a huge effort.  GitLab does support webhooks, so we will
  probably want to use webhooks to integrate merges and other events with
  updates to Roundup (e.g. to include pointers to commits, close issues,
  etc. similar to what is currently done).

* What happens to wiki.python.org?  Nothing!  While GitLab does support wikis
  in repositories, there's no reason for us to migration our Moin wikis.

* What happens to the existing GitHub mirrors?  We'd probably want to
  regenerate them once the official upstream branches are natively hosted in
  Git.  This may change commit ids, but after that, it should be easy to
  mirror the official Git branches and repositories far and wide.

* Where would the GitLab instance live?  Physically, in whatever hosting
  provider GitLab chooses.  We would point gitlab.python.org (or
  git.python.org?) to this host.


References
==========

.. [#openhub-stats] `Open Hub Statistics <https://www.openhub.net/repositories/compare>`
.. [#hg-git] `Hg-Git mercurial plugin <https://hg-git.github.io/>`
.. [#GitLab] `https://about.gitlab.com/`


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: