python-peps/pep-0507/index.html

424 lines
26 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 507 Migrate CPython to Git and GitLab | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0507/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 507 Migrate CPython to Git and GitLab | peps.python.org'>
<meta property="og:description" content="This PEP proposes migrating the repository hosting of CPython and the supporting repositories to Git. Further, it proposes adopting a hosted GitLab instance as the primary way of handling merge requests, code reviews, and code hosting. It is similar i...">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0507/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP proposes migrating the repository hosting of CPython and the supporting repositories to Git. Further, it proposes adopting a hosted GitLab instance as the primary way of handling merge requests, code reviews, and code hosting. It is similar i...">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 507</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 507 Migrate CPython to Git and GitLab</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Barry Warsaw &lt;barry&#32;&#97;t&#32;python.org&gt;</dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP describing or proposing a change to a Python community process, workflow or governance">Process</abbr></dd>
<dt class="field-even">Created<span class="colon">:</span></dt>
<dd class="field-even">30-Sep-2015</dd>
<dt class="field-odd">Post-History<span class="colon">:</span></dt>
<dd class="field-odd"><p></p></dd>
<dt class="field-even">Resolution<span class="colon">:</span></dt>
<dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/core-workflow/2016-January/000345.html">Core-Workflow message</a></dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#version-control-system">Version Control System</a></li>
<li><a class="reference internal" href="#repository-hosting">Repository Hosting</a></li>
<li><a class="reference internal" href="#code-review">Code Review</a></li>
<li><a class="reference internal" href="#gitlab-merge-requests">GitLab merge requests</a></li>
</ul>
</li>
<li><a class="reference internal" href="#criticism">Criticism</a><ul>
<li><a class="reference internal" href="#x-is-not-written-in-python">X is not written in Python</a></li>
<li><a class="reference internal" href="#mercurial-is-better-than-git">Mercurial is better than Git</a></li>
<li><a class="reference internal" href="#cpython-workflow-is-too-complicated">CPython Workflow is too Complicated</a></li>
</ul>
</li>
<li><a class="reference internal" href="#open-issues">Open issues</a></li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git. Further, it proposes adopting a
hosted GitLab instance as the primary way of handling merge requests,
code reviews, and code hosting. It is similar in intent to <a class="pep reference internal" href="../pep-0481/" title="PEP 481 Migrate CPython to Git, Github, and Phabricator">PEP 481</a>
but proposes an open source alternative to GitHub and omits the
proposal to run Phabricator. As with <a class="pep reference internal" href="../pep-0481/" title="PEP 481 Migrate CPython to Git, Github, and Phabricator">PEP 481</a>, this particular PEP is
offered as an alternative to <a class="pep reference internal" href="../pep-0474/" title="PEP 474 Creating forge.python.org">PEP 474</a> and <a class="pep reference internal" href="../pep-0462/" title="PEP 462 Core development workflow automation for CPython">PEP 462</a>.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>CPython is an open source project which relies on a number of
volunteers donating their time. As with any healthy, vibrant open
source project, it relies on attracting new volunteers as well as
retaining existing developers. Given that volunteer time is the most
scarce resource, providing a process that maximizes the efficiency of
contributors and reduces the friction for contributions, is of vital
importance for the long-term health of the project.</p>
<p>The current tool chain of the CPython project is a custom and unique
combination of tools. This has two critical implications:</p>
<ul class="simple">
<li>The unique nature of the tool chain means that contributors must
remember or relearn, the process, workflow, and tools whenever they
contribute to CPython, without the advantage of leveraging long-term
memory and familiarity they retain by working with other projects in
the FLOSS ecosystem. The knowledge they gain in working with
CPython is unlikely to be applicable to other projects.</li>
<li>The burden on the Python/PSF infrastructure team is much greater in
order to continue to maintain custom tools, improve them over time,
fix bugs, address security issues, and more generally adapt to new
standards in online software development with global collaboration.</li>
</ul>
<p>These limitations act as a barrier to contribution both for highly
engaged contributors (e.g. core Python developers) and especially for
more casual “drive-by” contributors, who care more about getting their
bug fix than learning a new suite of tools and workflows.</p>
<p>By proposing the adoption of both a different version control system
and a modern, well-maintained hosting solution, this PEP addresses
these limitations. It aims to enable a modern, well-understood
process that will carry CPython development for many years.</p>
<section id="version-control-system">
<h3><a class="toc-backref" href="#version-control-system" role="doc-backlink">Version Control System</a></h3>
<p>Currently the CPython and supporting repositories use Mercurial. As a
modern distributed version control system, it has served us well since
the migration from Subversion. However, when evaluating the VCS we
must consider the capabilities of the VCS itself as well as the
network effect and mindshare of the community around that VCS.</p>
<p>There are really only two real options for this, Mercurial and Git.
The technical capabilities of the two systems are largely equivalent,
therefore this PEP instead focuses on their social aspects.</p>
<p>It is not possible to get exact numbers for the number of projects or
people which are using a particular VCS, however we can infer this by
looking at several sources of information for what VCS projects are
using.</p>
<p>The Open Hub (previously Ohloh) statistics <a class="footnote-reference brackets" href="#openhub-stats" id="id1">[1]</a> show that
37% of the repositories indexed by The Open Hub are using Git (second
only to Subversion which has 48%) while Mercurial has just 2%, beating
only Bazaar which has 1%. This has Git being just over 18 times as
popular as Mercurial on The Open Hub.</p>
<p>Another source of information on VCS popularity is PyPI itself. This
source is more targeted at the Python community itself since it
represents projects developed for Python. Unfortunately PyPI does not
have a standard location for representing this information, so this
requires manual processing. If we limit our search to the top 100
projects on PyPI (ordered by download counts) we can see that 62% of
them use Git, while 22% of them use Mercurial, and 13% use something
else. This has Git being just under 3 times as popular as Mercurial
for the top 100 projects on PyPI.</p>
<p>These numbers back up the anecdotal evidence for Git as the far more
popular DVCS for open source projects. Choosing the more popular VCS
has a number of positive benefits.</p>
<p>For new contributors it increases the likelihood that they will have already
learned the basics of Git as part of working with another project or if they
are just now learning Git, that theyll be able to take that knowledge and
apply it to other projects. Additionally a larger community means more people
writing how to guides, answering questions, and writing articles about Git
which makes it easier for a new user to find answers and information about the
tool they are trying to learn and use. Given its popularity, there may also
be more auxiliary tooling written <em>around</em> Git. This increases options for
everything from GUI clients, helper scripts, repository hosting, etc.</p>
<p>Further, the adoption of Git as the proposed back-end repository
format doesnt prohibit the use of Mercurial by fans of that VCS!
Mercurial users have the <a class="footnote-reference brackets" href="#hg-git" id="id2">[2]</a> plugin which allows them to push
and pull from a Git server using the Mercurial front-end. Its a
well-maintained and highly functional plugin that seems to be
well-liked by Mercurial users.</p>
</section>
<section id="repository-hosting">
<h3><a class="toc-backref" href="#repository-hosting" role="doc-backlink">Repository Hosting</a></h3>
<p>Where and how the official repositories for CPython are hosted is in
someways determined by the choice of VCS. With Git there are several
options. In fact, once the repository is hosted in Git, branches can
be mirrored in many locations, within many free, open, and proprietary
code hosting sites.</p>
<p>Its still important for CPython to adopt a single, official
repository, with a web front-end that allows for many convenient and
common interactions entirely through the web, without always requiring
local VCS manipulations. These interactions include as a minimum,
code review with inline comments, branch diffing, CI integration, and
auto-merging.</p>
<p>This PEP proposes to adopt a <a class="footnote-reference brackets" href="#gitlab" id="id3">[3]</a> instance, run within the
python.org domain, accessible to and with ultimate control from the
PSF and the Python infrastructure team, but donated, hosted, and
primarily maintained by GitLab, Inc.</p>
<p>Why GitLab? Because it is a fully functional Git hosting system, that
sports modern web interactions, software workflows, and CI
integration. GitLabs Community Edition (CE) is open source software,
and thus is closely aligned with the principles of the CPython
community.</p>
</section>
<section id="code-review">
<h3><a class="toc-backref" href="#code-review" role="doc-backlink">Code Review</a></h3>
<p>Currently CPython uses a custom fork of Rietveld modified to not run
on Google App Engine and which is currently only really maintained by
one person. It is missing common features present in many modern code
review tools.</p>
<p>This PEP proposes to utilize GitLabs built-in merge requests and
online code review features to facilitate reviews of all proposed
changes.</p>
</section>
<section id="gitlab-merge-requests">
<h3><a class="toc-backref" href="#gitlab-merge-requests" role="doc-backlink">GitLab merge requests</a></h3>
<p>The normal workflow for a GitLab hosted project is to submit a <em>merge request</em>
asking that a feature or bug fix branch be merged into a target branch,
usually one or more of the stable maintenance branches or the next-version
master branch for new features. GitLabs merge requests are similar in form
and function to GitHubs pull requests, so anybody who is already familiar
with the latter should be able to immediately utilize the former.</p>
<p>Once submitted, a conversation about the change can be had between the
submitter and reviewer. This includes both general comments, and inline
comments attached to a particular line of the diff between the source and
target branches. Projects can also be configured to automatically run
continuous integration on the submitted branch, the results of which are
readily visible from the merge request page. Thus both the reviewer and
submitter can immediately see the results of the tests, making it much easier
to only land branches with passing tests. Each new push to the source branch
(e.g. to respond to a commenters feedback or to fix a failing test) results
in a new run of the CI, so that the state of the request always reflects the
latest commit.</p>
<p>Merge requests have a fairly major advantage over the older “submit a patch to
a bug tracker” model. They allow developers to work completely within the VCS
using standard VCS tooling, without requiring the creation of a patch file or
figuring out the right location to upload the patch to. This lowers the
barrier for sending a change to be reviewed.</p>
<p>Merge requests are far easier to review. For example, they provide nice
syntax highlighted diffs which can operate in either unified or side by side
views. They allow commenting inline and on the merge request as a whole and
they present that in a nice unified way which will also hide comments which no
longer apply. Comments can be hidden and revealed.</p>
<p>Actually merging a merge request is quite simple, if the source branch applies
cleanly to the target branch. A core reviewer simply needs to press the
“Merge” button for GitLab to automatically perform the merge. The source
branch can be optionally rebased, and once the merge is completed, the source
branch can be automatically deleted.</p>
<p>GitLab also has a good workflow for submitting pull requests to a project
completely through their web interface. This would enable the Python
documentation to have “Edit on GitLab” buttons on every page and people who
discover things like typos, inaccuracies, or just want to make improvements to
the docs they are currently reading. They can simply hit that button and get
an in browser editor that will let them make changes and submit a merge
request all from the comfort of their browser.</p>
</section>
</section>
<section id="criticism">
<h2><a class="toc-backref" href="#criticism" role="doc-backlink">Criticism</a></h2>
<section id="x-is-not-written-in-python">
<h3><a class="toc-backref" href="#x-is-not-written-in-python" role="doc-backlink">X is not written in Python</a></h3>
<p>One feature that the current tooling (Mercurial, Rietveld) has is that the
primary language for all of the pieces are written in Python. This PEP
focuses more on the <em>best</em> tools for the job and not necessarily on the <em>best</em>
tools that happen to be written in Python. Volunteer time is the most
precious resource for any open source project and we can best respect and
utilize that time by focusing on the benefits and downsides of the tools
themselves rather than what language their authors happened to write them in.</p>
<p>One concern is the ability to modify tools to work for us, however one of the
Goals here is to <em>not</em> modify software to work for us and instead adapt
ourselves to a more standardized workflow. This standardization pays off in
the ability to re-use tools out of the box freeing up developer time to
actually work on Python itself as well as enabling knowledge sharing between
projects.</p>
<p>However, if we do need to modify the tooling, Git itself is largely written in
C the same as CPython itself. It can also have commands written for it using
any language, including Python. GitLab itself is largely written in Ruby and
since it is Open Source software, we would have the ability to submit merge
requests to the upstream Community Edition, albeit in language potentially
unfamiliar to most Python programmers.</p>
</section>
<section id="mercurial-is-better-than-git">
<h3><a class="toc-backref" href="#mercurial-is-better-than-git" role="doc-backlink">Mercurial is better than Git</a></h3>
<p>Whether Mercurial or Git is better on a technical level is a highly subjective
opinion. This PEP does not state whether the mechanics of Git or Mercurial
are better, and instead focuses on the network effect that is available for
either option. While this PEP proposes switching to Git, Mercurial users are
not left completely out of the loop. By using the hg-git extension for
Mercurial, working with server-side Git repositories is fairly easy and
straightforward.</p>
</section>
<section id="cpython-workflow-is-too-complicated">
<h3><a class="toc-backref" href="#cpython-workflow-is-too-complicated" role="doc-backlink">CPython Workflow is too Complicated</a></h3>
<p>One sentiment that came out of previous discussions was that the multi-branch
model of CPython was too complicated for GitLab style merge requests. This
PEP disagrees with that sentiment.</p>
<p>Currently any particular change requires manually creating a patch for 2.7 and
3.x which wont change at all in this regards.</p>
<p>If someone submits a fix for the current stable branch (e.g. 3.5) the merge
request workflow can be used to create a request to merge the current stable
branch into the master branch, assuming there is no merge conflicts. As
always, merge conflicts must be manually and locally resolved. Because
developers also have the <em>option</em> of performing the merge locally, this
provides an improvement over the current situation where the merge <em>must</em>
always happen locally.</p>
<p>For fixes in the current development branch that must also be applied to
stable release branches, it is possible in many situations to locally cherry
pick and apply the change to other branches, with merge requests submitted for
each stable branch. It is also possible just cherry pick and complete the
merge locally. These are all accomplished with standard Git commands and
techniques, with the advantage that all such changes can go through the review
and CI test workflows, even for merges to stable branches. Minor changes may
be easily accomplished in the GitLab web editor.</p>
<p>No system can hide all the complexities involved in maintaining several long
lived branches. The only thing that the tooling can do is make it as easy as
possible to submit and commit changes.</p>
</section>
</section>
<section id="open-issues">
<h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open issues</a></h2>
<ul class="simple">
<li>What level of hosted support will GitLab offer? The PEP author has been in
contact with the GitLab CEO, with positive interest on their part. The
details of the hosting offer would have to be discussed.</li>
<li>What happens to Roundup and do we switch to the GitLab issue tracker?
Currently, this PEP is <em>not</em> suggesting we move from Roundup to GitLab
issues. We have way too much invested in Roundup right now and migrating
the data would be a huge effort. GitLab does support webhooks, so we will
probably want to use webhooks to integrate merges and other events with
updates to Roundup (e.g. to include pointers to commits, close issues,
etc. similar to what is currently done).</li>
<li>What happens to wiki.python.org? Nothing! While GitLab does support wikis
in repositories, theres no reason for us to migration our Moin wikis.</li>
<li>What happens to the existing GitHub mirrors? Wed probably want to
regenerate them once the official upstream branches are natively hosted in
Git. This may change commit ids, but after that, it should be easy to
mirror the official Git branches and repositories far and wide.</li>
<li>Where would the GitLab instance live? Physically, in whatever hosting
provider GitLab chooses. We would point gitlab.python.org (or
git.python.org?) to this host.</li>
</ul>
</section>
<section id="references">
<h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="openhub-stats" role="doc-footnote">
<dt class="label" id="openhub-stats">[<a href="#id1">1</a>]</dt>
<dd><a class="reference external" href="https://www.openhub.net/repositories/compare">Open Hub Statistics</a></aside>
<aside class="footnote brackets" id="hg-git" role="doc-footnote">
<dt class="label" id="hg-git">[<a href="#id2">2</a>]</dt>
<dd><a class="reference external" href="https://hg-git.github.io/">Hg-Git mercurial plugin</a></aside>
<aside class="footnote brackets" id="gitlab" role="doc-footnote">
<dt class="label" id="gitlab">[<a href="#id3">3</a>]</dt>
<dd><a class="reference external" href="https://about.gitlab.com">https://about.gitlab.com</a></aside>
</aside>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0507.rst">https://github.com/python/peps/blob/main/peps/pep-0507.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0507.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#version-control-system">Version Control System</a></li>
<li><a class="reference internal" href="#repository-hosting">Repository Hosting</a></li>
<li><a class="reference internal" href="#code-review">Code Review</a></li>
<li><a class="reference internal" href="#gitlab-merge-requests">GitLab merge requests</a></li>
</ul>
</li>
<li><a class="reference internal" href="#criticism">Criticism</a><ul>
<li><a class="reference internal" href="#x-is-not-written-in-python">X is not written in Python</a></li>
<li><a class="reference internal" href="#mercurial-is-better-than-git">Mercurial is better than Git</a></li>
<li><a class="reference internal" href="#cpython-workflow-is-too-complicated">CPython Workflow is too Complicated</a></li>
</ul>
</li>
<li><a class="reference internal" href="#open-issues">Open issues</a></li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0507.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>