321 lines
14 KiB
Plaintext
321 lines
14 KiB
Plaintext
PEP: 474
|
|
Title: Creating forge.python.org
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Nick Coghlan <ncoghlan@gmail.com>
|
|
Status: Draft
|
|
Type: Process
|
|
Content-Type: text/x-rst
|
|
Created: 19-Jul-2014
|
|
Post-History: 19-Jul-2014, 08-Jan-2015
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes setting up a new PSF provided resource, forge.python.org,
|
|
as a location for maintaining various supporting repositories
|
|
(such as the repository for Python Enhancement Proposals) in a way that is
|
|
more accessible to new contributors, and easier to manage for core
|
|
developers.
|
|
|
|
This PEP does *not* propose any changes to the core development workflow
|
|
for CPython itself (see PEP 462 in relation to that).
|
|
|
|
|
|
Proposal
|
|
========
|
|
|
|
This PEP proposes that an instance of the self-hosted Kallithea code
|
|
repository management system be deployed as "forge.python.org".
|
|
|
|
Individual repositories (such as the developer guide or the PEPs repository)
|
|
may then be migrated from the existing hg.python.org infrastructure to the
|
|
new forge.python.org infrastructure on a case by case basis. Each migration
|
|
will need to decide whether to retain a read-only mirror on hg.python.org,
|
|
or whether to just migrate wholesale to the new location.
|
|
|
|
In addition to supporting read-only mirrors on hg.python.org,
|
|
forge.python.org will also aim to support hosting mirrors on popular
|
|
proprietary hosting sites like GitHub and BitBucket. The aim will be to
|
|
allow users familiar with these sites to submit and discuss pull requests
|
|
using their preferred workflow, with forge.python.org automatically bringing
|
|
those contributions over to the master repository.
|
|
|
|
Given the availability and popularity of commercially backed "free for open
|
|
source projects" repository hosting services, this would not be a general
|
|
purpose hosting site for arbitrary Python projects. The initial focus will be
|
|
specifically on CPython and other repositories currently hosted on
|
|
hg.python.org. In the future, this could potentially be expanded to
|
|
consolidating other PSF managed repositories that are currently externally
|
|
hosted to gain access to a pull request based workflow, such as the
|
|
repository for the python.org Django application. As with the initial
|
|
migrations, any such future migrations would be considered on a case-by-case
|
|
basis, taking into account the preferences of the primary users of each
|
|
repository.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Currently, hg.python.org hosts more than just the core CPython repository,
|
|
it also hosts other repositories such as those for the CPython developer
|
|
guide and for Python Enhancement Proposals, along with various "sandbox"
|
|
repositories for core developer experimentation.
|
|
|
|
While the simple "pull request" style workflow made popular by code hosting
|
|
sites like GitHub and BitBucket isn't adequate for the complex branching
|
|
model needed for parallel maintenance and development of the various
|
|
CPython releases, it's a good fit for several of the ancillary projects
|
|
that surround CPython that we don't wish to move to a proprietary hosting
|
|
site.
|
|
|
|
The key requirements proposed for a PSF provided software forge are:
|
|
|
|
* MUST support simple "pull request" style workflows
|
|
* MUST support online editing for simple changes
|
|
* MUST be backed by an active development organisation (community or
|
|
commercial)
|
|
|
|
Additional recommended requirements that are satisfied by this proposal,
|
|
but may be negotiable if a sufficiently compelling alternative is presented:
|
|
|
|
* SHOULD support self-hosting on PSF infrastructure without ongoing fees
|
|
* SHOULD be a fully open source application written in Python
|
|
* SHOULD support Mercurial (for consistency with existing tooling)
|
|
* SHOULD support Git (to provide that option to users that prefer it)
|
|
* SHOULD allow users of git and Mercurial clients to transparently
|
|
collaborate on the same repository
|
|
* SHOULD be open to customisation to meet the needs of CPython core
|
|
development, including providing a potential path forward for the
|
|
proposed migration to a core reviewer model in PEP 462
|
|
|
|
|
|
The preference for self-hosting without ongoing fees rules out the
|
|
free-as-in-beer providers like GitHub and BitBucket, in addition to the
|
|
various proprietary source code management offerings.
|
|
|
|
The preference for Mercurial support not only rules out GitHub, but also
|
|
other Git-only solutions like GitLab and Gitorious.
|
|
|
|
The hard requirement for online editing support rules out the Apache
|
|
Allura/HgForge combination.
|
|
|
|
The preference for a fully open source solution rules out RhodeCode.
|
|
|
|
Of the various options considered by the author of this proposal, that
|
|
leaves `Kallithea SCM <https://kallithea-scm.org/>`__ as the proposed
|
|
foundation for a forge.python.org service.
|
|
|
|
Kallithea is a full GPLv3 application (derived from the clearly
|
|
and unambiguously GPLv3 licensed components of RhodeCode) that is being
|
|
developed under the auspices of the `Software Freedom Conservancy
|
|
<http://sfconservancy.org/news/2014/jul/04/kallithea-joins/>`__. The
|
|
Conservancy has `affirmed
|
|
<http://sfconservancy.org/blog/2014/jul/15/why-kallithea/>`__ that the
|
|
Kallithea codebase is completely and validly licensed under GPLv3. In
|
|
addition to their role in building the initial Kallithea community, the
|
|
Conservancy is also the legal home of both the Mercurial and Git projects.
|
|
Other SFC member projects that may be familiar to Python users include
|
|
Twisted, Gevent, BuildBot and PyPy.
|
|
|
|
|
|
Intended Benefits
|
|
==================
|
|
|
|
The primary benefit of deploying Kallithea as forge.python.org is that
|
|
supporting repositories such as the developer guide and the PEP repo could
|
|
potentially be managed using pull requests and online editing. This would be
|
|
much simpler than the current workflow which requires PEP editors and
|
|
other core developers to act as intermediaries to apply updates suggested
|
|
by other users.
|
|
|
|
The richer administrative functionality would also make it substantially
|
|
easier to grant users access to particular repositories for collaboration
|
|
purposes, without having to grant them general access to the entire
|
|
installation.
|
|
|
|
|
|
Sustaining Engineering Considerations
|
|
=====================================
|
|
|
|
Even with its current workflow, CPython itself remains one of the largest
|
|
open source projects in the world (in the
|
|
`top 2% <https://www.openhub.net/p/python/factoids#FactoidTeamSizeVeryLarge>`
|
|
of projects tracked on OpenHub). Unfortunately, we have been significantly
|
|
less effective at encouraging contributions to the projects that make up
|
|
CPython's workflow infrastructure, including ensuring that our installations
|
|
track upstream, and that wherever feasible, our own customisations are
|
|
contributed back to the original project.
|
|
|
|
As such, a core component of this proposal is to actively engage with the
|
|
upstream Kallithea community to lower the barriers to working with and on
|
|
the Kallithea SCM, as well as with the PSF Infrastructure team to ensure
|
|
the forge.python.org service integrates cleanly with the PSF's infrastructure
|
|
automation.
|
|
|
|
This approach aims to provide a number of key benefits:
|
|
|
|
* allowing those of us contributing to maintenance of this service to be
|
|
as productive as possible in the time we have available
|
|
* offering a compelling professional development opportunity to those
|
|
volunteers that choose to participate in maintenance of this service
|
|
* making the Kallithea project itself more attractive to other potential
|
|
users by making it as easy as possible to adopt, deploy and manage
|
|
* as a result of the above benefits, attracting sufficient contributors both
|
|
in the upstream Kallithea community, and within the CPython infrastructure
|
|
community, to allow the forge.python.org service to evolve effectively to
|
|
meet changing developer expectations
|
|
|
|
Some initial steps have already been taken to address these sustaining
|
|
engineering concerns:
|
|
|
|
* Tymoteusz Jankowski has been working with Donald Stufft to work out `what
|
|
would be involved <https://github.com/xliiv/psf-salt/tree/kallithea>`__
|
|
in deploying Kallithea using the PSF's Salt based infrastructure automation.
|
|
* Graham Dumpleton and I have been working on
|
|
`making it easy
|
|
<http://www.curiousefficiency.org/posts/2014/12/kallithea-on-openshift.html>
|
|
`__ to deploy
|
|
demonstration Kallithea instances to the free tier of Red Hat's open source
|
|
hosting service, OpenShift Online. (See the comments on that post, or the
|
|
`quickstart issue tracker
|
|
<https://github.com/ncoghlan/openshift-kallithea/issues/>`__ for links to
|
|
Graham's follow on work)
|
|
|
|
The next major step to be undertaken is to come up with a local development
|
|
workflow that allows contributors on Windows, Mac OS X and Linux to run
|
|
the Kallithea tests locally, without interfering with the operation of
|
|
their own system. The currently planned approach for this is to focus on
|
|
Vagrant, which is a popular automated virtual machine management system
|
|
specifically aimed at developers running local VMs for testing purposes.
|
|
The `Vagrant based development guidelines
|
|
<http://www.openshift.org/documentation/oo_deployment_guide_vagrant.html>`__
|
|
for
|
|
OpenShift Origin provide an extended example of the kind of workflow this
|
|
approach enables. It's also worth noting that Vagrant is one of the options
|
|
for working with a local build of the `main python.org website
|
|
<https://github.com/python/pythondotorg#using-vagrant>`__.
|
|
|
|
If these workflow proposals end up working well for Kallithea, they may also
|
|
be worth proposing for use by the upstream projects backing other PSF and
|
|
CPython infrastructure services, including Roundup, BuildBot, and the main
|
|
python.org web site.
|
|
|
|
|
|
Technical Concerns and Challenges
|
|
=================================
|
|
|
|
Introducing a new service into the CPython infrastructure presents a number
|
|
of interesting technical concerns and challenges. This section covers several
|
|
of the most significant ones.
|
|
|
|
User account management
|
|
-----------------------
|
|
|
|
Ideally we'd like to be able to offer a single account that spans all
|
|
python.org services, including Kallithea, Roundup/Rietveld, PyPI and the
|
|
back end for the new python.org site, but actually implementing that would
|
|
be a distinct infrastructure project, independent of this PEP.
|
|
|
|
For the initial rollout of forge.python.org, we will likely create yet another
|
|
identity silo within the PSF infrastructure. A potentially superior
|
|
alternative would be to add support for `python-social-auth
|
|
<https://python-social-auth.readthedocs.org>`__ to Kallithea, but actually
|
|
doing so would not be a requirement for the initial rollout of the service
|
|
(the main technical concern there is that Kallithea is a Pylons application
|
|
that has not yet been ported to Pyramid, so integration will require either
|
|
adding a Pylons backend to python-social-auth, or else embarking on the
|
|
Pyramid migration in Kallithea).
|
|
|
|
|
|
Breaking existing SSH access and links for Mercurial repositories
|
|
-----------------------------------------------------------------
|
|
|
|
This PEP proposes leaving the existing hg.python.org installation alone,
|
|
and setting up Kallithea on a new host. This approach minimises the risk
|
|
of interfering with the development of CPython itself (and any other
|
|
projects that don't migrate to the new software forge), but does make any
|
|
migrations of existing repos more disruptive (since existing checkouts will
|
|
break).
|
|
|
|
|
|
Integration with Roundup
|
|
------------------------
|
|
|
|
Kallithea provides configurable issue tracker integration. This will need
|
|
to be set up appropriately to integrate with the Roundup issue tracker at
|
|
bugs.python.org before the initial rollout of the forge.python.org service.
|
|
|
|
|
|
Accepting pull requests on GitHub and BitBucket
|
|
-----------------------------------------------
|
|
|
|
The initial rollout of forge.python.org would support publication of read-only
|
|
mirrors, both on hg.python.org and other services, as that is a relatively
|
|
straightforward operation that can be implemented in a commit hook.
|
|
|
|
While a highly desirable feature, accepting pull requests on external
|
|
services, and mirroring them as submissions to the master repositories on
|
|
forge.python.org is a more complex problem, and would likely not be included
|
|
as part of the initial rollout of the forge.python.org service.
|
|
|
|
|
|
Transparent Git and Mercurial interoperability
|
|
----------------------------------------------
|
|
|
|
Kallithea's native support for both Git and Mercurial offers an opportunity
|
|
to make it relatively straightforward for developers to use the client
|
|
of their choice to interact with repositories hosted on forge.python.org.
|
|
|
|
This transparent interoperability does *not* exist yet, but running our own
|
|
multi-VCS repository hosting service provides the opportunity to make this
|
|
capability a reality, rather than passively waiting for a proprietary
|
|
provider to deign to provide a feature that likely isn't in their commercial
|
|
interest. There's a significant misalignment of incentives between open
|
|
source communities and commercial providers in this particular area, as even
|
|
though offering VCS client choice can significantly reduce community friction
|
|
by eliminating the need for projects to make autocratic decisions that force
|
|
particular tooling choices on potential contributors, top down enforcement
|
|
of tool selection (regardless of developer preference) is currently still
|
|
the norm in the corporate and other organisational environments that produce
|
|
GitHub and Atlassian's paying customers.
|
|
|
|
Prior to acceptance, in the absence of transparent interoperability, this PEP
|
|
should propose specific recommendations for inclusion in the CPython
|
|
developer's guide for using git-hg to create pull requests against
|
|
forge.python.org hosted Mercurial repositories.
|
|
|
|
|
|
Future Implications for CPython Core Development
|
|
================================================
|
|
|
|
The workflow requirements for the main CPython development repository are
|
|
significantly more complex than those for the repositories being discussed
|
|
in this PEP. These concerns are covered in more detail in PEP 462.
|
|
|
|
Given Guido's recommendation to replace Rietveld with a more actively
|
|
maintained code review system, my current plan is to rewrite that PEP to
|
|
use Kallithea as the proposed glue layer, with enhanced Kallithea pull
|
|
requests eventually replacing the current practice of uploading patche files
|
|
directly to the issue tracker.
|
|
|
|
I've also started working with Pierre Yves-David on a `custom Mercurial
|
|
extension <https://bitbucket.org/ncoghlan/cpydev/src/default/cpyhg.py?at=default>`__
|
|
that automates some aspects of the CPython core development workflow.
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|