PEP 474: Updated forge.python.org proposal
This commit is contained in:
Nick Coghlan
parent
c4904f1ae8
commit
dbcb711913
242
pep-0474.txt
242
pep-0474.txt
|
|
@ -23,18 +23,11 @@ This PEP does *not* propose any changes to the core development workflow
|
|||
for CPython itself (see PEP 462 in relation to that).
|
||||
|
||||
|
||||
PEP Deferral
|
||||
============
|
||||
|
||||
This PEP is deferred largely because I don't currently have time to work on
|
||||
it. If anyone would like to take it over, let me know.
|
||||
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
This PEP proposes that, once the Kallithea project has made an official
|
||||
release, that a Kallithea instance be deployed as "forge.python.org".
|
||||
This PEP proposes that an instance of the self-hosted Kallithea code
|
||||
repository management system be deployed as "forge.python.org".
|
||||
|
||||
Individual repositories (such as the developer guide or the PEPs repository)
|
||||
may then be migrated from the existing hg.python.org infrastructure to the
|
||||
|
|
@ -42,9 +35,24 @@ new forge.python.org infrastructure on a case by case basis. Each migration
|
|||
will need to decide whether to retain a read-only mirror on hg.python.org,
|
||||
or whether to just migrate wholesale to the new location.
|
||||
|
||||
This would not be a general purpose hosting site for arbitrary Python
|
||||
projects, but a more narrowly focused site akin to the existing
|
||||
hg.python.org.
|
||||
In addition to supporting read-only mirrors on hg.python.org,
|
||||
forge.python.org will also aim to support hosting mirrors on popular
|
||||
proprietary hosting sites like GitHub and BitBucket. The aim will be to
|
||||
allow users familiar with these sites to submit and discuss pull requests
|
||||
using their preferred workflow, with forge.python.org automatically bringing
|
||||
those contributions over to the master repository.
|
||||
|
||||
Given the availability and popularity of commercially backed "free for open
|
||||
source projects" repository hosting services, this would not be a general
|
||||
purpose hosting site for arbitrary Python projects. The initial focus will be
|
||||
specifically on CPython and other repositories currently hosted on
|
||||
hg.python.org. In the future, this could potentially be expanded to
|
||||
consolidating other PSF managed repositories that are currently externally
|
||||
hosted to gain access to a pull request based workflow, such as the
|
||||
repository for the python.org Django application. As with the initial
|
||||
migrations, any such future migrations would be considered on a case-by-case
|
||||
basis, taking into account the preferences of the primary users of each
|
||||
repository.
|
||||
|
||||
|
||||
Rationale
|
||||
|
|
@ -64,36 +72,42 @@ site.
|
|||
|
||||
The key requirements proposed for a PSF provided software forge are:
|
||||
|
||||
* Must support self-hosting on PSF infrastructure without ongoing fees
|
||||
* Must support Mercurial (for consistency with existing tooling)
|
||||
* Must support simple "pull request" style workflows
|
||||
* Must support online editing for simple changes
|
||||
* MUST support simple "pull request" style workflows
|
||||
* MUST support online editing for simple changes
|
||||
* MUST be backed by an active development organisation (community or
|
||||
commercial)
|
||||
|
||||
Ideally, the chosen solution would also be a fully open source application
|
||||
written in Python.
|
||||
Additional recommended requirements that are satisfied by this proposal,
|
||||
but may be negotiable if a sufficiently compelling alternative is presented:
|
||||
|
||||
The requirement for self-hosting without ongoing fees rules out the
|
||||
* SHOULD support self-hosting on PSF infrastructure without ongoing fees
|
||||
* SHOULD be a fully open source application written in Python
|
||||
* SHOULD support Mercurial (for consistency with existing tooling)
|
||||
* SHOULD support Git (to provide that option to users that prefer it)
|
||||
* SHOULD allow users of git and Mercurial clients to transparently
|
||||
collaborate on the same repository
|
||||
* SHOULD be open to customisation to meet the needs of CPython core
|
||||
development, including providing a potential path forward for the
|
||||
proposed migration to a core reviewer model in PEP 462
|
||||
|
||||
|
||||
The preference for self-hosting without ongoing fees rules out the
|
||||
free-as-in-beer providers like GitHub and BitBucket, in addition to the
|
||||
various proprietary source code management offerings.
|
||||
|
||||
The requirement for Mercurial support not only rules out GitHub, but also
|
||||
The preference for Mercurial support not only rules out GitHub, but also
|
||||
other Git-only solutions like GitLab and Gitorious.
|
||||
|
||||
The requirement for online editing support rules out the Apache
|
||||
The hard requirement for online editing support rules out the Apache
|
||||
Allura/HgForge combination.
|
||||
|
||||
That leaves two main contenders from a technical perspective:
|
||||
The preference for a fully open source solution rules out RhodeCode.
|
||||
|
||||
* `RhodeCode <https://code.rhodecode.com/rhodecode>`__
|
||||
* `Kallithea SCM <https://kallithea-scm.org/>`__
|
||||
Of the various options considered by the author of this proposal, that
|
||||
leaves `Kallithea SCM <https://kallithea-scm.org/>`__ as the proposed
|
||||
foundation for a forge.python.org service.
|
||||
|
||||
The `legal uncertainty
|
||||
<http://sfconservancy.org/blog/2014/jul/15/why-kallithea/>`__ over the
|
||||
interaction between RhodeCode's current "Business Source" licensing model and
|
||||
the various GPL components it relies on currently make it unclear whether it
|
||||
is legally permissible to deploy it.
|
||||
|
||||
By contrast, Kallithea is a full GPLv3 application (derived from the clearly
|
||||
Kallithea is a full GPLv3 application (derived from the clearly
|
||||
and unambiguously GPLv3 licensed components of RhodeCode) that is being
|
||||
developed under the auspices of the `Software Freedom Conservancy
|
||||
<http://sfconservancy.org/news/2014/jul/04/kallithea-joins/>`__. The
|
||||
|
|
@ -106,7 +120,7 @@ Other SFC member projects that may be familiar to Python users include
|
|||
Twisted, Gevent, BuildBot and PyPy.
|
||||
|
||||
|
||||
Perceived Benefits
|
||||
Intended Benefits
|
||||
==================
|
||||
|
||||
The primary benefit of deploying Kallithea as forge.python.org is that
|
||||
|
|
@ -122,24 +136,97 @@ purposes, without having to grant them general access to the entire
|
|||
installation.
|
||||
|
||||
|
||||
Technical Challenges
|
||||
====================
|
||||
Sustaining Engineering Considerations
|
||||
=====================================
|
||||
|
||||
Even with its current workflow, CPython itself remains one of the largest
|
||||
open source projects in the world (in the
|
||||
`top 2% <https://www.openhub.net/p/python/factoids#FactoidTeamSizeVeryLarge>`
|
||||
of projects tracked on OpenHub). Unfortunately, we have been significantly
|
||||
less effective at encouraging contributions to the projects that make up
|
||||
CPython's workflow infrastructure, including ensuring that our installations
|
||||
track upstream, and that wherever feasible, our own customisations are
|
||||
contributed back to the original project.
|
||||
|
||||
As such, a core component of this proposal is to actively engage with the
|
||||
upstream Kallithea community to lower the barriers to working with and on
|
||||
the Kallithea SCM, as well as with the PSF Infrastructure team to ensure
|
||||
the forge.python.org service integrates cleanly with the PSF's infrastructure
|
||||
automation.
|
||||
|
||||
This approach aims to provide a number of key benefits:
|
||||
|
||||
* allowing those of us contributing to maintenance of this service to be
|
||||
as productive as possible in the time we have available
|
||||
* offering a compelling professional development opportunity to those
|
||||
volunteers that choose to participate in maintenance of this service
|
||||
* making the Kallithea project itself more attractive to other potential
|
||||
users by making it as easy as possible to adopt, deploy and manage
|
||||
* as a result of the above benefits, attracting sufficient contributors both
|
||||
in the upstream Kallithea community, and within the CPython infrastructure
|
||||
community, to allow the forge.python.org service to evolve effectively to
|
||||
meet changing developer expectations
|
||||
|
||||
Some initial steps have already been taken to address these sustaining
|
||||
engineering concerns:
|
||||
|
||||
* Tymoteusz Jankowski has been working with Donald Stufft to work out `what
|
||||
would be involved <https://github.com/xliiv/psf-salt/tree/kallithea>`__
|
||||
in deploying Kallithea using the PSF's Salt based infrastructure automation.
|
||||
* Graham Dumpleton and I have been working on
|
||||
`making it easy
|
||||
<http://www.curiousefficiency.org/posts/2014/12/kallithea-on-openshift.html>
|
||||
`__ to deploy
|
||||
demonstration Kallithea instances to the free tier of Red Hat's open source
|
||||
hosting service, OpenShift Online. (See the comments on that post, or the
|
||||
`quickstart issue tracker
|
||||
<https://github.com/ncoghlan/openshift-kallithea/issues/>`__ for links to
|
||||
Graham's follow on work)
|
||||
|
||||
The next major step to be undertaken is to come up with a local development
|
||||
workflow that allows contributors on Windows, Mac OS X and Linux to run
|
||||
the Kallithea tests locally, without interfering with the operation of
|
||||
their own system. The currently planned approach for this is to focus on
|
||||
Vagrant, which is a popular automated virtual machine management system
|
||||
specifically aimed at developers running local VMs for testing purposes.
|
||||
The `Vagrant based development guidelines
|
||||
<http://www.openshift.org/documentation/oo_deployment_guide_vagrant.html>`__
|
||||
for
|
||||
OpenShift Origin provide an extended example of the kind of workflow this
|
||||
approach enables. It's also worth noting that Vagrant is one of the options
|
||||
for working with a local build of the `main python.org website
|
||||
<https://github.com/python/pythondotorg#using-vagrant>`__.
|
||||
|
||||
If these workflow proposals end up working well for Kallithea, they may also
|
||||
be worth proposing for use by the upstream projects backing other PSF and
|
||||
CPython infrastructure services, including Roundup, BuildBot, and the main
|
||||
python.org web site.
|
||||
|
||||
|
||||
Technical Concerns and Challenges
|
||||
=================================
|
||||
|
||||
Introducing a new service into the CPython infrastructure presents a number
|
||||
of interesting technical concerns and challenges. This section covers several
|
||||
of the most significant ones.
|
||||
|
||||
User account management
|
||||
-----------------------
|
||||
|
||||
Ideally we'd be able to offer a single account that spans all python.org
|
||||
services, including Kallithea, Roundup/Rietveld, PyPI and the back end for
|
||||
the new python.org site, but actually implementing that would be a distinct
|
||||
infrastructure project, independent of this PEP.
|
||||
Ideally we'd like to be able to offer a single account that spans all
|
||||
python.org services, including Kallithea, Roundup/Rietveld, PyPI and the
|
||||
back end for the new python.org site, but actually implementing that would
|
||||
be a distinct infrastructure project, independent of this PEP.
|
||||
|
||||
A potentially simpler near term solution would be if Kallithea's user
|
||||
account management could be integrated with the existing account management
|
||||
in Roundup, similar to what was done for Rietveld. However, if that also
|
||||
turns out to be impractical in the near term, we would merely end up with
|
||||
another identity silo to be integrated at a later date, suggesting that
|
||||
this doesn't need to be considered a blocker for an initial Kallithea
|
||||
deployment.
|
||||
For the initial rollout of forge.python.org, we will likely create yet another
|
||||
identity silo within the PSF infrastructure. A potentially superior
|
||||
alternative would be to add support for `python-social-auth
|
||||
<https://python-social-auth.readthedocs.org>`__ to Kallithea, but actually
|
||||
doing so would not be a requirement for the initial rollout of the service
|
||||
(the main technical concern there is that Kallithea is a Pylons application
|
||||
that has not yet been ported to Pyramid, so integration will require either
|
||||
adding a Pylons backend to python-social-auth, or else embarking on the
|
||||
Pyramid migration in Kallithea).
|
||||
|
||||
|
||||
Breaking existing SSH access and links for Mercurial repositories
|
||||
|
|
@ -153,6 +240,71 @@ migrations of existing repos more disruptive (since existing checkouts will
|
|||
break).
|
||||
|
||||
|
||||
Integration with Roundup
|
||||
------------------------
|
||||
|
||||
Kallithea provides configurable issue tracker integration. This will need
|
||||
to be set up appropriately to integrate with the Roundup issue tracker at
|
||||
bugs.python.org before the initial rollout of the forge.python.org service.
|
||||
|
||||
|
||||
Accepting pull requests on GitHub and BitBucket
|
||||
-----------------------------------------------
|
||||
|
||||
The initial rollout of forge.python.org would support publication of read-only
|
||||
mirrors, both on hg.python.org and other services, as that is a relatively
|
||||
straightforward operation that can be implemented in a commit hook.
|
||||
|
||||
While a highly desirable feature, accepting pull requests on external
|
||||
services, and mirroring them as submissions to the master repositories on
|
||||
forge.python.org is a more complex problem, and would likely not be included
|
||||
as part of the initial rollout of the forge.python.org service.
|
||||
|
||||
|
||||
Transparent Git and Mercurial interoperability
|
||||
----------------------------------------------
|
||||
|
||||
Kallithea's native support for both Git and Mercurial offers an opportunity
|
||||
to make it relatively straightforward for developers to use the client
|
||||
of their choice to interact with repositories hosted on forge.python.org.
|
||||
|
||||
This transparent interoperability does *not* exist yet, but running our own
|
||||
multi-VCS repository hosting service provides the opportunity to make this
|
||||
capability a reality, rather than passively waiting for a proprietary
|
||||
provider to deign to provide a feature that likely isn't in their commercial
|
||||
interest. There's a significant misalignment of incentives between open
|
||||
source communities and commercial providers in this particular area, as even
|
||||
though offering VCS client choice can significantly reduce community friction
|
||||
by eliminating the need for projects to make autocratic decisions that force
|
||||
particular tooling choices on potential contributors, top down enforcement
|
||||
of tool selection (regardless of developer preference) is currently still
|
||||
the norm in the corporate and other organisational environments that produce
|
||||
GitHub and Atlassian's paying customers.
|
||||
|
||||
Prior to acceptance, in the absence of transparent interoperability, this PEP
|
||||
should propose specific recommendations for inclusion in the CPython
|
||||
developer's guide for using git-hg to create pull requests against
|
||||
forge.python.org hosted Mercurial repositories.
|
||||
|
||||
|
||||
Future Implications for CPython Core Development
|
||||
================================================
|
||||
|
||||
The workflow requirements for the main CPython development repository are
|
||||
significantly more complex than those for the repositories being discussed
|
||||
in this PEP. These concerns are covered in more detail in PEP 462.
|
||||
|
||||
Given Guido's recommendation to replace Reitveld with a more actively
|
||||
maintained code review system, my current plan is to rewrite that PEP to
|
||||
use Kallithea as the proposed glue layer, with enhanced Kallithea pull
|
||||
requests eventually replacing the current practice of uploading patche files
|
||||
directly to the issue tracker.
|
||||
|
||||
I've also started working with Pierre Yves-David on a `custom Mercurial
|
||||
extension <https://bitbucket.org/ncoghlan/cpydev/src/default/cpyhg.py?at=default>`__
|
||||
that automates some aspects of the CPython core development workflow.
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue