Update PEP 512 (#107)

This commit is contained in:
Brett Cannon 2016-09-23 10:59:32 -07:00 committed by GitHub
parent 94dbee096b
commit 0cd8619f18
1 changed files with 130 additions and 149 deletions

View File

@ -18,7 +18,8 @@ process from Mercurial [#hg]_ as hosted at
hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting
the minimum goals of this PEP should allow for the development
process of Python to be as productive as it currently is, and meeting
its extended goals should improve it.
its extended goals should improve the development process from its
status quo.
Rationale
@ -37,12 +38,14 @@ the basic steps were:
Rietveld code review tool [#rietveld]_.
6. Download the patch to make sure it still applies cleanly.
7. Run the test suite manually.
8. Commit the change manually.
9. If the change was for a bugfix release, merge into the
in-development branch.
10. Run the test suite manually again.
11. Commit the merge.
12. Push the changes.
8. Update the `NEWS`, `ACKS`, and "What's New" document as necessary
9. Pull changes to avoid a merge race.
10. Commit the change manually.
11. If the change was for a bugfix release, merge into the
in-development branch.
12. Run the test suite manually again.
13. Commit the merge.
14. Push the changes.
This is a very heavy, manual process for core developers. Even in the
simple case, you could only possibly skip the code review step, as you
@ -51,11 +54,11 @@ languishing on the issue tracker due to core developers not being
able to work through the backlog fast enough to keep up with
submissions. In turn, that led to a side-effect issue of discouraging
outside contribution due to frustration from lack of attention, which
is dangerous problem for an open source project as it runs counter to
having a viable future for the project. Simply accepting patches
uploaded to bugs.python.org [#b.p.o]_ is potentially simple for an
external contributor, it is as slow and burdensome as it gets for
a core developer to work with.
is dangerous problem for an open source project with no corporate
backing as it runs counter to having a viable future for the project.
While allowing patches to be uploaded to bugs.python.org [#b.p.o]_ is
potentially simple for an external contributor, it is as slow and
burdensome as it gets for a core developer to work with.
Hence the decision was made in late 2014 that a move to a new
development process was needed. A request for PEPs
@ -78,16 +81,16 @@ development process to GitHub. The key reasons for choosing GitHub
were [#reasons]_:
* Maintaining custom infrastructure has been a burden on volunteers
(e.g., a custom fork of Rietveld [#rietveld]_
that is not being maintained is currently being used).
(e.g., an unmaintained, custom fork of Rietveld [#rietveld]_
is currently being used).
* The custom workflow is very time-consuming for core developers
(not enough automated tooling built to help support it).
* The custom workflow is a hindrance to external contributors
(acts as a barrier of entry due to time required to ramp up on
development process).
development process unique to CPython itself).
* There is no feature differentiating GitLab from GitHub beyond
GitLab being open source.
* Familiarity with GitHub is far higher amongst core developers and
* Familiarity with GitHub is far higher among core developers and
external contributors than with GitLab.
* Our BDFL prefers GitHub (who would be the first person to tell
you that his opinion shouldn't matter, but the person making the
@ -95,36 +98,37 @@ were [#reasons]_:
the workflow of his own programming language to encourage his
continued participation).
There's even already an unofficial image to use to represent the
There's even already an unofficial logo to represent the
migration to GitHub [#pythocat]_.
The overarching goal of this migration is to improve the development
process to the extent that a core developer can go from external
contribution submission through all the steps leading to committing
said contribution all from within a browser on a tablet with WiFi
said contribution from within a browser on a tablet with WiFi
using *some* development process (this does not inherently mean
GitHub's default workflow). All of this will be done in such a way
that if an external contributor chooses not to use GitHub then they
will continue to have that option.
GitHub's default workflow). The final solution will also allow
an external contributor to contribute even if they chose not to use
GitHub (although there is not guarantee in feature parity).
Repositories to Migrate
=======================
While hg.python.org [#h.p.o]_ hosts many repositories, there are only
five key repositories that should move:
five key repositories that need to move:
1. devinabox [#devinabox-repo]_ (done)
2. benchmarks [#benchmarks-repo]_ (maybe)
2. benchmarks [#benchmarks-repo]_ (skipped)
3. peps [#peps-repo]_ (done)
4. devguide [#devguide-repo]_
4. devguide [#devguide-repo]_ (done)
5. cpython [#cpython-repo]_
The devinabox and benchmarks repositories are code-only.
The devinabox repository is code-only.
The peps and devguide repositories involve the generation of webpages.
And the cpython repository has special requirements for integration
with bugs.python.org [#b.p.o]_.
Migration Plan
==============
@ -140,10 +144,7 @@ Requirements for Code-Only Repositories
---------------------------------------
Completion of the requirements in this section will allow the
devinabox and benchmarks repositories to move to
GitHub. While devinabox has a sufficiently descriptive name, the
benchmarks repository does not; therefore, it will be named
"python-benchmarks".
devinabox repository to move to GitHub.
Create a 'Python core' team
@ -217,7 +218,7 @@ a negative label will be added to the pull request will be blocked
using GitHub's status API (e.g., a red label stating, "CLA not signed").
If a contributor lacks a bugs.python.org account, that will lead to
the negative label being used as well. Using a label for both
positive and negative cases provides a fallback notification if the
positive and negative cases provides a fallback signal if the
bot happens to fail, preventing potential false-positives or
false-negatives. It also allows for an easy way to trigger the bot
again by simply removing a CLA-related label (this is in contrast to
@ -226,7 +227,7 @@ triggered on code changes).
As no pre-existing bot exists to meet our needs, it will be hosted on
Heroku [#heroku]_ and written to target Python 3.5 to act as a
showcase for asynchronous programming. The code for the bot is hosted
showcase for asynchronous programming. The code for the bot is hosted
in the Knights Who Say Ni project [#ni]_.
@ -249,11 +250,6 @@ devguide [#devguide-repo]_ and peps [#peps-repo]_ repositories need
their respective processes updated to pull from their new Git
repositories.
The devguide repository might also need to be named
``python-devguide`` to make sure the repository is not ambiguous
when viewed in isolation from the
python organization [#github-python-org]_.
Requirements for the cpython Repository
---------------------------------------
@ -273,19 +269,8 @@ decided that a linear history is desired. People preferred having a
single commit representing a single change instead of having a set of
unrelated commits lead to a merge commit that represented a single
change. This means that the convenient "Merge" button in GitHub pull
requests is undesirable, as it creates a merge commit along with all
of the contributor's individual commits (this does not affect the
other repositories where the desire for a linear history doesn't
exist).
Luckily, Git [#git]_ does not require GitHub's workflow and so one can
be chosen which gives us a linear history by using Git's CLI. The
expectation is that all pull requests will be fast-forwarded and
rebased before being pushed to the master repository. This should
give proper attribution to the pull request author in the Git
history. This does have the consequence of losing some GitHub
features such as automatic closing of pull requests, link generation,
etc.
requests will be set to only do *squash* commits and not merge
commits.
A second set of recommended commands will also be written for
committing a contribution from a patch file uploaded to
@ -300,6 +285,7 @@ core developers is an open issue:
Linking pull requests to issues
'''''''''''''''''''''''''''''''
Historically, external contributions were attached to an issue on
bugs.python.org [#b.p.o]_ thanks to the fact that all external
contributions were uploaded as a file. For changes committed by a
@ -320,14 +306,12 @@ single fix solves multiple issues, but this is fairly rare and issues
can be merged into one using the ``Superseder`` field on the issue
tracker).
Association between a pull request and an issue will be done based on
detecting the regular expression``[Ii]ssue #(?P<bpo_id>\d+)``. If
this is specified in either the title or in the body of a message on
a pull request then connection will be made on
bugs.python.org [#b.p.o]_. A label will also be added to the pull
request to signify that the connection was made successfully. This
could lead to incorrect associations if the wrong issue or
referencing another issue was done, but these are rare occasions.
The association between a pull request and an issue will be done based
on detecting an issue number. If the issue is specified in either the
title or in the body of a message on a pull request then connection
will be made on bugs.python.org [#b.p.o]_. Some visible notification
-- e.g. label or message -- will be made to the pull request to
notify that the association was successfully made.
Notify the issue if a commit is made
@ -338,8 +322,8 @@ reflect this fact. This should work regardless of whether the commit
came from a pull request or a direct commit.
Update linking service for mapping commit IDs to URLs
'''''''''''''''''''''''''''''''''''''''''''''''''''''
Update the linking service for mapping commit IDs to URLs
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Currently you can use https://hg.python.org/lookup/ with a revision
ID from either the Subversion or Mercurial copies of the
@ -377,8 +361,8 @@ Optional, Planned Features
Once the cpython repository [#cpython-repo]_ is migrated, all
repositories will have been moved to GitHub [#github]_ and the
development process should be on equal footing as before. But a key
reason for this migration is to improve the development process,
development process should be on equal footing as before the move. But
a key reason for this migration is to improve the development process,
making it better than it has ever been. This section outlines some
plans on how to improve things.
@ -390,19 +374,54 @@ migration -- are tracked on their own wiki page [#tracker-plans]_.
Handling Misc/NEWS
''''''''''''''''''
Traditionally the ``Misc/NEWS`` file [#news-file]_ has been problematic
for changes which spanned Python releases. Oftentimes there will be
merge conflicts when committing a change between e.g., 3.5 and 3.6
only in the ``Misc/NEWS`` file. It's so common, in fact, that the
example instructions in the devguide explicitly mention how to
resolve conflicts in the ``Misc/NEWS`` file
Traditionally the ``Misc/NEWS`` file [#news-file]_ has been
problematic for changes which spanned Python releases. Oftentimes
there will be merge conflicts when committing a change between e.g.,
3.5 and 3.6 only in the ``Misc/NEWS`` file. It's so common, in fact,
that the example instructions in the devguide explicitly mention how
to resolve conflicts in the ``Misc/NEWS`` file
[#devguide-merge-across-branches]_. As part of our tool
modernization, working with the ``Misc/NEWS`` file will be
simplified.
There are currently two competing approaches to solving the
``Misc/NEWS`` problem which are discussed in an open issue:
`How to handle the Misc/NEWS file`_.
The planned approach is to use an individual file per news entry,
containing the text for the entry. In this scenario each feature
release would have its own directory for news entries and a separate
file would be created in that directory that was either named after
the issue it closed or a timestamp value (which prevents collisions).
Merges across branches would have no issue as the news entry file
would still be uniquely named and in the directory of the latest
version that contained the fix. A script would collect all news entry
files no matter what directory they reside in and create an
appropriate news file (the release directory can be ignored as the
mere fact that the file exists is enough to represent that the entry
belongs to the release). Classification can either be done by keyword
in the new entry file itself or by using subdirectories representing
each news entry classification in each release directory (or
classification of news entries could be dropped since critical
information is captured by the "What's New" documents which are
organized). The benefit of this approach is that it keeps the changes
with the code that was actually changed. It also ties the message to
being part of the commit which introduced the change. For a commit
made through the CLI, a script could be provided to help generate the
file. In a bot-driven scenario, the merge bot could have a way to
specify a specific news entry and create the file as part of its
flattened commit (while most likely also supporting using the first
line of the commit message if no specific news entry was specified).
If a web-based workflow is used then a status check could be used to
verify that a new entry file is in the pull request to act as a
reminder that the file is missing. Code for this approach has been
written previously for the Mercurial workflow at
http://bugs.python.org/issue18967. There is also tools from the
community like https://pypi.python.org/pypi/towncrier,
https://github.com/twisted/newsbuilder, and
http://docs.openstack.org/developer/reno/.
Discussions at the Sep 2016 Python core-dev sprints led to this
decision compared to the rejected approaches outlined in the
`Rejected Ideas` section of this PEP. The separate files approach
seems to have the right balance of flexibility and potential tooling
out of the various options while solving the motivating problem.
Handling Misc/ACKS
@ -452,13 +471,6 @@ disappear overnight.
Bot to handle pull request merging
''''''''''''''''''''''''''''''''''
As stated in the section entitled
"`Document steps to commit a pull request`_", the desire is to
maintain a linear history for cpython. Unfortunately,
Github's [#github]_ web-based workflow does not support a linear
history. Because of this, a bot should be written to substitute for
GitHub's in-browser commit abilities.
To start, the bot should accept commands to commit a pull request
against a list of branches. This allows for committing a pull request
that fixes a bug in multiple versions of Python.
@ -516,7 +528,10 @@ GitHub [#github]_ and yet still know when something occurs on GitHub
in terms of review comments on relevant pull requests. Current
thinking is to post a comment to bugs.python.org to the relevant
issue when at least one review comment has been made over a certain
period of time (e.g., 15 or 30 minutes). This keeps the email volume
period of time (e.g., 15 or 30 minutes, although with GitHub now
supporting
`reviews <https://help.github.com/articles/reviewing-changes-in-pull-requests/>`_
the time aspect may be unnecessary). This keeps the email volume
down for those that receive both GitHub and bugs.python.org email
notifications while still making sure that those only following
bugs.python.org know when there might be a review comment to address.
@ -541,6 +556,11 @@ such, it would be nice to set up appropriate webhooks to trigger
rebuilding the appropriate web content when the files they are based
on change instead of having to wait for, e.g., a cronjob to trigger.
This can partially be solved if the documentation is a Sphinx project
as then the site can have an unofficial mirror on
`Read the Docs <https://readthedocs.org/>`_, e.g.
http://cpython-devguide.readthedocs.io/.
Link web content back to files that it is generated from
''''''''''''''''''''''''''''''''''''''''''''''''''''''''
@ -608,8 +628,8 @@ filled in properly.
Status
======
Requirements for migrating the devinabox [#devinabox-repo]_ and
benchmarks [#benchmarks-repo]_ repositories:
Requirements for migrating the devinabox [#devinabox-repo]_
repository:
* Completed
@ -637,6 +657,7 @@ Repositories whose build steps need updating:
- peps [#peps-repo]_
- devguide [#devguide-repo]_
cpython repo [#cpython-repo]_
-----------------------------
@ -647,12 +668,12 @@ Required:
- `Update the devguide`_ to
`Document steps to commit a pull request`_
(https://github.com/python/devguide/milestone/1)
- `Update linking service for mapping commit IDs to URLs`_
- `Deprecate sys._mercurial`_
(http://bugs.python.org/issue27593)
- `Update the linking service for mapping commit IDs to URLs`_
- `Update PEP 101`_
- Email python-checkins for each commit (PR or direct)
(https://help.github.com/articles/managing-notifications-for-pushes-to-a-repository/)
- Message #python-dev for each commit (PR or direct)
(https://github.com/python/cpython/settings/hooks/new?service=irc)
* In progress
@ -660,11 +681,14 @@ Required:
(http://psf.upfronthosting.co.za/roundup/meta/issue589)
- `Notify the issue if a commit is made`_
(http://psf.upfronthosting.co.za/roundup/meta/issue590)
- `Deprecate sys._mercurial`_
(http://bugs.python.org/issue27593)
* Completed
- None
Optional features:
* Not started
@ -736,72 +760,6 @@ history will be kept implicitly, but it will need to make sure to
keep/add attribution.
How to handle the Misc/NEWS file
--------------------------------
There are three competing approaches to handling files like
``Misc/NEWS`` [#news-file]_. One is to add a news entry for issues on
bugs.python.org [#b.p.o]_. This would mean an issue that is marked
as "resolved" could not be closed until a news entry is added in the
"news" field in the issue tracker. The benefit of tying the news
entry to the issue is it makes sure that all changes worthy of a news
entry have an accompanying issue. It also makes classifying a news
entry automatic thanks to the Component field of the issue. The
Versions field of the issue also ties the news entry to which Python
releases were affected. A script would be written to query
bugs.python.org for relevant new entries for a release and to produce
the output needed to be checked into the code repository. This
approach is agnostic to whether a commit was done by CLI or bot. A
drawback is that there's a disconnect between the actual commit that
made the change and the news entry by having them live in separate
places (in this case, GitHub and bugs.python.org). This would mean
making a commit would then require remembering to go back to
bugs.python.org to add the news entry.
A competing approach is to use an individual file per news entry,
containing the text for the entry. In this scenario each feature
release would have its own directory for news entries and a separate
file would be created in that directory that was either named after
the issue it closed or a timestamp value (which prevents collisions).
Merges across branches would have no issue as the news entry file
would still be uniquely named and in the directory of the latest
version that contained the fix. A script would collect all news entry
files no matter what directory they reside in and create an
appropriate news file (the release directory can be ignored as the
mere fact that the file exists is enough to represent that the entry
belongs to the release). Classification can either be done by keyword
in the new entry file itself or by using subdirectories representing
each news entry classification in each release directory (or
classification of news entries could be dropped since critical
information is captured by the "What's New" documents which are
organized). The benefit of this approach is that it keeps the changes
with the code that was actually changed. It also ties the message to
being part of the commit which introduced the change. For a commit
made through the CLI, a script could be provided to help generate the
file. In a bot-driven scenario, the merge bot could have a way to
specify a specific news entry and create the file as part of its
flattened commit (while most likely also supporting using the first
line of the commit message if no specific news entry was specified).
If a web-based workflow is used then a status check could be used to
verify that a new entry file is in the pull request to act as a
reminder that the file is missing. Code for this approach has been
written previously for the Mercurial workflow at
http://bugs.python.org/issue18967. There is also tools from the
community like https://pypi.python.org/pypi/towncrier,
https://github.com/twisted/newsbuilder, and
http://docs.openstack.org/developer/reno/.
A yet third option is a merge script to handle the conflicts. This
approach allows for keeping the NEWS file as a single file. It does
run the risk, though, of failure and thus blocking a commit until it
can be manually resolved.
Discussions at the Sep 2016 Python core-dev sprints has led to the
decision that the separate files approach will be used. It seems to
have the right balance of flexibility and potential tooling out of the
three options while solving the motivating problem.
Naming the bots
---------------
@ -903,6 +861,29 @@ the first line of a commit message compared to that of a news entry
have different requirements in terms of brevity, what should be said,
etc.
Deriving ``Misc/NEWS`` from bugs.python.org
-------------------------------------------
A rejected solution to the ``NEWS`` file problem was to specify the
entry on bugs.python.org [#b.p.o]_. This would mean an issue that is
marked as "resolved" could not be closed until a news entry is added
in the "news" field in the issue tracker. The benefit of tying the
news entry to the issue is it makes sure that all changes worthy of a
news entry have an accompanying issue. It also makes classifying a
news entry automatic thanks to the Component field of the issue. The
Versions field of the issue also ties the news entry to which Python
releases were affected. A script would be written to query
bugs.python.org for relevant new entries for a release and to produce
the output needed to be checked into the code repository. This
approach is agnostic to whether a commit was done by CLI or bot. A
drawback is that there's a disconnect between the actual commit that
made the change and the news entry by having them live in separate
places (in this case, GitHub and bugs.python.org). This would mean
making a commit would then require remembering to go back to
bugs.python.org to add the news entry.
References
==========