python-peps/pep-0676.rst

PEP: 676
Title: PEP Infrastructure Process
Author: Adam Turner <python@quite.org.uk>
Sponsor: Mariatta <mariatta@python.org>
Discussions-To: https://discuss.python.org/t/10774
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 01-Nov-2021
Post-History: 23-Sep-2021, 30-Nov-2021


Abstract
========

This PEP addresses the infrastructure around rendering PEP files from
`reStructuredText`_ files to HTML webpages. We aim to specify a self-contained
and maintainable solution for PEP readers, authors, and editors.


Motivation
==========

As of November 2021, Python Enhancement Proposals (PEPs) are rendered in a
multi-system, multi-stage process. A continuous integration (CI) task runs a
`docutils`_ script to render all PEP files individually. The CI task then
uploads a tar archive to a server, where it is retrieved and rendered into the
`python.org`_ website periodically.

This places a constraint on the `python.org`_ website to handle raw HTML
uploads and handle PEP rendering, and makes the appropriate place to raise
issues unclear in some cases [1]_.

This PEP provides a specification for self-contained rendering of PEPs. This
would:

* reduce the amount of distributed configuration for supporting PEPs
* enable quality-of-life improvements for those who read, write, and review
  PEPs
* solve a number of outstanding issues, and lay the path for improvements
* save volunteer maintainers' time

We propose that PEPs are accessed through `peps.python.org`_ at the top-level
(for example `peps.python.org/pep-0008`_), and that all custom tooling to
support rendering PEPs is hosted in the `python/peps`_ repository.


Rationale
=========

Simplifying and Centralising Infrastructure
-------------------------------------------

As of November 2021, to locally render a PEP file, a PEP author or editor needs
to create a full local instance of the `python.org`_ website and run a number
of disparate scripts, following `documentation`_ that lives outside of the
`python/peps`_ repository.

By contrast, the proposed implementation provides a single `Makefile`_ and a
Python script to render all PEP files, with options to target a web-server or
the local filesystem.

Using a single repository to host all tooling will clarify where to raise
issues, reducing volunteer time spent in triage.

Simplified and centralised tooling may also reduce the barrier to entry to
further improvements, as the scope of the PEP rendering infrastructure is well
defined.


Quality-of-Life Improvements and Resolving Issues
-------------------------------------------------

There are several requests for additional features in reading PEPs, such as:

* syntax highlighting [2]_
* use of ``.. code-block::`` directives [2]_
* support for SVG images [3]_
* typographic quotation marks [4]_
* additional footer information [5]_
* intersphinx functionality [6]_

These are "easy wins" from this proposal, and would serve to improve the
quality-of-life for consumers of PEPs (including reviewers and writers).

For example, the current (as of November 2021) system runs periodically on a
schedule. This means that updates to PEPs cannot be circulated immediately,
reducing productivity. The reference implementation renders and publishes all
PEPs on every commit to the repository, solving the issue by design.

The reference implementation fixes several issues [7]_. For example:

* list styles are currently not respected by `python.org`_'s stylesheets
* support for updating images in PEPs is challenging in `python.org`_

Third-party providers such as `Read the Docs`_ or `Netlify`_ can enhance this
experience with features such as automatic rendering of pull requests.


Specification
=============

The proposed specification for rendering the PEP files to HTML is as per the
`reference implementation`_.

The rendered PEPs MUST be available at `peps.python.org`_. These SHOULD be
hosted as static files, and MAY be behind a content delivery network (CDN).

A service to render previews of pull requests SHOULD be provided. This service
MAY be integrated with the hosting and deployment solution.

The following redirect rules MUST be created for the `python.org`_ domain:

* ``/peps/``            -> https://peps.python.org/
* ``/dev/peps/``        -> https://peps.python.org/
* ``/peps/(.*)\.html``  -> https://peps.python.org/$1
* ``/dev/peps/(.*)``    -> https://peps.python.org/$1

The following nginx configuration would achieve this:

.. code-block:: nginx

    location ~ ^/dev/peps/?(.*)$ {
        return 308 https://peps.python.org/$1/;
    }

    location ~ ^/peps/(.*)\.html$ {
        return 308 https://peps.python.org/$1/;
    }

    location ^/(dev/)?peps(/.*)?$ {
        return 308 https://peps.python.org/;
    }

Redirects MUST be implemented to preserve `URL fragments`_ for backward
compatibility purposes.

Backwards Compatibility
=======================

Due to server-side redirects to the new canonical URLs, links in previously
published materials referring to the old URL schemes will be guaranteed to work.
All PEPs will continue to render correctly, and a custom stylesheet in the
reference implementation improves presentation for some elements (most notably
code blocks and block quotes). Therefore, this PEP presents no backwards
compatibility issues.


Security Implications
=====================

The main `python.org`_ website will no longer process raw HTML uploads,
closing a potential threat vector. PEP rendering and deployment processes will
use modern, well-maintained code and secure automated platforms, further
reducing the potential attack surface. Therefore, we see no negative security
impact.


How to Teach This
=================

The new canonical URLs will be publicised in the documentation. However, this
is mainly a backend infrastructure change, and there should be minimal
end-user impact. PEP 1 and PEP 12 will be updated as needed.


Reference Implementation
========================

The proposed implementation has been merged into the `python/peps`_ repository
in a series of pull requests [8]_. It uses the `Sphinx`_ documentation system
with a custom theme and extensions.

This already automatically renders all PEPs on every commit, and publishes them
to `python.github.io/peps`_.


Rejected Ideas
==============

It would likely be possible to amend the current (as of November 2021)
rendering process to include a subset of the quality-of-life improvements and
issue mitigations mentioned above. However, we do not believe that this would
solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering system and
import it into `python.org`_. We would argue that this would be the worst of
both worlds, as a great deal of complexity is added whilst none is removed.


Acknowledgements
================

- Hugo van Kemenade
- Pablo Galindo Salgado
- Éric Araujo
- Mariatta
- C.A.M. Gerlach


Footnotes
=========

.. _documentation: https://pythondotorg.readthedocs.io/pep_generation.html
.. _docutils: https://docutils.sourceforge.io
.. _Makefile: https://www.gnu.org/software/make/manual/make.html#Introduction
.. _Netlify: https://www.netlify.com/
.. _peps.python.org: https://peps.python.org/
.. _peps.python.org/pep-0008: https://peps.python.org/pep-0008/
.. _python.github.io/peps: https://python.github.io/peps
.. _python.org: https://www.python.org
.. _python/peps: https://github.com/python/peps
.. _Read the Docs: https://readthedocs.org
.. _reStructuredText: https://docutils.sourceforge.io/rst.html
.. _Sphinx: https://www.sphinx-doc.org/en/master/
.. _URL fragments: https://url.spec.whatwg.org/#concept-url-fragment

.. [1] For example,
       `pythondotorg#1024 <https://github.com/python/pythondotorg/issues/1204>`__,
       `pythondotorg#1038 <https://github.com/python/pythondotorg/issues/1038>`__,
       `pythondotorg#1387 <https://github.com/python/pythondotorg/issues/1387>`__,
       `pythondotorg#1388 <https://github.com/python/pythondotorg/issues/1388>`__,
       `pythondotorg#1393 <https://github.com/python/pythondotorg/issues/1393>`__,
       `pythondotorg#1564 <https://github.com/python/pythondotorg/issues/1564>`__,
       `pythondotorg#1913 <https://github.com/python/pythondotorg/issues/1913>`__,
.. [2] Requested: `pythondotorg#1063 <https://github.com/python/pythondotorg/pull/1063>`__,
       `pythondotorg#1206 <https://github.com/python/pythondotorg/issues/1206>`__,
       `pythondotorg#1638 <https://github.com/python/pythondotorg/pull/1638>`__,
       `peps#159 <https://github.com/python/peps/issues/159>`__,
       `comment in peps#1571 <https://github.com/python/peps/pull/1571#discussion_r478701944>`__,
       `peps#1577 <https://github.com/python/peps/pull/1577>`__,
.. [3] Requested: `peps#701 <https://github.com/python/peps/issues/701>`__
.. [4] Requested: `peps#165 <https://github.com/python/peps/issues/165>`__
.. [5] Requested: `pythondotorg#1564 <https://github.com/python/pythondotorg/issues/1564>`__
.. [6] Requested: `comment in peps#2 <https://github.com/python/peps/issues/2#issuecomment-339195595>`__
.. [7] As of November 2021, see
       `peps#1387 <https://github.com/python/peps/issues/1387>`__,
       `pythondotorg#824 <https://github.com/python/pythondotorg/issues/824>`__,
       `pythondotorg#1556 <https://github.com/python/pythondotorg/pull/1556>`__,
.. [8] Implementation PRs:
       `peps#1930 <https://github.com/python/peps/pull/1930>`__,
       `peps#1931 <https://github.com/python/peps/pull/1931>`__,
       `peps#1932 <https://github.com/python/peps/pull/1932>`__,
       `peps#1933 <https://github.com/python/peps/pull/1933>`__,
       `peps#1934 <https://github.com/python/peps/pull/1934>`__


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


..
 Local Variables:
 mode: indented-text
 indent-tabs-mode: nil
 sentence-end-double-space: t
 fill-column: 70
 coding: utf-8
 End: