python-peps/peps/pep-0306.rst

PEP: 306
Title: How to Change Python's Grammar
Version: $Revision$
Last-Modified: $Date$
Author: Michael Hudson <mwh@python.net>, Jack Diederich <jackdied@gmail.com>, Alyssa Coghlan <ncoghlan@gmail.com>, Benjamin Peterson <benjamin@python.org>
Status: Withdrawn
Type: Informational
Content-Type: text/x-rst
Created: 29-Jan-2003
Post-History: 30-Jan-2003


Note
====

This PEP has been moved to the Python dev guide [1]_.


Abstract
========

There's more to changing Python's grammar than editing
``Grammar/Grammar`` and ``Python/compile.c``.  This PEP aims to be a
checklist of places that must also be fixed.

It is probably incomplete.  If you see omissions, just add them if
you can -- you are not going to offend the author's sense of
ownership.  Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Python
grammar hacking, for several reasons.


Rationale
=========

People are getting this wrong all the time; it took well over a
year before someone noticed [2]_ that adding the floor division
operator (``//``) broke the ``parser`` module.


Checklist
=========

- ``Grammar/Grammar``: OK, you'd probably worked this one out :)

- ``Parser/Python.asdl`` may need changes to match the ``Grammar``.  Run
  ``make`` to regenerate ``Include/Python-ast.h`` and
  ``Python/Python-ast.c``.

- ``Python/ast.c`` will need changes to create the AST objects
  involved with the ``Grammar`` change.  ``Lib/compiler/ast.py`` will
  need matching changes to the pure-python AST objects.

- ``Parser/pgen`` needs to be rerun to regenerate ``Include/graminit.h``
  and ``Python/graminit.c``. (make should handle this for you.)

- ``Python/symbtable.c``: This handles the symbol collection pass
  that happens immediately before the compilation pass.

- ``Python/compile.c``: You will need to create or modify the
  ``compiler_*`` functions to generate opcodes for your productions.

- You may need to regenerate ``Lib/symbol.py`` and/or ``Lib/token.py``
  and/or ``Lib/keyword.py``.

- The ``parser`` module.  Add some of your new syntax to ``test_parser``,
  bang on ``Modules/parsermodule.c`` until it passes.

- Add some usage of your new syntax to ``test_grammar.py``.

- The ``compiler`` package.  A good test is to compile the standard
  library and test suite with the ``compiler`` package and then check
  it runs.  Note that this only needs to be done in Python 2.x.

- If you've gone so far as to change the token structure of
  Python, then the ``Lib/tokenizer.py`` library module will need to
  be changed.

- Certain changes may require tweaks to the library module
  ``pyclbr``.

- Documentation must be written!

- After everything's been checked in, you're likely to see a new
  change to ``Python/Python-ast.c``.  This is because this
  (generated) file contains the SVN version of the source from
  which it was generated.  There's no way to avoid this; you just
  have to submit this file separately.


References
==========

.. [1] CPython Developer's Guide: Changing CPython's Grammar
       https://devguide.python.org/grammar/

.. [2] SF Bug #676521, parser module validation failure
       https://bugs.python.org/issue676521


Copyright
=========

This document has been placed in the public domain.



..
  Local Variables:
  mode: indented-text
  indent-tabs-mode: nil
  sentence-end-double-space: t
  fill-column: 70
  End: