PEP: 306 Title: How to Change Python's Grammar Version: $Revision$ Last-Modified: $Date$ Author: Michael Hudson , Jack Diederich , Nick Coghlan , Benjamin Peterson Status: Withdrawn Type: Informational Content-Type: text/x-rst Created: 29-Jan-2003 Post-History: 30-Jan-2003 Note ==== This PEP has been moved to the Python dev guide [1]_. Abstract ======== There's more to changing Python's grammar than editing ``Grammar/Grammar`` and ``Python/compile.c``. This PEP aims to be a checklist of places that must also be fixed. It is probably incomplete. If you see omissions, just add them if you can -- you are not going to offend the author's sense of ownership. Otherwise submit a bug or patch and assign it to mwh. This PEP is not intended to be an instruction manual on Python grammar hacking, for several reasons. Rationale ========= People are getting this wrong all the time; it took well over a year before someone noticed [2]_ that adding the floor division operator (``//``) broke the ``parser`` module. Checklist ========= - ``Grammar/Grammar``: OK, you'd probably worked this one out :) - ``Parser/Python.asdl`` may need changes to match the ``Grammar``. Run ``make`` to regenerate ``Include/Python-ast.h`` and ``Python/Python-ast.c``. - ``Python/ast.c`` will need changes to create the AST objects involved with the ``Grammar`` change. ``Lib/compiler/ast.py`` will need matching changes to the pure-python AST objects. - ``Parser/pgen`` needs to be rerun to regenerate ``Include/graminit.h`` and ``Python/graminit.c``. (make should handle this for you.) - ``Python/symbtable.c``: This handles the symbol collection pass that happens immediately before the compilation pass. - ``Python/compile.c``: You will need to create or modify the ``compiler_*`` functions to generate opcodes for your productions. - You may need to regenerate ``Lib/symbol.py`` and/or ``Lib/token.py`` and/or ``Lib/keyword.py``. - The ``parser`` module. Add some of your new syntax to ``test_parser``, bang on ``Modules/parsermodule.c`` until it passes. - Add some usage of your new syntax to ``test_grammar.py``. - The ``compiler`` package. A good test is to compile the standard library and test suite with the ``compiler`` package and then check it runs. Note that this only needs to be done in Python 2.x. - If you've gone so far as to change the token structure of Python, then the ``Lib/tokenizer.py`` library module will need to be changed. - Certain changes may require tweaks to the library module ``pyclbr``. - Documentation must be written! - After everything's been checked in, you're likely to see a new change to ``Python/Python-ast.c``. This is because this (generated) file contains the SVN version of the source from which it was generated. There's no way to avoid this; you just have to submit this file separately. References ========== .. [1] CPython Developer's Guide: Changing CPython's Grammar https://devguide.python.org/grammar/ .. [2] SF Bug #676521, parser module validation failure https://bugs.python.org/issue676521 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: