PEP 511

* Fix the API: use sys.get/set_ast_transformers()
* elaborate changes on importlib, try to be more concrete
* add more examples of usage of AST transformers

pep-0511.txt

@@ -37,15 +37,40 @@ importing modules, see described use cases below.
 Usage 1: AST optimizer
 ----------------------
 
+Python 3.6 optimizes the code using a peephole optimizer. By
+definition, a peephole optimizer has a narrow view of the code and so
+can only implement basic optimizations. The optimizer rewrites the
+bytecode. It is difficult to enhance it, because it written in C.
+
 Transforming an Abstract Syntax Tree (AST) is a convenient
 way to implement an optimizer. It's easier to work on the AST than
 working on the bytecode, AST contains more information and is more high
 level.
 
-Python 3.6 optimizes the code using a peephole optimizer. By
-definition, a peephole optimizer has a narrow view of the code and so
-can only implement basic optimizations. The optimizer rewrites the
-bytecode. It is difficult to enhance it, because it written in C.
+Example of optimizations which can be implemented with an AST optimizer:
+
+* `Copy propagation
+  <https://en.wikipedia.org/wiki/Copy_propagation>`_:
+  replace ``x=1; y=x`` with ``x=1; y=1``
+* `Constant folding
+  <https://en.wikipedia.org/wiki/Constant_folding>`_:
+  replace ``1+1`` with ``2``
+* `Dead code elimination
+  <https://en.wikipedia.org/wiki/Dead_code_elimination>`_
+
+Using guards (see the PEP 510), it is possible to implement a much wider choice
+of optimizations. Examples:
+
+* Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used
+  as iterable
+* `Loop unrolling <https://en.wikipedia.org/wiki/Loop_unrolling>`_
+* Call pure builtins: replace ``len("abc")`` with ``3``
+* Copy used builtin symbols to constants
+
+See also `optimizations implemented in fatoptimizer
+<https://fatoptimizer.readthedocs.org/en/latest/optimizations.html>`_, a
+static optimizer for Python 3.6.
+
 
 Usage 2: Preprocessor
 ---------------------
@@ -53,14 +78,17 @@ Usage 2: Preprocessor
 A preprocessor can be easily implemented with an AST transformer. A
 preprocessor has various and different usages. Examples:
 
-* Remove debug code (like assertions and logs) to make the code faster to run
+* Remove debug code like assertions and logs to make the code faster to run
   it for production.
 * `Tail-call Optimization <https://en.wikipedia.org/wiki/Tail_call>`_
 * Add profiling code
-* Lazy macro create a memoizing thunk.
-
-Examples extending or changing the Python language:
-
+* `Lazy evaluation <https://en.wikipedia.org/wiki/Lazy_evaluation>`_:
+  see `lazy_python <https://github.com/llllllllll/lazy_python>`_
+  (bytecode transformer) and `lazy macro of MacroPy
+  <https://github.com/lihaoyi/macropy#lazy>`_ (AST transformer)
+* Change dictionary literals into collection.OrderedDict instances
+* Declare constants: see `@asconstants of codetransformer
+  <https://pypi.python.org/pypi/codetransformer>`_
 * Domain Specific Language (DSL) like SQL queries. The
   Python language itself doesn't need to be modified. Previous attempts to
   implement DSL for SQL like `PEP 335 - Overloadable Boolean Operators
@@ -69,7 +97,8 @@ Examples extending or changing the Python language:
 * String Interpolation, but `PEP 498 -- Literal String Interpolation
   <https://www.python.org/dev/peps/pep-0498/>`_ was merged into Python 3.6.
 
-MacroPy has a much longer list of examples and use cases.
+`MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
+examples and use cases.
 
 
 Use Cases
@@ -147,38 +176,25 @@ Changes
 
 This PEP proposes to add an API to register AST transformers.
 
-A new ``-o OPTIM_TAG`` command line option is added to only load
-transformed code: it changes the name of searched ``.pyc`` files. If the
-``.pyc`` file of a module is missing and the ``.py`` is available, an
-``ImportError`` exception is raised import if the AST transformers
-required to transform the code are missing. The import behaviour with
-the default optimizer tag (``'opt'``) is unchanged.
-
 The transformation can done ahead of time. It allows to implement
 powerful but expensive transformations.
 
-API to support AST transformers:
 
-* Add ``sys.ast_transformers``: list of AST transformers used to rewrite
-  an AST tree. The list is empty by default: no AST transformer.
-* Add ``sys.implementation.ast_transformers``: name of AST
-  transformers registered in ``sys.ast_transformers``
-* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
-  The default optimization tag is ``'opt'``.
-* Use the optimizer tag in ``.pyc`` filenames in ``importlib``.
-  Remove also the special case for the optimizer level ``0`` with the
-  default optimizer tag ``'opt'`` to simplify the code.
-* Add a new ``-o OPTIM_TAG`` command line option to set
-  ``sys.implementation.optim_tag``
+API for AST transformers
+------------------------
 
-.. note::
-   FIXME: There is a design issue: ``sys.ast_transformers`` and
-   ``sys.implementation.ast_transformers`` can be inconsistent.
-   ``sys.implementation.ast_transformers`` is required at runtime in
-   some corner cases to have specific code depending if a specific AST
-   transformer was used. Do you have a better suggestion?
+Add new functions to register AST transformers:
 
-API of an AST transformer (from ``sys.ast_transformers``):
+* ``sys.set_ast_transformers(transformers)``: set the list of AST
+  transformers
+* ``sys.get_ast_transformers()``: get the list of AST
+  transformers.
+
+The order of AST transformers matter. Running transformer A and then
+transformer B can give a different output than running transformer B an
+then transformer A.
+
+API of an AST transformer:
 
 * An AST transformer is a callable object with the prototype::
 
@@ -202,9 +218,55 @@ API of an AST transformer (from ``sys.ast_transformers``):
 .. note::
    It would be nice to pass the fully qualified name of a module in the
    *context*  when an AST transformer is used to transform a module, but
-   it looks like the information is not available currently.
+   it looks like the information is not available in
+   ``PyParser_ASTFromStringObject()``.
 
-AST transformer changes:
+
+Optimizer tag
+-------------
+
+Changes:
+
+* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
+  The default optimization tag is ``'opt'``.
+* Add a new ``-o OPTIM_TAG`` command line option to set
+  ``sys.implementation.optim_tag``
+
+Changes on ``importlib``:
+
+* ``importlib`` uses ``sys.implementation.optim_tag`` to build the
+  ``.pyc`` filename to importing modules, instead of always using
+  ``opt``. Remove also the special case for the optimizer level ``0``
+  with the default optimizer tag ``'opt'`` to simplify the code.
+* When loading a module, if the ``.pyc`` file is missing but the ``.py``
+  is available, the ``.py`` is only used if AST optimizers have the same
+  optimizer tag than the current tag, otherwise an ``ImportError``
+  exception is raised.
+
+Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
+be compiled to import a module::
+
+    def get_ast_optim_tag():
+        transformers = sys.get_ast_transformers()
+        if not transformers:
+            return 'opt'
+        return '-'.join(transformer.name for transformer in transformers)
+
+    def use_py():
+        return (get_ast_transformers() == sys.implementation.optim_tag)
+
+The order of ``sys.get_ast_transformers()`` matter. For example, the
+``fat`` transformer followed by the ``pythran`` transformer gives the
+optimizer tag ``fat-pythran``.
+
+The behaviour of the ``importlib`` module is unchanged with the default
+optimizer tag (``'opt'``).
+
+
+AST enhancements
+----------------
+
+Enhancements to simplify the implementation of AST transformers:
 
 * Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the
   transformed AST. ``PyCF_ONLY_AST`` returns the AST before the
@@ -215,9 +277,8 @@ AST transformer changes:
   frozenset items.
 * ``PyCodeObject.co_lnotab``: line number delta becomes signed to
   support moving instructions (note: need to modify MAGIC_NUMBER in
-  importlib). Implemented in the `issue #26107: code.co_lnotab: use
-  signed line number delta to support moving instructions in an
-  optimizer <https://bugs.python.org/issue26107>`_
+  importlib). Implemented in the `issue #26107
+  <https://bugs.python.org/issue26107>`_
 * Enhance the bytecode compiler to support ``tuple`` and ``frozenset``
   constants. Currently, ``tuple`` and ``frozenset`` constants are
   created by the peephole transformer, after the bytecode compilation.
@@ -273,13 +334,16 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
     class ASTTransformer:
         name = "knights_who_say_ni"
 
+        def __init__(self):
+            self.transformer = KnightsWhoSayNi()
+
         def __call__(self, tree, context):
-            KnightsWhoSayNi().visit(tree)
+            self.transformer.visit(tree)
             return tree
 
 
     # register the AST transformer
-    sys.ast_transformers.append(ASTTransformer())
+    sys.set_ast_transformers([ASTTransformer()])
 
     # execute code which will be transformed by ast_transformer()
     exec("print('Hello World!')")