PEP 511: add bytecode transformers
This commit is contained in:
Victor Stinner
parent
51f79940e1
commit
5af754e05f
107
pep-0511.txt
107
pep-0511.txt
|
|
@ -1,5 +1,5 @@
|
|||
PEP: 511
|
||||
Title: API for AST transformers
|
||||
Title: API for code transformers
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Victor Stinner <victor.stinner@gmail.com>
|
||||
|
|
@ -12,12 +12,12 @@ Python-Version: 3.6
|
|||
Abstract
|
||||
========
|
||||
|
||||
Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
|
||||
command line option to change ``.pyc`` filenames. Raise an
|
||||
Propose an API to register AST and bytecode transformers. Add also ``-o
|
||||
OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
|
||||
``ImportError`` exception on import if the ``.pyc`` file is missing and
|
||||
the AST transformers required to transform the code are missing.
|
||||
AST transformers are not needed code transformed ahead of time (loaded
|
||||
from ``.pyc`` files).
|
||||
the code transformers required to transform the code are missing. code
|
||||
transformers are not needed code transformed ahead of time (loaded from
|
||||
``.pyc`` files).
|
||||
|
||||
|
||||
Rationale
|
||||
|
|
@ -132,18 +132,18 @@ obfuscator, deobfuscator, and user-assisted decompiler.
|
|||
Use Cases
|
||||
=========
|
||||
|
||||
This section give examples of use cases explaining when and how AST
|
||||
This section give examples of use cases explaining when and how code
|
||||
transformers will be used.
|
||||
|
||||
Interactive interpreter
|
||||
-----------------------
|
||||
|
||||
It will be possible to use AST transformers with the interactive
|
||||
It will be possible to use code transformers with the interactive
|
||||
interpreter which is popular in Python and commonly used to demonstrate
|
||||
Python.
|
||||
|
||||
The code is transformed at runtime and so the interpreter can be slower
|
||||
when expensive AST transformers are used.
|
||||
when expensive code transformers are used.
|
||||
|
||||
Build a transformed package
|
||||
---------------------------
|
||||
|
|
@ -153,7 +153,7 @@ It will be possible to build a package of the transformed code.
|
|||
A transformer can have a configuration. The configuration is not stored
|
||||
in the package.
|
||||
|
||||
All ``.pyc`` files of the package must be transformed with the same AST
|
||||
All ``.pyc`` files of the package must be transformed with the same code
|
||||
transformers and the same transformers configuration.
|
||||
|
||||
It is possible to build different ``.pyc`` files using different
|
||||
|
|
@ -181,7 +181,7 @@ If a package does not contain any ``.pyc`` files of the current
|
|||
optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are
|
||||
created during the installation.
|
||||
|
||||
AST transformers of the optimizer tag are required. Otherwise, the
|
||||
Code transformers of the optimizer tag are required. Otherwise, the
|
||||
installation fails with an error.
|
||||
|
||||
|
||||
|
|
@ -191,10 +191,10 @@ Execute transformed code
|
|||
It will be possible to execute transformed code.
|
||||
|
||||
Raise an ``ImportError`` exception on import if the ``.pyc`` file of the
|
||||
current optimizer tag is missing and the AST transformers required to
|
||||
current optimizer tag is missing and the code transformers required to
|
||||
transform the code are missing.
|
||||
|
||||
The interesting point here is that AST transformers are not needed to
|
||||
The interesting point here is that code transformers are not needed to
|
||||
execute the transformed code if all required ``.pyc`` files are already
|
||||
available.
|
||||
|
||||
|
|
@ -202,46 +202,57 @@ available.
|
|||
Changes
|
||||
=======
|
||||
|
||||
This PEP proposes to add an API to register AST transformers.
|
||||
This PEP proposes to add an API to register code transformers.
|
||||
|
||||
The transformation can done ahead of time. It allows to implement
|
||||
powerful but expensive transformations.
|
||||
|
||||
|
||||
API for AST transformers
|
||||
------------------------
|
||||
API for code transformers
|
||||
-------------------------
|
||||
|
||||
Add new functions to register AST transformers:
|
||||
Add new functions to register code transformers:
|
||||
|
||||
* ``sys.set_ast_transformers(transformers)``: set the list of AST
|
||||
transformers
|
||||
* ``sys.get_ast_transformers()``: get the list of AST
|
||||
* ``sys.set_code_transformers(transformers)``: set the list of code
|
||||
transformers and update ``sys.implementation.optim_tag``
|
||||
* ``sys.get_code_transformers()``: get the list of code
|
||||
transformers.
|
||||
|
||||
The order of AST transformers matter. Running transformer A and then
|
||||
The order of code transformers matter. Running transformer A and then
|
||||
transformer B can give a different output than running transformer B an
|
||||
then transformer A.
|
||||
|
||||
API of an AST transformer:
|
||||
API of an code transformer:
|
||||
|
||||
* An AST transformer is a callable object with the prototype::
|
||||
* An code transformer is a class with ``ast_transformer()`` and/or
|
||||
``bytecode_transformer()`` methods and a ``name`` attribute. Example::
|
||||
|
||||
def ast_transformer(tree, context):
|
||||
...
|
||||
return tree
|
||||
class MyTransformer:
|
||||
name = 'my_transformer'
|
||||
|
||||
where *tree* is an AST tree and *context* is an object with a
|
||||
``filename`` attribute (``str``). New attributes may be added to
|
||||
*context* in the future.
|
||||
def bytecode_transformer(self, bytecode, ...):
|
||||
...
|
||||
return tree
|
||||
|
||||
* It must return an AST tree.
|
||||
* It must have a ``name`` attribute (``str``): short string used to identify an
|
||||
optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters:
|
||||
``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used
|
||||
to join AST transformer names to build the optimizer tag.
|
||||
* The transformer is called after the creation of the AST by the parser
|
||||
and before the compilation to bytecode
|
||||
* It can modify the AST tree in place, or create a new AST tree.
|
||||
def ast_transformer(self, tree, context):
|
||||
...
|
||||
return tree
|
||||
|
||||
``bytecode_transformer()``: *bytecode* is bytes objects.
|
||||
|
||||
``ast_transformer()``: *tree* is an AST tree and *context* is an
|
||||
object with a ``filename`` attribute (``str``). New attributes may be
|
||||
added to *context* in the future. It must return an AST tree. It can
|
||||
modify the AST tree in place, or create a new AST tree.
|
||||
|
||||
* The ``name`` attribute (``str``) must be a short string used to
|
||||
identify an optimizer. The name must not contain ``.`` (dot), ``-``
|
||||
(dash) characters or directory separators: ``.`` is used to separated
|
||||
fields in a ``.pyc`` filename and ``-`` is used to join AST
|
||||
transformer names to build the optimizer tag.
|
||||
* The AST transformer is called after the creation of the AST by the
|
||||
parser and before the compilation to bytecode
|
||||
* The bytecode transformer is after the compilation to bytecode
|
||||
|
||||
.. note::
|
||||
It would be nice to pass the fully qualified name of a module in the
|
||||
|
|
@ -267,23 +278,23 @@ Changes on ``importlib``:
|
|||
``opt``. Remove also the special case for the optimizer level ``0``
|
||||
with the default optimizer tag ``'opt'`` to simplify the code.
|
||||
* When loading a module, if the ``.pyc`` file is missing but the ``.py``
|
||||
is available, the ``.py`` is only used if AST optimizers have the same
|
||||
optimizer tag than the current tag, otherwise an ``ImportError``
|
||||
is available, the ``.py`` is only used if code optimizers have the
|
||||
same optimizer tag than the current tag, otherwise an ``ImportError``
|
||||
exception is raised.
|
||||
|
||||
Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
|
||||
be compiled to import a module::
|
||||
|
||||
def get_ast_optim_tag():
|
||||
transformers = sys.get_ast_transformers()
|
||||
def transformers_tag():
|
||||
transformers = sys.get_code_transformers()
|
||||
if not transformers:
|
||||
return 'opt'
|
||||
return '-'.join(transformer.name for transformer in transformers)
|
||||
|
||||
def use_py():
|
||||
return (get_ast_transformers() == sys.implementation.optim_tag)
|
||||
return (transformers_tag() == sys.implementation.optim_tag)
|
||||
|
||||
The order of ``sys.get_ast_transformers()`` matter. For example, the
|
||||
The order of ``sys.get_code_transformers()`` matter. For example, the
|
||||
``fat`` transformer followed by the ``pythran`` transformer gives the
|
||||
optimizer tag ``fat-pythran``.
|
||||
|
||||
|
|
@ -345,7 +356,7 @@ With the ``'fat'`` optimizer tag:
|
|||
|
||||
|
||||
AST transformer
|
||||
----------------
|
||||
---------------
|
||||
|
||||
Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
|
||||
|
||||
|
|
@ -365,13 +376,15 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
|
|||
def __init__(self):
|
||||
self.transformer = KnightsWhoSayNi()
|
||||
|
||||
def __call__(self, tree, context):
|
||||
def ast_transformer(self, tree, context):
|
||||
self.transformer.visit(tree)
|
||||
return tree
|
||||
|
||||
|
||||
# register the AST transformer
|
||||
sys.set_ast_transformers([ASTTransformer()])
|
||||
# append our AST transformer after existing transformers
|
||||
transformers = sys.get_code_transformers()
|
||||
transformers.append(ASTTransformer())
|
||||
sys.set_code_transformers(transformers)
|
||||
|
||||
# execute code which will be transformed by ast_transformer()
|
||||
exec("print('Hello World!')")
|
||||
|
|
|
|||
Loading…
Reference in New Issue