From 5af754e05fb87f5de0dcc94a62461f84f7892070 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 15 Jan 2016 12:54:09 +0100 Subject: [PATCH] PEP 511: add bytecode transformers --- pep-0511.txt | 107 +++++++++++++++++++++++++++++---------------------- 1 file changed, 60 insertions(+), 47 deletions(-) diff --git a/pep-0511.txt b/pep-0511.txt index 1cffa46f0..ddef9bdcd 100644 --- a/pep-0511.txt +++ b/pep-0511.txt @@ -1,5 +1,5 @@ PEP: 511 -Title: API for AST transformers +Title: API for code transformers Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner @@ -12,12 +12,12 @@ Python-Version: 3.6 Abstract ======== -Propose an API to support AST transformers. Add also ``-o OPTIM_TAG`` -command line option to change ``.pyc`` filenames. Raise an +Propose an API to register AST and bytecode transformers. Add also ``-o +OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an ``ImportError`` exception on import if the ``.pyc`` file is missing and -the AST transformers required to transform the code are missing. -AST transformers are not needed code transformed ahead of time (loaded -from ``.pyc`` files). +the code transformers required to transform the code are missing. code +transformers are not needed code transformed ahead of time (loaded from +``.pyc`` files). Rationale @@ -132,18 +132,18 @@ obfuscator, deobfuscator, and user-assisted decompiler. Use Cases ========= -This section give examples of use cases explaining when and how AST +This section give examples of use cases explaining when and how code transformers will be used. Interactive interpreter ----------------------- -It will be possible to use AST transformers with the interactive +It will be possible to use code transformers with the interactive interpreter which is popular in Python and commonly used to demonstrate Python. The code is transformed at runtime and so the interpreter can be slower -when expensive AST transformers are used. +when expensive code transformers are used. Build a transformed package --------------------------- @@ -153,7 +153,7 @@ It will be possible to build a package of the transformed code. A transformer can have a configuration. The configuration is not stored in the package. -All ``.pyc`` files of the package must be transformed with the same AST +All ``.pyc`` files of the package must be transformed with the same code transformers and the same transformers configuration. It is possible to build different ``.pyc`` files using different @@ -181,7 +181,7 @@ If a package does not contain any ``.pyc`` files of the current optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are created during the installation. -AST transformers of the optimizer tag are required. Otherwise, the +Code transformers of the optimizer tag are required. Otherwise, the installation fails with an error. @@ -191,10 +191,10 @@ Execute transformed code It will be possible to execute transformed code. Raise an ``ImportError`` exception on import if the ``.pyc`` file of the -current optimizer tag is missing and the AST transformers required to +current optimizer tag is missing and the code transformers required to transform the code are missing. -The interesting point here is that AST transformers are not needed to +The interesting point here is that code transformers are not needed to execute the transformed code if all required ``.pyc`` files are already available. @@ -202,46 +202,57 @@ available. Changes ======= -This PEP proposes to add an API to register AST transformers. +This PEP proposes to add an API to register code transformers. The transformation can done ahead of time. It allows to implement powerful but expensive transformations. -API for AST transformers ------------------------- +API for code transformers +------------------------- -Add new functions to register AST transformers: +Add new functions to register code transformers: -* ``sys.set_ast_transformers(transformers)``: set the list of AST - transformers -* ``sys.get_ast_transformers()``: get the list of AST +* ``sys.set_code_transformers(transformers)``: set the list of code + transformers and update ``sys.implementation.optim_tag`` +* ``sys.get_code_transformers()``: get the list of code transformers. -The order of AST transformers matter. Running transformer A and then +The order of code transformers matter. Running transformer A and then transformer B can give a different output than running transformer B an then transformer A. -API of an AST transformer: +API of an code transformer: -* An AST transformer is a callable object with the prototype:: +* An code transformer is a class with ``ast_transformer()`` and/or + ``bytecode_transformer()`` methods and a ``name`` attribute. Example:: - def ast_transformer(tree, context): - ... - return tree + class MyTransformer: + name = 'my_transformer' - where *tree* is an AST tree and *context* is an object with a - ``filename`` attribute (``str``). New attributes may be added to - *context* in the future. + def bytecode_transformer(self, bytecode, ...): + ... + return tree -* It must return an AST tree. -* It must have a ``name`` attribute (``str``): short string used to identify an - optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters: - ``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used - to join AST transformer names to build the optimizer tag. -* The transformer is called after the creation of the AST by the parser - and before the compilation to bytecode -* It can modify the AST tree in place, or create a new AST tree. + def ast_transformer(self, tree, context): + ... + return tree + + ``bytecode_transformer()``: *bytecode* is bytes objects. + + ``ast_transformer()``: *tree* is an AST tree and *context* is an + object with a ``filename`` attribute (``str``). New attributes may be + added to *context* in the future. It must return an AST tree. It can + modify the AST tree in place, or create a new AST tree. + +* The ``name`` attribute (``str``) must be a short string used to + identify an optimizer. The name must not contain ``.`` (dot), ``-`` + (dash) characters or directory separators: ``.`` is used to separated + fields in a ``.pyc`` filename and ``-`` is used to join AST + transformer names to build the optimizer tag. +* The AST transformer is called after the creation of the AST by the + parser and before the compilation to bytecode +* The bytecode transformer is after the compilation to bytecode .. note:: It would be nice to pass the fully qualified name of a module in the @@ -267,23 +278,23 @@ Changes on ``importlib``: ``opt``. Remove also the special case for the optimizer level ``0`` with the default optimizer tag ``'opt'`` to simplify the code. * When loading a module, if the ``.pyc`` file is missing but the ``.py`` - is available, the ``.py`` is only used if AST optimizers have the same - optimizer tag than the current tag, otherwise an ``ImportError`` + is available, the ``.py`` is only used if code optimizers have the + same optimizer tag than the current tag, otherwise an ``ImportError`` exception is raised. Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can be compiled to import a module:: - def get_ast_optim_tag(): - transformers = sys.get_ast_transformers() + def transformers_tag(): + transformers = sys.get_code_transformers() if not transformers: return 'opt' return '-'.join(transformer.name for transformer in transformers) def use_py(): - return (get_ast_transformers() == sys.implementation.optim_tag) + return (transformers_tag() == sys.implementation.optim_tag) -The order of ``sys.get_ast_transformers()`` matter. For example, the +The order of ``sys.get_code_transformers()`` matter. For example, the ``fat`` transformer followed by the ``pythran`` transformer gives the optimizer tag ``fat-pythran``. @@ -345,7 +356,7 @@ With the ``'fat'`` optimizer tag: AST transformer ----------------- +--------------- Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``:: @@ -365,13 +376,15 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``:: def __init__(self): self.transformer = KnightsWhoSayNi() - def __call__(self, tree, context): + def ast_transformer(self, tree, context): self.transformer.visit(tree) return tree - # register the AST transformer - sys.set_ast_transformers([ASTTransformer()]) + # append our AST transformer after existing transformers + transformers = sys.get_code_transformers() + transformers.append(ASTTransformer()) + sys.set_code_transformers(transformers) # execute code which will be transformed by ast_transformer() exec("print('Hello World!')")