iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 511 

* bytecode => code
* more fun with bytecode
* enhance the Changes section
This commit is contained in:
Victor Stinner 2016-01-15 16:18:00 +01:00
parent 5af754e05f
commit 77c59a0cdd
1 changed files with 190 additions and 68 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

258
pep-0511.txt
View File

@ -12,10 +12,11 @@ Python-Version: 3.6
Abstract
========
Propose an API to register AST and bytecode transformers. Add also ``-o
OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
``ImportError`` exception on import if the ``.pyc`` file is missing and
the code transformers required to transform the code are missing. code
Propose an API to register bytecode and AST transformers. Add also ``-o
OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o
noopt`` disables the peephole optimizer. Raise an ``ImportError``
exception on import if the ``.pyc`` file is missing and the code
transformers required to transform the code are missing. code
transformers are not needed code transformed ahead of time (loaded from
``.pyc`` files).
@ -39,16 +40,14 @@ Writing an optimizer or a preprocessor is out of the scope of this PEP.
Usage 1: AST optimizer
----------------------
Python 3.6 optimizes the code using a peephole optimizer. By
definition, a peephole optimizer has a narrow view of the code and so
can only implement basic optimizations. The optimizer rewrites the
bytecode. It is difficult to enhance it, because it written in C.
Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It's easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.
Since the optimization can done ahead of time, complex but slow
optimizations can be implemented.
Example of optimizations which can be implemented with an AST optimizer:
* `Copy propagation
@ -101,7 +100,9 @@ Usage 2: Preprocessor
---------------------
A preprocessor can be easily implemented with an AST transformer. A
preprocessor has various and different usages. Examples:
preprocessor has various and different usages.
Some examples:
* Remove debug code like assertions and logs to make the code faster to run
it for production.
@ -125,10 +126,49 @@ preprocessor has various and different usages. Examples:
`MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
examples and use cases.
This PEP does not add any new code transformer. Using a code transformer
will require an external module and to register it manually.
See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python
obfuscator, deobfuscator, and user-assisted decompiler.
Usage 3: Disable all optimization
---------------------------------
Ned Batchelder asked to add an option to disable the peephole optimizer
because it makes code coverage more difficult to implement. See the
discussion on the python-ideas mailing list: `Disable all peephole
optimizations
<https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_.
This PEP adds a new ``-o noopt`` command line option to disable the
peephole optimizer. In Python, it's as easy as::
sys.set_code_transformers([])
It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add
mechanism to disable optimizations.
Usage 4: Write new bytecode optimizers in Python
------------------------------------------------
Python 3.6 optimizes the code using a peephole optimizer. By
definition, a peephole optimizer has a narrow view of the code and so
can only implement basic optimizations. The optimizer rewrites the
bytecode. It is difficult to enhance it, because it written in C.
With this PEP, it becomes possible to implement a new bytecode optimizer
in pure Python and experiment new optimizations.
Some optimizations are easier to implement on the AST like constant
folding, but optimizations on the bytecode are still useful. For
example, when the AST is compiled to bytecode, useless jumps can be
emited because the compiler is naive and does not try to optimize
anything.
Use Cases
=========
@ -199,17 +239,85 @@ execute the transformed code if all required ``.pyc`` files are already
available.
Code transformer API
====================
A code transformer is a class with ``ast_transformer()`` and/or
``code_transformer()`` methods (API described below) and a ``name``
attribute.
For efficiency, do not define a ``code_transformer()`` or
``ast_transformer()`` method if it does nothing.
The ``name`` attribute (``str``) must be a short string used to identify
an optimizer. It is used to build a ``.pyc`` filename. The name must not
contain dots (``'.'``), dashes (``'-'``) or directory separators: dots
are used to separated fields in a ``.pyc`` filename and dashes areused
to join code transformer names to build the optimizer tag.
.. note::
It would be nice to pass the fully qualified name of a module in the
*context* when an AST transformer is used to transform a module on
import, but it looks like the information is not available in
``PyParser_ASTFromStringObject()``.
code_transformer()
------------------
Prototype::
def code_transformer(code, consts, names, lnotab, context):
...
return (code, consts, names, lnotab)
Parameters:
* *code*: the bytecode (``bytes``)
* *consts*: a sequence of constants
* *names*: tuple of variable names
* *lnotab*: table mapping instruction offsets to line numbers (``bytes``)
The code transformer is run after the compilation to bytecode
ast_transformer()
------------------
Prototype::
def ast_transformer(tree, context):
...
return tree
Parameters:
* *tree*: an AST tree
* *context*: an object with a ``filename`` attribute (``str``)
It must return an AST tree. It can modify the AST tree in place, or
create a new AST tree.
The AST transformer is called after the creation of the AST by the
parser and before the compilation to bytecode. New attributes may be
added to *context* in the future.
Changes
=======
This PEP proposes to add an API to register code transformers.
In short, add:
The transformation can done ahead of time. It allows to implement
powerful but expensive transformations.
* ``-o OPTIM_TAG`` command line option
* ``ast.Constant``
* ``ast.PyCF_TRANSFORMED_AST``
* ``sys.get_code_transformers()``
* ``sys.implementation.optim_tag``
* ``sys.set_code_transformers(transformers)``
API for code transformers
-------------------------
API to get/set code transformers
--------------------------------
Add new functions to register code transformers:
@ -222,43 +330,15 @@ The order of code transformers matter. Running transformer A and then
transformer B can give a different output than running transformer B an
then transformer A.
API of an code transformer:
Example to prepend a new code transformer::
* An code transformer is a class with ``ast_transformer()`` and/or
``bytecode_transformer()`` methods and a ``name`` attribute. Example::
transformers = sys.get_code_transformers()
transformers.insert(0, new_cool_transformer)
sys.set_code_transformers(transformers)
class MyTransformer:
name = 'my_transformer'
def bytecode_transformer(self, bytecode, ...):
...
return tree
def ast_transformer(self, tree, context):
...
return tree
``bytecode_transformer()``: *bytecode* is bytes objects.
``ast_transformer()``: *tree* is an AST tree and *context* is an
object with a ``filename`` attribute (``str``). New attributes may be
added to *context* in the future. It must return an AST tree. It can
modify the AST tree in place, or create a new AST tree.
* The ``name`` attribute (``str``) must be a short string used to
identify an optimizer. The name must not contain ``.`` (dot), ``-``
(dash) characters or directory separators: ``.`` is used to separated
fields in a ``.pyc`` filename and ``-`` is used to join AST
transformer names to build the optimizer tag.
* The AST transformer is called after the creation of the AST by the
parser and before the compilation to bytecode
* The bytecode transformer is after the compilation to bytecode
.. note::
It would be nice to pass the fully qualified name of a module in the
*context* when an AST transformer is used to transform a module, but
it looks like the information is not available in
``PyParser_ASTFromStringObject()``.
All AST tranformers are run sequentially (ex: the second transformer
gets the input of the first transformer), and then all bytecode
transformers are run sequentially.
Optimizer tag
@ -269,7 +349,7 @@ Changes:
* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
The default optimization tag is ``'opt'``.
* Add a new ``-o OPTIM_TAG`` command line option to set
``sys.implementation.optim_tag``
``sys.implementation.optim_tag``.
Changes on ``importlib``:
@ -302,6 +382,19 @@ The behaviour of the ``importlib`` module is unchanged with the default
optimizer tag (``'opt'``).
Peephole optimizer
------------------
By default, ``sys.implementation.optim_tag`` is ``opt`` and
``sys.get_code_transformers()`` returns a list of one code transformer:
the peephole optimizer (optimize the bytecode).
Use ``-o noopt`` to disable the peephole optimizer. In this case, the
optimizer tag is ``noopt`` and no code transformer is registered.
Using the ``-o opt`` option has not effect.
AST enhancements
----------------
@ -326,8 +419,8 @@ Enhancements to simplify the implementation of AST transformers:
node type
Example
=======
Examples
========
.pyc filenames
--------------
@ -355,10 +448,40 @@ With the ``'fat'`` optimizer tag:
=========================== ==================
Bytecode transformer
--------------------
Scary bytecode transformer replacing all strings with
``"Ni! Ni! Ni!"``::
import sys
class BytecodeTransformer:
name = "knights_who_say_ni"
def code_transformer(self, code, consts, names, lnotab, context):
consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const
for const in consts]
return (code, consts, names, lnotab)
# replace existing code transformers with our bytecode transformer
sys.set_code_transformers([BytecodeTransformer()])
# execute code which will be transformed by ast_transformer()
exec("print('Hello World!')")
Output::
Ni! Ni! Ni!
AST transformer
---------------
Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
Similary to the bytecode transformer example, the AST transformer also
replaces all strings with ``"Ni! Ni! Ni!"``::
import ast
import sys
@ -381,10 +504,8 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
return tree
# append our AST transformer after existing transformers
transformers = sys.get_code_transformers()
transformers.append(ASTTransformer())
sys.set_code_transformers(transformers)
# replace existing code transformers with the new AST transformer
sys.set_code_transformers([ASTTransformer()])
# execute code which will be transformed by ast_transformer()
exec("print('Hello World!')")
@ -397,8 +518,8 @@ Output::
Other Python implementations
============================
The PEP 511 should be implemented be all Python implementation. The AST
emited by the parser is not specified.
The PEP 511 should be implemented by all Python implementation, but the
bytecode and the AST are not standardized.
By the way, even between minor version of CPython, there are changes on
the AST API. There are differences, but only minor differences. It is
@ -432,8 +553,13 @@ implementing various optimizations. Most interesting optimizations break
the Python semantics since no guard is used to disable optimization if
something changes.
Issue #17515: `Add sys.setasthook() to allow to use a custom AST
optimizer <https://bugs.python.org/issue17515>`_.
In 2015, Victor Stinner wrote the `fatoptimizer
<http://fatoptimizer.readthedocs.org/>`_ project, an AST optimizer
specializing functions using guards.
The Issue #17515 `"Add sys.setasthook() to allow to use a custom AST"
optimizer <https://bugs.python.org/issue17515>`_ was a first attempt of
API for code transformers, but specific to AST.
Python Preprocessors
@ -448,8 +574,8 @@ Python Preprocessors
preprocessor directives in Python, like ``#define`` and ``#ifdef``
Modify the bytecode
-------------------
Bytecode transformers
---------------------
* `codetransformer <https://pypi.python.org/pypi/codetransformer>`_:
Bytecode transformers for CPython inspired by the ``ast`` module’s
@ -465,10 +591,6 @@ Modify the bytecode
See also:
* `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_
* `Issue #2506 <https://bugs.python.org/issue2506>`_: Add mechanism to
disable optimizations
* `[Python-ideas] Disable all peephole optimizations
<https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_
Copyright