iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
python-peps/pep-0511/index.html

757 lines
54 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 511 – API for code transformers | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0511/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 511 – API for code transformers | peps.python.org'>
<meta property="og:description" content="Propose an API to register bytecode and AST transformers. Add also -o OPTIM_TAG command line option to change .pyc filenames, -o noopt disables the peephole optimizer. Raise an ImportError exception on import if the .pyc file is missing and the code tra...">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0511/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="Propose an API to register bytecode and AST transformers. Add also -o OPTIM_TAG command line option to change .pyc filenames, -o noopt disables the peephole optimizer. Raise an ImportError exception on import if the .pyc file is missing and the code tra...">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 511</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 511 – API for code transformers</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Victor Stinner &lt;vstinner&#32;&#97;t&#32;python.org&gt;</dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-even">Created<span class="colon">:</span></dt>
<dd class="field-even">04-Jan-2016</dd>
<dt class="field-odd">Python-Version<span class="colon">:</span></dt>
<dd class="field-odd">3.6</dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#usage-1-ast-optimizer">Usage 1: AST optimizer</a></li>
<li><a class="reference internal" href="#usage-2-preprocessor">Usage 2: Preprocessor</a></li>
<li><a class="reference internal" href="#usage-3-disable-all-optimization">Usage 3: Disable all optimization</a></li>
<li><a class="reference internal" href="#usage-4-write-new-bytecode-optimizers-in-python">Usage 4: Write new bytecode optimizers in Python</a></li>
</ul>
</li>
<li><a class="reference internal" href="#use-cases">Use Cases</a><ul>
<li><a class="reference internal" href="#interactive-interpreter">Interactive interpreter</a></li>
<li><a class="reference internal" href="#build-a-transformed-package">Build a transformed package</a></li>
<li><a class="reference internal" href="#install-a-package-containing-transformed-pyc-files">Install a package containing transformed .pyc files</a></li>
<li><a class="reference internal" href="#build-pyc-files-when-installing-a-package">Build .pyc files when installing a package</a></li>
<li><a class="reference internal" href="#execute-transformed-code">Execute transformed code</a></li>
</ul>
</li>
<li><a class="reference internal" href="#code-transformer-api">Code transformer API</a><ul>
<li><a class="reference internal" href="#code-transformer-method">code_transformer() method</a></li>
<li><a class="reference internal" href="#ast-transformer-method">ast_transformer() method</a></li>
</ul>
</li>
<li><a class="reference internal" href="#changes">Changes</a><ul>
<li><a class="reference internal" href="#api-to-get-set-code-transformers">API to get/set code transformers</a></li>
<li><a class="reference internal" href="#optimizer-tag">Optimizer tag</a></li>
<li><a class="reference internal" href="#peephole-optimizer">Peephole optimizer</a></li>
<li><a class="reference internal" href="#ast-enhancements">AST enhancements</a></li>
</ul>
</li>
<li><a class="reference internal" href="#examples">Examples</a><ul>
<li><a class="reference internal" href="#pyc-filenames">.pyc filenames</a></li>
<li><a class="reference internal" href="#bytecode-transformer">Bytecode transformer</a></li>
<li><a class="reference internal" href="#ast-transformer">AST transformer</a></li>
</ul>
</li>
<li><a class="reference internal" href="#other-python-implementations">Other Python implementations</a></li>
<li><a class="reference internal" href="#discussion">Discussion</a></li>
<li><a class="reference internal" href="#prior-art">Prior Art</a><ul>
<li><a class="reference internal" href="#ast-optimizers">AST optimizers</a></li>
<li><a class="reference internal" href="#python-preprocessors">Python Preprocessors</a></li>
<li><a class="reference internal" href="#bytecode-transformers">Bytecode transformers</a></li>
</ul>
</li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="rejection-notice">
<h2><a class="toc-backref" href="#rejection-notice" role="doc-backlink">Rejection Notice</a></h2>
<p>This PEP was rejected by its author.</p>
<p>This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.</p>
<p>This PEP was also seen as a nice tool to experiment new Python features,
but it is already possible to experiment them without the PEP, only with
importlib hooks. If a feature becomes useful, it should be directly part
of Python, instead of depending on an third party Python module.</p>
<p>Finally, this PEP was driven was the FAT Python optimization project
which was abandoned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.</p>
</section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>Propose an API to register bytecode and AST transformers. Add also <code class="docutils literal notranslate"><span class="pre">-o</span>
<span class="pre">OPTIM_TAG</span></code> command line option to change <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> filenames, <code class="docutils literal notranslate"><span class="pre">-o</span>
<span class="pre">noopt</span></code> disables the peephole optimizer. Raise an <code class="docutils literal notranslate"><span class="pre">ImportError</span></code>
exception on import if the <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> file is missing and the code
transformers required to transform the code are missing. code
transformers are not needed code transformed ahead of time (loaded from
<code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files).</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>Python does not provide a standard way to transform the code. Projects
transforming the code use various hooks. The MacroPy project uses an
import hook: it adds its own module finder in <code class="docutils literal notranslate"><span class="pre">sys.meta_path</span></code> to
hook its AST transformer. Another option is to monkey-patch the
builtin <code class="docutils literal notranslate"><span class="pre">compile()</span></code> function. There are even more options to
hook a code transformer.</p>
<p>Python 3.4 added a <code class="docutils literal notranslate"><span class="pre">compile_source()</span></code> method to
<code class="docutils literal notranslate"><span class="pre">importlib.abc.SourceLoader</span></code>. But code transformation is wider than
just importing modules, see described use cases below.</p>
<p>Writing an optimizer or a preprocessor is out of the scope of this PEP.</p>
<section id="usage-1-ast-optimizer">
<h3><a class="toc-backref" href="#usage-1-ast-optimizer" role="doc-backlink">Usage 1: AST optimizer</a></h3>
<p>Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It’s easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.</p>
<p>Since the optimization can done ahead of time, complex but slow
optimizations can be implemented.</p>
<p>Example of optimizations which can be implemented with an AST optimizer:</p>
<ul class="simple">
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Copy_propagation">Copy propagation</a>:
replace <code class="docutils literal notranslate"><span class="pre">x=1;</span> <span class="pre">y=x</span></code> with <code class="docutils literal notranslate"><span class="pre">x=1;</span> <span class="pre">y=1</span></code></li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Constant_folding">Constant folding</a>:
replace <code class="docutils literal notranslate"><span class="pre">1+1</span></code> with <code class="docutils literal notranslate"><span class="pre">2</span></code></li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Dead_code_elimination">Dead code elimination</a></li>
</ul>
<p>Using guards (see <a class="pep reference internal" href="../pep-0510/" title="PEP 510 – Specialize functions with guards">PEP 510</a>), it is possible to
implement a much wider choice of optimizations. Examples:</p>
<ul class="simple">
<li>Simplify iterable: replace <code class="docutils literal notranslate"><span class="pre">range(3)</span></code> with <code class="docutils literal notranslate"><span class="pre">(0,</span> <span class="pre">1,</span> <span class="pre">2)</span></code> when used
as iterable</li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Loop_unrolling">Loop unrolling</a></li>
<li>Call pure builtins: replace <code class="docutils literal notranslate"><span class="pre">len(&quot;abc&quot;)</span></code> with <code class="docutils literal notranslate"><span class="pre">3</span></code></li>
<li>Copy used builtin symbols to constants</li>
<li>See also <a class="reference external" href="https://fatoptimizer.readthedocs.org/en/latest/optimizations.html">optimizations implemented in fatoptimizer</a>,
a static optimizer for Python 3.6.</li>
</ul>
<p>The following issues can be implemented with an AST optimizer:</p>
<ul class="simple">
<li><a class="reference external" href="https://bugs.python.org/issue1346238">Issue #1346238</a>: A constant folding
optimization pass for the AST</li>
<li><a class="reference external" href="http://bugs.python.org/issue2181">Issue #2181</a>:
optimize out local variables at end of function</li>
<li><a class="reference external" href="http://bugs.python.org/issue2499">Issue #2499</a>:
Fold unary + and not on constants</li>
<li><a class="reference external" href="http://bugs.python.org/issue4264">Issue #4264</a>:
Patch: optimize code to use LIST_APPEND instead of calling list.append</li>
<li><a class="reference external" href="http://bugs.python.org/issue7682">Issue #7682</a>:
Optimisation of if with constant expression</li>
<li><a class="reference external" href="https://bugs.python.org/issue10399">Issue #10399</a>: AST
Optimization: inlining of function calls</li>
<li><a class="reference external" href="http://bugs.python.org/issue11549">Issue #11549</a>:
Build-out an AST optimizer, moving some functionality out of the
peephole optimizer</li>
<li><a class="reference external" href="http://bugs.python.org/issue17068">Issue #17068</a>:
peephole optimization for constant strings</li>
<li><a class="reference external" href="http://bugs.python.org/issue17430">Issue #17430</a>:
missed peephole optimization</li>
</ul>
</section>
<section id="usage-2-preprocessor">
<h3><a class="toc-backref" href="#usage-2-preprocessor" role="doc-backlink">Usage 2: Preprocessor</a></h3>
<p>A preprocessor can be easily implemented with an AST transformer. A
preprocessor has various and different usages.</p>
<p>Some examples:</p>
<ul class="simple">
<li>Remove debug code like assertions and logs to make the code faster to
run it for production.</li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Tail_call">Tail-call Optimization</a></li>
<li>Add profiling code</li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Lazy_evaluation">Lazy evaluation</a>:
see <a class="reference external" href="https://github.com/llllllllll/lazy_python">lazy_python</a>
(bytecode transformer) and <a class="reference external" href="https://github.com/lihaoyi/macropy#lazy">lazy macro of MacroPy</a> (AST transformer)</li>
<li>Change dictionary literals into collection.OrderedDict instances</li>
<li>Declare constants: see <a class="reference external" href="https://pypi.python.org/pypi/codetransformer">&#64;asconstants of codetransformer</a></li>
<li>Domain Specific Language (DSL) like SQL queries. The
Python language itself doesn’t need to be modified. Previous attempts
to implement DSL for SQL like <a class="pep reference internal" href="../pep-0335/" title="PEP 335 – Overloadable Boolean Operators">PEP 335 - Overloadable Boolean
Operators</a> was rejected.</li>
<li>Pattern Matching of functional languages</li>
<li>String Interpolation, but <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>
was merged into Python
3.6.</li>
</ul>
<p><a class="reference external" href="https://github.com/lihaoyi/macropy">MacroPy</a> has a long list of
examples and use cases.</p>
<p>This PEP does not add any new code transformer. Using a code transformer
will require an external module and to register it manually.</p>
<p>See also <a class="reference external" href="https://bitbucket.org/namn/pyxfuscator">PyXfuscator</a>: Python
obfuscator, deobfuscator, and user-assisted decompiler.</p>
</section>
<section id="usage-3-disable-all-optimization">
<h3><a class="toc-backref" href="#usage-3-disable-all-optimization" role="doc-backlink">Usage 3: Disable all optimization</a></h3>
<p>Ned Batchelder asked to add an option to disable the peephole optimizer
because it makes code coverage more difficult to implement. See the
discussion on the python-ideas mailing list: <a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2014-May/027893.html">Disable all peephole
optimizations</a>.</p>
<p>This PEP adds a new <code class="docutils literal notranslate"><span class="pre">-o</span> <span class="pre">noopt</span></code> command line option to disable the
peephole optimizer. In Python, it’s as easy as:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sys</span><span class="o">.</span><span class="n">set_code_transformers</span><span class="p">([])</span>
</pre></div>
</div>
<p>It will fix the <a class="reference external" href="https://bugs.python.org/issue2506">Issue #2506</a>: Add
mechanism to disable optimizations.</p>
</section>
<section id="usage-4-write-new-bytecode-optimizers-in-python">
<h3><a class="toc-backref" href="#usage-4-write-new-bytecode-optimizers-in-python" role="doc-backlink">Usage 4: Write new bytecode optimizers in Python</a></h3>
<p>Python 3.6 optimizes the code using a peephole optimizer. By
definition, a peephole optimizer has a narrow view of the code and so
can only implement basic optimizations. The optimizer rewrites the
bytecode. It is difficult to enhance it, because it written in C.</p>
<p>With this PEP, it becomes possible to implement a new bytecode optimizer
in pure Python and experiment new optimizations.</p>
<p>Some optimizations are easier to implement on the AST like constant
folding, but optimizations on the bytecode are still useful. For
example, when the AST is compiled to bytecode, useless jumps can be
emitted because the compiler is naive and does not try to optimize
anything.</p>
</section>
</section>
<section id="use-cases">
<h2><a class="toc-backref" href="#use-cases" role="doc-backlink">Use Cases</a></h2>
<p>This section give examples of use cases explaining when and how code
transformers will be used.</p>
<section id="interactive-interpreter">
<h3><a class="toc-backref" href="#interactive-interpreter" role="doc-backlink">Interactive interpreter</a></h3>
<p>It will be possible to use code transformers with the interactive
interpreter which is popular in Python and commonly used to demonstrate
Python.</p>
<p>The code is transformed at runtime and so the interpreter can be slower
when expensive code transformers are used.</p>
</section>
<section id="build-a-transformed-package">
<h3><a class="toc-backref" href="#build-a-transformed-package" role="doc-backlink">Build a transformed package</a></h3>
<p>It will be possible to build a package of the transformed code.</p>
<p>A transformer can have a configuration. The configuration is not stored
in the package.</p>
<p>All <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files of the package must be transformed with the same code
transformers and the same transformers configuration.</p>
<p>It is possible to build different <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files using different
optimizer tags. Example: <code class="docutils literal notranslate"><span class="pre">fat</span></code> for the default configuration and
<code class="docutils literal notranslate"><span class="pre">fat_inline</span></code> for a different configuration with function inlining
enabled.</p>
<p>A package can contain <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files with different optimizer tags.</p>
</section>
<section id="install-a-package-containing-transformed-pyc-files">
<h3><a class="toc-backref" href="#install-a-package-containing-transformed-pyc-files" role="doc-backlink">Install a package containing transformed .pyc files</a></h3>
<p>It will be possible to install a package which contains transformed
<code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files.</p>
<p>All <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files with any optimizer tag contained in the package are
installed, not only for the current optimizer tag.</p>
</section>
<section id="build-pyc-files-when-installing-a-package">
<h3><a class="toc-backref" href="#build-pyc-files-when-installing-a-package" role="doc-backlink">Build .pyc files when installing a package</a></h3>
<p>If a package does not contain any <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files of the current
optimizer tag (or some <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files are missing), the <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> are
created during the installation.</p>
<p>Code transformers of the optimizer tag are required. Otherwise, the
installation fails with an error.</p>
</section>
<section id="execute-transformed-code">
<h3><a class="toc-backref" href="#execute-transformed-code" role="doc-backlink">Execute transformed code</a></h3>
<p>It will be possible to execute transformed code.</p>
<p>Raise an <code class="docutils literal notranslate"><span class="pre">ImportError</span></code> exception on import if the <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> file of the
current optimizer tag is missing and the code transformers required to
transform the code are missing.</p>
<p>The interesting point here is that code transformers are not needed to
execute the transformed code if all required <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files are already
available.</p>
</section>
</section>
<section id="code-transformer-api">
<h2><a class="toc-backref" href="#code-transformer-api" role="doc-backlink">Code transformer API</a></h2>
<p>A code transformer is a class with <code class="docutils literal notranslate"><span class="pre">ast_transformer()</span></code> and/or
<code class="docutils literal notranslate"><span class="pre">code_transformer()</span></code> methods (API described below) and a <code class="docutils literal notranslate"><span class="pre">name</span></code>
attribute.</p>
<p>For efficiency, do not define a <code class="docutils literal notranslate"><span class="pre">code_transformer()</span></code> or
<code class="docutils literal notranslate"><span class="pre">ast_transformer()</span></code> method if it does nothing.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">name</span></code> attribute (<code class="docutils literal notranslate"><span class="pre">str</span></code>) must be a short string used to identify
an optimizer. It is used to build a <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> filename. The name must not
contain dots (<code class="docutils literal notranslate"><span class="pre">'.'</span></code>), dashes (<code class="docutils literal notranslate"><span class="pre">'-'</span></code>) or directory separators: dots
are used to separated fields in a <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> filename and dashes areused
to join code transformer names to build the optimizer tag.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>It would be nice to pass the fully qualified name of a module in the
<em>context</em> when an AST transformer is used to transform a module on
import, but it looks like the information is not available in
<code class="docutils literal notranslate"><span class="pre">PyParser_ASTFromStringObject()</span></code>.</p>
</div>
<section id="code-transformer-method">
<h3><a class="toc-backref" href="#code-transformer-method" role="doc-backlink">code_transformer() method</a></h3>
<p>Prototype:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">code_transformer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">code</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="o">...</span>
<span class="n">new_code</span> <span class="o">=</span> <span class="o">...</span>
<span class="o">...</span>
<span class="k">return</span> <span class="n">new_code</span>
</pre></div>
</div>
<p>Parameters:</p>
<ul class="simple">
<li><em>code</em>: code object</li>
<li><em>context</em>: an object with an <em>optimize</em> attribute (<code class="docutils literal notranslate"><span class="pre">int</span></code>), the optimization
level (0, 1 or 2). The value of the <em>optimize</em> attribute comes from the
<em>optimize</em> parameter of the <code class="docutils literal notranslate"><span class="pre">compile()</span></code> function, it is equal to
<code class="docutils literal notranslate"><span class="pre">sys.flags.optimize</span></code> by default.</li>
</ul>
<p>Each implementation of Python can add extra attributes to <em>context</em>. For
example, on CPython, <em>context</em> will also have the following attribute:</p>
<ul class="simple">
<li><em>interactive</em> (<code class="docutils literal notranslate"><span class="pre">bool</span></code>): true if in interactive mode</li>
</ul>
<p>XXX add more flags?</p>
<p>XXX replace flags int with a sub-namespace, or with specific attributes?</p>
<p>The method must return a code object.</p>
<p>The code transformer is run after the compilation to bytecode</p>
</section>
<section id="ast-transformer-method">
<h3><a class="toc-backref" href="#ast-transformer-method" role="doc-backlink">ast_transformer() method</a></h3>
<p>Prototype:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">ast_transformer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tree</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">return</span> <span class="n">tree</span>
</pre></div>
</div>
<p>Parameters:</p>
<ul class="simple">
<li><em>tree</em>: an AST tree</li>
<li><em>context</em>: an object with a <code class="docutils literal notranslate"><span class="pre">filename</span></code> attribute (<code class="docutils literal notranslate"><span class="pre">str</span></code>)</li>
</ul>
<p>It must return an AST tree. It can modify the AST tree in place, or
create a new AST tree.</p>
<p>The AST transformer is called after the creation of the AST by the
parser and before the compilation to bytecode. New attributes may be
added to <em>context</em> in the future.</p>
</section>
</section>
<section id="changes">
<h2><a class="toc-backref" href="#changes" role="doc-backlink">Changes</a></h2>
<p>In short, add:</p>
<ul class="simple">
<li>-o OPTIM_TAG command line option</li>
<li>sys.implementation.optim_tag</li>
<li>sys.get_code_transformers()</li>
<li>sys.set_code_transformers(transformers)</li>
<li>ast.PyCF_TRANSFORMED_AST</li>
</ul>
<section id="api-to-get-set-code-transformers">
<h3><a class="toc-backref" href="#api-to-get-set-code-transformers" role="doc-backlink">API to get/set code transformers</a></h3>
<p>Add new functions to register code transformers:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">sys.set_code_transformers(transformers)</span></code>: set the list of code
transformers and update <code class="docutils literal notranslate"><span class="pre">sys.implementation.optim_tag</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">sys.get_code_transformers()</span></code>: get the list of code
transformers.</li>
</ul>
<p>The order of code transformers matter. Running transformer A and then
transformer B can give a different output than running transformer B an
then transformer A.</p>
<p>Example to prepend a new code transformer:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">transformers</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">get_code_transformers</span><span class="p">()</span>
<span class="n">transformers</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">new_cool_transformer</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">set_code_transformers</span><span class="p">(</span><span class="n">transformers</span><span class="p">)</span>
</pre></div>
</div>
<p>All AST transformers are run sequentially (ex: the second transformer
gets the input of the first transformer), and then all bytecode
transformers are run sequentially.</p>
</section>
<section id="optimizer-tag">
<h3><a class="toc-backref" href="#optimizer-tag" role="doc-backlink">Optimizer tag</a></h3>
<p>Changes:</p>
<ul class="simple">
<li>Add <code class="docutils literal notranslate"><span class="pre">sys.implementation.optim_tag</span></code> (<code class="docutils literal notranslate"><span class="pre">str</span></code>): optimization tag.
The default optimization tag is <code class="docutils literal notranslate"><span class="pre">'opt'</span></code>.</li>
<li>Add a new <code class="docutils literal notranslate"><span class="pre">-o</span> <span class="pre">OPTIM_TAG</span></code> command line option to set
<code class="docutils literal notranslate"><span class="pre">sys.implementation.optim_tag</span></code>.</li>
</ul>
<p>Changes on <code class="docutils literal notranslate"><span class="pre">importlib</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">importlib</span></code> uses <code class="docutils literal notranslate"><span class="pre">sys.implementation.optim_tag</span></code> to build the
<code class="docutils literal notranslate"><span class="pre">.pyc</span></code> filename to importing modules, instead of always using
<code class="docutils literal notranslate"><span class="pre">opt</span></code>. Remove also the special case for the optimizer level <code class="docutils literal notranslate"><span class="pre">0</span></code>
with the default optimizer tag <code class="docutils literal notranslate"><span class="pre">'opt'</span></code> to simplify the code.</li>
<li>When loading a module, if the <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> file is missing but the <code class="docutils literal notranslate"><span class="pre">.py</span></code>
is available, the <code class="docutils literal notranslate"><span class="pre">.py</span></code> is only used if code optimizers have the
same optimizer tag than the current tag, otherwise an <code class="docutils literal notranslate"><span class="pre">ImportError</span></code>
exception is raised.</li>
</ul>
<p>Pseudo-code of a <code class="docutils literal notranslate"><span class="pre">use_py()</span></code> function to decide if a <code class="docutils literal notranslate"><span class="pre">.py</span></code> file can
be compiled to import a module:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">transformers_tag</span><span class="p">():</span>
<span class="n">transformers</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">get_code_transformers</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">transformers</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">&#39;noopt&#39;</span>
<span class="k">return</span> <span class="s1">&#39;-&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">transformer</span><span class="o">.</span><span class="n">name</span>
<span class="k">for</span> <span class="n">transformer</span> <span class="ow">in</span> <span class="n">transformers</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">use_py</span><span class="p">():</span>
<span class="k">return</span> <span class="p">(</span><span class="n">transformers_tag</span><span class="p">()</span> <span class="o">==</span> <span class="n">sys</span><span class="o">.</span><span class="n">implementation</span><span class="o">.</span><span class="n">optim_tag</span><span class="p">)</span>
</pre></div>
</div>
<p>The order of <code class="docutils literal notranslate"><span class="pre">sys.get_code_transformers()</span></code> matter. For example, the
<code class="docutils literal notranslate"><span class="pre">fat</span></code> transformer followed by the <code class="docutils literal notranslate"><span class="pre">pythran</span></code> transformer gives the
optimizer tag <code class="docutils literal notranslate"><span class="pre">fat-pythran</span></code>.</p>
<p>The behaviour of the <code class="docutils literal notranslate"><span class="pre">importlib</span></code> module is unchanged with the default
optimizer tag (<code class="docutils literal notranslate"><span class="pre">'opt'</span></code>).</p>
</section>
<section id="peephole-optimizer">
<h3><a class="toc-backref" href="#peephole-optimizer" role="doc-backlink">Peephole optimizer</a></h3>
<p>By default, <code class="docutils literal notranslate"><span class="pre">sys.implementation.optim_tag</span></code> is <code class="docutils literal notranslate"><span class="pre">opt</span></code> and
<code class="docutils literal notranslate"><span class="pre">sys.get_code_transformers()</span></code> returns a list of one code transformer:
the peephole optimizer (optimize the bytecode).</p>
<p>Use <code class="docutils literal notranslate"><span class="pre">-o</span> <span class="pre">noopt</span></code> to disable the peephole optimizer. In this case, the
optimizer tag is <code class="docutils literal notranslate"><span class="pre">noopt</span></code> and no code transformer is registered.</p>
<p>Using the <code class="docutils literal notranslate"><span class="pre">-o</span> <span class="pre">opt</span></code> option has not effect.</p>
</section>
<section id="ast-enhancements">
<h3><a class="toc-backref" href="#ast-enhancements" role="doc-backlink">AST enhancements</a></h3>
<p>Enhancements to simplify the implementation of AST transformers:</p>
<ul class="simple">
<li>Add a new compiler flag <code class="docutils literal notranslate"><span class="pre">PyCF_TRANSFORMED_AST</span></code> to get the
transformed AST. <code class="docutils literal notranslate"><span class="pre">PyCF_ONLY_AST</span></code> returns the AST before the
transformers.</li>
</ul>
</section>
</section>
<section id="examples">
<h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2>
<section id="pyc-filenames">
<h3><a class="toc-backref" href="#pyc-filenames" role="doc-backlink">.pyc filenames</a></h3>
<p>Example of <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> filenames of the <code class="docutils literal notranslate"><span class="pre">os</span></code> module.</p>
<p>With the default optimizer tag <code class="docutils literal notranslate"><span class="pre">'opt'</span></code>:</p>
<table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head">.pyc filename</th>
<th class="head">Optimization level</th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.opt-0.pyc</span></code></td>
<td>0</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.opt-1.pyc</span></code></td>
<td>1</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.opt-2.pyc</span></code></td>
<td>2</td>
</tr>
</tbody>
</table>
<p>With the <code class="docutils literal notranslate"><span class="pre">'fat'</span></code> optimizer tag:</p>
<table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head">.pyc filename</th>
<th class="head">Optimization level</th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.fat-0.pyc</span></code></td>
<td>0</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.fat-1.pyc</span></code></td>
<td>1</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">os.cpython-36.fat-2.pyc</span></code></td>
<td>2</td>
</tr>
</tbody>
</table>
</section>
<section id="bytecode-transformer">
<h3><a class="toc-backref" href="#bytecode-transformer" role="doc-backlink">Bytecode transformer</a></h3>
<p>Scary bytecode transformer replacing all strings with
<code class="docutils literal notranslate"><span class="pre">&quot;Ni!</span> <span class="pre">Ni!</span> <span class="pre">Ni!&quot;</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">types</span>
<span class="k">class</span> <span class="nc">BytecodeTransformer</span><span class="p">:</span>
<span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;knights_who_say_ni&quot;</span>
<span class="k">def</span> <span class="nf">code_transformer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">code</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">consts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;Ni! Ni! Ni!&#39;</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">const</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="n">const</span>
<span class="k">for</span> <span class="n">const</span> <span class="ow">in</span> <span class="n">code</span><span class="o">.</span><span class="n">co_consts</span><span class="p">]</span>
<span class="k">return</span> <span class="n">types</span><span class="o">.</span><span class="n">CodeType</span><span class="p">(</span><span class="n">code</span><span class="o">.</span><span class="n">co_argcount</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_kwonlyargcount</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_nlocals</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_stacksize</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_flags</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_code</span><span class="p">,</span>
<span class="nb">tuple</span><span class="p">(</span><span class="n">consts</span><span class="p">),</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_names</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_varnames</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_filename</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_name</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_firstlineno</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_lnotab</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_freevars</span><span class="p">,</span>
<span class="n">code</span><span class="o">.</span><span class="n">co_cellvars</span><span class="p">)</span>
<span class="c1"># replace existing code transformers with the new bytecode transformer</span>
<span class="n">sys</span><span class="o">.</span><span class="n">set_code_transformers</span><span class="p">([</span><span class="n">BytecodeTransformer</span><span class="p">()])</span>
<span class="c1"># execute code which will be transformed by code_transformer()</span>
<span class="n">exec</span><span class="p">(</span><span class="s2">&quot;print(&#39;Hello World!&#39;)&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>Output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Ni! Ni! Ni!
</pre></div>
</div>
</section>
<section id="ast-transformer">
<h3><a class="toc-backref" href="#ast-transformer" role="doc-backlink">AST transformer</a></h3>
<p>Similarly to the bytecode transformer example, the AST transformer also
replaces all strings with <code class="docutils literal notranslate"><span class="pre">&quot;Ni!</span> <span class="pre">Ni!</span> <span class="pre">Ni!&quot;</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">ast</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">class</span> <span class="nc">KnightsWhoSayNi</span><span class="p">(</span><span class="n">ast</span><span class="o">.</span><span class="n">NodeTransformer</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">visit_Str</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
<span class="n">node</span><span class="o">.</span><span class="n">s</span> <span class="o">=</span> <span class="s1">&#39;Ni! Ni! Ni!&#39;</span>
<span class="k">return</span> <span class="n">node</span>
<span class="k">class</span> <span class="nc">ASTTransformer</span><span class="p">:</span>
<span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;knights_who_say_ni&quot;</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">transformer</span> <span class="o">=</span> <span class="n">KnightsWhoSayNi</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">ast_transformer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tree</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">transformer</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="k">return</span> <span class="n">tree</span>
<span class="c1"># replace existing code transformers with the new AST transformer</span>
<span class="n">sys</span><span class="o">.</span><span class="n">set_code_transformers</span><span class="p">([</span><span class="n">ASTTransformer</span><span class="p">()])</span>
<span class="c1"># execute code which will be transformed by ast_transformer()</span>
<span class="n">exec</span><span class="p">(</span><span class="s2">&quot;print(&#39;Hello World!&#39;)&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>Output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Ni! Ni! Ni!
</pre></div>
</div>
</section>
</section>
<section id="other-python-implementations">
<h2><a class="toc-backref" href="#other-python-implementations" role="doc-backlink">Other Python implementations</a></h2>
<p>The <a class="pep reference internal" href="../pep-0511/" title="PEP 511 – API for code transformers">PEP 511</a> should be implemented by all Python implementation, but the
bytecode and the AST are not standardized.</p>
<p>By the way, even between minor version of CPython, there are changes on
the AST API. There are differences, but only minor differences. It is
quite easy to write an AST transformer which works on Python 2.7 and
Python 3.5 for example.</p>
</section>
<section id="discussion">
<h2><a class="toc-backref" href="#discussion" role="doc-backlink">Discussion</a></h2>
<ul class="simple">
<li><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2016-January/037884.html">[Python-ideas] PEP 511: API for code transformers</a>
(January 2016)</li>
<li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2012-August/121286.html">[Python-Dev] AST optimizer implemented in Python</a>
(August 2012)</li>
</ul>
</section>
<section id="prior-art">
<h2><a class="toc-backref" href="#prior-art" role="doc-backlink">Prior Art</a></h2>
<section id="ast-optimizers">
<h3><a class="toc-backref" href="#ast-optimizers" role="doc-backlink">AST optimizers</a></h3>
<p>The Issue #17515 <a class="reference external" href="https://bugs.python.org/issue17515">“Add sys.setasthook() to allow to use a custom AST”
optimizer</a> was a first attempt of
API for code transformers, but specific to AST.</p>
<p>In 2015, Victor Stinner wrote the <a class="reference external" href="http://fatoptimizer.readthedocs.org/">fatoptimizer</a> project, an AST optimizer
specializing functions using guards.</p>
<p>In 2014, Kevin Conway created the <a class="reference external" href="http://pycc.readthedocs.org/">PyCC</a>
optimizer.</p>
<p>In 2012, Victor Stinner wrote the <a class="reference external" href="https://bitbucket.org/haypo/astoptimizer/">astoptimizer</a> project, an AST optimizer
implementing various optimizations. Most interesting optimizations break
the Python semantics since no guard is used to disable optimization if
something changes.</p>
<p>In 2011, Eugene Toder proposed to rewrite some peephole optimizations in
a new AST optimizer: issue #11549, <a class="reference external" href="https://bugs.python.org/issue11549">Build-out an AST optimizer, moving
some functionality out of the peephole optimizer</a>. The patch adds <code class="docutils literal notranslate"><span class="pre">ast.Lit</span></code> (it
was proposed to rename it to <code class="docutils literal notranslate"><span class="pre">ast.Literal</span></code>).</p>
</section>
<section id="python-preprocessors">
<h3><a class="toc-backref" href="#python-preprocessors" role="doc-backlink">Python Preprocessors</a></h3>
<ul class="simple">
<li><a class="reference external" href="https://github.com/lihaoyi/macropy">MacroPy</a>: MacroPy is an
implementation of Syntactic Macros in the Python Programming Language.
MacroPy provides a mechanism for user-defined functions (macros) to
perform transformations on the abstract syntax tree (AST) of a Python
program at import time.</li>
<li><a class="reference external" href="https://code.google.com/p/pypreprocessor/">pypreprocessor</a>: C-style
preprocessor directives in Python, like <code class="docutils literal notranslate"><span class="pre">#define</span></code> and <code class="docutils literal notranslate"><span class="pre">#ifdef</span></code></li>
</ul>
</section>
<section id="bytecode-transformers">
<h3><a class="toc-backref" href="#bytecode-transformers" role="doc-backlink">Bytecode transformers</a></h3>
<ul class="simple">
<li><a class="reference external" href="https://pypi.python.org/pypi/codetransformer">codetransformer</a>:
Bytecode transformers for CPython inspired by the <code class="docutils literal notranslate"><span class="pre">ast</span></code> module’s
<code class="docutils literal notranslate"><span class="pre">NodeTransformer</span></code>.</li>
<li><a class="reference external" href="http://code.google.com/p/byteplay/">byteplay</a>: Byteplay lets you
convert Python code objects into equivalent objects which are easy to
play with, and lets you convert those objects back into living Python
code objects. It’s useful for applying crazy transformations on Python
functions, and is also useful in learning Python byte code
intricacies. See <a class="reference external" href="http://wiki.python.org/moin/ByteplayDoc">byteplay documentation</a>.</li>
</ul>
<p>See also:</p>
<ul class="simple">
<li><a class="reference external" href="http://pypi.python.org/pypi/BytecodeAssembler">BytecodeAssembler</a></li>
</ul>
</section>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0511.rst">https://github.com/python/peps/blob/main/peps/pep-0511.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0511.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#usage-1-ast-optimizer">Usage 1: AST optimizer</a></li>
<li><a class="reference internal" href="#usage-2-preprocessor">Usage 2: Preprocessor</a></li>
<li><a class="reference internal" href="#usage-3-disable-all-optimization">Usage 3: Disable all optimization</a></li>
<li><a class="reference internal" href="#usage-4-write-new-bytecode-optimizers-in-python">Usage 4: Write new bytecode optimizers in Python</a></li>
</ul>
</li>
<li><a class="reference internal" href="#use-cases">Use Cases</a><ul>
<li><a class="reference internal" href="#interactive-interpreter">Interactive interpreter</a></li>
<li><a class="reference internal" href="#build-a-transformed-package">Build a transformed package</a></li>
<li><a class="reference internal" href="#install-a-package-containing-transformed-pyc-files">Install a package containing transformed .pyc files</a></li>
<li><a class="reference internal" href="#build-pyc-files-when-installing-a-package">Build .pyc files when installing a package</a></li>
<li><a class="reference internal" href="#execute-transformed-code">Execute transformed code</a></li>
</ul>
</li>
<li><a class="reference internal" href="#code-transformer-api">Code transformer API</a><ul>
<li><a class="reference internal" href="#code-transformer-method">code_transformer() method</a></li>
<li><a class="reference internal" href="#ast-transformer-method">ast_transformer() method</a></li>
</ul>
</li>
<li><a class="reference internal" href="#changes">Changes</a><ul>
<li><a class="reference internal" href="#api-to-get-set-code-transformers">API to get/set code transformers</a></li>
<li><a class="reference internal" href="#optimizer-tag">Optimizer tag</a></li>
<li><a class="reference internal" href="#peephole-optimizer">Peephole optimizer</a></li>
<li><a class="reference internal" href="#ast-enhancements">AST enhancements</a></li>
</ul>
</li>
<li><a class="reference internal" href="#examples">Examples</a><ul>
<li><a class="reference internal" href="#pyc-filenames">.pyc filenames</a></li>
<li><a class="reference internal" href="#bytecode-transformer">Bytecode transformer</a></li>
<li><a class="reference internal" href="#ast-transformer">AST transformer</a></li>
</ul>
</li>
<li><a class="reference internal" href="#other-python-implementations">Other Python implementations</a></li>
<li><a class="reference internal" href="#discussion">Discussion</a></li>
<li><a class="reference internal" href="#prior-art">Prior Art</a><ul>
<li><a class="reference internal" href="#ast-optimizers">AST optimizers</a></li>
<li><a class="reference internal" href="#python-preprocessors">Python Preprocessors</a></li>
<li><a class="reference internal" href="#bytecode-transformers">Bytecode transformers</a></li>
</ul>
</li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0511.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink