PEP 3147: Resolve uses of the default role (#3385)

This commit is contained in:
Adam Turner 2023-09-01 20:21:21 +01:00 committed by GitHub
parent ac2c284aea
commit eecda12e5a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 89 additions and 89 deletions

View File

@ -30,7 +30,7 @@ CPython compiles its source code into "byte code", and for performance
reasons, it caches this byte code on the file system whenever the reasons, it caches this byte code on the file system whenever the
source file has changes. This makes loading of Python modules much source file has changes. This makes loading of Python modules much
faster because the compilation phase can be bypassed. When your faster because the compilation phase can be bypassed. When your
source file is `foo.py`, CPython caches the byte code in a `foo.pyc` source file is ``foo.py``, CPython caches the byte code in a ``foo.pyc``
file right next to the source. file right next to the source.
Byte code files contain two 32-bit big-endian numbers followed by the Byte code files contain two 32-bit big-endian numbers followed by the
@ -59,11 +59,11 @@ with Python 2.6 being the default.
This causes a conflict for third party Python source files installed This causes a conflict for third party Python source files installed
by the system, because you cannot compile a single Python source file by the system, because you cannot compile a single Python source file
for more than one Python version at a time. When Python finds a `pyc` for more than one Python version at a time. When Python finds a ``pyc``
file with a non-matching magic number, it falls back to the slower file with a non-matching magic number, it falls back to the slower
process of recompiling the source. Thus if your system installed a process of recompiling the source. Thus if your system installed a
`/usr/share/python/foo.py`, two different versions of Python would ``/usr/share/python/foo.py``, two different versions of Python would
fight over the `pyc` file and rewrite it each time the source is fight over the ``pyc`` file and rewrite it each time the source is
compiled. (The standard library is unaffected by this, since multiple compiled. (The standard library is unaffected by this, since multiple
versions of the stdlib *are* installed on such distributions..) versions of the stdlib *are* installed on such distributions..)
@ -99,7 +99,7 @@ Proposal
Python's import machinery is extended to write and search for byte Python's import machinery is extended to write and search for byte
code cache files in a single directory inside every Python package code cache files in a single directory inside every Python package
directory. This directory will be called `__pycache__`. directory. This directory will be called ``__pycache__``.
Further, pyc file names will contain a magic string (called a "tag") Further, pyc file names will contain a magic string (called a "tag")
that differentiates the Python version they were compiled for. This that differentiates the Python version they were compiled for. This
@ -107,36 +107,36 @@ allows multiple byte compiled cache files to co-exist for a single
Python source file. Python source file.
The magic tag is implementation defined, but should contain the The magic tag is implementation defined, but should contain the
implementation name and a version number shorthand, e.g. `cpython-32`. implementation name and a version number shorthand, e.g. ``cpython-32``.
It must be unique among all versions of Python, and whenever the magic It must be unique among all versions of Python, and whenever the magic
number is bumped, a new magic tag must be defined. An example `pyc` number is bumped, a new magic tag must be defined. An example ``pyc``
file for Python 3.2 is thus `foo.cpython-32.pyc`. file for Python 3.2 is thus ``foo.cpython-32.pyc``.
The magic tag is available in the `imp` module via the `get_tag()` The magic tag is available in the ``imp`` module via the ``get_tag()``
function. This is parallel to the `imp.get_magic()` function. function. This is parallel to the ``imp.get_magic()`` function.
This scheme has the added benefit of reducing the clutter in a Python This scheme has the added benefit of reducing the clutter in a Python
package directory. package directory.
When a Python source file is imported for the first time, a When a Python source file is imported for the first time, a
`__pycache__` directory will be created in the package directory, if ``__pycache__`` directory will be created in the package directory, if
one does not already exist. The pyc file for the imported source will one does not already exist. The pyc file for the imported source will
be written to the `__pycache__` directory, using the magic-tag be written to the ``__pycache__`` directory, using the magic-tag
formatted name. If either the creation of the `__pycache__` directory formatted name. If either the creation of the ``__pycache__`` directory
or the pyc file inside that fails, the import will still succeed, just or the pyc file inside that fails, the import will still succeed, just
as it does in a pre-:pep:`3147` world. as it does in a pre-:pep:`3147` world.
If the py source file is missing, the pyc file inside `__pycache__` If the py source file is missing, the pyc file inside ``__pycache__``
will be ignored. This eliminates the problem of accidental stale pyc will be ignored. This eliminates the problem of accidental stale pyc
file imports. file imports.
For backward compatibility, Python will still support pyc-only For backward compatibility, Python will still support pyc-only
distributions, however it will only do so when the pyc file lives in distributions, however it will only do so when the pyc file lives in
the directory where the py file *would* have been, i.e. not in the the directory where the py file *would* have been, i.e. not in the
`__pycache__` directory. pyc file outside of `__pycache__` will only ``__pycache__`` directory. pyc file outside of ``__pycache__`` will only
be imported if the py source file is missing. be imported if the py source file is missing.
Tools such as `py_compile` [15]_ and `compileall` [16]_ will be Tools such as ``py_compile`` [15]_ and ``compileall`` [16]_ will be
extended to create :pep:`3147` formatted layouts automatically, but will extended to create :pep:`3147` formatted layouts automatically, but will
have an option to create pyc-only distribution layouts. have an option to create pyc-only distribution layouts.
@ -146,8 +146,8 @@ Examples
What would this look like in practice? What would this look like in practice?
Let's say we have a Python package named `alpha` which contains a Let's say we have a Python package named ``alpha`` which contains a
sub-package name `beta`. The source directory layout before byte sub-package name ``beta``. The source directory layout before byte
compilation might look like this:: compilation might look like this::
alpha/ alpha/
@ -218,7 +218,7 @@ As you can see, as long as the Python version identifier string is
unique, any number of pyc files can co-exist. These identifier unique, any number of pyc files can co-exist. These identifier
strings are described in more detail below. strings are described in more detail below.
A nice property of this layout is that the `__pycache__` directories A nice property of this layout is that the ``__pycache__`` directories
can generally be ignored, such that a normal directory listing would can generally be ignored, such that a normal directory listing would
show something like this:: show something like this::
@ -239,50 +239,50 @@ This is much less cluttered than even today's Python.
Python behavior Python behavior
=============== ===============
When Python searches for a module to import (say `foo`), it may find When Python searches for a module to import (say ``foo``), it may find
one of several situations. As per current Python rules, the term one of several situations. As per current Python rules, the term
"matching pyc" means that the magic number matches the current "matching pyc" means that the magic number matches the current
interpreter's magic number, and the source file's timestamp matches interpreter's magic number, and the source file's timestamp matches
the timestamp in the `pyc` file exactly. the timestamp in the ``pyc`` file exactly.
Case 0: The steady state Case 0: The steady state
------------------------ ------------------------
When Python is asked to import module `foo`, it searches for a When Python is asked to import module ``foo``, it searches for a
`foo.py` file (or `foo` package, but that's not important for this ``foo.py`` file (or ``foo`` package, but that's not important for this
discussion) along its `sys.path`. If found, Python looks to see if discussion) along its ``sys.path``. If found, Python looks to see if
there is a matching `__pycache__/foo.<magic>.pyc` file, and if so, there is a matching ``__pycache__/foo.<magic>.pyc`` file, and if so,
that `pyc` file is loaded. that ``pyc`` file is loaded.
Case 1: The first import Case 1: The first import
------------------------ ------------------------
When Python locates the `foo.py`, if the `__pycache__/foo.<magic>.pyc` When Python locates the ``foo.py``, if the ``__pycache__/foo.<magic>.pyc``
file is missing, Python will create it, also creating the file is missing, Python will create it, also creating the
`__pycache__` directory if necessary. Python will parse and byte ``__pycache__`` directory if necessary. Python will parse and byte
compile the `foo.py` file and save the byte code in compile the ``foo.py`` file and save the byte code in
`__pycache__/foo.<magic>.pyc`. ``__pycache__/foo.<magic>.pyc``.
Case 2: The second import Case 2: The second import
------------------------- -------------------------
When Python is asked to import module `foo` a second time (in a When Python is asked to import module ``foo`` a second time (in a
different process of course), it will again search for the `foo.py` different process of course), it will again search for the ``foo.py``
file along its `sys.path`. When Python locates the `foo.py` file, it file along its ``sys.path``. When Python locates the ``foo.py`` file, it
looks for a matching `__pycache__/foo.<magic>.pyc` and finding this, looks for a matching ``__pycache__/foo.<magic>.pyc`` and finding this,
it reads the byte code and continues as usual. it reads the byte code and continues as usual.
Case 3: __pycache__/foo.<magic>.pyc with no source Case 3: __pycache__/foo.<magic>.pyc with no source
--------------------------------------------------- ---------------------------------------------------
It's possible that the `foo.py` file somehow got removed, while It's possible that the ``foo.py`` file somehow got removed, while
leaving the cached pyc file still on the file system. If the leaving the cached pyc file still on the file system. If the
`__pycache__/foo.<magic>.pyc` file exists, but the `foo.py` file used ``__pycache__/foo.<magic>.pyc`` file exists, but the ``foo.py`` file used
to create it does not, Python will raise an `ImportError` when asked to create it does not, Python will raise an ``ImportError`` when asked
to import foo. In other words, Python will not import a pyc file from to import foo. In other words, Python will not import a pyc file from
the cache directory unless the source file exists. the cache directory unless the source file exists.
@ -291,8 +291,8 @@ Case 4: legacy pyc files and source-less imports
------------------------------------------------ ------------------------------------------------
Python will ignore all legacy pyc files when a source file exists next Python will ignore all legacy pyc files when a source file exists next
to it. In other words, if a `foo.pyc` file exists next to the to it. In other words, if a ``foo.pyc`` file exists next to the
`foo.py` file, the pyc file will be ignored in all cases ``foo.py`` file, the pyc file will be ignored in all cases
In order to continue to support source-less distributions though, if In order to continue to support source-less distributions though, if
the source file is missing, Python will import a lone pyc file if it the source file is missing, Python will import a lone pyc file if it
@ -302,9 +302,9 @@ lives where the source file would have been.
Case 5: read-only file systems Case 5: read-only file systems
------------------------------ ------------------------------
When the source lives on a read-only file system, or the `__pycache__` When the source lives on a read-only file system, or the ``__pycache__``
directory or pyc file cannot otherwise be written, all the same rules directory or pyc file cannot otherwise be written, all the same rules
apply. This is also the case when `__pycache__` happens to be written apply. This is also the case when ``__pycache__`` happens to be written
with permissions which do not allow for writing containing pyc files. with permissions which do not allow for writing containing pyc files.
@ -324,9 +324,9 @@ Alternative Python implementations
Alternative Python implementations such as Jython [11]_, IronPython Alternative Python implementations such as Jython [11]_, IronPython
[12]_, PyPy [13]_, Pynie [14]_, and Unladen Swallow can also use the [12]_, PyPy [13]_, Pynie [14]_, and Unladen Swallow can also use the
`__pycache__` directory to store whatever compilation artifacts make ``__pycache__`` directory to store whatever compilation artifacts make
sense for their platforms. For example, Jython could store the class sense for their platforms. For example, Jython could store the class
file for the module in `__pycache__/foo.jython-32.class`. file for the module in ``__pycache__/foo.jython-32.class``.
Implementation strategy Implementation strategy
@ -336,7 +336,7 @@ This feature is targeted for Python 3.2, solving the problem for those
and all future versions. It may be back-ported to Python 2.7. and all future versions. It may be back-ported to Python 2.7.
Vendors are free to backport the changes to earlier distributions as Vendors are free to backport the changes to earlier distributions as
they see fit. For backports of this feature to Python 2, when the they see fit. For backports of this feature to Python 2, when the
`-U` flag is used, a file such as `foo.cpython-27u.pyc` can be ``-U`` flag is used, a file such as ``foo.cpython-27u.pyc`` can be
written. written.
@ -360,9 +360,9 @@ The easiest way to detect whether your version of Python provides PEP
__file__ __file__
--------- ---------
In Python 3, when you import a module, its `__file__` attribute points In Python 3, when you import a module, its ``__file__`` attribute points
to its source `py` file (in Python 2, it points to the `pyc` file). A to its source ``py`` file (in Python 2, it points to the ``pyc`` file). A
package's `__file__` points to the `py` file for its `__init__.py`. package's ``__file__`` points to the ``py`` file for its ``__init__.py``.
E.g.:: E.g.::
>>> import foo >>> import foo
@ -373,64 +373,64 @@ E.g.::
>>> baz.__file__ >>> baz.__file__
'baz/__init__.py' 'baz/__init__.py'
Nothing in this PEP would change the semantics of `__file__`. Nothing in this PEP would change the semantics of ``__file__``.
This PEP proposes the addition of an `__cached__` attribute to This PEP proposes the addition of an ``__cached__`` attribute to
modules, which will always point to the actual `pyc` file that was modules, which will always point to the actual ``pyc`` file that was
read or written. When the environment variable read or written. When the environment variable
`$PYTHONDONTWRITEBYTECODE` is set, or the `-B` option is given, or if ``$PYTHONDONTWRITEBYTECODE`` is set, or the ``-B`` option is given, or if
the source lives on a read-only filesystem, then the `__cached__` the source lives on a read-only filesystem, then the ``__cached__``
attribute will point to the location that the `pyc` file *would* have attribute will point to the location that the ``pyc`` file *would* have
been written to if it didn't exist. This location of course includes been written to if it didn't exist. This location of course includes
the `__pycache__` subdirectory in its path. the ``__pycache__`` subdirectory in its path.
For alternative Python implementations which do not support `pyc` For alternative Python implementations which do not support ``pyc``
files, the `__cached__` attribute may point to whatever information files, the ``__cached__`` attribute may point to whatever information
makes sense. E.g. on Jython, this might be the `.class` file for the makes sense. E.g. on Jython, this might be the ``.class`` file for the
module: `__pycache__/foo.jython-32.class`. Some implementations may module: ``__pycache__/foo.jython-32.class``. Some implementations may
use multiple compiled files to create the module, in which case use multiple compiled files to create the module, in which case
`__cached__` may be a tuple. The exact contents of `__cached__` are ``__cached__`` may be a tuple. The exact contents of ``__cached__`` are
Python implementation specific. Python implementation specific.
It is recommended that when nothing sensible can be calculated, It is recommended that when nothing sensible can be calculated,
implementations should set the `__cached__` attribute to `None`. implementations should set the ``__cached__`` attribute to ``None``.
py_compile and compileall py_compile and compileall
------------------------- -------------------------
Python comes with two modules, `py_compile` [15]_ and `compileall` Python comes with two modules, ``py_compile`` [15]_ and ``compileall``
[16]_ which support compiling Python modules external to the built-in [16]_ which support compiling Python modules external to the built-in
import machinery. `py_compile` in particular has intimate knowledge import machinery. ``py_compile`` in particular has intimate knowledge
of byte compilation, so these will be updated to understand the new of byte compilation, so these will be updated to understand the new
layout. The `-b` flag is added to `compileall` for writing legacy layout. The ``-b`` flag is added to ``compileall`` for writing legacy
`.pyc` byte-compiled file path names. ``.pyc`` byte-compiled file path names.
bdist_wininst and the Windows installer bdist_wininst and the Windows installer
--------------------------------------- ---------------------------------------
These tools also compile modules explicitly on installation. If they These tools also compile modules explicitly on installation. If they
do not use `py_compile` and `compileall`, then they would also have to do not use ``py_compile`` and ``compileall``, then they would also have to
be modified to understand the new layout. be modified to understand the new layout.
File extension checks File extension checks
--------------------- ---------------------
There exists some code which checks for files ending in `.pyc` and There exists some code which checks for files ending in ``.pyc`` and
simply chops off the last character to find the matching `.py` file. simply chops off the last character to find the matching ``.py`` file.
This code will obviously fail once this PEP is implemented. This code will obviously fail once this PEP is implemented.
To support this use case, we'll add two new methods to the `imp` To support this use case, we'll add two new methods to the ``imp``
package [17]_: package [17]_:
* `imp.cache_from_source(py_path)` -> `pyc_path` * ``imp.cache_from_source(py_path)`` -> ``pyc_path``
* `imp.source_from_cache(pyc_path)` -> `py_path` * ``imp.source_from_cache(pyc_path)`` -> ``py_path``
Alternative implementations are free to override these functions to Alternative implementations are free to override these functions to
return reasonable values based on their own support for this PEP. return reasonable values based on their own support for this PEP.
These methods are allowed to return `None` when the implementation (or These methods are allowed to return ``None`` when the implementation (or
:pep:`302` loader in effect) for whatever reason cannot calculate :pep:`302` loader in effect) for whatever reason cannot calculate
the appropriate file name. They should not raise exceptions. the appropriate file name. They should not raise exceptions.
@ -443,15 +443,15 @@ possible to backport this PEP. However, in Python 3.2 (and possibly
2.7), this behavior will be turned on by default, and in fact, it will 2.7), this behavior will be turned on by default, and in fact, it will
replace the old behavior. Backports will need to support the old replace the old behavior. Backports will need to support the old
layout by default. We suggest supporting :pep:`3147` through the use of layout by default. We suggest supporting :pep:`3147` through the use of
an environment variable called `$PYTHONENABLECACHEDIR` or the command an environment variable called ``$PYTHONENABLECACHEDIR`` or the command
line switch `-Xenablecachedir` to enable the feature. line switch ``-Xenablecachedir`` to enable the feature.
Makefiles and other dependency tools Makefiles and other dependency tools
------------------------------------ ------------------------------------
Makefiles and other tools which calculate dependencies on `.pyc` files Makefiles and other tools which calculate dependencies on ``.pyc`` files
(e.g. to byte-compile the source if the `.pyc` is missing) will have (e.g. to byte-compile the source if the ``.pyc`` is missing) will have
to be updated to check the new paths. to be updated to check the new paths.
@ -465,7 +465,7 @@ were considered and rejected during the PEP's development.
Hexadecimal magic tags Hexadecimal magic tags
---------------------- ----------------------
pyc files inside of the `__pycache__` directories contain a magic tag pyc files inside of the ``__pycache__`` directories contain a magic tag
in their file names. These are mnemonic tags for the actual magic in their file names. These are mnemonic tags for the actual magic
numbers used by the importer. We could have used the hexadecimal numbers used by the importer. We could have used the hexadecimal
representation [10]_ of the binary magic number as a unique representation [10]_ of the binary magic number as a unique
@ -485,15 +485,15 @@ PEP 304
There is some overlap between the goals of this PEP and :pep:`304`, There is some overlap between the goals of this PEP and :pep:`304`,
which has been withdrawn. However :pep:`304` would allow a user to which has been withdrawn. However :pep:`304` would allow a user to
create a shadow file system hierarchy in which to store `pyc` files. create a shadow file system hierarchy in which to store ``pyc`` files.
This concept of a shadow hierarchy for `pyc` files could be used to This concept of a shadow hierarchy for ``pyc`` files could be used to
satisfy the aims of this PEP. Although the :pep:`304` does not indicate satisfy the aims of this PEP. Although the :pep:`304` does not indicate
why it was withdrawn, shadow directories have a number of problems. why it was withdrawn, shadow directories have a number of problems.
The location of the shadow `pyc` files would not be easily discovered The location of the shadow ``pyc`` files would not be easily discovered
and would depend on the proper and consistent use of the and would depend on the proper and consistent use of the
`$PYTHONBYTECODE` environment variable both by the system and by end ``$PYTHONBYTECODE`` environment variable both by the system and by end
users. There are also global implications, meaning that while the users. There are also global implications, meaning that while the
system might want to shadow `pyc` files, users might not want to, but system might want to shadow ``pyc`` files, users might not want to, but
the PEP defines only an all-or-nothing approach. the PEP defines only an all-or-nothing approach.
As an example of the problem, a common (though fragile) Python idiom As an example of the problem, a common (though fragile) Python idiom
@ -503,9 +503,9 @@ for locating data files is to do something like this::
import foo.bar import foo.bar
data_file = join(dirname(foo.bar.__file__), 'my.dat') data_file = join(dirname(foo.bar.__file__), 'my.dat')
This would be problematic since `foo.bar.__file__` will give the This would be problematic since ``foo.bar.__file__`` will give the
location of the `pyc` file in the shadow directory, and it may not be location of the ``pyc`` file in the shadow directory, and it may not be
possible to find the `my.dat` file relative to the source directory possible to find the ``my.dat`` file relative to the source directory
from there. from there.
@ -513,11 +513,11 @@ Fat byte compilation files
-------------------------- --------------------------
An earlier version of this PEP described "fat" Python byte code files. An earlier version of this PEP described "fat" Python byte code files.
These files would contain the equivalent of multiple `pyc` files in a These files would contain the equivalent of multiple ``pyc`` files in a
single `pyf` file, with a lookup table keyed off the appropriate magic single ``pyf`` file, with a lookup table keyed off the appropriate magic
number. This was an extensible file format so that the first 5 number. This was an extensible file format so that the first 5
parallel Python implementations could be supported fairly efficiently, parallel Python implementations could be supported fairly efficiently,
but with extension lookup tables available to scale `pyf` byte code but with extension lookup tables available to scale ``pyf`` byte code
objects as large as necessary. objects as large as necessary.
The fat byte compilation files were fairly complex, and inherently The fat byte compilation files were fairly complex, and inherently
@ -541,7 +541,7 @@ tools that are dependent on the file extension.
.pyc .pyc
---- ----
A proposal was floated to call the `__pycache__` directory `.pyc` or A proposal was floated to call the ``__pycache__`` directory ``.pyc`` or
some other dot-file name. This would have the effect on \*nix systems some other dot-file name. This would have the effect on \*nix systems
of hiding the directory. There are many reasons why this was of hiding the directory. There are many reasons why this was
rejected by the BDFL [20]_ including the fact that dot-files are only rejected by the BDFL [20]_ including the fact that dot-files are only
@ -616,7 +616,7 @@ ACKNOWLEDGMENTS
Barry Warsaw's original idea was for fat Python byte code files. Barry Warsaw's original idea was for fat Python byte code files.
Martin von Loewis reviewed an early draft of the PEP and suggested the Martin von Loewis reviewed an early draft of the PEP and suggested the
simplification to store traditional `pyc` and `pyo` files in a simplification to store traditional ``pyc`` and ``pyo`` files in a
directory. Many other people reviewed early versions of this PEP and directory. Many other people reviewed early versions of this PEP and
provided useful feedback including but not limited to: provided useful feedback including but not limited to: