327 lines
14 KiB
ReStructuredText
327 lines
14 KiB
ReStructuredText
PEP: 0648
|
|
Title: Extensible customizations of the interpreter at startup
|
|
Author: Mario Corchero <mariocj89@gmail.com>
|
|
Sponsor: Pablo Galindo
|
|
BDFL-Delegate: XXXX
|
|
Discussions-To: https://discuss.python.org/t/pep-648-extensible-customizations-of-the-interpreter-at-startup/6403
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 30-Dec-2020
|
|
Python-Version: 3.10
|
|
Post-History: python-ideas: 16th Dec. python-dev: 18th Dec.
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes supporting extensible customization of the interpreter by
|
|
allowing users to install scripts that will be executed at startup.
|
|
|
|
Motivation
|
|
==========
|
|
|
|
System administrators, tools that repackage the interpreter and some
|
|
libraries need to customize aspects of the interpreter at startup time.
|
|
|
|
This is usually achieved via ``sitecustomize.py`` for system administrators
|
|
whilst libraries rely on exploiting ``pth`` files. This PEP proposes a way of
|
|
achieving the same in a more user-friendly and structured way.
|
|
|
|
Limitations of ``pth`` files
|
|
----------------------------
|
|
|
|
If a library needs to perform any customization before an import or that
|
|
relates to the general working of the interpreter, they often rely on the
|
|
fact that ``pth`` files, which are loaded at startup and implemented via the
|
|
site module [#site]_, can include Python code that will be executed when the
|
|
``pth`` file is evaluated.
|
|
|
|
Note that ``pth`` files were originally developed to just add additional
|
|
directories to ``sys.path``, but they may also contain lines which start
|
|
with "import", which will be passed to ``exec()``. Users have exploited this
|
|
feature to allow the customizations that they needed. See setuptools
|
|
[#setuptools]_ or betterexceptions [#betterexceptions]_ as examples.
|
|
|
|
Using ``pth`` files for this purpose is far from ideal for library developers,
|
|
as they need to inject code into a single line preceded by an import, making
|
|
it rather unreadable. Library developers following that practice will usually
|
|
create a module that performs all actions on import, as done by
|
|
betterexceptions [#betterexceptions]_, but the approach is still not really
|
|
user friendly.
|
|
|
|
Additionally, it is also non-ideal for users of the interpreter if they want
|
|
to inspect what is being executed at Python startup as they need to review
|
|
all the ``pth`` files for potential code execution which can be spread across
|
|
all site paths. Most of those ``pth`` files will be "legitimate" ``pth``
|
|
files that just modify the path, answering the question of "what is changing
|
|
my interpreter at startup" a rather complex one.
|
|
|
|
Lastly, there have been multiple suggestions for removing code execution from
|
|
``pth`` files, see [#bpo-24534]_ and [#bpo-33944]_.
|
|
|
|
Limitations of ``sitecustomize.py``
|
|
-----------------------------------
|
|
|
|
Whilst sitecustomize is an acceptable solution, it assumes a single person is
|
|
in charge of the system and the interpreter. If both the system administrator
|
|
and the responsibility of provisioning the interpreter want to add
|
|
customizations at the interpreter startup they need to agree on the contents
|
|
of the file and combine all the changes. This is not a major limitation
|
|
though, and it is not the main driver of this change. Should the change
|
|
happen, it will also improve the situation for these users, as rather than
|
|
having a ``sitecustomize.py`` which performs all those actions, they can have
|
|
custom isolated files named after the features they want to enhance. As an
|
|
example, Ubuntu could change their current ``sitecustomize.py`` to just be
|
|
``ubuntu_apport_python_hook``. This not only better represents its intent but
|
|
also gives users of the interpreter a better understanding of the
|
|
modifications happening on their interpreter.
|
|
|
|
Benefits of ``__sitecustomize__``
|
|
---------------------------------
|
|
|
|
Having a structured way of injecting custom startup scripts will allow
|
|
supporting the cases presented above in a better way. It will result in both
|
|
maintainers and users having a better experience as detailed, and allow
|
|
CPython to deprecate and eventually remove code execution from ``pth`` files
|
|
as desired in the previously mentioned bpos. Additionally, these solutions
|
|
provide a unique way to support all use-cases that before have been fulfilled
|
|
via the misuse of ``pth`` files, ``sitecustomize.py`` and
|
|
``usercustomize.py``. The use of a ``__sitecustomize__`` will allow for
|
|
packages, tools and system admins to inject scripts that will be loaded at
|
|
startups through an easy-to-understand mechanism.
|
|
|
|
Rationale
|
|
=========
|
|
|
|
This PEP proposes supporting extensible customization of the interpreter at
|
|
startup by allowing users to install scripts into a folder named
|
|
``__sitecustomize__`` located in any site path. Those scripts will be executed
|
|
at startup time. The implementation will take advantage of the fact that all
|
|
site paths are already being walked to look for ``pth`` files to include also a
|
|
check for ``__sitecustomize__`` folders and execute all scripts within them.
|
|
|
|
The ``site`` module will expose an option on its main function that allows
|
|
listing all scripts that will be executed, which will allow users to quickly
|
|
see all customizations that affect an interpreter.
|
|
|
|
We will also work with packaging build backends to facilitate the installation
|
|
of these files.
|
|
|
|
Why ``__sitecustomize__``
|
|
-------------------------
|
|
|
|
The name aims to follow the already existing concept of ``sitecustomize.py``.
|
|
As the folder will be within ``sys.path``, given that it is located in site
|
|
paths, we choose to use double underscore around its name, to prevent
|
|
colliding with the already existing ``sitecustomize.py``.
|
|
|
|
Disabling start scripts
|
|
-----------------------
|
|
|
|
In some scenarios, like when the startup time is key, it might be desired to
|
|
disable this option altogether. Whilst we could add a new flag to do so, we
|
|
think that the already existing flag ``-S`` [#s-flag]_ is already good
|
|
enough, as it disables all ``site``-related manipulation. If the flag is
|
|
passed in, ``__sitecustomize__`` will not be used.
|
|
|
|
Order of execution
|
|
------------------
|
|
|
|
The scripts in ``__sitecustomize__`` will be executed in file name sorted order
|
|
after the evaluation of ``pth`` files. We considered executing them in random
|
|
order, but that could result in different results depending on how the
|
|
interpreter chooses to pick up those files. So even if it won't be a good
|
|
practice to rely on other files being executed, we think that is better than
|
|
having randomly different results on interpreter startup.
|
|
We chose to run the scripts after the ``pth`` files in case a user needs to
|
|
add items to the path before running a script.
|
|
|
|
Note the execution happens after the handling of ``pth`` files for each of the
|
|
site paths and that the ``__sitecustomize__`` folder need to be in site paths
|
|
and not in just any importable path.
|
|
|
|
Impact on startup time
|
|
----------------------
|
|
|
|
If an interpreter is not using this mechanism, the impact on performance is
|
|
expected to be minimal as this PEP just adds a check for
|
|
``__sitecustomize__`` when ``site.py`` is walking the site paths looking for
|
|
``pth`` files. This impact will be reduced in the future as we will remove
|
|
two other imports: "sitecustomize.py" and "usercustomize.py".
|
|
|
|
If the user has custom scripts, we think that the impact on the performance
|
|
of walking each of the folders is acceptable, as the user wants to use this
|
|
feature. If they need to run a time-sensitive application, they can always
|
|
use ``-S`` to disable this entirely.
|
|
|
|
Running "./python -c pass" with perf on 50 iterations, repeating 50 times the
|
|
command on each and getting the geometric mean on a commodity laptop did not
|
|
reveal any substantial raise on CPU time.
|
|
|
|
Failure handling
|
|
----------------
|
|
|
|
Any error on any of the scripts will not be logged unless the interpreter is
|
|
run in verbose mode and it should not stop the evaluation of other scripts.
|
|
The user will just receive a message in stderr saying that the script failed to
|
|
be executed and that verbose mode can be used to get more information. This
|
|
behaviour follows the one already existing for ``sitecustomize.py``.
|
|
|
|
Scripts naming convention
|
|
-------------------------
|
|
|
|
Packages will be encouraged to include the name of the package within the name
|
|
of the script to avoid collisions between packages. The only requirement on the
|
|
filename is that it ends in ``.py`` for the interpreter to execute them.
|
|
|
|
Relationship with sitecustomize and usercustomize
|
|
-------------------------------------------------
|
|
|
|
The existing logic for ``sitecustomize.py`` and ``usercustomize.py`` will be left
|
|
as is, later deprecated and scheduled for removal. Once ``__sitecustomize__`` is
|
|
supported, it will provide better integration for all existing users, and even
|
|
if it will indeed require a migration for system administrators, we expect the
|
|
effort required to be minimal, it will just require moving and renaming the
|
|
current ``sitecustomize.py`` into the new provided folder.
|
|
|
|
Identifying all installed scripts
|
|
---------------------------------
|
|
|
|
To facilitate debugging of the Python startup, a new option will be added to
|
|
the main of the site module to list all scripts that will be executed as part
|
|
of the ``__sitecustomize__`` initialization.
|
|
|
|
How to teach this
|
|
=================
|
|
|
|
This can be documented and taught as simple as saying that the interpreter
|
|
will try to look for the ``__sitecustomize__`` folder at startup in its site
|
|
paths and if it finds any scripts with ``.py`` extension, it will then
|
|
execute it one by one.
|
|
|
|
For system administrators and tools that package the interpreter, we can now
|
|
recommend placing files in ``__sitecustomize__`` as they used to place
|
|
``sitecustomize.py``. Being more comfortable on that their content won't be
|
|
overridden by the next person, as they can provide with specific files to
|
|
handle the logic they want to customize.
|
|
|
|
Library developers should be able to specify a new argument on tools like
|
|
setuptools that will inject those new files. Something like
|
|
``sitecustomize_scripts=["scripts/betterexceptions.py"]``, which allows them to
|
|
add those. Should the build backend not support that, they can manually
|
|
install them as they used to do with ``pth`` files. We will recommend them to
|
|
include the name of the package as part of the script's name.
|
|
|
|
Backward compatibility
|
|
======================
|
|
|
|
We propose to add support for ``__sitecustomize__`` in the next release of
|
|
Python, add a warning on the three next releases on the deprecation and
|
|
future removal of ``sitecustomize.py``, ``usercustomize.py`` and code execution
|
|
in ``pth`` files, and remove it after maintainers have had 4 releases to
|
|
migrate. Ignoring those lines in pth files.
|
|
|
|
Whilst the existing ``sitecutzomize.py`` mechanism was created targeting
|
|
System Administrators that placed it in a site path, the file could be
|
|
actually placed anywhere in the path at the time that the interpreter was
|
|
starting up. The new mechanism does not allow for users to place
|
|
``__sitecustomize__`` folders anywhere in the path, but only in site paths.
|
|
System administrators can recover a similar behavior to ``sitecustomize.py``
|
|
if they need it by adding a custom script in ``__sitecustomize__`` which just
|
|
imports ``sitecustomize`` as a migration path.
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
An initial implementation that passes the CPython test suite is available for
|
|
evaluation [#reference-implementation]_.
|
|
|
|
This implementation is just for the reviewer to play with and check potential
|
|
issues that this PEP could generate.
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
Do nothing
|
|
----------
|
|
|
|
Whilst the current status "works" it presents the issues listed in the
|
|
motivation. After analysing the impact of this change, we believe it is worth
|
|
given the enhanced experience it brings.
|
|
|
|
Formalize using ``pth`` files
|
|
-----------------------------
|
|
|
|
Another option would be to just glorify and document the usage of ``pth`` files
|
|
to inject code at startup code, but that is a suboptimal experience for users
|
|
as listed in the motivation.
|
|
|
|
Making ``__sitecustomize__`` a namespace package
|
|
------------------------------------------------
|
|
|
|
We considered making the folder a namespace package and just import all the
|
|
modules within it, which allowed searching across all paths in ``sys.path``
|
|
at initialization time and provided a way to declare dependencies between
|
|
scripts by importing each other. This was rejected for multiple reasons:
|
|
|
|
1. This was unnecessarily broadening the list of paths where arbitrary scripts
|
|
are executed.
|
|
2. The logic brought additional complexity, like what to do if a package were
|
|
to install an ``__init__.py`` file in one of the locations.
|
|
3. It's cheaper to search for ``__sitecustomize__`` as we are looking for
|
|
``pth`` files already in the site paths compared to performing an actual
|
|
import of a namespace package.
|
|
|
|
Support for shutdown custom scripts
|
|
-----------------------------------
|
|
|
|
``init.d`` users might be tempted to implement this feature in a way that users
|
|
could also add code at shutdown, but extra support for that is not needed, as
|
|
Python users can already do that via ``atexit``.
|
|
|
|
Using entry_points
|
|
------------------
|
|
|
|
We considered extending the use of entry points to allow specifying scripts
|
|
that should be executed at startup but we discarded that solution due to two
|
|
main reasons. The first one being impact on startup time. This approach will
|
|
require scanning all packages distribution information to just execute a
|
|
handful of files. This has an impact on performance even if the user is not
|
|
using the feature and such impact growths linearly with the number of packages
|
|
installed in the environment. The second reason was that the proposed
|
|
implementation in this PEP offers a single solution for startup customization
|
|
for packages and system administrators. Additionally, if the main objective of
|
|
entry points is to make it easy for libraries to install scripts at startup,
|
|
that can still be added and make the build backends just install the files
|
|
within the ``__sitecustomize__`` folder.
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the CC0-1.0-Universal
|
|
license, whichever is more permissive.
|
|
|
|
References
|
|
==========
|
|
|
|
.. [#bpo-24534]
|
|
https://bugs.python.org/issue24534
|
|
|
|
.. [#bpo-33944]
|
|
https://bugs.python.org/issue33944
|
|
|
|
.. [#s-flag]
|
|
https://docs.python.org/3/using/cmdline.html#id3
|
|
|
|
.. [#setuptools]
|
|
https://github.com/pypa/setuptools/blob/b6bbe236ed0689f50b5148f1172510b975687e62/setup.py#L100
|
|
|
|
.. [#betterexceptions]
|
|
https://github.com/Qix-/better-exceptions/blob/7b417527757d555faedc354c86d3b6fe449200c2/better_exceptions_hook.pth#L1
|
|
|
|
.. [#reference-implementation]
|
|
https://github.com/mariocj89/cpython/tree/pu/__sitecustomize__
|
|
|
|
.. [#site]
|
|
https://docs.python.org/3/library/site.html
|