From 388568986b9d7d3248fac30ea62005e3e0febbd1 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 14 Oct 2024 19:54:23 +0100 Subject: [PATCH] PEP 762: REPL-acing the default REPL (#4048) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Pablo Galindo Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Jelle Zijlstra Co-authored-by: Carol Willing Co-authored-by: Łukasz Langa Co-authored-by: Emily Morehouse --- .github/CODEOWNERS | 1 + peps/pep-0762.rst | 281 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 282 insertions(+) create mode 100644 peps/pep-0762.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index a06a5f3d4..40e46a229 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -639,6 +639,7 @@ peps/pep-0758.rst @pablogsal @brettcannon peps/pep-0759.rst @warsaw peps/pep-0760.rst @pablogsal @brettcannon peps/pep-0761.rst @sethmlarson @hugovk +peps/pep-0762.rst @pablogsal @ambv @lysnikolaou @emilyemorehouse # ... peps/pep-0777.rst @warsaw # ... diff --git a/peps/pep-0762.rst b/peps/pep-0762.rst new file mode 100644 index 000000000..c3dc8295f --- /dev/null +++ b/peps/pep-0762.rst @@ -0,0 +1,281 @@ +PEP: 762 +Title: REPL-acing the default REPL +Author: Pablo Galindo Salgado , Łukasz Langa , Lysandros Nikolaou , Emily Morehouse-Valcarcel +Sponsor: Pablo Galindo Salgado +Status: Final +Type: Informational +Created: 11-Oct-2024 +Python-Version: 3.13 + +Abstract +======== + +One of Python’s core strengths is its interactive mode, also known as the +Read-Eval-Print Loop (REPL), or the Python console, or the Python shell. This +PEP describes the new implementation of this functionality written in Python. +The new REPL released in Python 3.13 aims to provide modern features expected by +today's users, such as multi-line editing, syntax highlighting, custom commands, +and an overall improved interactive experience. + +Motivation +========== + +Up to Python 3.12, the interactive shell of CPython was written in C as a +special mode of the parser. It was therefore difficult to maintain and extend. +It relied on the existence of GNU readline (or an equivalent) for basic +functionality like cursor movement and history tracking. Python compiled without +this library provided an interactive mode of very limited capability. On the +other hand, Python compiled with readline outsourced decisions and configuration +around user input in ways that made extending it difficult. + +This complexity has deterred contributions and has made it challenging to +implement new features. As a result, the CPython interactive shell has seen +minimal changes, falling behind user expectations for modern equivalents. + +Many features that users have come to expect from modern REPLs were absent in +the previous version. Some examples of these features include multi-line editing +and history, custom commands, syntax highlighting, or ergonomic handling of copy +and paste. The lack of these features greatly impacts the user experience of +many groups of users of CPython, in particular in environments where users don’t +control dependencies and cannot install their own packages. This is especially +common for users learning the language and educators. + +Addressing such issues with the C implementation would require complex +workarounds, such as AST matching of commands, which would add prohibitive +complexity to the codebase. + +With the new REPL written in Python, we are addressing these limitations while +also bringing CPython's interactive experience more in line with modern +expectations and capabilities. + +Rationale +========= + +Implementing the new REPL in Python, rather than C, has significantly lowered +the barrier to entry for contributors. This change has made it easier to test, +validate, and modify the REPL, leading to increased community involvement and +faster feature development. The improved accessibility of the codebase is +expected to result in a more rapidly evolving and user-responsive REPL. + +Instead of writing a Python REPL from scratch, we decided to base the +implementation of the new REPL on `PyREPL `_. +This decision was driven by several key factors. First and foremost, +developing a terminal application that works consistently across different +operating systems and terminal emulators is a complex undertaking. +By adopting PyREPL, which has been battle-tested in the PyPy project, +we can leverage existing, proven code rather than starting from scratch. + +Sharing a codebase with PyPy for the REPL implementation offers mutual benefits +to both projects. It allows for shared maintenance efforts, faster bug fixes, +and feature improvements that can benefit users of both CPython and PyPy. This +collaboration can lead to a more robust and feature-rich REPL for the entire +Python ecosystem. + +The previous REPL written in C leveraged the “readline” or “editline” libraries +as a backend to allow certain features such as navigation, history preservation +and recall, autocompletion, and configurable keyboard behavior. PyREPL does not +use those libraries, implementing most of the other functionality directly as +part of the shell. The main missing functionality (configurability of input) is +outweighed by the benefits of the new architecture. The configuration files for +these libraries (e.g. inputrc) are complex and include features that PyREPL +doesn’t plan to implement, making it infeasible to transparently add support for +them in the new shell. Using “readline” or “editline” in PyREPL would be +prohibitively complex due to multi-line editing handling and multiplatform +support. + +Although this means that existing readline/editline configurations will not be +compatible with the PyREPL, we believe the enhanced features and improved +extensibility are an overall win. See “Backwards Compatibility” for discussion +of continued support for customized workflows. + +The previous REPL made it challenging to properly implement custom commands, +which is a very common feature of interactive shells. For instance, the ``exit`` +command was implemented as a method call of a custom object injected in the +global namespace, leading to unintuitive behavior that often confuses users when +they simply type ``exit``, as the interpreter prompts them to the supposedly +correct usage ``exit()``. + +Specification +============= + +PyREPL is implemented as a new private Python module called :mod:`!_pyrepl`, existing +alongside the current C implementation. In its first implementation, it +introduces the following key features: + +1. Multi-line History and Editing: Users are able to navigate and edit their + command history across multiple lines, improving the ability to refine and reuse + complex blocks of code. + + Editing multi-line blocks provides automatic indentation using four spaces, which + is consistent with :pep:`8` recommendations. When a line ending with a colon is + encountered, the following line is automatically indented utilizing the + indentation pattern that is inferred from the first line that contains + indentation. Lines are indented with four spaces, and tabs are converted into + spaces. + + Users can access history of commands by using :kbd:`up` and :kbd:`down` arrows. Within + a multi-line entry, the arrow keys navigate line-by-line within the block before + moving to the next history entry. The :kbd:`down` arrow works in reverse, navigating + from older entries to the most recent. + + History can be searched forward (using :kbd:`Ctrl+S`) and in reverse (using :kbd:`Ctrl+R`) + using a custom-specified substring query. It can also be searched with a prefix + query by entering the prefix into a shell line and using PageUp and PageDown + keys. + +2. Copying and Pasting: in supported terminal emulators, bracketed pasting + capability is discovered and used by PyREPL. This allows for transparent pasting + of blocks of code without immediate execution or invalid automatic indentation. + + For terminal emulators that don’t support this mode, a dedicated paste mode is + implemented to allow for easy insertion of multi-line code snippets without + triggering immediate execution or indentation issues. + + Users enter manual paste mode by hitting the :kbd:`F3` key. The prompt changes from + ``>>>`` to ``(paste)`` where users can paste contents from their clipboard or + manually type as desired. Once the content is ready, hitting :kbd:`F3` exits paste + mode. Then, pressing Enter executes the block. + + Users can enter multiple commands on a single input when using paste mode, which + will help paste code from other sources. + + To copy blocks of code without the leading command prompts and without the + output of the commands, users can enter the history view via the :kbd:`F2` key. This + mode uses a pager to display history of executed commands without the prompts + and output. + +3. Help via :kbd:`F1`. + + Access to the standard Help module is accessible via a Custom Command ``help`` + (see below) or via the :kbd:`F1` key. Hit :kbd:`F1` to enter help mode. When you're done, hit + :kbd:`F1` or a standard command (``q``, ``quit`` or ``exit``) to exit. + + Browsing interactive help does not retain command history. + +4. Custom Commands: The REPL supports the implementation of custom commands, + such as ``exit``, in a more natural and user-friendly manner, avoiding the current + function call workaround. + + The initial list of custom commands includes: + + * ``exit`` + * ``quit`` + * ``copyright`` + * ``help`` + * ``clear`` + + Commands are available as long as there is no name conflict with a variable in a + reachable scope. For example, after assigning ``exit = 1``, the variable will + take precedence over PyREPL commands. ``del exit`` in this case will remove the + conflict and the command will function again. + +5. Colors: the prompts as well as certain elements of the output, like exception + tracebacks, are now colored. Colors can be disabled using the standard + ``NO_COLOR`` environment variable, or forced by using the standard + ``FORCE_COLOR`` environment variable. A Python-specific environment variable is + also available called ``PYTHON_COLORS``. The initial implementation in Python + 3.13 does not offer customization of the color theme. + +These features are significantly enhancing the interactive Python experience, +bringing it more in line with modern development environments and user +expectations. The implementation is in Python, offering several advantages: + +1. Easier Testing and Validation: Writing tests for Python code is dramatically + simpler and more straightforward than for C code, allowing for more + comprehensive test coverage of all existing and old features. + +2. Lower Contribution Barrier: Python's accessibility compared to C has been + encouraging more community contributions, leading to faster feature development + and bug fixes. + +3. Flexibility: A Python implementation is easier to extend and modify, + improving developer velocity on new features and improvements by core developers + and contributors alike. + +Backwards Compatibility +======================= + +The PyREPL implementation is designed to maintain full backward compatibility +with existing Python code as the old basic REPL will be preserved as a fallback +and is available on demand, in case custom workflows require it. It will also be +used in cases where the new REPL cannot be used due to environmental constraints +or other issues. Users have the option to explicitly choose the old basic REPL +by setting the environment variable ``PYTHON_BASIC_REPL`` to 1. This ensures +that users can continue using the familiar interface and capabilities if they +prefer, or if they encounter any issues with the new implementation. + +It's important to emphasize that the introduction of PyREPL does not remove any +existing functionality. Any functionality of the old basic REPL unavailable in +PyREPL is preserved and maintained in the old basic REPL that can be used by +users as a fallback. + +In particular, users wanting to continue using their custom input configuration +in ``inputrc`` or ``editrc`` files can continue using the old basic REPL. + +The authors do not expect any PyREPL functionality to be ported to the old basic +REPL. Similarly, ``inputrc`` and ``editrc`` support is explicitly not planned in +PyREPL. Those configuration files are provided by and parsed by “readline” and +“editline” libraries, and their functional scope does not match the +functionality PyREPL is targeting. + +To facilitate a smooth transition, `clear documentation `_ +is provided on how to switch between PyREPL and the old basic REPL. + +This approach ensures that while we're introducing significant improvements with +the new REPL, we're not forcing any immediate changes on users who rely on the +current implementation. The fallback mechanism and user choice option provide a +safety net that allows for gradual adoption of the new REPL while maintaining +all existing functionality. + +Security Implications +===================== + +There are no security implications derived from this proposal. + +How to Teach This +================= + +The introduction of PyREPL is accompanied by documentation and tutorials. Key +areas of focus for education will include: + +1. Detailed explanations on using multi-line editing, paste mode, and other new + features. + +2. Custom commands (existing and new). + +3. How to switch to the new REPL, including any + differences from the previous readline/editline-based configuration. + +Rejected Ideas +============== + +Several alternative approaches were considered and ultimately rejected: + +1. Extending the current C implementation: While this would maintain maximum + backwards compatibility, it was deemed too complex and would not address the + fundamental limitations described ut supra. + +2. Developing a new REPL from scratch: This approach was rejected due to the + complexity of creating a cross-platform terminal application and the desire to + leverage existing, proven code. + +3. Using other existing REPL implementations: The authors looked at several + alternatives like `IPython `_, + `bpython `_, + `ptpython `_, and + `xonsh `_. While all the above are impressive projects, + in the end PyREPL was chosen for its combination of maturity, feature set, + and lack of additional dependencies. Another key factor was the alignment + with PyPy's implementation. + + +Acknowledgments +=============== + +Thanks to Diego Russo for providing feedback on drafts of this PEP. + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive.