python-peps/peps/pep-0762.rst

PEP: 762
Title: REPL-acing the default REPL
Author: Pablo Galindo Salgado <pablogsal@python.org>, Łukasz Langa <lukasz@python.org>, Lysandros Nikolaou <lisandrosnik@gmail.com>, Emily Morehouse-Valcarcel <emily@python.org>
Sponsor: Pablo Galindo Salgado 
Status: Final
Type: Informational
Created: 11-Oct-2024
Python-Version: 3.13

Abstract
========

One of Python’s core strengths is its interactive mode, also known as the
Read-Eval-Print Loop (REPL), or the Python console, or the Python shell. This
PEP describes the new implementation of this functionality written in Python.
The new REPL released in Python 3.13 aims to provide modern features expected by
today's users, such as multi-line editing, syntax highlighting, custom commands,
and an overall improved interactive experience.

Motivation
==========

Up to Python 3.12, the interactive shell of CPython was written in C as a
special mode of the parser. It was therefore difficult to maintain and extend.
It relied on the existence of GNU readline (or an equivalent) for basic
functionality like cursor movement and history tracking. Python compiled without
this library provided an interactive mode of very limited capability. On the
other hand, Python compiled with readline outsourced decisions and configuration
around user input in ways that made extending it difficult.

This complexity has deterred contributions and has made it challenging to
implement new features. As a result, the CPython interactive shell has seen
minimal changes, falling behind user expectations for modern equivalents.

Many features that users have come to expect from modern REPLs were absent in
the previous version. Some examples of these features include multi-line editing
and history, custom commands, syntax highlighting, or ergonomic handling of copy
and paste. The lack of these features greatly impacts the user experience of
many groups of users of CPython, in particular in environments where users don’t
control dependencies and cannot install their own packages. This is especially
common for users learning the language and educators.

Addressing such issues with the C implementation would require complex
workarounds, such as AST matching of commands, which would add prohibitive
complexity to the codebase.

With the new REPL written in Python, we are addressing these limitations while
also bringing CPython's interactive experience more in line with modern
expectations and capabilities.

Rationale
=========

Implementing the new REPL in Python, rather than C, has significantly lowered
the barrier to entry for contributors. This change has made it easier to test,
validate, and modify the REPL, leading to increased community involvement and
faster feature development. The improved accessibility of the codebase is
expected to result in a more rapidly evolving and user-responsive REPL.

Instead of writing a Python REPL from scratch, we decided to base the
implementation of the new REPL on `PyREPL <https://github.com/pypy/pypy/tree/d102094b863ce49b7af030dcb0cecaac515d97c6/lib_pypy/pyrepl>`_.
This decision was driven by several key factors. First and foremost,
developing a terminal application that works consistently across different
operating systems and terminal emulators is a complex undertaking.
By adopting PyREPL, which has been battle-tested in the PyPy project,
we can leverage existing, proven code rather than starting from scratch.

Sharing a codebase with PyPy for the REPL implementation offers mutual benefits
to both projects. It allows for shared maintenance efforts, faster bug fixes,
and feature improvements that can benefit users of both CPython and PyPy. This
collaboration can lead to a more robust and feature-rich REPL for the entire
Python ecosystem.

The previous REPL written in C leveraged the “readline” or “editline” libraries
as a backend to allow certain features such as navigation, history preservation
and recall, autocompletion, and configurable keyboard behavior. PyREPL does not
use those libraries, implementing most of the other functionality directly as
part of the shell. The main missing functionality (configurability of input) is
outweighed by the benefits of the new architecture. The configuration files for
these libraries (e.g. inputrc) are complex and include features that PyREPL
doesn’t plan to implement, making it infeasible to transparently add support for
them in the new shell. Using “readline” or “editline” in PyREPL would be
prohibitively complex due to multi-line editing handling and multiplatform
support.

Although this means that existing readline/editline configurations will not be
compatible with the PyREPL, we believe the enhanced features and improved
extensibility are an overall win. See “Backwards Compatibility” for discussion
of continued support for customized workflows.

The previous REPL made it challenging to properly implement custom commands,
which is a very common feature of interactive shells. For instance, the ``exit``
command was implemented as a method call of a custom object injected in the
global namespace, leading to unintuitive behavior that often confuses users when
they simply type ``exit``, as the interpreter prompts them to the supposedly
correct usage ``exit()``.

Specification
=============

PyREPL is implemented as a new private Python module called :mod:`!_pyrepl`, existing
alongside the current C implementation. In its first implementation, it
introduces the following key features:

1. Multi-line History and Editing: Users are able to navigate and edit their
   command history across multiple lines, improving the ability to refine and reuse
   complex blocks of code.

   Editing multi-line blocks provides automatic indentation using four spaces, which
   is consistent with :pep:`8` recommendations. When a line ending with a colon is
   encountered, the following line is automatically indented utilizing the
   indentation pattern that is inferred from the first line that contains
   indentation. Lines are indented with four spaces, and tabs are converted into
   spaces.

   Users can access history of commands by using :kbd:`up` and :kbd:`down` arrows. Within
   a multi-line entry, the arrow keys navigate line-by-line within the block before
   moving to the next history entry. The :kbd:`down` arrow works in reverse, navigating
   from older entries to the most recent.

   History can be searched forward (using :kbd:`Ctrl+S`) and in reverse (using :kbd:`Ctrl+R`)
   using a custom-specified substring query. It can also be searched with a prefix
   query by entering the prefix into a shell line and using PageUp and PageDown
   keys.

2. Copying and Pasting: in supported terminal emulators, bracketed pasting
   capability is discovered and used by PyREPL. This allows for transparent pasting
   of blocks of code without immediate execution or invalid automatic indentation.
   
   For terminal emulators that don’t support this mode, a dedicated paste mode is
   implemented to allow for easy insertion of multi-line code snippets without
   triggering immediate execution or indentation issues.
   
   Users enter manual paste mode by hitting the :kbd:`F3` key. The prompt changes from
   ``>>>`` to ``(paste)`` where users can paste contents from their clipboard or
   manually type as desired. Once the content is ready, hitting :kbd:`F3` exits paste
   mode. Then, pressing Enter executes the block.
   
   Users can enter multiple commands on a single input when using paste mode, which
   will help paste code from other sources.
   
   To copy blocks of code without the leading command prompts and without the
   output of the commands, users can enter the history view via the :kbd:`F2` key. This
   mode uses a pager to display history of executed commands without the prompts
   and output.

3. Help via :kbd:`F1`.

   Access to the standard Help module is accessible via a Custom Command ``help``
   (see below) or via the :kbd:`F1` key. Hit :kbd:`F1` to enter help mode. When you're done, hit
   :kbd:`F1` or a standard command (``q``, ``quit`` or ``exit``) to exit.
   
   Browsing interactive help does not retain command history.

4. Custom Commands: The REPL supports the implementation of custom commands,
   such as ``exit``, in a more natural and user-friendly manner, avoiding the current
   function call workaround.
   
   The initial list of custom commands includes:

   * ``exit``
   * ``quit``
   * ``copyright``
   * ``help``
   * ``clear``
   
   Commands are available as long as there is no name conflict with a variable in a
   reachable scope. For example, after assigning ``exit = 1``, the variable will
   take precedence over PyREPL commands. ``del exit`` in this case will remove the
   conflict and the command will function again.

5. Colors: the prompts as well as certain elements of the output, like exception
   tracebacks, are now colored. Colors can be disabled using the standard
   ``NO_COLOR`` environment variable, or forced by using the standard
   ``FORCE_COLOR`` environment variable. A Python-specific environment variable is
   also available called ``PYTHON_COLORS``. The initial implementation in Python
   3.13 does not offer customization of the color theme.

These features are significantly enhancing the interactive Python experience,
bringing it more in line with modern development environments and user
expectations. The implementation is in Python, offering several advantages:

1. Easier Testing and Validation: Writing tests for Python code is dramatically
   simpler and more straightforward than for C code, allowing for more
   comprehensive test coverage of all existing and old features.

2. Lower Contribution Barrier: Python's accessibility compared to C has been
   encouraging more community contributions, leading to faster feature development
   and bug fixes.

3. Flexibility: A Python implementation is easier to extend and modify,
   improving developer velocity on new features and improvements by core developers
   and contributors alike.

Backwards Compatibility
=======================

The PyREPL implementation is designed to maintain full backward compatibility
with existing Python code as the old basic REPL will be preserved as a fallback
and is available on demand, in case custom workflows require it. It will also be
used in cases where the new REPL cannot be used due to environmental constraints
or other issues.  Users have the option to explicitly choose the old basic REPL
by setting the environment variable ``PYTHON_BASIC_REPL`` to 1. This ensures
that users can continue using the familiar interface and capabilities if they
prefer, or if they encounter any issues with the new implementation.

It's important to emphasize that the introduction of PyREPL does not remove any
existing functionality. Any functionality of the old basic REPL unavailable in
PyREPL is preserved and maintained in the old basic REPL that can be used by
users as a fallback.

In particular, users wanting to continue using their custom input configuration
in ``inputrc`` or ``editrc`` files can continue using the old basic REPL.

The authors do not expect any PyREPL functionality to be ported to the old basic
REPL. Similarly, ``inputrc`` and ``editrc`` support is explicitly not planned in
PyREPL. Those configuration files are provided by and parsed by “readline” and
“editline” libraries, and their functional scope does not match the
functionality PyREPL is targeting.

To facilitate a smooth transition, `clear documentation <https://docs.python.org/3.13/tutorial/appendix.html#interactive-mode>`_
is provided on how to switch between PyREPL and the old basic REPL.

This approach ensures that while we're introducing significant improvements with
the new REPL, we're not forcing any immediate changes on users who rely on the
current implementation. The fallback mechanism and user choice option provide a
safety net that allows for gradual adoption of the new REPL while maintaining
all existing functionality.

Security Implications
=====================

There are no security implications derived from this proposal.

How to Teach This
=================

The introduction of PyREPL is accompanied by documentation and tutorials. Key
areas of focus for education will include:

1. Detailed explanations on using multi-line editing, paste mode, and other new
   features.

2. Custom commands (existing and new).

3. How to switch to the new REPL, including any
   differences from the previous readline/editline-based configuration.

Rejected Ideas
==============

Several alternative approaches were considered and ultimately rejected:

1. Extending the current C implementation: While this would maintain maximum
   backwards compatibility, it was deemed too complex and would not address the
   fundamental limitations described ut supra.

2. Developing a new REPL from scratch: This approach was rejected due to the
   complexity of creating a cross-platform terminal application and the desire to
   leverage existing, proven code.

3. Using other existing REPL implementations: The authors looked at several
   alternatives like `IPython <https://ipython.org/>`_,
   `bpython <https://bpython-interpreter.org/>`_,
   `ptpython <https://github.com/prompt-toolkit/ptpython>`_, and
   `xonsh <https://xon.sh/>`_.  While all the above are impressive projects,
   in the end PyREPL was chosen for its combination of maturity, feature set,
   and lack of additional dependencies.  Another key factor was the alignment
   with PyPy's implementation.

   
Acknowledgments
===============

Thanks to Diego Russo for providing feedback on drafts of this PEP.

Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.