python-peps/peps/pep-0675.rst

PEP: 675
Title: Arbitrary Literal String Type
Author: Pradeep Kumar Srinivasan <gohanpra@gmail.com>, Graham Bleaney <gbleaney@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/VB74EHNM4RODDFM64NEEEBJQVAUAWIAW/
Status: Final
Type: Standards Track
Topic: Typing
Created: 30-Nov-2021
Python-Version: 3.11
Post-History: 07-Feb-2022
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/XEOOSSPNYPGZ5NXOJFPLXG2BTN7EVRT5/

.. canonical-typing-spec:: :ref:`typing:literalstring` and
                           :py:data:`typing.LiteralString`

Abstract
========

There is currently no way to specify, using type annotations, that a
function parameter can be of any literal string type. We have to
specify a precise literal string type, such as
``Literal["foo"]``. This PEP introduces a supertype of literal string
types: ``LiteralString``. This allows a function to accept arbitrary
literal string types, such as ``Literal["foo"]`` or
``Literal["bar"]``.


Motivation
==========

Powerful APIs that execute SQL or shell commands often recommend that
they be invoked with literal strings, rather than arbitrary user
controlled strings. There is no way to express this recommendation in
the type system, however, meaning security vulnerabilities sometimes
occur when developers fail to follow it. For example, a naive way to
look up a user record from a database is to accept a user id and
insert it into a predefined SQL query:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)
        ...  # Transform data to a User object and return it

    query_user(conn, "user123")  # OK.

However, the user-controlled data ``user_id`` is being mixed with the
SQL command string, which means a malicious user could run arbitrary
SQL commands:

.. code-block::
   :class: bad

    # Delete the table.
    query_user(conn, "user123; DROP TABLE data;")

    # Fetch all users (since 1 = 1 is always true).
    query_user(conn, "user123 OR 1 = 1")


To prevent such SQL injection attacks, SQL APIs offer parameterized
queries, which separate the executed query from user-controlled data
and make it impossible to run arbitrary queries. For example, with
`sqlite3 <https://docs.python.org/3/library/sqlite3.html>`_, our
original function would be written safely as a query with parameters:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = "SELECT * FROM data WHERE user_id = ?"
        conn.execute(query, (user_id,))
        ...

The problem is that there is no way to enforce this
discipline. sqlite3's own `documentation
<https://docs.python.org/3/library/sqlite3.html>`_ can only admonish
the reader to not dynamically build the ``sql`` argument from external
input; the API's authors cannot express that through the type
system. Users can (and often do) still use a convenient f-string as
before and leave their code vulnerable to SQL injection.

Existing tools, such as the popular security linter `Bandit
<https://github.com/PyCQA/bandit/blob/aac3f16f45648a7756727286ba8f8f0cf5e7d408/bandit/plugins/django_sql_injection.py#L102>`_,
attempt to detect unsafe external data used in SQL APIs, by inspecting
the AST or by other semantic pattern-matching. These tools, however,
preclude common idioms like storing a large multi-line query in a
variable before executing it, adding literal string modifiers to the
query based on some conditions, or transforming the query string using
a function. (We survey existing tools in the `Rejected Alternatives`_
section.) For example, many tools will detect a false positive issue
in this benign snippet:


::

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """
        if limit:
            query += " LIMIT 1"

        conn.execute(query, (user_id,))

We want to forbid harmful execution of user-controlled data while
still allowing benign idioms like the above and not requiring extra
user work.

To meet this goal, we introduce the ``LiteralString`` type, which only
accepts string values that are known to be made of literals. This is a
generalization of the ``Literal["foo"]`` type from :pep:`586`.
A string of type
``LiteralString`` cannot contain user-controlled data. Thus, any API
that only accepts ``LiteralString`` will be immune to injection
vulnerabilities (with `pragmatic limitations <Appendix B:
Limitations_>`_).

Since we want the ``sqlite3`` ``execute`` method to disallow strings
built with user input, we would make its `typeshed stub
<https://github.com/python/typeshed/blob/1c88ceeee924ec6cfe05dd4865776b49fec299e6/stdlib/sqlite3/dbapi2.pyi#L153>`_
accept a ``sql`` query that is of type ``LiteralString``:

::

    from typing import LiteralString

    def execute(self, sql: LiteralString, parameters: Iterable[str] = ...) -> Cursor: ...


This successfully forbids our unsafe SQL example. The variable
``query`` below is inferred to have type ``str``, since it is created
from a format string using ``user_id``, and cannot be passed to
``execute``:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)  # Error: Expected LiteralString, got str.
        ...

The method remains flexible enough to allow our more complicated
example:

::

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        # This is a literal string.
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """

        if limit:
            # Still has type LiteralString because we added a literal string.
            query += " LIMIT 1"

        conn.execute(query, (user_id,))  # OK

Notice that the user did not have to change their SQL code at all. The
type checker was able to infer the literal string type and complain
only in case of violations.

``LiteralString`` is also useful in other cases where we want strict
command-data separation, such as when building shell commands or when
rendering a string into an HTML response without escaping (see
`Appendix A: Other Uses`_). Overall, this combination of strictness
and flexibility makes it easy to enforce safer API usage in sensitive
code without burdening users.

Usage statistics
----------------

In a sample of open-source projects using ``sqlite3``, we found that
``conn.execute`` was called `~67% of the time
<https://grep.app/search?q=conn%5C.execute%5C%28%5Cs%2A%5B%27%22%5D&regexp=true&filter[lang][0]=Python>`_
with a safe string literal and `~33% of the time
<https://grep.app/search?current=3&q=conn%5C.execute%5C%28%5Ba-zA-Z_%5D%2B%5C%29&regexp=true&filter[lang][0]=Python>`_
with a potentially unsafe, local string variable. Using this PEP's
literal string type along with a type checker would prevent the unsafe
portion of that 33% of cases (ie. the ones where user controlled data
is incorporated into the query), while seamlessly allowing the safe
ones to remain.

Rationale
=========

Firstly, why use *types* to prevent security vulnerabilities?

Warning users in documentation is insufficient - most users either
never see these warnings or ignore them. Using an existing dynamic or
static analysis approach is too restrictive - these prevent natural
idioms, as we saw in the `Motivation`_ section (and will discuss more
extensively in the `Rejected Alternatives`_ section). The typing-based
approach in this PEP strikes a user-friendly balance between
strictness and flexibility.

Runtime approaches do not work because, at runtime, the query string
is a plain ``str``. While we could prevent some exploits using
heuristics, such as regex-filtering for obviously malicious payloads,
there will always be a way to work around them (perfectly
distinguishing good and bad queries reduces to the halting problem).

Static approaches, such as checking the AST to see if the query string
is a literal string expression, cannot tell when a string is assigned
to an intermediate variable or when it is transformed by a benign
function. This makes them overly restrictive.

The type checker, surprisingly, does better than both because it has
access to information not available in the runtime or static analysis
approaches. Specifically, the type checker can tell us whether an
expression has a literal string type, say ``Literal["foo"]``. The type
checker already propagates types across variable assignments or
function calls.

In the current type system itself, if the SQL or shell command
execution function only accepted three possible input strings, our job
would be done. We would just say:

::

    def execute(query: Literal["foo", "bar", "baz"]) -> None: ...

But, of course, ``execute`` can accept *any* possible query. How do we
ensure that the query does not contain an arbitrary, user-controlled
string?

We want to specify that the value must be of some type
``Literal[<...>]`` where ``<...>`` is some string. This is what
``LiteralString`` represents. ``LiteralString`` is the "supertype" of
all literal string types. In effect, this PEP just introduces a type
in the type hierarchy between ``Literal["foo"]`` and ``str``. Any
particular literal string, such as ``Literal["foo"]`` or
``Literal["bar"]``, is compatible with ``LiteralString``, but not the
other way around. The "supertype" of ``LiteralString`` itself is
``str``. So, ``LiteralString`` is compatible with ``str``, but not the
other way around.

Note that a ``Union`` of literal types is naturally compatible with
``LiteralString`` because each element of the ``Union`` is individually
compatible with ``LiteralString``. So, ``Literal["foo", "bar"]`` is
compatible with ``LiteralString``.

However, recall that we don't just want to represent exact literal
queries. We also want to support composition of two literal strings,
such as ``query + " LIMIT 1"``. This too is possible with the above
concept. If ``x`` and ``y`` are two values of type ``LiteralString``,
then ``x + y`` will also be of type compatible with
``LiteralString``. We can reason about this by looking at specific
instances such as ``Literal["foo"]`` and ``Literal["bar"]``; the value
of the added string ``x + y`` can only be ``"foobar"``, which has type
``Literal["foobar"]`` and is thus compatible with
``LiteralString``. The same reasoning applies when ``x`` and ``y`` are
unions of literal types; the result of pairwise adding any two literal
types from ``x`` and ``y`` respectively is a literal type, which means
that the overall result is a ``Union`` of literal types and is thus
compatible with ``LiteralString``.

In this way, we are able to leverage Python's concept of a ``Literal``
string type to specify that our API can only accept strings that are
known to be constructed from literals. More specific details follow in
the remaining sections.

Specification
=============


Runtime Behavior
----------------

We propose adding ``LiteralString`` to ``typing.py``, with an
implementation similar to ``typing.NoReturn``.

Note that ``LiteralString`` is a special form used solely for type
checking. There is no expression for which ``type(<expr>)`` will
produce ``LiteralString`` at runtime. So, we do not specify in the
implementation that it is a subclass of ``str``.


Valid Locations for ``LiteralString``
-----------------------------------------

``LiteralString`` can be used where any other type can be used:

::

    variable_annotation: LiteralString

    def my_function(literal_string: LiteralString) -> LiteralString: ...

    class Foo:
        my_attribute: LiteralString

    type_argument: List[LiteralString]

    T = TypeVar("T", bound=LiteralString)

It cannot be nested within unions of ``Literal`` types:

::

    bad_union: Literal["hello", LiteralString]  # Not OK
    bad_nesting: Literal[LiteralString]  # Not OK


Type Inference
--------------

.. _inferring_literal_string:


Inferring ``LiteralString``
'''''''''''''''''''''''''''

Any literal string type is compatible with ``LiteralString``. For
example, ``x: LiteralString = "foo"`` is valid because ``"foo"`` is
inferred to be of type ``Literal["foo"]``.

As per the `Rationale`_, we also infer ``LiteralString`` in the
following cases:

+ Addition: ``x + y`` is of type ``LiteralString`` if both ``x`` and
  ``y`` are compatible with ``LiteralString``.

+ Joining: ``sep.join(xs)`` is of type ``LiteralString`` if ``sep``'s
  type is compatible with ``LiteralString`` and ``xs``'s type is
  compatible with ``Iterable[LiteralString]``.

+ In-place addition: If ``s`` has type ``LiteralString`` and ``x`` has
  type compatible with ``LiteralString``, then ``s += x`` preserves
  ``s``'s type as ``LiteralString``.

+ String formatting: An f-string has type ``LiteralString`` if and only
  if its constituent expressions are literal strings. ``s.format(...)``
  has type ``LiteralString`` if and only if ``s`` and the arguments have
  types compatible with ``LiteralString``.

+ Literal-preserving methods: In `Appendix C <PEP 675 Appendix C_>`_,
  we have provided an exhaustive list of ``str`` methods that preserve the
  ``LiteralString`` type.

In all other cases, if one or more of the composed values has a
non-literal type ``str``, the composition of types will have type
``str``. For example, if ``s`` has type ``str``, then ``"hello" + s``
has type ``str``. This matches the pre-existing behavior of type
checkers.

``LiteralString`` is compatible with the type ``str``. It inherits all
methods from ``str``. So, if we have a variable ``s`` of type
``LiteralString``, it is safe to write ``s.startswith("hello")``.

Some type checkers refine the type of a string when doing an equality
check:

::

    def foo(s: str) -> None:
        if s == "bar":
            reveal_type(s)  # => Literal["bar"]

Such a refined type in the if-block is also compatible with
``LiteralString`` because its type is ``Literal["bar"]``.


Examples
''''''''

See the examples below to help clarify the above rules:

::


    literal_string: LiteralString
    s: str = literal_string  # OK

    literal_string: LiteralString = s  # Error: Expected LiteralString, got str.
    literal_string: LiteralString = "hello"  # OK

Addition of literal strings:

::

    def expect_literal_string(s: LiteralString) -> None: ...

    expect_literal_string("foo" + "bar")  # OK
    expect_literal_string(literal_string + "bar")  # OK

    literal_string2: LiteralString
    expect_literal_string(literal_string + literal_string2)  # OK

    plain_string: str
    expect_literal_string(literal_string + plain_string)  # Not OK.

Join using literal strings:

::

    expect_literal_string(",".join(["foo", "bar"]))  # OK
    expect_literal_string(literal_string.join(["foo", "bar"]))  # OK
    expect_literal_string(literal_string.join([literal_string, literal_string2]))  # OK

    xs: List[LiteralString]
    expect_literal_string(literal_string.join(xs)) # OK
    expect_literal_string(plain_string.join([literal_string, literal_string2]))
    # Not OK because the separator has type 'str'.

In-place addition using literal strings:

::

    literal_string += "foo"  # OK
    literal_string += literal_string2  # OK
    literal_string += plain_string # Not OK

Format strings using literal strings:

::

    literal_name: LiteralString
    expect_literal_string(f"hello {literal_name}")
    # OK because it is composed from literal strings.

    expect_literal_string("hello {}".format(literal_name))  # OK

    expect_literal_string(f"hello")  # OK

    username: str
    expect_literal_string(f"hello {username}")
    # NOT OK. The format-string is constructed from 'username',
    # which has type 'str'.

    expect_literal_string("hello {}".format(username))  # Not OK

Other literal types, such as literal integers, are not compatible with ``LiteralString``:

::

    some_int: int
    expect_literal_string(some_int)  # Error: Expected LiteralString, got int.

    literal_one: Literal[1] = 1
    expect_literal_string(literal_one)  # Error: Expected LiteralString, got Literal[1].


We can call functions on literal strings:

::

    def add_limit(query: LiteralString) -> LiteralString:
        return query + " LIMIT = 1"

    def my_query(query: LiteralString, user_id: str) -> None:
        sql_connection().execute(add_limit(query), (user_id,))  # OK

Conditional statements and expressions work as expected:

::

    def return_literal_string() -> LiteralString:
        return "foo" if condition1() else "bar"  # OK

    def return_literal_str2(literal_string: LiteralString) -> LiteralString:
        return "foo" if condition1() else literal_string  # OK

    def return_literal_str3() -> LiteralString:
        if condition1():
            result: Literal["foo"] = "foo"
        else:
            result: LiteralString = "bar"

        return result  # OK


Interaction with TypeVars and Generics
''''''''''''''''''''''''''''''''''''''

TypeVars can be bound to ``LiteralString``:

::

    from typing import Literal, LiteralString, TypeVar

    TLiteral = TypeVar("TLiteral", bound=LiteralString)

    def literal_identity(s: TLiteral) -> TLiteral:
        return s

    hello: Literal["hello"] = "hello"
    y = literal_identity(hello)
    reveal_type(y)  # => Literal["hello"]

    s: LiteralString
    y2 = literal_identity(s)
    reveal_type(y2)  # => LiteralString

    s_error: str
    literal_identity(s_error)
    # Error: Expected TLiteral (bound to LiteralString), got str.


``LiteralString`` can be used as a type argument for generic classes:

::

    class Container(Generic[T]):
        def __init__(self, value: T) -> None:
            self.value = value

    literal_string: LiteralString = "hello"
    x: Container[LiteralString] = Container(literal_string)  # OK

    s: str
    x_error: Container[LiteralString] = Container(s)  # Not OK

Standard containers like ``List`` work as expected:

::

    xs: List[LiteralString] = ["foo", "bar", "baz"]


Interactions with Overloads
'''''''''''''''''''''''''''

Literal strings and overloads do not need to interact in a special
way: the existing rules work fine. ``LiteralString`` can be used as a
fallback overload where a specific ``Literal["foo"]`` type does not
match:

::

    @overload
    def foo(x: Literal["foo"]) -> int: ...
    @overload
    def foo(x: LiteralString) -> bool: ...
    @overload
    def foo(x: str) -> str: ...

    x1: int = foo("foo")  # First overload.
    x2: bool = foo("bar")  # Second overload.
    s: str
    x3: str = foo(s)  # Third overload.


Backwards Compatibility
=======================

We propose adding ``typing_extensions.LiteralString`` for use in
earlier Python versions.

As :pep:`PEP 586 mentions
<586#backwards-compatibility>`,
type checkers "should feel free to experiment with more sophisticated
inference techniques". So, if the type checker infers a literal string
type for an unannotated variable that is initialized with a literal
string, the following example should be OK:

::

    x = "hello"
    expect_literal_string(x)
    # OK, because x is inferred to have type 'Literal["hello"]'.

This enables precise type checking of idiomatic SQL query code without
annotating the code at all (as seen in the `Motivation`_ section
example).

However, like :pep:`586`, this PEP does not mandate the above inference
strategy. In case the type checker doesn't infer ``x`` to have type
``Literal["hello"]``, users can aid the type checker by explicitly
annotating it as ``x: LiteralString``:

::

    x: LiteralString = "hello"
    expect_literal_string(x)


Rejected Alternatives
=====================

Why not use tool X?
-------------------

Tools to catch issues such as SQL injection seem to come in three
flavors: AST based, function level analysis, and taint flow analysis.

**AST-based tools**: `Bandit
<https://github.com/PyCQA/bandit/blob/aac3f16f45648a7756727286ba8f8f0cf5e7d408/bandit/plugins/django_sql_injection.py#L102>`_
has a plugin to warn when SQL queries are not literal
strings. The problem is that many perfectly safe SQL
queries are dynamically built out of string literals, as shown in the
`Motivation`_ section. At the
AST level, the resultant SQL query is not going to appear as a string
literal anymore and is thus indistinguishable from a potentially
malicious string. To use these tools would require significantly
restricting developers' ability to build SQL queries. ``LiteralString``
can provide similar safety guarantees with fewer restrictions.

**Semgrep and pyanalyze**: Semgrep supports a more sophisticated
function level analysis, including `constant propagation
<https://semgrep.dev/docs/writing-rules/data-flow/#constant-propagation>`_
within a function. This allows us to prevent injection attacks while
permitting some forms of safe dynamic SQL queries within a
function. `pyanalyze
<https://github.com/quora/pyanalyze/blob/afcb58cd3e967e4e3fea9e57bb18b6b1d9d42ed7/README.md#extending-pyanalyze>`_
has a similar extension. But neither handles function calls that
construct and return safe SQL queries. For example, in the code sample
below, ``build_insert_query`` is a helper function to create a query
that inserts multiple values into the corresponding columns. Semgrep
and pyanalyze forbid this natural usage whereas ``LiteralString``
handles it with no burden on the programmer:

::

    def build_insert_query(
        table: LiteralString
        insert_columns: Iterable[LiteralString],
    ) -> LiteralString:
        sql = "INSERT INTO " + table

        column_clause = ", ".join(insert_columns)
        value_clause = ", ".join(["?"] * len(insert_columns))

        sql += f" ({column_clause}) VALUES ({value_clause})"
        return sql

    def insert_data(
        conn: Connection,
        kvs_to_insert: Dict[LiteralString, str]
    ) -> None:
        query = build_insert_query("data", kvs_to_insert.keys())
        conn.execute(query, kvs_to_insert.values())

    # Example usage
    data_to_insert = {
        "column_1": value_1,  # Note: values are not literals
        "column_2": value_2,
        "column_3": value_3,
    }
    insert_data(conn, data_to_insert)


**Taint flow analysis**: Tools such as `Pysa
<https://pyre-check.org/docs/pysa-basics/>`_ or `CodeQL
<https://codeql.github.com/>`_ are capable of tracking data flowing
from a user controlled input into a SQL query. These tools are
powerful but involve considerable overhead in setting up the tool in
CI, defining "taint" sinks and sources, and teaching developers how to
use them. They also usually take longer to run than a type checker
(minutes instead of seconds), which means feedback is not
immediate. Finally, they move the burden of preventing vulnerabilities
on to library users instead of allowing the libraries themselves to
specify precisely how their APIs must be called (as is possible with
``LiteralString``).

One final reason to prefer using a new type over a dedicated tool is
that type checkers are more widely used than dedicated security
tooling; for example, MyPy was downloaded `over 7 million times
<https://www.pypistats.org/packages/mypy>`_ in Jan 2022 vs `less than
2 million times <https://www.pypistats.org/packages/bandit>`_ for
Bandit. Having security protections built right into type checkers
will mean that more developers benefit from them.


Why not use a ``NewType`` for ``str``?
--------------------------------------

Any API for which ``LiteralString`` would be suitable could instead be
updated to accept a different type created within the Python type
system, such as ``NewType("SafeSQL", str)``:

::

    SafeSQL = NewType("SafeSQL", str)

    def execute(self, sql: SafeSQL, parameters: Iterable[str] = ...) -> Cursor: ...

    execute(SafeSQL("SELECT * FROM data WHERE user_id = ?"), user_id)  # OK

    user_query: str
    execute(user_query)  # Error: Expected SafeSQL, got str.


Having to create a new type to call an API might give some developers
pause and encourage more caution, but it doesn't guarantee that
developers won't just turn a user controlled string into the new type,
and pass it into the modified API anyway:

::

    query = f"SELECT * FROM data WHERE user_id = f{user_id}"
    execute(SafeSQL(query))  # No error!

We are back to square one with the problem of preventing arbitrary
inputs to ``SafeSQL``. This is not a theoretical concern
either. Django uses the above approach with ``SafeString`` and
`mark_safe
<https://docs.djangoproject.com/en/dev/_modules/django/utils/safestring/#SafeString>`_. Issues
such as `CVE-2020-13596
<https://github.com/django/django/commit/2dd4d110c159d0c81dff42eaead2c378a0998735>`_
show how this technique can `fail
<https://nvd.nist.gov/vuln/detail/CVE-2020-13596>`_.

Also note that this requires invasive changes to the source code
(wrapping the query with ``SafeSQL``) whereas ``LiteralString``
requires no such changes. Users can remain oblivious to it as long as
they pass in literal strings to sensitive APIs.

Why not try to emulate Trusted Types?
-------------------------------------

`Trusted Types
<https://w3c.github.io/webappsec-trusted-types/dist/spec/>`_ is a W3C
specification for preventing DOM-based Cross Site Scripting (XSS). XSS
occurs when dangerous browser APIs accept raw user-controlled
strings. The specification modifies these APIs to accept only the
"Trusted Types" returned by designated sanitizing functions. These
sanitizing functions must take in a potentially malicious string and
validate it or render it benign somehow, for example by verifying that
it is a valid URL or HTML-encoding it.

It can be tempting to assume porting the concept of Trusted Types to
Python could solve the problem. The fundamental difference, however,
is that the output of a Trusted Types sanitizer is usually intended
*to not be executable code*. Thus it's easy to HTML encode the input,
strip out dangerous tags, or otherwise render it inert. With a SQL
query or shell command, the end result *still needs to be executable
code*. There is no way to write a sanitizer that can reliably figure
out which parts of an input string are benign and which ones are
potentially malicious.

Runtime Checkable ``LiteralString``
-----------------------------------

The ``LiteralString`` concept could be extended beyond static type
checking to be a runtime checkable property of ``str`` objects. This
would provide some benefits, such as allowing frameworks to raise
errors on dynamic strings. Such runtime errors would be a more robust
defense mechanism than type errors, which can potentially be
suppressed, ignored, or never even seen if the author does not use a
type checker.

This extension to the ``LiteralString`` concept would dramatically
increase the scope of the proposal by requiring changes to one of the
most fundamental types in Python. While runtime taint checking on
strings, similar to Perl's `taint <https://metacpan.org/pod/Taint>`_,
has been `considered <https://bugs.python.org/issue500698>`_ and
`attempted <https://github.com/felixgr/pytaint>`_ in the past, and
others may consider it in the future, such extensions are out of scope
for this PEP.


Rejected Names
--------------

We considered a variety of names for the literal string type and
solicited ideas on `typing-sig
<https://mail.python.org/archives/list/typing-sig@python.org/thread/VB74EHNM4RODDFM64NEEEBJQVAUAWIAW/>`_.
Some notable alternatives were:

+ ``Literal[str]``: This is a natural extension of the
  ``Literal["foo"]`` type name, but typing-sig `objected
  <https://mail.python.org/archives/list/typing-sig@python.org/message/2ZQO4NTJEI42KTRJDBL77MNANEXOW7UI/>`_
  that users could mistake this for the literal type of the ``str``
  class.

+ ``LiteralStr``: This is shorter than ``LiteralString`` but looks
  weird to the PEP authors.

+ ``LiteralDerivedString``: This (along with
  ``MadeFromLiteralString``) best captures the technical meaning of
  the type. It represents not just the type of literal expressions,
  such as ``"foo"``, but also that of expressions composed from
  literals, such as ``"foo" + "bar"``. However, both names seem wordy.

+ ``StringLiteral``: Users might confuse this with the existing
  concept of `"string literals"
  <https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals>`_
  where the string exists as a syntactic token in the source code,
  whereas our concept is more general.

+ ``SafeString``: While this comes close to our intended meaning, it
  may mislead users into thinking that the string has been sanitized in
  some way, perhaps by escaping HTML tags or shell-related special
  characters.

+ ``ConstantStr``: This does not capture the idea of composing literal
  strings.

+ ``StaticStr``: This suggests that the string is statically
  computable, i.e., computable without running the program, which is
  not true. The literal string may vary based on runtime flags, as
  seen in the `Motivation`_ examples.

+ ``LiteralOnly[str]``: This has the advantage of being extensible to
  other literal types, such as ``bytes`` or ``int``. However, we did
  not find the extensibility worth the loss of readability.

Overall, there was no clear winner on typing-sig over a long period,
so we decided to tip the scales in favor of ``LiteralString``.


``LiteralBytes``
----------------

We could generalize literal byte types, such as ``Literal[b"foo"]``,
to ``LiteralBytes``. However, literal byte types are used much less
frequently than literal string types and we did not find much user
demand for ``LiteralBytes``, so we decided not to include it in this
PEP. Others may, however, consider it in future PEPs.


Reference Implementation
========================

This is implemented in Pyre v0.9.8 and is actively being used.

The implementation simply extends the type checker with
``LiteralString`` as a supertype of literal string types.

To support composition via addition, join, etc., it was sufficient to
overload the stubs for ``str`` in Pyre's copy of typeshed.


Appendix A: Other Uses
======================

To simplify the discussion and require minimal security knowledge, we
focused on SQL injections throughout the PEP. ``LiteralString``,
however, can also be used to prevent many other kinds of `injection
vulnerabilities <https://owasp.org/www-community/Injection_Flaws>`_.

Command Injection
-----------------

APIs such as ``subprocess.run`` accept a string which can be run as a
shell command:

::

    subprocess.run(f"echo 'Hello {name}'", shell=True)

If user-controlled data is included in the command string, the code is
vulnerable to "command injection"; i.e., an attacker can run malicious
commands. For example, a value of ``' && rm -rf / #`` would result in
the following destructive command being run:

::

    echo 'Hello ' && rm -rf / #'

This vulnerability could be prevented by updating ``run`` to only
accept ``LiteralString`` when used in ``shell=True`` mode. Here is one
simplified stub:

::

    def run(command: LiteralString, *args: str, shell: bool=...): ...

Cross Site Scripting (XSS)
--------------------------

Most popular Python web frameworks, such as Django, use a templating
engine to produce HTML from user data. These templating languages
auto-escape user data before inserting it into the HTML template and
thus prevent cross site scripting (XSS) vulnerabilities.

But a common way to `bypass auto-escaping
<https://django.readthedocs.io/en/stable/ref/templates/language.html#how-to-turn-it-off>`_
and render HTML as-is is to use functions like ``mark_safe`` in
`Django
<https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.safestring.mark_safe>`_
or ``do_mark_safe`` in `Jinja2
<https://github.com/pallets/jinja/blob/077b7918a7642ff6742fe48a32e54d7875140894/src/jinja2/filters.py#L1264>`_,
which cause XSS vulnerabilities:

.. code-block::
   :class: bad

    dangerous_string = django.utils.safestring.mark_safe(f"<script>{user_input}</script>")
    return(dangerous_string)

This vulnerability could be prevented by updating ``mark_safe`` to
only accept ``LiteralString``:

.. code-block::
   :class: good

    def mark_safe(s: LiteralString) -> str: ...

Server Side Template Injection (SSTI)
-------------------------------------

Templating frameworks, such as Jinja, allow Python expressions which
will be evaluated and substituted into the rendered result:

::

    template_str = "There are {{ len(values) }} values: {{ values }}"
    template = jinja2.Template(template_str)
    template.render(values=[1, 2])
    # Result: "There are 2 values: [1, 2]"

If an attacker controls all or part of the template string, they can
insert expressions which execute arbitrary code and `compromise
<https://www.onsecurity.io/blog/server-side-template-injection-with-jinja2/>`_
the application:

.. code-block::
   :class: bad

    malicious_str = "{{''.__class__.__base__.__subclasses__()[408]('rm - rf /',shell=True)}}"
    template = jinja2.Template(malicious_str)
    template.render()
    # Result: The shell command 'rm - rf /' is run

Template injection exploits like this could be prevented by updating
the ``Template`` API to only accept ``LiteralString``:

.. code-block::
   :class: good

    class Template:
        def __init__(self, source: LiteralString): ...


Logging Format String Injection
-------------------------------

Logging frameworks often allow their input strings to contain
formatting directives. At its worst, allowing users to control the
logged string has led to `CVE-2021-44228
<https://nvd.nist.gov/vuln/detail/CVE-2021-44228>`_ (colloquially
known as ``log4shell``), which has been described as the `"most
critical vulnerability of the last decade"
<https://www.theguardian.com/technology/2021/dec/10/software-flaw-most-critical-vulnerability-log-4-shell>`_.
While no Python frameworks are currently known to be vulnerable to a
similar attack, the built-in logging framework does provide formatting
options which are vulnerable to Denial of Service attacks from
externally controlled logging strings. The following example
illustrates a simple denial of service scenario:

.. code-block::
   :class: bad

    external_string = "%(foo)999999999s"
    ...
    # Tries to add > 1GB of whitespace to the logged string:
    logger.info(f'Received: {external_string}', some_dict)

This kind of attack could be prevented by requiring that the format
string passed to the logger be a ``LiteralString`` and that all
externally controlled data be passed separately as arguments (as
proposed in `Issue 46200 <https://bugs.python.org/issue46200>`_):

.. code-block::
   :class: good

    def info(msg: LiteralString, *args: object) -> None:
        ...


Appendix B: Limitations
=======================

There are a number of ways ``LiteralString`` could still fail to
prevent users from passing strings built from non-literal data to an
API:

1. If the developer does not use a type checker or does not add type
annotations, then violations will go uncaught.

2. ``cast(LiteralString, non_literal_string)`` could be used to lie to
the type checker and allow a dynamic string value to masquerade as a
``LiteralString``. The same goes for a variable that has type ``Any``.

3. Comments such as ``# type: ignore`` could be used to ignore
warnings about non-literal strings.

4. Trivial functions could be constructed to convert a ``str`` to a
``LiteralString``:

.. code-block::
   :class: bad

    def make_literal(s: str) -> LiteralString:
        letters: Dict[str, LiteralString] = {
            "A": "A",
            "B": "B",
            ...
        }
        output: List[LiteralString] = [letters[c] for c in s]
        return "".join(output)


We could mitigate the above using linting, code review, etc., but
ultimately a clever, malicious developer attempting to circumvent the
protections offered by ``LiteralString`` will always succeed. The
important thing to remember is that ``LiteralString`` is not intended
to protect against *malicious* developers; it is meant to protect
against benign developers accidentally using sensitive APIs in a
dangerous way (without getting in their way otherwise).

Without ``LiteralString``, the best enforcement tool API authors have
is documentation, which is easily ignored and often not seen. With
``LiteralString``, API misuse requires conscious thought and artifacts
in the code that reviewers and future developers can notice.

.. _PEP 675 Appendix C:

Appendix C: ``str`` methods that preserve ``LiteralString``
===========================================================

The ``str`` class has several methods that would benefit from
``LiteralString``. For example, users might expect
``"hello".capitalize()`` to have the type ``LiteralString`` similar to
the other examples we have seen in the `Inferring LiteralString
<inferring_literal_string_>`_ section. Inferring the type
``LiteralString`` is correct because the string is not an arbitrary
user-supplied string - we know that it has the type
``Literal["HELLO"]``, which is compatible with ``LiteralString``. In
other words, the ``capitalize`` method preserves the ``LiteralString``
type. There are several other ``str`` methods that preserve
``LiteralString``.

We propose updating the stub for ``str`` in typeshed so that the
methods are overloaded with the ``LiteralString``-preserving
versions. This means type checkers do not have to hardcode
``LiteralString`` behavior for each method. It also lets us easily
support new methods in the future by updating the typeshed stub.

For example, to preserve literal types for the ``capitalize`` method,
we would change the stub as below:

::

    # before
    def capitalize(self) -> str: ...

    # after
    @overload
    def capitalize(self: LiteralString) -> LiteralString: ...
    @overload
    def capitalize(self) -> str: ...

The downside of changing the ``str`` stub is that the stub becomes
more complicated and can make error messages harder to
understand. Type checkers may need to special-case ``str`` to make
error messages understandable for users.

Below is an exhaustive list of ``str`` methods which, when called with
arguments of type ``LiteralString``, must be treated as returning a
``LiteralString``. If this PEP is accepted, we will update these
method signatures in typeshed:

::

    @overload
    def capitalize(self: LiteralString) -> LiteralString: ...
    @overload
    def capitalize(self) -> str: ...

    @overload
    def casefold(self: LiteralString) -> LiteralString: ...
    @overload
    def casefold(self) -> str: ...

    @overload
    def center(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def center(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    if sys.version_info >= (3, 8):
        @overload
        def expandtabs(self: LiteralString, tabsize: SupportsIndex = ...) -> LiteralString: ...
        @overload
        def expandtabs(self, tabsize: SupportsIndex = ...) -> str: ...

    else:
        @overload
        def expandtabs(self: LiteralString, tabsize: int = ...) -> LiteralString: ...
        @overload
        def expandtabs(self, tabsize: int = ...) -> str: ...

    @overload
    def format(self: LiteralString, *args: LiteralString, **kwargs: LiteralString) -> LiteralString: ...
    @overload
    def format(self, *args: str, **kwargs: str) -> str: ...

    @overload
    def join(self: LiteralString, __iterable: Iterable[LiteralString]) -> LiteralString: ...
    @overload
    def join(self, __iterable: Iterable[str]) -> str: ...

    @overload
    def ljust(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def ljust(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    @overload
    def lower(self: LiteralString) -> LiteralString: ...
    @overload
    def lower(self) -> LiteralString: ...

    @overload
    def lstrip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def lstrip(self, __chars: str | None = ...) -> str: ...

    @overload
    def partition(self: LiteralString, __sep: LiteralString) -> tuple[LiteralString, LiteralString, LiteralString]: ...
    @overload
    def partition(self, __sep: str) -> tuple[str, str, str]: ...

    @overload
    def replace(self: LiteralString, __old: LiteralString, __new: LiteralString, __count: SupportsIndex = ...) -> LiteralString: ...
    @overload
    def replace(self, __old: str, __new: str, __count: SupportsIndex = ...) -> str: ...

    if sys.version_info >= (3, 9):
        @overload
        def removeprefix(self: LiteralString, __prefix: LiteralString) -> LiteralString: ...
        @overload
        def removeprefix(self, __prefix: str) -> str: ...

        @overload
        def removesuffix(self: LiteralString, __suffix: LiteralString) -> LiteralString: ...
        @overload
        def removesuffix(self, __suffix: str) -> str: ...

    @overload
    def rjust(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def rjust(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    @overload
    def rpartition(self: LiteralString, __sep: LiteralString) -> tuple[LiteralString, LiteralString, LiteralString]: ...
    @overload
    def rpartition(self, __sep: str) -> tuple[str, str, str]: ...

    @overload
    def rsplit(self: LiteralString, sep: LiteralString | None = ..., maxsplit: SupportsIndex = ...) -> list[LiteralString]: ...
    @overload
    def rsplit(self, sep: str | None = ..., maxsplit: SupportsIndex = ...) -> list[str]: ...

    @overload
    def rstrip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def rstrip(self, __chars: str | None = ...) -> str: ...

    @overload
    def split(self: LiteralString, sep: LiteralString | None = ..., maxsplit: SupportsIndex = ...) -> list[LiteralString]: ...
    @overload
    def split(self, sep: str | None = ..., maxsplit: SupportsIndex = ...) -> list[str]: ...

    @overload
    def splitlines(self: LiteralString, keepends: bool = ...) -> list[LiteralString]: ...
    @overload
    def splitlines(self, keepends: bool = ...) -> list[str]: ...

    @overload
    def strip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def strip(self, __chars: str | None = ...) -> str: ...

    @overload
    def swapcase(self: LiteralString) -> LiteralString: ...
    @overload
    def swapcase(self) -> str: ...

    @overload
    def title(self: LiteralString) -> LiteralString: ...
    @overload
    def title(self) -> str: ...

    @overload
    def upper(self: LiteralString) -> LiteralString: ...
    @overload
    def upper(self) -> str: ...

    @overload
    def zfill(self: LiteralString, __width: SupportsIndex) -> LiteralString: ...
    @overload
    def zfill(self, __width: SupportsIndex) -> str: ...

    @overload
    def __add__(self: LiteralString, __s: LiteralString) -> LiteralString: ...
    @overload
    def __add__(self, __s: str) -> str: ...

    @overload
    def __iter__(self: LiteralString) -> Iterator[str]: ...
    @overload
    def __iter__(self) -> Iterator[str]: ...

    @overload
    def __mod__(self: LiteralString, __x: Union[LiteralString, Tuple[LiteralString, ...]]) -> str: ...
    @overload
    def __mod__(self, __x: Union[str, Tuple[str, ...]]) -> str: ...

    @overload
    def __mul__(self: LiteralString, __n: SupportsIndex) -> LiteralString: ...
    @overload
    def __mul__(self, __n: SupportsIndex) -> str: ...

    @overload
    def __repr__(self: LiteralString) -> LiteralString: ...
    @overload
    def __repr__(self) -> str: ...

    @overload
    def __rmul__(self: LiteralString, n: SupportsIndex) -> LiteralString: ...
    @overload
    def __rmul__(self, n: SupportsIndex) -> str: ...

    @overload
    def __str__(self: LiteralString) -> LiteralString: ...
    @overload
    def __str__(self) -> str: ...


Appendix D: Guidelines for using ``LiteralString`` in Stubs
===========================================================

Libraries that do not contain type annotations within their source may
specify type stubs in Typeshed. Libraries written in other languages,
such as those for machine learning, may also provide Python type
stubs. This means the type checker cannot verify that the type
annotations match the source code and must trust the type stub. Thus,
authors of type stubs need to be careful when using ``LiteralString``,
since a function may falsely appear to be safe when it is not.

We recommend the following guidelines for using ``LiteralString`` in stubs:

+ If the stub is for a pure function, we recommend using ``LiteralString``
  in the return type of the function or of its overloads only if all
  the corresponding parameters have literal types (i.e.,
  ``LiteralString`` or ``Literal["a", "b"]``).

  ::

      # OK
      @overload
      def my_transform(x: LiteralString, y: Literal["a", "b"]) -> LiteralString: ...
      @overload
      def my_transform(x: str, y: str) -> str: ...

      # Not OK
      @overload
      def my_transform(x: LiteralString, y: str) -> LiteralString: ...
      @overload
      def my_transform(x: str, y: str) -> str: ...

+ If the stub is for a ``staticmethod``, we recommend the same
  guideline as above.

+ If the stub is for any other kind of method, we recommend against
  using ``LiteralString`` in the return type of the method or any of
  its overloads. This is because, even if all the explicit parameters
  have type ``LiteralString``, the object itself may be created using
  user data and thus the return type may be user-controlled.

+ If the stub is for a class attribute or global variable, we also
  recommend against using ``LiteralString`` because the untyped code
  may write arbitrary values to the attribute.

However, we leave the final call to the library author. They may use
``LiteralString`` if they feel confident that the string returned by
the method or function or the string stored in the attribute is
guaranteed to have a literal type - i.e., the string is created by
applying only literal-preserving ``str`` operations to a string
literal.

Note that these guidelines do not apply to inline type annotations
since the type checker can verify that, say, a method returning
``LiteralString`` does in fact return an expression of that type.


Resources
=========

Literal String Types in Scala
-----------------------------

Scala `uses
<https://www.scala-lang.org/api/2.13.x/scala/Singleton.html>`_
``Singleton`` as the supertype for singleton types, which includes
literal string types, such as ``"foo"``. ``Singleton`` is Scala's
generalized analogue of this PEP's ``LiteralString``.

Tamer Abdulradi showed how Scala's literal string types can be used
for "Preventing SQL injection at compile time", Scala Days talk
`Literal types: What are they good for?
<https://slideslive.com/38907881/literal-types-what-they-are-good-for>`_
(slides 52 to 68).

Thanks
------

Thanks to the following people for their feedback on the PEP:

Edward Qiu, Jia Chen, Shannon Zhu, Gregory P. Smith, Никита Соболев,
CAM Gerlach, Arie Bovenberg, David Foster, and Shengye Wan

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.