From b2ce9f86f5612c1448fd4ea1040270dd1964d4ee Mon Sep 17 00:00:00 2001 From: Inada Naoki Date: Sat, 30 Jan 2021 18:18:19 +0900 Subject: [PATCH] PEP 597: Add EncodingWarning (#1788) --- pep-0597.rst | 173 +++++++++++++++++++++++---------------------------- 1 file changed, 79 insertions(+), 94 deletions(-) diff --git a/pep-0597.rst b/pep-0597.rst index 65068c521..a931115ed 100644 --- a/pep-0597.rst +++ b/pep-0597.rst @@ -1,6 +1,6 @@ PEP: 597 -Title: Soft deprecation of default encoding -Last-Modified: 23-Jun-2020 +Title: Add optional EncodingWarning +Last-Modified: 30-Jan-2021 Author: Inada Naoki Discussions-To: https://discuss.python.org/t/3880 Status: Draft @@ -13,16 +13,13 @@ Python-Version: 3.10 Abstract ======== -This PEP proposes: +Add a new warning category ``EncodingWarning``. It is emitted when +``encoding`` option is ommitted and the default encoding is locale +encoding. -* ``TextIOWrapper`` raises a ``PendingDeprecationWarning`` when the - ``encoding`` option is not specified and dev mode is enabled. - -* Add ``encoding="locale"`` option to ``TextIOWrapper``. It behaves - like ``encoding=None`` but don't raise a warning. - -* Add ``io.LOCALE_ENCODING = "locale"`` constant to avoid confusing - ``LookupError``. +The warning is disabled by default. New ``-X warn_encoding`` +commandline option and ``PYTHONWARNENCODING`` environment variable +are used to enable the warnings. Motivation @@ -67,108 +64,123 @@ But this change will affect many applications and libraries. Many ``DeprecationWarning`` will be raised if we start raising the warning by default. It will be too noisy. -While this PEP doesn't cover the change, this PEP will help to reduce -the number of ``DeprecationWarning`` in the future. +While this PEP doesn't cover the change, this PEP will help to +reduce the number of ``DeprecationWarning`` in the future. Specification ============= -Raising a PendingDeprecationWarning ---------------------------------------- +``EncodingWarning`` +-------------------- -``TextIOWrapper`` raises the ``PendingDeprecationWarning`` when the -``encoding`` option is omitted and dev mode is enabled. +Add new ``EncodingWarning`` warning class which is a subclass of +``Warning``. It is used to warn when ``encoding`` option is omitted +and the default encoding is locale-specific. + + +Options to enable the warning +------------------------------ + +``-X warn_encoding`` option and the ``PYTHONWARNENCODING`` +environment variable are added. They are used to enable the +``EncodingWarning``. + +``sys.flags.encoding_warning`` is also added. It is a boolean flag +represents ``EncodingWarning`` is enabled. + +When the option is enabled, ``io.TextIOWrapper()``, ``open()``, and +other modules emit ``EncodingWarning`` when ``encoding`` is omitted. ``encoding="locale"`` option ---------------------------- -When ``encoding="locale"`` is specified to the ``TextIOWrapper``, it -behaves same to ``encoding=None`` except it doesn't raise warning. -In detail, the encoding is chosen by this order: +``io.TextIOWrapper`` accepts ``encoding="locale"`` option. It means +same to current ``encoding=None``. But ``io.TextIOWrapper`` doesn't +emit ``EncodingWarning`` when ``encoding="locale"`` is specified. -1. ``os.device_encoding(buffer.fileno())`` -2. ``locale.getpreferredencoding(False)`` +Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can +be used to avoid confusing ``LookupError: unknown encoding: locale`` +error when the code is run in old Python accidentally. -This option can be used to use the locale encoding explicitly and -suppress the ``PendingDeprecationWarning``. +The constant can be used to test that ``encoding="locale"`` option is +supported too. For example, +.. code-block:: -``io.LOCALE_ENCODING`` ----------------------- - -``io`` module has ``io.LOCALE_ENCODING = "locale"`` constant. This -constant can be used to avoid confusing ``LookupError: unknown -encoding: locale`` error when the code is run in Python older than -3.10 accidentally. - -The constant can be used to test that ``encoding="locale"`` option -is supported too. - -:: - - # Want to suppress the Warning in dev mode but still need support + # Want to suppress an EncodingWarning but still need support # old Python versions. locale_encoding = getattr(io, "LOCALE_ENCODING", None) with open(filename, encoding=locale_encoding) as f: ... -``io.text_encoding`` --------------------- +``io.text_encoding()`` +----------------------- -``TextIOWrapper`` is used indirectly in most cases. For example, -``open``, and ``pathlib.Path.read_text()`` use it. Warning to these -functions doesn't make sense. Callers of these functions should be -warned instead. +``io.text_encoding()`` is a helper function for functions having +``encoding=None`` option and pass it to ``io.TextIOWrapper()`` or +``open()``. -``io.text_encoding(encoding, stacklevel=1)`` is a helper function for -it. Pure Python implementation will be like this:: +Pure Python implementation will be like this:: def text_encoding(encoding, stacklevel=1): - """ - Helper function to choose the text encoding. + """Helper function to choose the text encoding. - When encoding is not None, just return it. - Otherwise, return the default text encoding ("locale" for now), - and raise a PendingDeprecationWarning in dev mode. + When *encoding* is not None, just return it. + Otherwise, return the default text encoding (i.e., "locale"). - This function can be used in APIs having encoding=None option. - But please consider encoding="utf-8" for new APIs. + This function emits EncodingWarning if *encoding* is None and + sys.flags.encoding_warning is true. + + This function can be used in APIs having encoding=None option + and pass it to TextIOWrapper or open. + But please consider using encoding="utf-8" for new APIs. """ if encoding is None: - if sys.flags.dev_mode: + if sys.flags.encoding_warning: import warnings - warnings.warn( - "'encoding' option is not specified. The default encoding " - "might be changed to 'utf-8' in the future", - PendingDeprecationWarning, stacklevel + 2) + warnings.warn("'encoding' option is omitted", + EncodingWarning, stacklevel + 2) encoding = LOCALE_ENCODING return encoding -``pathlib.Path.read_text()`` can use this function like this:: +For example, ``pathlib.Path.read_text()`` can use the function like: + +.. code-block:: def read_text(self, encoding=None, errors=None): - """ - Open the file in text mode, read it, and close the file. - """ encoding = io.text_encoding(encoding) with self.open(mode='r', encoding=encoding, errors=errors) as f: return f.read() -subprocess module doesn't warn ------------------------------- +``subprocess`` module +---------------------- -While the subprocess module uses TextIOWrapper, it doesn't raise -``PendingDeprecationWarning``. It uses the ``io.LOCALE_ENCODING`` -by default. +The default encoding for pipe in the subprocess module is changed +to ``io.LOCALE_ENCODING``. In other words, subprocess module doesn't +emit the ``EncodingWarning``. + +The default encoding for PIPE is relating to the encoding of the +stdio than the default encoding of ``TextIOWrapper``. So this PEP +doesn't propose to emit the warning for pipes. Rationale ========= +Opt-in warning +--------------- + +Although ``DeprecationWarning`` is supressed by default, emitting +``DeprecationWarning`` alwasy when ``encoding`` option is omitted +would be too noisy. + +Noisy warnings may leads developers to dismiss the ``DeprecationWarning``. + + "locale" is not a codec alias ----------------------------- @@ -180,33 +192,6 @@ when ``encoding=None``. This behavior can not be implemented in the codec. -Use a PendingDeprecationWarning -------------------------------- - -This PEP doesn't cover changing the default encoding to UTF-8. -So we use ``PendingDeprecationWarning`` instead of -``DeprecationWarning`` for now. - - -Raise warning only in dev mode ------------------------------- - -This PEP will produce a huge amount of ``PendingDeprecationWarning``. -It will be too noisy for most Python developers. - -We need to fix all warnings in the standard library. We need to wait -pip and major dev tools like ``pytest`` fix warnings before raising -this warning by default. - - -subprocess module doesn't warn ------------------------------- - -The default encoding for PIPE is relating to the encoding of the -stdio than the default encoding of ``TextIOWrapper``. So this PEP -doesn't propose to raise warning from the subprocess module. - - Reference Implementation ========================