PEP: 597 Title: Add PYTHONTEXTENCODING environment variable Author: Inada Naoki Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 05-Jun-2019 Python-Version: 3.9 Abstract ======== Currently, ``TextIOWrapper`` uses ``locale.getpreferredencoding(False)`` (hereinafter called "locale encoding") when ``encoding`` is not specified. This PEP proposes adding ``PYTHONTEXTENCODING`` environment variable to override the default text encoding since Python 3.9. The goal of this PEP is providing "UTF-8 by default" experience to Windows users, because macOS, Linux, Android, iOS users use UTF-8 by default already. Motivation ========== UTF-8 is the best encoding for saving unicode text -------------------------------------------------- String in Python 3 is unicode. Encoding valid unicode strings with UTF-8 should not fail. On the other hand, most locale encoding used in Windows can not save all valid unicode string. It will cause UnicodeEncodeError or it may not round-trip. User may lost their data in such case. UTF-8 is the best encoding for saving text when user don't specify any encoding. Using UTF-8 by default is easier on new programmers --------------------------------------------------- Python is one of the most popular first programming languages. New programmers may not know about encoding. When they download text data written in UTF-8 from the Internet, they are forced to learn about encoding. Popular text editors like VS Code or Atom use UTF-8 by default. Even Microsoft Notepad uses UTF-8 by default since the Windows 10 May 2019 Update. (Note that Python 3.9 will be released in 2021.) Additionally, the default encoding of Python source files is UTF-8. We can assume new Python programmers who don't know about encoding use editors which use UTF-8 by default. It would be nice if new programmers are not forced to learn about encoding until they need to handle text files encoded in encoding other than UTF-8. People assume it is always UTF-8 -------------------------------- Package authors using macOS or Linux may forget that the default encoding is not always UTF-8. For example, ``long_description = open("README.md").read()`` in ``setup.py`` is a common mistake. If there is at least one emoji or any other non-ASCII character in the ``README.md`` file, many Windows users cannot install the package due to a ``UnicodeDecodeError``. Consistent with default encoding -------------------------------- Python has ``sys.defaultencoding()`` which is always "UTF-8". ``str.encode()`` uses "UTF-8" when encoding is omitted. Using "UTF-8" for text files are consistent with it. It makes Python more easy to learn language. Specification ============= ``PYTHONTEXTENCODING`` environment variable ------------------------------------------- ``PYTHONTEXTENCODING`` environment variable can be used to specify the default text encoding. Unlike ``PYTHONIOENCODING``, it doesn't accept error handler. ``PYTHONIOENCODING`` support it because changing error handler of stdio was difficult. But it is not true for regular files. ``sys.gettextencoding()`` ------------------------- When ``PYTHONTEXTENCODING`` is specified, this function return it. When it is not specified, this function returns ``locale.getpreferredencoding(False)``. ``encoding="locale"`` option ---------------------------- ``TextIOWrapper`` now accepts ``encoding="locale"`` option. "locale" is not real encoding or alias. This is just a shortcut of ``encoding=locale.getpreferredencoding(False)``. Changes in stdlibs ------------------ ``TextIOWrapper`` uses ``sys.gettextencoding()`` where ``locale.getpreferredencoding(False)`` is used. But ``stdin``, ``stdout``, and ``stderr`` continue to respect locale encoding as well. ``PYTHONIOENCODING`` can be used to override their encoding. Pipes and TTY should use the "locale" encoding. UTF-8 mode [1]_ can be used to override these encoding: * ``subprocess`` and ``os.popen`` use the "locale" encoding because the subprocess will use the locale encoding. * ``getpass.getpass`` uses the "locale" encoding when using TTY. All other code using the default encoding are not modified. They can be overridden by ``PYTHONTEXTENCODING``. This is an incomplete list: * ``lzma.open``, ``gzip.open``, ``bz2.open``, ``ZipFile.read_text`` * ``socket.makefile`` * ``tempfile.TemporaryFile``, ``tempfile.NamedTemporaryFile`` * ``trace.CoverageResults.write_results_file`` Rationale ========= Why not just enable UTF-8 mode by default? ------------------------------------------ This PEP is not mutually exclusive to UTF-8 mode. If we enable UTF-8 mode by default, even people using Windows will forget the default encoding is not always UTF-8. More scripts will be written assuming the default encoding is UTF-8. So changing the default encoding of text files to UTF-8 would be better even if UTF-8 mode is enabled by default at some point. Why is "locale" not an alias codec? ----------------------------------- For backward compatibility, ``io.TextIOWrapper`` calls ``locale.getpreferredencoding(False)`` every time when ``encoding="locale"`` is specified. It respects changing locale after Python startup. Why not change std(in|out|err) encoding too? -------------------------------------------- Even when the locale encoding is not UTF-8, there can be many UTF-8 text files. These files could be downloaded from the Internet or written by modern text editors. On the other hand, terminal encoding is assumed to be the same as locale encoding. And other tools are assumed to read and write the locale encoding as well. std(in|out|err) are likely to be connected to a terminal or other tools. So the locale encoding should be respected. Anyway, ``PYTHONIOENCODING`` can be used to change these encodings. Reference Implementation ======================== To be written. Rejected Ideas ============== Change the default text encoding -------------------------------- Previous version of this PEP tried to change the default encoding to UTF-8. But we should have deprecation period long enough. Between the deprecation period, users can not change the default text encoding. And there are many difficulty there: * Omitting ``encoding`` option is very common. * If we raise ``DeprecationWarning`` always, it will be too noisy. * We can not assume how user use it. Complicated heuritics may be needed to raise ``DeprecationWarning`` only when it is really needed. * Users of legacy systems may dismiss warning. * They may not check the warning. * They may upgrade Python from 2.7 after 2020. Additionally, Microsoft is improving UTF-8 support of Windows 10 recently. There are no public plan for future UTF-8 support yet. But Python may be able to change the default encoding without painful deprecation period in the future. Open Issues =========== Easy way to set ``PYTHONTEXTENCODING`` -------------------------------------- UTF-8 is the best encoding for new users. But setting environment variables is not easy enough to new users. It would be helpfule if Python on Windows can provide easy way to set ``PYTHONTEXTENCODING=UTF-8`` even after Python is installed. Commandline option ------------------ If there is reasonable use case for changing default text encoding per process, command line option should be considered. C-API ----- The default text encoding should be able to configured from C. This will be considered when writing reference Implementation. Additionally, C-API like ``PySys_GetTextEncoding()`` should be considered too. References ========== .. [1] PEP 540, Add a new UTF-8 Mode (https://www.python.org/dev/peps/pep-0540/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: