python-peps/pep-0597.rst

PEP: 597
Title: Add PYTHONTEXTENCODING environment variable
Author: Inada Naoki  <songofacandy@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 05-Jun-2019
Python-Version: 3.9


Abstract
========

Currently, ``TextIOWrapper`` uses ``locale.getpreferredencoding(False)``
(hereinafter called "locale encoding") when ``encoding`` is not
specified.

This PEP proposes adding ``PYTHONTEXTENCODING`` environment
variable to override the default text encoding since Python 3.9.

The goal of this PEP is providing "UTF-8 by default" experience to
Windows users, because macOS, Linux, Android, iOS users use UTF-8
by default already.


Motivation
==========

UTF-8 is the best encoding for saving unicode text
--------------------------------------------------

String in Python 3 is unicode.  Encoding valid unicode strings with
UTF-8 should not fail.

On the other hand, most locale encoding used in Windows can not
save all valid unicode string.  It will cause UnicodeEncodeError
or it may not round-trip.  User may lost their data in such case.

UTF-8 is the best encoding for saving text when user don't specify
any encoding.


Using UTF-8 by default is easier on new programmers
---------------------------------------------------

Python is one of the most popular first programming languages.

New programmers may not know about encoding.  When they download text
data written in UTF-8 from the Internet, they are forced to learn
about encoding.

Popular text editors like VS Code or Atom use UTF-8 by default.
Even Microsoft Notepad uses UTF-8 by default since the Windows 10 May
2019 Update.  (Note that Python 3.9 will be released in 2021.)

Additionally, the default encoding of Python source files is UTF-8.
We can assume new Python programmers who don't know about encoding
use editors which use UTF-8 by default.

It would be nice if new programmers are not forced to learn about
encoding until they need to handle text files encoded in encoding
other than UTF-8.


People assume it is always UTF-8
--------------------------------

Package authors using macOS or Linux may forget that the default
encoding is not always UTF-8.

For example, ``long_description = open("README.md").read()`` in
``setup.py`` is a common mistake.  If there is at least one emoji or
any other non-ASCII character in the ``README.md`` file, many Windows
users cannot install the package due to a ``UnicodeDecodeError``.


Consistent with default encoding
--------------------------------

Python has ``sys.defaultencoding()`` which is always "UTF-8".
``str.encode()`` uses "UTF-8" when encoding is omitted.

Using "UTF-8" for text files are consistent with it.  It makes Python
more easy to learn language.


Specification
=============

``PYTHONTEXTENCODING`` environment variable
-------------------------------------------

``PYTHONTEXTENCODING`` environment variable can be used to specify the
default text encoding.

Unlike ``PYTHONIOENCODING``, it doesn't accept error handler.
``PYTHONIOENCODING`` support it because changing error handler of
stdio was difficult.  But it is not true for regular files.


``sys.gettextencoding()``
-------------------------

When ``PYTHONTEXTENCODING`` is specified, this function return it.

When it is not specified, this function returns
``locale.getpreferredencoding(False)``.


``encoding="locale"`` option
----------------------------

``TextIOWrapper`` now accepts ``encoding="locale"`` option.

"locale" is not real encoding or alias.
This is just a shortcut of
``encoding=locale.getpreferredencoding(False)``.


Changes in stdlibs
------------------

``TextIOWrapper`` uses ``sys.gettextencoding()`` where
``locale.getpreferredencoding(False)`` is used.

But ``stdin``, ``stdout``, and ``stderr`` continue to respect
locale encoding as well.  ``PYTHONIOENCODING`` can be used to
override their encoding.

Pipes and TTY should use the "locale" encoding.  UTF-8 mode [1]_
can be used to override these encoding:

* ``subprocess`` and ``os.popen`` use the "locale" encoding because
  the subprocess will use the locale encoding.
* ``getpass.getpass`` uses the "locale" encoding when using TTY.

All other code using the default encoding are not modified.
They can be overridden by ``PYTHONTEXTENCODING``.
This is an incomplete list:

* ``lzma.open``, ``gzip.open``, ``bz2.open``, ``ZipFile.read_text``
* ``socket.makefile``
* ``tempfile.TemporaryFile``, ``tempfile.NamedTemporaryFile``
* ``trace.CoverageResults.write_results_file``


Rationale
=========

Why not just enable UTF-8 mode by default?
------------------------------------------

This PEP is not mutually exclusive to UTF-8 mode.

If we enable UTF-8 mode by default, even people using Windows will
forget the default encoding is not always UTF-8.  More scripts will
be written assuming the default encoding is UTF-8.

So changing the default encoding of text files to UTF-8 would be
better even if UTF-8 mode is enabled by default at some point.


Why is "locale" not an alias codec?
-----------------------------------

For backward compatibility, ``io.TextIOWrapper`` calls
``locale.getpreferredencoding(False)`` every time when
``encoding="locale"`` is specified.

It respects changing locale after Python startup.


Why not change std(in|out|err) encoding too?
--------------------------------------------

Even when the locale encoding is not UTF-8, there can be many UTF-8
text files.  These files could be downloaded from the Internet or
written by modern text editors.

On the other hand, terminal encoding is assumed to be the same as
locale encoding.  And other tools are assumed to read and write the
locale encoding as well.

std(in|out|err) are likely to be connected to a terminal or other
tools. So the locale encoding should be respected.

Anyway, ``PYTHONIOENCODING`` can be used to change these encodings.


Reference Implementation
========================

To be written.


Rejected Ideas
==============

Change the default text encoding
--------------------------------

Previous version of this PEP tried to change the default encoding
to UTF-8.

But we should have deprecation period long enough.  Between the
deprecation period, users can not change the default text encoding.

And there are many difficulty there:

* Omitting ``encoding`` option is very common.

  * If we raise ``DeprecationWarning`` always, it will be too noisy.
  * We can not assume how user use it.  Complicated heuritics may be
    needed to raise ``DeprecationWarning`` only when it is really
    needed.

* Users of legacy systems may dismiss warning.

  * They may not check the warning.
  * They may upgrade Python from 2.7 after 2020.


Additionally, Microsoft is improving UTF-8 support of Windows 10
recently.

There are no public plan for future UTF-8 support yet.  But Python may
be able to change the default encoding without painful deprecation
period in the future.


Open Issues
===========

Easy way to set ``PYTHONTEXTENCODING``
--------------------------------------

UTF-8 is the best encoding for new users.  But setting environment
variables is not easy enough to new users.

It would be helpfule if Python on Windows can provide easy way to set
``PYTHONTEXTENCODING=UTF-8`` even after Python is installed.


Commandline option
------------------

If there is reasonable use case for changing default text encoding
per process, command line option should be considered.


C-API
-----

The default text encoding should be able to configured from C.
This will be considered when writing reference Implementation.

Additionally, C-API like ``PySys_GetTextEncoding()`` should be
considered too.


References
==========

.. [1] PEP 540, Add a new UTF-8 Mode
   (https://www.python.org/dev/peps/pep-0540/)


Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: