Add PEP 540: Add a new UTF-8 mode
This commit is contained in:
Victor Stinner
parent
043254687a
commit
9780f3ab43
|
|
@ -0,0 +1,286 @@
|
|||
PEP: 540
|
||||
Title: Add a new UTF-8 mode
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Victor Stinner <victor.stinner@gmail.com>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 5-January-2016
|
||||
Python-Version: 3.7
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
Add a new UTF-8 mode, opt-in option to use UTF-8 for operating system
|
||||
data instead of the locale encoding. Add ``-X utf8`` command line option
|
||||
and ``PYTHONUTF8`` environment variable.
|
||||
|
||||
|
||||
Context
|
||||
=======
|
||||
|
||||
Locale and operating system data
|
||||
--------------------------------
|
||||
|
||||
Python uses the ``LC_CTYPE`` locale to decide how to encode and decode
|
||||
data from/to the operating system:
|
||||
|
||||
* file content
|
||||
* command line arguments: ``sys.argv``
|
||||
* standard streams: ``sys.stdin``, ``sys.stdout``, ``sys.stderr``
|
||||
* environment variables: ``os.environ``
|
||||
* filenames: ``os.listdir(str)`` for example
|
||||
* pipes: ``subprocess.Popen`` using ``subprocess.PIPE`` for example
|
||||
* error messages
|
||||
* name of a timezone
|
||||
* user name, terminal name: ``os``, ``grp`` and ``pwd`` modules
|
||||
* host name, UNIX socket path: see the ``socket`` module
|
||||
* etc.
|
||||
|
||||
At startup, Python calls ``setlocale(LC_CTYPE, "")`` to use the user
|
||||
``LC_CTYPE`` locale and then store the locale encoding,
|
||||
``sys.getfilesystemencoding()``. In the whole lifetime of a Python process,
|
||||
the same encoding and error handler are used to encode and decode data
|
||||
from/to the operating system.
|
||||
|
||||
.. note::
|
||||
In some corner case, the *current* ``LC_CTYPE`` locale must be used
|
||||
instead of ``sys.getfilesystemencoding()``. For example, the ``time``
|
||||
module uses the *current* ``LC_CTYPE`` locale to decode timezone
|
||||
names.
|
||||
|
||||
|
||||
The POSIX locale and its encoding
|
||||
---------------------------------
|
||||
|
||||
The following environment variables are used to configure the locale, in
|
||||
this preference order:
|
||||
|
||||
* ``LC_ALL``, most important variable
|
||||
* ``LC_CTYPE``
|
||||
* ``LANG``
|
||||
|
||||
The POSIX locale,also known as "the C locale", is used:
|
||||
|
||||
* if the first set variable is set to ``"C"``
|
||||
* if all these variables are unset, for example when a program is
|
||||
started in an empty environment.
|
||||
|
||||
The encoding of the POSIX locale must be ASCII or a superset of ASCII.
|
||||
|
||||
On Linux, the POSIX locale uses the ASCII encoding.
|
||||
|
||||
On FreeBSD and Solaris, ``nl_langinfo(CODESET)`` announces an alias of
|
||||
the ASCII encoding, whereas ``mbstowcs()`` and ``wcstombs()`` functions
|
||||
use the ISO 8859-1 encoding (Latin1) in practice. The problem is that
|
||||
``os.fsencode()`` and ``os.fsdecode()`` use
|
||||
``locale.getpreferredencoding()`` codec. For example, if command line
|
||||
arguments are decoded by ``mbstowcs()`` and encoded back by
|
||||
``os.fsencode()``, an ``UnicodeEncodeError`` exception is raised instead
|
||||
of retrieving the original byte string.
|
||||
|
||||
To fix this issue, Python now checks since Python 3.4 if ``mbstowcs()``
|
||||
really uses the ASCII encoding if the the ``LC_CTYPE`` uses the the
|
||||
POSIX locale and ``nl_langinfo(CODESET)`` returns ``"ASCII"`` (or an
|
||||
alias to ASCII). If not (the effective encoding is not ASCII), Python
|
||||
uses its own ASCII codec instead of using ``mbstowcs()`` and
|
||||
``wcstombs()`` functions for operating system data.
|
||||
|
||||
See the `POSIX locale (2016 Edition)
|
||||
<http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html>`_.
|
||||
|
||||
|
||||
C.UTF-8 and C.utf8 locales
|
||||
--------------------------
|
||||
|
||||
Some operating systems provide a variant of the POSIX locale using the
|
||||
UTF-8 encoding:
|
||||
|
||||
* Fedora 25: ``"C.utf8"`` or ``"C.UTF-8"``
|
||||
* Debian (eglibc 2.13-1, 2011): ``"C.UTF-8"``
|
||||
* HP-UX: ``"C.utf8"``
|
||||
|
||||
It was proposed to add a ``C.UTF-8`` locale to glibc: `glibc C.UTF-8
|
||||
proposal <https://sourceware.org/glibc/wiki/Proposals/C.UTF-8>`_.
|
||||
|
||||
|
||||
Popularity of the UTF-8 encoding
|
||||
--------------------------------
|
||||
|
||||
Python 3 uses UTF-8 by default for Python source files.
|
||||
|
||||
On Mac OS X, Windows and Android, Python always use UTF-8 for operating
|
||||
system data instead of the locale encoding. For Windows, see the `PEP
|
||||
529: Change Windows filesystem encoding to UTF-8
|
||||
<https://www.python.org/dev/peps/pep-0529/>`_.
|
||||
|
||||
On Linux, UTF-8 became the defacto standard encoding by default,
|
||||
replacing legacy encodings like ISO 8859-1 or ShiftJIS. For example,
|
||||
using different encodings for filenames and standard streams is likely
|
||||
to create mojibake, so UTF-8 is now used *everywhere*.
|
||||
|
||||
The UTF-8 encoding is the default encoding of XML and JSON file format.
|
||||
In January 2017, UTF-8 was used in `more than 88% of web pages
|
||||
<https://w3techs.com/technologies/details/en-utf8/all/all>`_ (HTML,
|
||||
Javascript, CSS, etc.).
|
||||
|
||||
See `utf8everywhere.org <http://utf8everywhere.org/>`_ for more general
|
||||
information on the UTF-8 codec.
|
||||
|
||||
.. note::
|
||||
Some applications and operating systems (especially Windows) use Byte
|
||||
Order Markers (BOM) to indicate the used Unicode encoding: UTF-7,
|
||||
UTF-8, UTF-16-LE, etc. BOM are not well supported and rarely used in
|
||||
Python.
|
||||
|
||||
|
||||
Old data stored in different encodings and surrogateescape
|
||||
----------------------------------------------------------
|
||||
|
||||
Even if UTF-8 became the defacto standard, there are still systems in
|
||||
the wild which don't use UTF-8. And there are a lot of data stored in
|
||||
different encodings. For example, an old USB key using the ext3
|
||||
filesystem with filenames encoded to ISO 8859-1.
|
||||
|
||||
The Linux kernel and the libc don't decode filenames: a filename is used
|
||||
as a raw array of bytes. The common solution to support any filename is
|
||||
to store filenames as bytes and don't try to decode them. When displayed to
|
||||
stdout, mojibake is displayed if the filename and the terminal don't use
|
||||
the same encoding.
|
||||
|
||||
Python 3 promotes Unicode everywhere including filenames. A solution to
|
||||
support filenames not decodable from the locale encoding was found: the
|
||||
``surrogateescape`` error handler (`PEP 393
|
||||
<https://www.python.org/dev/peps/pep-0393/>`_), store undecodable bytes
|
||||
as surrogate characters. This error handler is used by default for
|
||||
operating system data, by ``os.fsdecode()`` and ``os.fsencode()`` for
|
||||
example (except on Windows which uses the ``strict`` error handler).
|
||||
|
||||
|
||||
Standard streams
|
||||
----------------
|
||||
|
||||
Python uses the locale encoding for standard streams: stdin, stdout and
|
||||
stderr. The ``strict`` error handler is used by stdin and stdout to
|
||||
prevent mojibake.
|
||||
|
||||
The ``backslashreplace`` error handler is used by stderr to avoid
|
||||
Unicode encode error when displaying non-ASCII text. It is especially
|
||||
useful when the POSIX locale is used, because this locale usually uses
|
||||
the ASCII encoding.
|
||||
|
||||
The problem is that operating system data like filenames are decoded
|
||||
using the ``surrogateescape`` error handler (PEP 393). Displaying a
|
||||
filename to stdout raises an Unicode encode error if the filename
|
||||
contains an undecoded byte stored as a surrogate character.
|
||||
|
||||
Python 3.6 now uses ``surrogateescape`` for stdin and stdout if the
|
||||
POSIX locale is used: `issue #19977 <http://bugs.python.org/issue19977>`_. The
|
||||
idea is to passthrough operating system data even if it means mojibake, because
|
||||
most UNIX applications work like that. Most UNIX applications store filenames
|
||||
as bytes, usually simply because bytes are first-citizen class in the used
|
||||
programming language, whereas Unicode is badly supported.
|
||||
|
||||
.. note::
|
||||
The encoding and/or the error handler of standard streams can be
|
||||
overriden with the ``PYTHONIOENCODING`` environment variable.
|
||||
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
Add a new UTF-8 mode, opt-in option to use UTF-8 for operating system data
|
||||
instead of the locale encoding:
|
||||
|
||||
* Add ``-X utf8`` command line option
|
||||
* Add ``PYTHONUTF8=1`` environment variable
|
||||
|
||||
Add also a strict UTF-8 mode, enabled by ``-X utf8=strict`` or
|
||||
``PYTHONUTF8=strict``.
|
||||
|
||||
The UTF-8 mode changes the default encoding and error handler used by
|
||||
open(), os.fsdecode(), os.fsencode(), sys.stdin, sys.stdout and
|
||||
sys.stderr:
|
||||
|
||||
============================ ======================= ======================= ====================== ======================
|
||||
Function Default, other locales Default, POSIX locale UTF-8 UTF-8 Strict
|
||||
============================ ======================= ======================= ====================== ======================
|
||||
open() locale/strict locale/strict UTF-8/surrogateescape UTF-8/strict
|
||||
os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape UTF-8/surrogateescape UTF-8/strict
|
||||
sys.stdin locale/strict locale/surrogateescape UTF-8/surrogateescape UTF-8/strict
|
||||
sys.stdout locale/strict locale/surrogateescape UTF-8/surrogateescape UTF-8/strict
|
||||
sys.stderr locale/backslashreplace locale/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace
|
||||
============================ ======================= ======================= ====================== ======================
|
||||
|
||||
The UTF-8 mode is disabled by default to keep hard Unicode errors when
|
||||
encoding or decoding operating system data failed, and to keep the
|
||||
backward compatibility. The user is responsible to enable explicitly the
|
||||
UTF-8 mode, and so is better prepared for mojibake than if the UTF-8
|
||||
mode would be enabled *by default*.
|
||||
|
||||
The UTF-8 mode should be used on systems known to be configured with
|
||||
UTF-8 where most applications speak UTF-8. It prevents Unicode errors if
|
||||
the user overrides a locale *by mistake* or if a Python program is
|
||||
started with no locale configured (and so with the POSIX locale).
|
||||
|
||||
Most UNIX applications handle operating system data as bytes, so
|
||||
``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables have a
|
||||
limited impact on how these data are handled by the application.
|
||||
|
||||
The Python UTF-8 mode should help to make Python more interoperable with
|
||||
the other UNIX applications in the system assuming that *UTF-8* is used
|
||||
everywhere and that users *expect* UTF-8.
|
||||
|
||||
Ignoring ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables in
|
||||
Python is more convenient, since they are more commonly misconfigured
|
||||
*by mistake* (configured to use an encoding different than UTF-8,
|
||||
whereas the system uses UTF-8), rather than being misconfigured by intent.
|
||||
|
||||
|
||||
Backward Compatibility
|
||||
======================
|
||||
|
||||
Since the UTF-8 mode is disabled by default, it has no impact on the
|
||||
backward compatibility. The new UTF-8 mode must be enabled explicitly.
|
||||
|
||||
|
||||
Alternatives
|
||||
============
|
||||
|
||||
Always use UTF-8
|
||||
----------------
|
||||
|
||||
Python already always use the UTF-8 encoding on Mac OS X, Android and Windows.
|
||||
Since UTF-8 became the defacto encoding, it makes sense to always use it on all
|
||||
platforms with any locale.
|
||||
|
||||
The risk is to introduce mojibake if the locale uses a different encoding,
|
||||
especially for locales other than the POSIX locale.
|
||||
|
||||
|
||||
Force UTF-8 for the POSIX locale
|
||||
--------------------------------
|
||||
|
||||
An alternative to always using UTF-8 in any case is to only use UTF-8 when the
|
||||
``LC_CTYPE`` locale is the POSIX locale.
|
||||
|
||||
The `PEP 538: Coercing the legacy C locale to C.UTF-8
|
||||
<https://www.python.org/dev/peps/pep-0538/>`_ of Nick Coghlan proposes to
|
||||
implement that using the ``C.UTF-8`` locale.
|
||||
|
||||
|
||||
Related Work
|
||||
============
|
||||
|
||||
Perl has a ``-C`` command line option and a ``PERLUNICODE`` environment
|
||||
varaible to force UTF-8: see `perlrun
|
||||
<http://perldoc.perl.org/perlrun.html>`_. It is possible to configure
|
||||
UTF-8 per standard stream, on input and output streams, etc.
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
Loading…
Reference in New Issue