2016-12-27 21:31:21 -05:00
|
|
|
|
PEP: 538
|
2017-03-05 02:35:19 -05:00
|
|
|
|
Title: Coercing the legacy C locale to a UTF-8 based locale
|
2016-12-27 21:31:21 -05:00
|
|
|
|
Version: $Revision$
|
|
|
|
|
Last-Modified: $Date$
|
|
|
|
|
Author: Nick Coghlan <ncoghlan@gmail.com>
|
2017-03-09 21:13:53 -05:00
|
|
|
|
BDFL-Delegate: Barry Warsaw
|
2016-12-27 21:31:21 -05:00
|
|
|
|
Status: Draft
|
|
|
|
|
Type: Standards Track
|
|
|
|
|
Content-Type: text/x-rst
|
|
|
|
|
Created: 28-Dec-2016
|
|
|
|
|
Python-Version: 3.7
|
2017-01-07 07:02:17 -05:00
|
|
|
|
Post-History: 03-Jan-2017 (linux-sig),
|
2017-03-05 02:35:19 -05:00
|
|
|
|
07-Jan-2017 (python-ideas),
|
|
|
|
|
05-Mar-2017 (python-dev)
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
|
========
|
|
|
|
|
|
|
|
|
|
An ongoing challenge with Python 3 on \*nix systems is the conflict between
|
|
|
|
|
needing to use the configured locale encoding by default for consistency with
|
2017-03-13 01:06:48 -04:00
|
|
|
|
other locale-aware components in the same process subprocesses,
|
2017-01-20 09:13:24 -05:00
|
|
|
|
and the fact that the standard C locale (as defined in POSIX:2001) typically
|
|
|
|
|
implies a default text encoding of ASCII, which is entirely inadequate for the
|
2017-01-07 02:04:39 -05:00
|
|
|
|
development of networked services and client applications in a multilingual
|
|
|
|
|
world.
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
PEP 540 proposes a change to CPython's handling of the legacy C locale such
|
|
|
|
|
that CPython will assume the use of UTF-8 in such environments, rather than
|
|
|
|
|
persisting with the demonstrably problematic assumption of ASCII as an
|
|
|
|
|
appropriate encoding for communicating with operating system interfaces.
|
2017-03-05 02:29:54 -05:00
|
|
|
|
This is a good approach for cases where network encoding interoperability
|
|
|
|
|
is a more important concern than local encoding interoperability.
|
2017-01-20 09:13:24 -05:00
|
|
|
|
|
|
|
|
|
However, it comes at the cost of making CPython's encoding assumptions diverge
|
2017-03-13 01:06:48 -04:00
|
|
|
|
from those of other locale-aware components in the same process, as well as
|
|
|
|
|
those of components running in subprocesses that share the same environment.
|
2017-01-20 09:13:24 -05:00
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
It also requires changes to the internals of how CPython itself works, rather
|
|
|
|
|
than using existing configuration settings that are supported by Python
|
|
|
|
|
versions prior to Python 3.7.
|
|
|
|
|
|
|
|
|
|
Accordingly, this PEP proposes that independently of the UTF-8 mode proposed
|
|
|
|
|
in PEP 540, the way the CPython implementation handles the default C locale be
|
|
|
|
|
changed such that:
|
|
|
|
|
|
|
|
|
|
* unless the new ``PYTHONCOERCECLOCALE`` environment variable is set to ``0``,
|
|
|
|
|
the standalone CPython binary will automatically attempt to coerce the ``C``
|
|
|
|
|
locale to the first available locale out of ``C.UTF-8``, ``C.utf8``, or
|
|
|
|
|
``UTF-8``
|
|
|
|
|
* if the locale is successfully coerced, and PEP 540 is not accepted, then
|
|
|
|
|
``PYTHONIOENCODING`` (if not otherwise set) will be set to
|
|
|
|
|
``utf-8:surrogateescape``.
|
|
|
|
|
* if the locale is successfully coerced, and PEP 540 *is* accepted, then
|
|
|
|
|
``PYTHONUTF8`` (if not otherwise set) will be set to ``1``
|
2017-01-07 02:04:39 -05:00
|
|
|
|
* if the subsequent runtime initialization process detects that the legacy
|
2017-01-20 09:13:24 -05:00
|
|
|
|
``C`` locale remains active (e.g. none of ``C.UTF-8``, ``C.utf8`` or ``UTF-8``
|
2017-03-13 01:06:48 -04:00
|
|
|
|
are available, or the runtime is embedded in an application other than the
|
|
|
|
|
main CPython binary), locale coercion is not explicitly disabled, and the
|
|
|
|
|
``PYTHONUTF8`` feature defined in PEP 540 is also disabled (or not
|
|
|
|
|
implemented), it will emit a warning on stderr that use of the legacy
|
|
|
|
|
``C`` locale's default ASCII text encoding may cause various Unicode
|
|
|
|
|
compatibility issues
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
With this change, any \*nix platform that does *not* offer at least one of the
|
2017-01-20 09:13:24 -05:00
|
|
|
|
``C.UTF-8``, ``C.utf8`` or ``UTF-8`` locales as part of its standard
|
2017-01-07 02:04:39 -05:00
|
|
|
|
configuration would only be considered a fully supported platform for CPython
|
2017-03-05 02:29:54 -05:00
|
|
|
|
3.7+ deployments when either the new ``PYTHONUTF8`` mode defined in PEP 540 is
|
|
|
|
|
used, or else a suitable locale other than the default ``C`` locale is
|
|
|
|
|
configured explicitly (e.g. `en_AU.UTF-8`, ``zh_CN.gb18030``).
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
Redistributors (such as Linux distributions) with a narrower target audience
|
2017-01-07 02:34:43 -05:00
|
|
|
|
than the upstream CPython development team may also choose to opt in to this
|
2017-03-05 02:29:54 -05:00
|
|
|
|
locale coercion behaviour for the Python 3.6.x series by applying the necessary
|
|
|
|
|
changes as a downstream patch when first introducing Python 3.6.0.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
|
2017-01-03 00:19:37 -05:00
|
|
|
|
Background
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
While the CPython interpreter is starting up, it may need to convert from
|
|
|
|
|
the ``char *`` format to the ``wchar_t *`` format, or from one of those formats
|
2017-01-20 09:13:24 -05:00
|
|
|
|
to ``PyUnicodeObject *``, in a way that's consistent with the locale settings
|
|
|
|
|
of the overall system. It handles these cases by relying on the operating
|
|
|
|
|
system to do the conversion and then ensuring that the text encoding name
|
|
|
|
|
reported by ``sys.getfilesystemencoding()`` matches the encoding used during
|
|
|
|
|
this early bootstrapping process.
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
On Apple platforms (including both Mac OS X and iOS), this is straightforward,
|
|
|
|
|
as Apple guarantees that these operations will always use UTF-8 to do the
|
|
|
|
|
conversion.
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
|
|
|
|
On Windows, the limitations of the ``mbcs`` format used by default in these
|
|
|
|
|
conversions proved sufficiently problematic that PEP 528 and PEP 529 were
|
|
|
|
|
implemented to bypass the operating system supplied interfaces for binary data
|
|
|
|
|
handling and force the use of UTF-8 instead.
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
On Android, many components, including CPython, already assume the use of UTF-8
|
|
|
|
|
as the system encoding, regardless of the locale setting. However, this isn't
|
|
|
|
|
the case for all components, and the discrepancy can cause problems in some
|
|
|
|
|
situations (for example, when using the GNU readline module [16_]).
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
On non-Apple and non-Android \*nix systems, these operations are handled using
|
|
|
|
|
the C locale system in glibc, which has the following characteristics [4_]:
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
|
|
|
|
* by default, all processes start in the ``C`` locale, which uses ``ASCII``
|
|
|
|
|
for these conversions. This is almost never what anyone doing multilingual
|
2017-01-07 02:04:39 -05:00
|
|
|
|
text processing actually wants (including CPython and C/C++ GUI frameworks).
|
2017-01-03 00:19:37 -05:00
|
|
|
|
* calling ``setlocale(LC_ALL, "")`` reconfigures the active locale based on
|
|
|
|
|
the locale categories configured in the current process environment
|
|
|
|
|
* if the locale requested by the current environment is unknown, or no specific
|
|
|
|
|
locale is configured, then the default ``C`` locale will remain active
|
|
|
|
|
|
|
|
|
|
The specific locale category that covers the APIs that CPython depends on is
|
|
|
|
|
``LC_CTYPE``, which applies to "classification and conversion of characters,
|
|
|
|
|
and to multibyte and wide characters" [5_]. Accordingly, CPython includes the
|
|
|
|
|
following key calls to ``setlocale``:
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
* in the main ``python`` binary, CPython calls ``setlocale(LC_ALL, "")`` to
|
|
|
|
|
configure the entire C locale subsystem according to the process environment.
|
|
|
|
|
It does this prior to making any calls into the shared CPython library
|
2017-01-03 00:19:37 -05:00
|
|
|
|
* in ``Py_Initialize``, CPython calls ``setlocale(LC_CTYPE, "")``, such that
|
|
|
|
|
the configured locale settings for that category *always* match those set in
|
|
|
|
|
the environment. It does this unconditionally, and it *doesn't* revert the
|
|
|
|
|
process state change in ``Py_Finalize``
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
(This summary of the locale handling omits several technical details related
|
|
|
|
|
to exactly where and when the text encoding declared as part of the locale
|
|
|
|
|
settings is used - see PEP 540 for further discussion, as these particular
|
|
|
|
|
details matter more when decoupling CPython from the declared C locale than
|
|
|
|
|
they do when overriding the locale with one based on UTF-8)
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
|
|
|
|
These calls are usually sufficient to provide sensible behaviour, but they can
|
|
|
|
|
still fail in the following cases:
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
* SSH environment forwarding means that SSH clients may sometimes forward
|
2017-01-07 02:04:39 -05:00
|
|
|
|
client locale settings to servers that don't have that locale installed. This
|
2017-03-05 02:29:54 -05:00
|
|
|
|
leads to CPython running in the default ASCII-based C locale
|
2017-01-03 00:19:37 -05:00
|
|
|
|
* some process environments (such as Linux containers) may not have any
|
2017-01-07 02:04:39 -05:00
|
|
|
|
explicit locale configured at all. As with unknown locales, this leads to
|
|
|
|
|
CPython running in the default ASCII-based C locale
|
|
|
|
|
|
|
|
|
|
The simplest way to deal with this problem for currently released versions of
|
|
|
|
|
CPython is to explicitly set a more sensible locale when launching the
|
|
|
|
|
application. For example::
|
|
|
|
|
|
|
|
|
|
LC_ALL=C.UTF-8 LANG=C.UTF-8 python3 ...
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
The ``C.UTF-8`` locale is a full locale definition that uses ``UTF-8`` for the
|
|
|
|
|
``LC_CTYPE`` category, and the same settings as the ``C`` locale for all other
|
|
|
|
|
categories (including ``LC_COLLATE``). It is offered by a number of Linux
|
|
|
|
|
distributions (including Debian, Ubuntu, Fedora, Alpine and Android) as an
|
|
|
|
|
alternative to the ASCII-based C locale.
|
|
|
|
|
|
|
|
|
|
Mac OS X and other \*BSD systems have taken a different approach, and instead
|
|
|
|
|
of offering a ``C.UTF-8`` locale, instead offer a partial ``UTF-8`` locale that
|
|
|
|
|
only defines the ``LC_CTYPE`` category. On such systems, the preferred
|
|
|
|
|
environmental locale adjustment is to set ``LC_CTYPE=UTF-8`` rather than to set
|
|
|
|
|
``LC_ALL`` or ``LANG``. [17_]
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
In the specific case of Docker containers and similar technologies, the
|
|
|
|
|
appropriate locale setting can be specified directly in the container image
|
|
|
|
|
definition.
|
|
|
|
|
|
|
|
|
|
Another common failure case is developers specifying ``LANG=C`` in order to
|
|
|
|
|
see otherwise translated user interface messages in English, rather than the
|
2017-03-13 01:06:48 -04:00
|
|
|
|
more narrowly scoped ``LC_MESSAGES=C`` or ``LANGUAGE=en``.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Relationship with other PEPs
|
|
|
|
|
============================
|
|
|
|
|
|
|
|
|
|
This PEP shares a common problem statement with PEP 540 (improving Python 3's
|
2017-03-05 02:29:54 -05:00
|
|
|
|
behaviour in the default C locale), but diverges markedly in the proposed
|
2017-01-07 02:04:39 -05:00
|
|
|
|
solution:
|
|
|
|
|
|
|
|
|
|
* PEP 540 proposes to entirely decouple CPython's default text encoding from
|
|
|
|
|
the C locale system in that case, allowing text handling inconsistencies to
|
2017-03-13 01:06:48 -04:00
|
|
|
|
arise between CPython and other locale-ware components running in the same
|
|
|
|
|
process and in subprocesses. This approach aims to make CPython behave less
|
|
|
|
|
like a locale-aware application, and more like locale-independent language
|
2017-01-07 02:04:39 -05:00
|
|
|
|
runtimes like the JVM, .NET CLR, Go, Node.js, and Rust
|
2017-01-20 09:13:24 -05:00
|
|
|
|
* this PEP proposes to override the legacy C locale with a more recently
|
2017-01-07 02:04:39 -05:00
|
|
|
|
defined locale that uses UTF-8 as its default text encoding. This means that
|
|
|
|
|
the text encoding override will apply not only to CPython, but also to any
|
2017-03-13 01:06:48 -04:00
|
|
|
|
locale-aware extension modules loaded into the current process, as well as to
|
|
|
|
|
locale-aware applications invoked in subprocesses that inherit their
|
2017-01-07 02:04:39 -05:00
|
|
|
|
environment from the parent process. This approach aims to retain CPython's
|
2017-03-13 01:06:48 -04:00
|
|
|
|
traditional strong support for integration with other locale-aware components
|
|
|
|
|
while also actively helping to push forward the adoption and standardisation
|
|
|
|
|
of the C.UTF-8 locale as a Unicode-aware replacement for the legacy C locale
|
|
|
|
|
in the wider C/C++ ecosystem
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
After reviewing both PEPs, it became clear that they didn't actually conflict
|
|
|
|
|
at a technical level, and the proposal in PEP 540 offered a superior option in
|
2017-01-20 09:35:51 -05:00
|
|
|
|
cases where no suitable locale was available, as well as offering a better
|
2017-01-20 09:13:24 -05:00
|
|
|
|
reference behaviour for platforms where the notion of a "locale encoding"
|
|
|
|
|
doesn't make sense (for example, embedded systems running MicroPython rather
|
2017-01-20 09:35:51 -05:00
|
|
|
|
than the CPython reference interpreter).
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Meanwhile, this PEP offered improved compatibility with other locale-aware
|
|
|
|
|
components, and an approach more amenable to being backported to Python 3.6
|
|
|
|
|
by downstream redistributors.
|
2017-03-05 02:29:54 -05:00
|
|
|
|
|
|
|
|
|
As a result, this PEP was amended to refer to PEP 540 as a complementary
|
|
|
|
|
solution that offered improved behaviour both when locale coercion triggered,
|
|
|
|
|
as well as when none of the standard UTF-8 based locales were available.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
The availability of PEP 540 also meant that the ``LC_CTYPE=en_US.UTF-8`` legacy
|
|
|
|
|
fallback was removed from the list of UTF-8 locales tried as a coercion target,
|
|
|
|
|
with CPython instead relying solely on the proposed PYTHONUTF8 mode in such
|
|
|
|
|
cases.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Motivation
|
|
|
|
|
==========
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
While Linux container technologies like Docker, Kubernetes, and OpenShift are
|
|
|
|
|
best known for their use in web service development, the related container
|
|
|
|
|
formats and execution models are also being adopted for Linux command line
|
|
|
|
|
application development. Technologies like Gnome Flatpak [7_] and
|
|
|
|
|
Ubunty Snappy [8_] further aim to bring these same techniques to Linux GUI
|
|
|
|
|
application development.
|
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
When using Python 3 for application development in these contexts, it isn't
|
|
|
|
|
uncommon to see text encoding related errors akin to the following::
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
$ docker run --rm fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
Unable to decode the command from the command line:
|
|
|
|
|
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed
|
|
|
|
|
$ docker run --rm ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
Unable to decode the command from the command line:
|
|
|
|
|
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed
|
|
|
|
|
|
|
|
|
|
Even though the same command is likely to work fine when run locally::
|
|
|
|
|
|
|
|
|
|
$ python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
The source of the problem can be seen by instead running the ``locale`` command
|
|
|
|
|
in the three environments::
|
|
|
|
|
|
|
|
|
|
$ locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=en_AU.UTF-8
|
|
|
|
|
LC_CTYPE="en_AU.UTF-8"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
$ docker run --rm fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=
|
|
|
|
|
LC_CTYPE="POSIX"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
$ docker run --rm ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=
|
|
|
|
|
LANGUAGE=
|
|
|
|
|
LC_CTYPE="POSIX"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
|
|
|
|
|
In this particular example, we can see that the host system locale is set to
|
|
|
|
|
"en_AU.UTF-8", so CPython uses UTF-8 as the default text encoding. By contrast,
|
|
|
|
|
the base Docker images for Fedora and Debian don't have any specific locale
|
|
|
|
|
set, so they use the POSIX locale by default, which is an alias for the
|
|
|
|
|
ASCII-based default C locale.
|
|
|
|
|
|
|
|
|
|
The simplest way to get Python 3 (regardless of the exact version) to behave
|
|
|
|
|
sensibly in Fedora and Debian based containers is to run it in the ``C.UTF-8``
|
|
|
|
|
locale that both distros provide::
|
|
|
|
|
|
|
|
|
|
$ docker run --rm -e LANG=C.UTF-8 fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
$ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
$ docker run --rm -e LANG=C.UTF-8 fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=C.UTF-8
|
|
|
|
|
LC_CTYPE="C.UTF-8"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
$ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=C.UTF-8
|
|
|
|
|
LANGUAGE=
|
|
|
|
|
LC_CTYPE="C.UTF-8"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
|
|
|
|
|
The Alpine Linux based Python images provided by Docker, Inc, already use the
|
|
|
|
|
C.UTF-8 locale by default::
|
|
|
|
|
|
|
|
|
|
$ docker run --rm python:3 python3 -c 'print("ℙƴ☂ℌøἤ")'
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
$ docker run --rm python:3 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
|
|
|
|
|
LANG=C.UTF-8
|
|
|
|
|
LANGUAGE=
|
|
|
|
|
LC_CTYPE="C.UTF-8"
|
|
|
|
|
LC_ALL=
|
|
|
|
|
|
|
|
|
|
Similarly, for custom container images (i.e. those adding additional content on
|
|
|
|
|
top of a base distro image), a more suitable locale can be set in the image
|
|
|
|
|
definition so everything just works by default. However, it would provide a much
|
|
|
|
|
nicer and more consistent user experience if CPython were able to just deal
|
|
|
|
|
with this problem automatically rather than relying on redistributors or end
|
|
|
|
|
users to handle it through system configuration changes.
|
|
|
|
|
|
|
|
|
|
While the glibc developers are working towards making the C.UTF-8 locale
|
|
|
|
|
universally available for use by glibc based applications like CPython [6_],
|
|
|
|
|
this unfortunately doesn't help on platforms that ship older versions of glibc
|
|
|
|
|
without that feature, and also don't provide C.UTF-8 as an on-disk locale the
|
2017-01-20 09:13:24 -05:00
|
|
|
|
way Debian and Fedora do. For these platforms, the mechanism proposed in
|
|
|
|
|
PEP 540 at least allows CPython itself to behave sensibly, albeit without any
|
|
|
|
|
mechanism to get other C/C++ components that decode binary streams as text to
|
|
|
|
|
do the same.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
|
2017-01-07 20:54:24 -05:00
|
|
|
|
Design Principles
|
|
|
|
|
=================
|
|
|
|
|
|
|
|
|
|
The above motivation leads to the following core design principles for the
|
|
|
|
|
proposed solution:
|
|
|
|
|
|
|
|
|
|
* if a locale other than the default C locale is explicitly configured, we'll
|
|
|
|
|
continue to respect it
|
|
|
|
|
* if we're changing the locale setting without an explicit config option, we'll
|
|
|
|
|
emit a warning on stderr that we're doing so rather than silently changing
|
|
|
|
|
the process configuration. This will alert application and system integrators
|
|
|
|
|
to the change, even if they don't closely follow the PEP process or Python
|
|
|
|
|
release announcements. However, to minimize the chance of introducing new
|
|
|
|
|
problems for end users, we'll do this *without* using the warnings system, so
|
|
|
|
|
even running with ``-Werror`` won't turn it into a runtime exception
|
2017-03-05 02:29:54 -05:00
|
|
|
|
* any changes made will use *existing* configuration options
|
2017-01-07 20:54:24 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Minimizing the negative impact on systems currently correctly configured to
|
2017-01-20 09:13:24 -05:00
|
|
|
|
use GB-18030 or another partially ASCII compatible universal encoding leads to
|
|
|
|
|
an additional design principle:
|
2017-01-07 20:54:24 -05:00
|
|
|
|
|
|
|
|
|
* if a UTF-8 based Linux container is run on a host that is explicitly
|
|
|
|
|
configured to use a non-UTF-8 encoding, and tries to exchange locally
|
|
|
|
|
encoded data with that host rather than exchanging explicitly UTF-8 encoded
|
2017-01-20 09:13:24 -05:00
|
|
|
|
data, CPython will endeavour to correctly round-trip host provided data that
|
|
|
|
|
is concatenated or split solely at common ASCII compatible code points, but
|
|
|
|
|
may otherwise emit nonsensical results.
|
2017-01-07 20:54:24 -05:00
|
|
|
|
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Specification
|
|
|
|
|
=============
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
|
|
|
|
To better handle the cases where CPython would otherwise end up attempting
|
2017-01-07 02:04:39 -05:00
|
|
|
|
to operate in the ``C`` locale, this PEP proposes that CPython automatically
|
|
|
|
|
attempt to coerce the legacy ``C`` locale to a UTF-8 based locale when it is
|
|
|
|
|
run as a standalone command line application.
|
|
|
|
|
|
|
|
|
|
It further proposes to emit a warning on stderr if the legacy ``C`` locale
|
|
|
|
|
is in effect at the point where the language runtime itself is initialized,
|
2017-03-13 01:06:48 -04:00
|
|
|
|
the explicit environmental flag to disable locale coercion is not set, and
|
|
|
|
|
the PEP 540 UTF-8 encoding override is also disabled, in order to warn
|
2017-01-20 09:13:24 -05:00
|
|
|
|
system and application integrators that they're running CPython in an
|
|
|
|
|
unsupported configuration.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Legacy C locale coercion in the standalone Python interpreter binary
|
|
|
|
|
--------------------------------------------------------------------
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
When run as a standalone application, CPython has the opportunity to
|
|
|
|
|
reconfigure the C locale before any locale dependent operations are executed
|
|
|
|
|
in the process.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
This means that it can change the locale settings not only for the CPython
|
2017-03-13 01:06:48 -04:00
|
|
|
|
runtime, but also for any other locale-aware components running in the current
|
2017-01-07 02:04:39 -05:00
|
|
|
|
process (e.g. as part of extension modules), as well as in subprocesses that
|
|
|
|
|
inherit their environment from the current process.
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
After calling ``setlocale(LC_ALL, "")`` to initialize the locale settings in
|
|
|
|
|
the current process, the main interpreter binary will be updated to include
|
|
|
|
|
the following call::
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
const char *ctype_loc = setlocale(LC_CTYPE, NULL);
|
|
|
|
|
|
|
|
|
|
This cryptic invocation is the API that C provides to query the current locale
|
|
|
|
|
setting without changing it. Given that query, it is possible to check for
|
|
|
|
|
exactly the ``C`` locale with ``strcmp``::
|
|
|
|
|
|
|
|
|
|
ctype_loc != NULL && strcmp(ctype_loc, "C") == 0 # true only in the C locale
|
|
|
|
|
|
2017-01-07 07:14:20 -05:00
|
|
|
|
This call also returns ``"C"`` when either no particular locale is set, or the
|
|
|
|
|
nominal locale is set to an alias for the ``C`` locale (such as ``POSIX``).
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Given this information, CPython can then attempt to coerce the locale to one
|
|
|
|
|
that uses UTF-8 rather than ASCII as the default encoding.
|
|
|
|
|
|
|
|
|
|
Three such locales will be tried:
|
|
|
|
|
|
|
|
|
|
* ``C.UTF-8`` (available at least in Debian, Ubuntu, and Fedora 25+, and
|
|
|
|
|
expected to be available by default in a future version of glibc)
|
|
|
|
|
* ``C.utf8`` (available at least in HP-UX)
|
2017-01-20 09:13:24 -05:00
|
|
|
|
* ``UTF-8`` (available in at least some \*BSD variants)
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
For ``C.UTF-8`` and ``C.utf8``, the coercion will be implemented by actually
|
|
|
|
|
setting the ``LANG`` and ``LC_ALL`` environment variables to the candidate
|
|
|
|
|
locale name, such that future calls to ``setlocale()`` will see them, as will
|
|
|
|
|
other components looking for those settings (such as GUI development
|
|
|
|
|
frameworks).
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
For the platforms where it is defined, ``UTF-8`` is a partial locale that only
|
|
|
|
|
defines the ``LC_CTYPE`` category. Accordingly, only the ``LC_CTYPE``
|
|
|
|
|
environment variable would be set when using this fallback option.
|
|
|
|
|
|
|
|
|
|
To adjust automatically to future changes in locale availability, these checks
|
|
|
|
|
will be implemented at runtime on all platforms other than Mac OS X and Windows,
|
|
|
|
|
rather than attempting to determine which locales to try at compile time.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
If the locale settings are changed successfully, and the ``PYTHONIOENCODING``
|
|
|
|
|
environment variable is currently unset, then it will be forced to
|
|
|
|
|
``PYTHONIOENCODING=utf-8:surrogateescape``.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
When this locale coercion is activated, the following warning will be
|
|
|
|
|
printed on stderr, with the warning containing whichever locale was
|
|
|
|
|
successfully configured::
|
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Python detected LC_CTYPE=C: LC_ALL & LANG coerced to C.UTF-8 (set another
|
2017-01-07 07:20:23 -05:00
|
|
|
|
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
When falling back to the ``UTF-8`` locale, the message would be slightly
|
|
|
|
|
different::
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Python detected LC_CTYPE=C: LC_CTYPE coerced to UTF-8 (set another locale
|
2017-01-07 07:20:23 -05:00
|
|
|
|
or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
In combination with PEP 540, this locale coercion will mean that the standard
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Python binary *and* locale-aware extensions should once again "just work"
|
2017-01-20 09:13:24 -05:00
|
|
|
|
in the three main failure cases we're aware of (missing locale
|
2017-01-07 03:19:44 -05:00
|
|
|
|
settings, SSH forwarding of unknown locales, and developers explicitly
|
|
|
|
|
requesting ``LANG=C``), as long as the target platform provides at least one
|
|
|
|
|
of the candidate UTF-8 based environments.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
If none of the candidate locales are successfully configured, then
|
|
|
|
|
initialization will continue in the C locale and the Unicode compatibility
|
|
|
|
|
warning described in the next section will be emitted just as it would for
|
|
|
|
|
any other application.
|
|
|
|
|
|
|
|
|
|
If ``PYTHONCOERCECLOCALE=0`` is explicitly set, initialization will continue in
|
|
|
|
|
the C locale and the Unicode compatibility warning described in the next
|
|
|
|
|
section will be automatically suppressed.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
The interpreter will always check for the ``PYTHONCOERCECLOCALE`` environment
|
2017-03-13 01:06:48 -04:00
|
|
|
|
variable at startup (even when running under the ``-E`` or ``-I`` switches),
|
|
|
|
|
as the locale coercion check necessarily takes place before any command line
|
|
|
|
|
argument processing. For consistency, the runtime check to determine whether
|
|
|
|
|
or not to suppress the locale compatibility warning will be similarly
|
|
|
|
|
independent of these settings.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Changes to the runtime initialization process
|
|
|
|
|
---------------------------------------------
|
2017-01-03 00:19:37 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
By the time that ``Py_Initialize`` is called, arbitrary locale-dependent
|
|
|
|
|
operations may have taken place in the current process. This means that
|
|
|
|
|
by the time it is called, it is *too late* to switch to a different locale -
|
|
|
|
|
doing so would introduce inconsistencies in decoded text, even in the context
|
|
|
|
|
of the standalone Python interpreter binary.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Accordingly, when ``Py_Initialize`` is called and CPython detects that the
|
2017-03-13 01:06:48 -04:00
|
|
|
|
configured locale is still the default ``C`` locale, ``PYTHONCOERCECLOCALE=0``
|
|
|
|
|
is set, *and* the ``PYTHONUTF8`` feature from PEP 540 is disabled (or not
|
|
|
|
|
implemented), the following warning will be issued::
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
|
2017-03-13 01:06:48 -04:00
|
|
|
|
encoding), which may cause Unicode compatibility problems. Using C.UTF-8,
|
2017-03-05 02:29:54 -05:00
|
|
|
|
C.utf8, or UTF-8 (if available) as alternative Unicode-compatible
|
|
|
|
|
locales is recommended.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
In this case, no actual change will be made to the locale settings.
|
|
|
|
|
|
|
|
|
|
Instead, the warning informs both system and application integrators that
|
|
|
|
|
they're running Python 3 in a configuration that we don't expect to work
|
|
|
|
|
properly.
|
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
The second sentence providing recommendations may eventually be conditionally
|
|
|
|
|
compiled based on the operating system (e.g. recommending ``LC_CTYPE=UTF-8``
|
|
|
|
|
on \*BSD systems), but the initial implementation will just use the common
|
|
|
|
|
generic message shown above.
|
2017-01-20 09:13:24 -05:00
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
New build-time configuration options
|
|
|
|
|
------------------------------------
|
|
|
|
|
|
|
|
|
|
While both of the above behaviours would be enabled by default, they would
|
|
|
|
|
also have new associated configuration options and preprocessor definitions
|
|
|
|
|
for the benefit of redistributors that want to override those default settings.
|
|
|
|
|
|
|
|
|
|
The locale coercion behaviour would be controlled by the flag
|
|
|
|
|
``--with[out]-c-locale-coercion``, which would set the ``PY_COERCE_C_LOCALE``
|
|
|
|
|
preprocessor definition.
|
|
|
|
|
|
|
|
|
|
The locale warning behaviour would be controlled by the flag
|
|
|
|
|
``--with[out]-c-locale-warning``, which would set the ``PY_WARN_ON_C_LOCALE``
|
|
|
|
|
preprocessor definition.
|
|
|
|
|
|
|
|
|
|
On platforms where they would have no effect (e.g. Mac OS X, iOS, Android,
|
|
|
|
|
Windows) these preprocessor variables would always be undefined.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 03:19:44 -05:00
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
Platform Support Changes
|
|
|
|
|
========================
|
|
|
|
|
|
|
|
|
|
A new "Legacy C Locale" section will be added to PEP 11 that states:
|
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
* as of CPython 3.7, \*nix platforms are expected to provide at least one of
|
|
|
|
|
``C.UTF-8`` (full locale), ``C.utf8`` (full locale) or ``UTF-8`` (
|
|
|
|
|
``LC_CTYPE``-only locale) as an alternative to the legacy ``C`` locale.
|
|
|
|
|
Any Unicode related integration problems that occur only in that locale and
|
|
|
|
|
cannot be reproduced in an appropriately configured non-ASCII locale will be
|
|
|
|
|
closed as "won't fix".
|
|
|
|
|
|
|
|
|
|
If PEP 540 is also implemented, then this section would instead state:
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
* as of CPython 3.7, the legacy C locale is only supported when operating in
|
|
|
|
|
"UTF-8" mode. Any Unicode handling issues that occur only in that locale
|
|
|
|
|
and cannot be reproduced in an appropriately configured non-ASCII locale will
|
|
|
|
|
be closed as "won't fix"
|
|
|
|
|
* as of CPython 3.7, \*nix platforms are expected to provide at least one of
|
|
|
|
|
``C.UTF-8`` (full locale), ``C.utf8`` (full locale) or ``UTF-8`` (
|
|
|
|
|
``LC_CTYPE``-only locale) as an alternative to the legacy ``C`` locale.
|
2017-03-13 01:06:48 -04:00
|
|
|
|
Any Unicode related integration problems with other locale-aware components
|
|
|
|
|
that occur only in that locale and cannot be reproduced in an appropriately
|
|
|
|
|
configured non-ASCII locale will be closed as "won't fix".
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Rationale
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Improving the handling of the C locale
|
|
|
|
|
--------------------------------------
|
|
|
|
|
|
|
|
|
|
It has been clear for some time that the C locale's default encoding of
|
|
|
|
|
``ASCII`` is entirely the wrong choice for development of modern networked
|
|
|
|
|
services. Newer languages like Rust and Go have eschewed that default entirely,
|
|
|
|
|
and instead made it a deployment requirement that systems be configured to use
|
|
|
|
|
UTF-8 as the text encoding for operating system interfaces. Similarly, Node.js
|
|
|
|
|
assumes UTF-8 by default (a behaviour inherited from the V8 JavaScript engine)
|
|
|
|
|
and requires custom build settings to indicate it should use the system
|
2017-01-03 00:19:37 -05:00
|
|
|
|
locale settings for locale-aware operations. Both the JVM and the .NET CLR
|
|
|
|
|
use UTF-16-LE as their primary encoding for passing text between applications
|
|
|
|
|
and the underlying platform.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
The challenge for CPython has been the fact that in addition to being used for
|
|
|
|
|
network service development, it is also extensively used as an embedded
|
|
|
|
|
scripting language in larger applications, and as a desktop application
|
|
|
|
|
development language, where it is more important to be consistent with other
|
2017-03-13 01:06:48 -04:00
|
|
|
|
locale-aware components sharing the same process, as well as with the user's
|
|
|
|
|
desktop locale settings, than it is with the emergent conventions of modern
|
|
|
|
|
network service development.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
The core premise of this PEP is that for *all* of these use cases, the
|
|
|
|
|
assumption of ASCII implied by the default "C" locale is the wrong choice,
|
|
|
|
|
and furthermore that the following assumptions are valid:
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
* in desktop application use cases, the process locale will *already* be
|
|
|
|
|
configured appropriately, and if it isn't, then that is an operating system
|
2017-01-20 09:13:24 -05:00
|
|
|
|
or embedding application level problem that needs to be reported to and
|
|
|
|
|
resolved by the operating system provider or application developer
|
2016-12-27 21:31:21 -05:00
|
|
|
|
* in network service development use cases (especially those based on Linux
|
|
|
|
|
containers), the process locale may not be configured *at all*, and if it
|
|
|
|
|
isn't, then the expectation is that components will impose their own default
|
|
|
|
|
encoding the way Rust, Go and Node.js do, rather than trusting the legacy C
|
|
|
|
|
default encoding of ASCII the way CPython currently does
|
|
|
|
|
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
Defaulting to "surrogateescape" error handling on the standard IO streams
|
|
|
|
|
-------------------------------------------------------------------------
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
By coercing the locale away from the legacy C default and its assumption of
|
|
|
|
|
ASCII as the preferred text encoding, this PEP also disables the implicit use
|
|
|
|
|
of the "surrogateescape" error handler on the standard IO streams that was
|
2017-01-20 09:13:24 -05:00
|
|
|
|
introduced in Python 3.5 ([15_]), as well as the automatic use of
|
|
|
|
|
``surrogateescape`` when operating in PEP 540's UTF-8 mode.
|
2017-01-07 20:54:24 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
Rather than introducing yet another configuration option to address that,
|
|
|
|
|
this PEP proposes to use the existing ``PYTHONIOENCODING`` setting to ensure
|
|
|
|
|
that the ``surrogateescape`` handler is enabled when the interpreter is
|
|
|
|
|
required to make assumptions regarding the expected filesystem encoding.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
The aim of this behaviour is to attempt to ensure that operating system
|
|
|
|
|
provided text values are typically able to be transparently passed through a
|
|
|
|
|
Python 3 application even if it is incorrect in assuming that that text has
|
|
|
|
|
been encoded as UTF-8.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
In particular, GB 18030 [12_] is a Chinese national text encoding standard
|
2017-01-20 09:13:24 -05:00
|
|
|
|
that handles all Unicode code points, that is formally incompatible with both
|
|
|
|
|
ASCII and UTF-8, but will nevertheless often tolerate processing as surrogate
|
|
|
|
|
escaped data - the points where GB 18030 reuses ASCII byte values in an
|
|
|
|
|
incompatible way are likely to be invalid in UTF-8, and will therefore be
|
|
|
|
|
escaped and opaque to string processing operations that split on or search for
|
|
|
|
|
the relevant ASCII code points. Operations that don't involve splitting on or
|
|
|
|
|
searching for particular ASCII or Unicode code point values are almost
|
|
|
|
|
certain to work correctly.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
Similarly, Shift-JIS [13_] and ISO-2022-JP [14_] remain in widespread use in
|
2017-01-20 09:13:24 -05:00
|
|
|
|
Japan, and are incompatible with both ASCII and UTF-8, but will tolerate text
|
|
|
|
|
processing operations that don't involve splitting on or searching for
|
|
|
|
|
particular ASCII or Unicode code point values.
|
|
|
|
|
|
|
|
|
|
As an example, consider two files, one encoded with UTF-8 (the default encoding
|
|
|
|
|
for ``en_AU.UTF-8``), and one encoded with GB-18030 (the default encoding for
|
|
|
|
|
``zh_CN.gb18030``)::
|
|
|
|
|
|
|
|
|
|
$ python3 -c 'open("utf8.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("utf-8"))'
|
2017-02-06 09:02:06 -05:00
|
|
|
|
$ python3 -c 'open("gb18030.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("gb18030"))'
|
2017-01-20 09:13:24 -05:00
|
|
|
|
|
|
|
|
|
On disk, we can see that these are two very different files::
|
|
|
|
|
|
|
|
|
|
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "rb").read().strip()); \
|
|
|
|
|
print("GB18030:", open("gb18030.txt", "rb").read().strip())'
|
|
|
|
|
UTF-8: b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4\n'
|
|
|
|
|
GB18030: b'\x816\xbd6\x810\x9d0\x817\xa29\x816\xbc4\x810\x8b3\x816\x8d6\n'
|
|
|
|
|
|
|
|
|
|
That nevertheless can both be rendered correctly to the terminal as long as
|
|
|
|
|
they're decoded prior to printing::
|
|
|
|
|
|
|
|
|
|
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
|
|
|
|
|
print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())'
|
|
|
|
|
UTF-8: ℙƴ☂ℌøἤ
|
|
|
|
|
GB18030: ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
By contrast, if we just pass along the raw bytes, as ``cat`` and similar C/C++
|
|
|
|
|
utilities will tend to do::
|
|
|
|
|
|
|
|
|
|
$ LANG=en_AU.UTF-8 cat utf8.txt gb18030.txt
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
<20>6<EFBFBD>6<EFBFBD>0<EFBFBD>0<EFBFBD>7<EFBFBD>9<EFBFBD>6<EFBFBD>4<EFBFBD>0<EFBFBD>3<EFBFBD>6<EFBFBD>6
|
|
|
|
|
|
|
|
|
|
Even setting a specifically Chinese locale won't help in getting the
|
|
|
|
|
GB-18030 encoded file rendered correctly::
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
$ LANG=zh_CN.gb18030 cat utf8.txt gb18030.txt
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
<20>6<EFBFBD>6<EFBFBD>0<EFBFBD>0<EFBFBD>7<EFBFBD>9<EFBFBD>6<EFBFBD>4<EFBFBD>0<EFBFBD>3<EFBFBD>6<EFBFBD>6
|
|
|
|
|
|
|
|
|
|
The problem is that the *terminal* encoding setting remains UTF-8, regardless
|
|
|
|
|
of the nominal locale. A GB18030 terminal can be emulated using the ``iconv``
|
|
|
|
|
utility::
|
|
|
|
|
|
|
|
|
|
$ cat utf8.txt gb18030.txt | iconv -f GB18030 -t UTF-8
|
|
|
|
|
鈩櫰粹槀鈩屆羔激
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
This reverses the problem, such that the GB18030 file is rendered correctly,
|
|
|
|
|
but the UTF-8 file has been converted to unrelated hanzi characters, rather than
|
|
|
|
|
the expected rendering of "Python" as non-ASCII characters.
|
|
|
|
|
|
|
|
|
|
With the emulated GB18030 terminal encoding, assuming UTF-8 in Python results
|
|
|
|
|
in *both* files being displayed incorrectly::
|
|
|
|
|
|
|
|
|
|
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
|
|
|
|
|
print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
|
|
|
|
|
| iconv -f GB18030 -t UTF-8
|
|
|
|
|
UTF-8: 鈩櫰粹槀鈩屆羔激
|
|
|
|
|
GB18030: 鈩櫰粹槀鈩屆羔激
|
|
|
|
|
|
|
|
|
|
However, setting the locale correctly means that the emulated GB18030 terminal
|
|
|
|
|
now displays both files as originally intended::
|
|
|
|
|
|
|
|
|
|
$ LANG=zh_CN.gb18030 \
|
|
|
|
|
python3 -c 'print("UTF-8: ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
|
|
|
|
|
print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
|
|
|
|
|
| iconv -f GB18030 -t UTF-8
|
|
|
|
|
UTF-8: ℙƴ☂ℌøἤ
|
|
|
|
|
GB18030: ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
The rationale for retaining ``surrogateescape`` as the default IO encoding is
|
|
|
|
|
that it will preserve the following helpful behaviour in the C locale::
|
|
|
|
|
|
|
|
|
|
$ cat gb18030.txt \
|
|
|
|
|
| LANG=C python3 -c "import sys; print(sys.stdin.read())" \
|
|
|
|
|
| iconv -f GB18030 -t UTF-8
|
|
|
|
|
ℙƴ☂ℌøἤ
|
|
|
|
|
|
|
|
|
|
Rather than reverting to the exception seen when a UTF-8 based locale is
|
|
|
|
|
explicitly configured::
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
$ cat gb18030.txt \
|
|
|
|
|
| python3 -c "import sys; print(sys.stdin.read())" \
|
|
|
|
|
| iconv -f GB18030 -t UTF-8
|
|
|
|
|
Traceback (most recent call last):
|
|
|
|
|
File "<string>", line 1, in <module>
|
|
|
|
|
File "/usr/lib64/python3.5/codecs.py", line 321, in decode
|
|
|
|
|
(result, consumed) = self._buffer_decode(data, self.errors, final)
|
|
|
|
|
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte
|
|
|
|
|
|
|
|
|
|
Note: an alternative to setting ``PYTHONIOENCODING`` as the PEP currently
|
|
|
|
|
proposes would be to instead *always* default to ``surrogateescape`` on the
|
|
|
|
|
standard streams, and require the use of ``PYTHONIOENCODING=:strict`` to request
|
|
|
|
|
text encoding validation during stream processing. Adopting such an approach
|
|
|
|
|
would bring Python 3 more into line with typical C/C++ tools that pass along
|
|
|
|
|
the raw bytes without checking them for conformance to their nominal encoding,
|
|
|
|
|
and would hence also make the last example display the desired output::
|
|
|
|
|
|
|
|
|
|
$ cat gb18030.txt \
|
|
|
|
|
| PYTHONIOENCODING=:surrogateescape python3 -c "import sys; print(sys.stdin.read())" \
|
|
|
|
|
| iconv -f GB18030 -t UTF-8
|
|
|
|
|
ℙƴ☂ℌøἤ
|
2017-01-07 20:54:24 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
However, such a change would have broader implications than the C locale
|
|
|
|
|
specific changes currently proposed, so it is considered out of scope for this
|
|
|
|
|
PEP.
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
Dropping official support for ASCII based text handling in the legacy C locale
|
|
|
|
|
------------------------------------------------------------------------------
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
We've been trying to get strict bytes/text separation to work reliably in the
|
|
|
|
|
legacy C locale for over a decade at this point. Not only haven't we been able
|
|
|
|
|
to get it to work, neither has anyone else - the only viable alternatives
|
|
|
|
|
identified have been to pass the bytes along verbatim without eagerly decoding
|
2017-01-20 09:13:24 -05:00
|
|
|
|
them to text (C/C++, Python 2.x, Ruby, etc), or else to ignore the nominal
|
|
|
|
|
C/C++ locale encoding entirely and assume the use of either UTF-8 (PEP 540,
|
|
|
|
|
Rust, Go, Node.js, etc) or UTF-16-LE (JVM, .NET CLR).
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
While this PEP ensures that developers that genuinely need to do so can still
|
|
|
|
|
opt-in to running their Python code in the legacy C locale (either by setting
|
|
|
|
|
PYTHONCOERCECLOCALE=0 or running a custom build that sets
|
|
|
|
|
``--without-c-locale-coercion``), it also makes it clear that we *don't*
|
|
|
|
|
expect Python 3's Unicode handling to be completely reliable in that
|
|
|
|
|
configuration, and the recommended alternative is to use a more appropriate
|
|
|
|
|
locale setting (or PEP 540's UTF-8 mode, if that is available).
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Providing implicit locale coercion only when running standalone
|
|
|
|
|
---------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
Over the course of Python 3.x development, multiple attempts have been made
|
|
|
|
|
to improve the handling of incorrect locale settings at the point where the
|
|
|
|
|
Python interpreter is initialised. The problem that emerged is that this is
|
|
|
|
|
ultimately *too late* in the interpreter startup process - data such as command
|
|
|
|
|
line arguments and the contents of environment variables may have already been
|
|
|
|
|
retrieved from the operating system and processed under the incorrect ASCII
|
|
|
|
|
text encoding assumption well before ``Py_Initialize`` is called.
|
|
|
|
|
|
|
|
|
|
The problems created by those inconsistencies were then even harder to diagnose
|
|
|
|
|
and debug than those created by believing the operating system's claim that
|
|
|
|
|
ASCII was a suitable encoding to use for operating system interfaces. This was
|
|
|
|
|
the case even for the default CPython binary, let alone larger C/C++
|
|
|
|
|
applications that embed CPython as a scripting engine.
|
|
|
|
|
|
|
|
|
|
The approach proposed in this PEP handles that problem by moving the locale
|
|
|
|
|
coercion as early as possible in the interpreter startup sequence when running
|
|
|
|
|
standalone: it takes place directly in the C-level ``main()`` function, even
|
|
|
|
|
before calling in to the `Py_Main()`` library function that implements the
|
|
|
|
|
features of the CPython interpreter CLI.
|
|
|
|
|
|
|
|
|
|
The ``Py_Initialize`` API then only gains an explicit warning (emitted on
|
|
|
|
|
``stderr``) when it detects use of the ``C`` locale, and relies on the
|
|
|
|
|
embedding application to specify something more reasonable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Querying LC_CTYPE for C locale detection
|
|
|
|
|
----------------------------------------
|
|
|
|
|
|
|
|
|
|
``LC_CTYPE`` is the actual locale category that CPython relies on to drive the
|
|
|
|
|
implicit decoding of environment variables, command line arguments, and other
|
|
|
|
|
text values received from the operating system.
|
|
|
|
|
|
|
|
|
|
As such, it makes sense to check it specifically when attempting to determine
|
|
|
|
|
whether or not the current locale configuration is likely to cause Unicode
|
|
|
|
|
handling problems.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Setting both LANG & LC_ALL for C.UTF-8 locale coercion
|
|
|
|
|
------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
Python is often used as a glue language, integrating other C/C++ ABI compatible
|
|
|
|
|
components in the current process, and components written in arbitrary
|
|
|
|
|
languages in subprocesses.
|
|
|
|
|
|
|
|
|
|
Setting ``LC_ALL`` to ``C.UTF-8`` imposes a locale setting override on all
|
2017-03-13 01:06:48 -04:00
|
|
|
|
locale-aware components in the current process and in any subprocesses that
|
|
|
|
|
inherit the current environment. This is important to handle cases where the
|
|
|
|
|
problem has arisen from a setting like ``LC_CTYPE=UTF-8`` being provided on a
|
|
|
|
|
system where no ``UTF-8`` locale is defined (e.g. when a Mac OS X ssh client is
|
2017-01-20 09:13:24 -05:00
|
|
|
|
configured to forward locale settings, and the user logs into a Linux server).
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
Setting ``LANG`` to ``C.UTF-8`` ensures that even components that only check
|
|
|
|
|
the ``LANG`` fallback for their locale settings will still use ``C.UTF-8``.
|
|
|
|
|
|
|
|
|
|
Together, these should ensure that when the locale coercion is activated, the
|
|
|
|
|
switch to the C.UTF-8 locale will be applied consistently across the current
|
|
|
|
|
process and any subprocesses that inherit the current environment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Allowing restoration of the legacy behaviour
|
|
|
|
|
--------------------------------------------
|
|
|
|
|
|
|
|
|
|
The CPython command line interpreter is often used to investigate faults that
|
|
|
|
|
occur in other applications that embed CPython, and those applications may still
|
|
|
|
|
be using the C locale even after this PEP is implemented.
|
|
|
|
|
|
|
|
|
|
Providing a simple on/off switch for the locale coercion behaviour makes it
|
|
|
|
|
much easier to reproduce the behaviour of such applications for debugging
|
|
|
|
|
purposes, as well as making it easier to reproduce the behaviour of older 3.x
|
|
|
|
|
runtimes even when running a version with this change applied.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Implementation
|
|
|
|
|
==============
|
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
A draft implementation of the change (including test cases and documentation)
|
|
|
|
|
is linked from issue 28180 [1_], which is an end user request that
|
2017-01-20 09:13:24 -05:00
|
|
|
|
``sys.getfilesystemencoding()`` default to ``utf-8`` rather than ``ascii``.
|
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
This patch is now being maintained as the ``pep538-coerce-c-locale`` feature
|
|
|
|
|
branch [18_] in Nick Coghlan's fork of the CPython repository on GitHub.
|
|
|
|
|
|
|
|
|
|
NOTE: As discussed in [1_], the currently posted draft implementation has some
|
2017-03-13 02:03:31 -04:00
|
|
|
|
known issues on Android. It is also missing the ``configure.ac`` adjustments
|
|
|
|
|
to ensure that ``PY_COERCE_C_LOCALE`` and ``PY_WARN_ON_C_LOCALE`` are always
|
|
|
|
|
undefined on Mac OS X.
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
Backporting to earlier Python 3 releases
|
|
|
|
|
========================================
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
Backporting to Python 3.6.0
|
|
|
|
|
---------------------------
|
|
|
|
|
|
|
|
|
|
If this PEP is accepted for Python 3.7, redistributors backporting the change
|
|
|
|
|
specifically to their initial Python 3.6.0 release will be both allowed and
|
|
|
|
|
encouraged. However, such backports should only be undertaken either in
|
2017-01-20 09:13:24 -05:00
|
|
|
|
conjunction with the changes needed to also provide a suitable locale by
|
|
|
|
|
default, or else specifically for platforms where such a locale is already
|
2017-01-07 02:04:39 -05:00
|
|
|
|
consistently available.
|
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
At least the Fedora project is planning to pursue this approach for the
|
|
|
|
|
upcoming Fedora 26 release [19_].
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
|
|
|
|
|
Backporting to other 3.x releases
|
|
|
|
|
---------------------------------
|
|
|
|
|
|
|
|
|
|
While the proposed behavioural change is seen primarily as a bug fix addressing
|
|
|
|
|
Python 3's current misbehaviour in the default ASCII-based C locale, it still
|
2017-01-20 09:13:24 -05:00
|
|
|
|
represents a reasonably significant change in the way CPython interacts with
|
2017-01-07 02:04:39 -05:00
|
|
|
|
the C locale system. As such, while some redistributors may still choose to
|
|
|
|
|
backport it to even earlier Python 3.x releases based on the needs and
|
|
|
|
|
interests of their particular user base, this wouldn't be encouraged as a
|
|
|
|
|
general practice.
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
However, configuring Python 3 *environments* (such as base container
|
|
|
|
|
images) to use these configuration settings by default is both allowed
|
|
|
|
|
and recommended.
|
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
Acknowledgements
|
|
|
|
|
================
|
|
|
|
|
|
|
|
|
|
The locale coercion approach proposed in this PEP is inspired directly by
|
|
|
|
|
Armin Ronacher's handling of this problem in the ``click`` command line
|
|
|
|
|
utility development framework [2_]::
|
|
|
|
|
|
|
|
|
|
$ LANG=C python3 -c 'import click; cli = click.command()(lambda:None); cli()'
|
|
|
|
|
Traceback (most recent call last):
|
|
|
|
|
...
|
|
|
|
|
RuntimeError: Click will abort further execution because Python 3 was
|
|
|
|
|
configured to use ASCII as encoding for the environment. Either run this
|
|
|
|
|
under Python 2 or consult http://click.pocoo.org/python3/ for mitigation
|
|
|
|
|
steps.
|
|
|
|
|
|
|
|
|
|
This system supports the C.UTF-8 locale which is recommended.
|
|
|
|
|
You might be able to resolve your issue by exporting the
|
|
|
|
|
following environment variables:
|
|
|
|
|
|
|
|
|
|
export LC_ALL=C.UTF-8
|
|
|
|
|
export LANG=C.UTF-8
|
|
|
|
|
|
|
|
|
|
The change was originally proposed as a downstream patch for Fedora's
|
|
|
|
|
system Python 3.6 package [3_], and then reformulated as a PEP for Python 3.7
|
|
|
|
|
with a section allowing for backports to earlier versions by redistributors.
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
The initial draft was posted to the Python Linux SIG for discussion [10_] and
|
|
|
|
|
then amended based on both that discussion and Victor Stinner's work in
|
|
|
|
|
PEP 540 [11_].
|
|
|
|
|
|
|
|
|
|
The "ℙƴ☂ℌøἤ" string used in the Unicode handling examples throughout this PEP
|
|
|
|
|
is taken from Ned Batchelder's excellent "Pragmatic Unicode" presentation [9_].
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
Stephen Turnbull has long provided valuable insight into the text encoding
|
|
|
|
|
handling challenges he regularly encounters at the University of Tsukuba
|
|
|
|
|
(筑波大学).
|
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
|
|
|
|
References
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
.. [1] CPython: sys.getfilesystemencoding() should default to utf-8
|
|
|
|
|
(http://bugs.python.org/issue28180)
|
|
|
|
|
|
|
|
|
|
.. [2] Locale configuration required for click applications under Python 3
|
|
|
|
|
(http://click.pocoo.org/5/python3/#python-3-surrogate-handling)
|
|
|
|
|
|
|
|
|
|
.. [3] Fedora: force C.UTF-8 when Python 3 is run under the C locale
|
|
|
|
|
(https://bugzilla.redhat.com/show_bug.cgi?id=1404918)
|
|
|
|
|
|
2017-01-03 00:19:37 -05:00
|
|
|
|
.. [4] GNU C: How Programs Set the Locale
|
|
|
|
|
( https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html)
|
|
|
|
|
|
|
|
|
|
.. [5] GNU C: Locale Categories
|
|
|
|
|
(https://www.gnu.org/software/libc/manual/html_node/Locale-Categories.html)
|
|
|
|
|
|
2017-01-07 02:04:39 -05:00
|
|
|
|
.. [6] glibc C.UTF-8 locale proposal
|
|
|
|
|
(https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)
|
|
|
|
|
|
|
|
|
|
.. [7] GNOME Flatpak
|
|
|
|
|
(http://flatpak.org/)
|
|
|
|
|
|
|
|
|
|
.. [8] Ubuntu Snappy
|
|
|
|
|
(https://www.ubuntu.com/desktop/snappy)
|
|
|
|
|
|
|
|
|
|
.. [9] Pragmatic Unicode
|
|
|
|
|
(http://nedbatchelder.com/text/unipain.html)
|
|
|
|
|
|
|
|
|
|
.. [10] linux-sig discussion of initial PEP draft
|
|
|
|
|
(https://mail.python.org/pipermail/linux-sig/2017-January/000014.html)
|
|
|
|
|
|
|
|
|
|
.. [11] Feedback notes from linux-sig discussion and PEP 540
|
|
|
|
|
(https://github.com/python/peps/issues/171)
|
|
|
|
|
|
|
|
|
|
.. [12] GB 18030
|
|
|
|
|
(https://en.wikipedia.org/wiki/GB_18030)
|
|
|
|
|
|
|
|
|
|
.. [13] Shift-JIS
|
|
|
|
|
(https://en.wikipedia.org/wiki/Shift_JIS)
|
|
|
|
|
|
|
|
|
|
.. [14] ISO-2022
|
|
|
|
|
(https://en.wikipedia.org/wiki/ISO/IEC_2022)
|
2016-12-27 21:31:21 -05:00
|
|
|
|
|
2017-01-07 20:54:24 -05:00
|
|
|
|
.. [15] Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale
|
|
|
|
|
(https://bugs.python.org/issue19977)
|
|
|
|
|
|
2017-01-20 09:13:24 -05:00
|
|
|
|
.. [16] test_readline.test_nonascii fails on Android
|
|
|
|
|
(http://bugs.python.org/issue28997)
|
|
|
|
|
|
|
|
|
|
.. [17] UTF-8 locale discussion on "locale.getdefaultlocale() fails on Mac OS X with default language set to English"
|
|
|
|
|
(http://bugs.python.org/issue18378#msg215215)
|
|
|
|
|
|
2017-03-05 02:29:54 -05:00
|
|
|
|
.. [18] GitHub branch diff for ``ncoghlan:pep538-coerce-c-locale``
|
|
|
|
|
(https://github.com/python/cpython/compare/master...ncoghlan:pep538-coerce-c-locale)
|
|
|
|
|
|
2017-03-13 01:06:48 -04:00
|
|
|
|
.. [19] Fedora 26 change proposal for locale coercion backport
|
|
|
|
|
(https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale)
|
|
|
|
|
|
2016-12-27 21:31:21 -05:00
|
|
|
|
Copyright
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
This document has been placed in the public domain under the terms of the
|
|
|
|
|
CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
..
|
|
|
|
|
Local Variables:
|
|
|
|
|
mode: indented-text
|
|
|
|
|
indent-tabs-mode: nil
|
|
|
|
|
sentence-end-double-space: t
|
|
|
|
|
fill-column: 70
|
|
|
|
|
coding: utf-8
|
|
|
|
|
End:
|