284 lines
10 KiB
Plaintext
284 lines
10 KiB
Plaintext
PEP: 261
|
||
Title: Support for "wide" Unicode characters
|
||
Version: $Revision$
|
||
Author: paulp@activestate.com (Paul Prescod)
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Created: 27-Jun-2001
|
||
Python-Version: 2.2
|
||
Post-History: 27-Jun-2001
|
||
|
||
|
||
Abstract
|
||
|
||
Python 2.1 unicode characters can have ordinals only up to 2**16 -1.
|
||
This range corresponds to a range in Unicode known as the Basic
|
||
Multilingual Plane. There are now characters in Unicode that live
|
||
on other "planes". The largest addressable character in Unicode
|
||
has the ordinal 17 * 2**16 - 1 (0x10ffff). For readability, we
|
||
will call this TOPCHAR and call characters in this range "wide
|
||
characters".
|
||
|
||
|
||
Glossary
|
||
|
||
Character
|
||
|
||
Used by itself, means the addressable units of a Python
|
||
Unicode string.
|
||
|
||
Code point
|
||
|
||
A code point is an integer between 0 and TOPCHAR.
|
||
If you imagine Unicode as a mapping from integers to
|
||
characters, each integer is a code point. But the
|
||
integers between 0 and TOPCHAR that do not map to
|
||
characters are also code points. Some will someday
|
||
be used for characters. Some are guaranteed never
|
||
to be used for characters.
|
||
|
||
Codec
|
||
|
||
A set of functions for translating between physical
|
||
encodings (e.g. on disk or coming in from a network)
|
||
into logical Python objects.
|
||
|
||
Encoding
|
||
|
||
Mechanism for representing abstract characters in terms of
|
||
physical bits and bytes. Encodings allow us to store
|
||
Unicode characters on disk and transmit them over networks
|
||
in a manner that is compatible with other Unicode software.
|
||
|
||
Surrogate pair
|
||
|
||
Two physical characters that represent a single logical
|
||
character. Part of a convention for representing 32-bit
|
||
code points in terms of two 16-bit code points.
|
||
|
||
Unicode string
|
||
|
||
A Python type representing a sequence of code points with
|
||
"string semantics" (e.g. case conversions, regular
|
||
expression compatibility, etc.) Constructed with the
|
||
unicode() function.
|
||
|
||
|
||
Proposed Solution
|
||
|
||
One solution would be to merely increase the maximum ordinal
|
||
to a larger value. Unfortunately the only straightforward
|
||
implementation of this idea is to use 4 bytes per character.
|
||
This has the effect of doubling the size of most Unicode
|
||
strings. In order to avoid imposing this cost on every
|
||
user, Python 2.2 will allow the 4-byte implementation as a
|
||
build-time option. Users can choose whether they care about
|
||
wide characters or prefer to preserve memory.
|
||
|
||
The 4-byte option is called "wide Py_UNICODE". The 2-byte option
|
||
is called "narrow Py_UNICODE".
|
||
|
||
Most things will behave identically in the wide and narrow worlds.
|
||
|
||
* unichr(i) for 0 <= i < 2**16 (0x10000) always returns a
|
||
length-one string.
|
||
|
||
* unichr(i) for 2**16 <= i <= TOPCHAR will return a
|
||
length-one string on wide Python builds. On narrow builds it will
|
||
raise ValueError.
|
||
|
||
ISSUE
|
||
|
||
Python currently allows \U literals that cannot be
|
||
represented as a single Python character. It generates two
|
||
Python characters known as a "surrogate pair". Should this
|
||
be disallowed on future narrow Python builds?
|
||
|
||
Pro:
|
||
|
||
Python already the construction of a surrogate pair
|
||
for a large unicode literal character escape sequence.
|
||
This is basically designed as a simple way to construct
|
||
"wide characters" even in a narrow Python build. It is also
|
||
somewhat logical considering that the Unicode-literal syntax
|
||
is basically a short-form way of invoking the unicode-escape
|
||
codec.
|
||
|
||
Con:
|
||
|
||
Surrogates could be easily created this way but the user
|
||
still needs to be careful about slicing, indexing, printing
|
||
etc. Therefore some have suggested that Unicode
|
||
literals should not support surrogates.
|
||
|
||
|
||
ISSUE
|
||
|
||
Should Python allow the construction of characters that do
|
||
not correspond to Unicode code points? Unassigned Unicode
|
||
code points should obviously be legal (because they could
|
||
be assigned at any time). But code points above TOPCHAR are
|
||
guaranteed never to be used by Unicode. Should we allow access
|
||
to them anyhow?
|
||
|
||
Pro:
|
||
|
||
If a Python user thinks they know what they're doing why
|
||
should we try to prevent them from violating the Unicode
|
||
spec? After all, we don't stop 8-bit strings from
|
||
containing non-ASCII characters.
|
||
|
||
Con:
|
||
|
||
Codecs and other Unicode-consuming code will have to be
|
||
careful of these characters which are disallowed by the
|
||
Unicode specification.
|
||
|
||
* ord() is always the inverse of unichr()
|
||
|
||
* There is an integer value in the sys module that describes the
|
||
largest ordinal for a character in a Unicode string on the current
|
||
interpreter. sys.maxunicode is 2**16-1 (0xffff) on narrow builds
|
||
of Python and TOPCHAR on wide builds.
|
||
|
||
ISSUE: Should there be distinct constants for accessing
|
||
TOPCHAR and the real upper bound for the domain of
|
||
unichr (if they differ)? There has also been a
|
||
suggestion of sys.unicodewidth which can take the
|
||
values 'wide' and 'narrow'.
|
||
|
||
* every Python Unicode character represents exactly one Unicode code
|
||
point (i.e. Python Unicode Character = Abstract Unicode character).
|
||
|
||
* codecs will be upgraded to support "wide characters"
|
||
(represented directly in UCS-4, and as variable-length sequences
|
||
in UTF-8 and UTF-16). This is the main part of the implementation
|
||
left to be done.
|
||
|
||
* There is a convention in the Unicode world for encoding a 32-bit
|
||
code point in terms of two 16-bit code points. These are known
|
||
as "surrogate pairs". Python's codecs will adopt this convention
|
||
and encode 32-bit code points as surrogate pairs on narrow Python
|
||
builds.
|
||
|
||
ISSUE
|
||
|
||
Should there be a way to tell codecs not to generate
|
||
surrogates and instead treat wide characters as
|
||
errors?
|
||
|
||
Pro:
|
||
|
||
I might want to write code that works only with
|
||
fixed-width characters and does not have to worry about
|
||
surrogates.
|
||
|
||
|
||
Con:
|
||
|
||
No clear proposal of how to communicate this to codecs.
|
||
|
||
* there are no restrictions on constructing strings that use
|
||
code points "reserved for surrogates" improperly. These are
|
||
called "isolated surrogates". The codecs should disallow reading
|
||
these from files, but you could construct them using string
|
||
literals or unichr().
|
||
|
||
|
||
Implementation
|
||
|
||
There is a new define:
|
||
|
||
#define Py_UNICODE_SIZE 2
|
||
|
||
To test whether UCS2 or UCS4 is in use, the derived macro
|
||
Py_UNICODE_WIDE should be used, which is defined when UCS-4 is in
|
||
use.
|
||
|
||
There is a new configure option:
|
||
|
||
--enable-unicode=ucs2 configures a narrow Py_UNICODE, and uses
|
||
wchar_t if it fits
|
||
--enable-unicode=ucs4 configures a wide Py_UNICODE, and uses
|
||
wchar_t if it fits
|
||
--enable-unicode same as "=ucs2"
|
||
--disable-unicode entirely remove the Unicode functionality.
|
||
|
||
It is also proposed that one day --enable-unicode will just
|
||
default to the width of your platforms wchar_t.
|
||
|
||
Windows builds will be narrow for a while based on the fact that
|
||
there have been few requests for wide characters, those requests
|
||
are mostly from hard-core programmers with the ability to buy
|
||
their own Python and Windows itself is strongly biased towards
|
||
16-bit characters.
|
||
|
||
|
||
Notes
|
||
|
||
This PEP does NOT imply that people using Unicode need to use a
|
||
4-byte encoding for their files on disk or sent over the network.
|
||
It only allows them to do so. For example, ASCII is still a
|
||
legitimate (7-bit) Unicode-encoding.
|
||
|
||
It has been proposed that there should be a module that handles
|
||
surrogates in narrow Python builds for programmers. If someone
|
||
wants to implement that, it will be another PEP. It might also be
|
||
combined with features that allow other kinds of character-,
|
||
word- and line- based indexing.
|
||
|
||
|
||
Rejected Suggestions
|
||
|
||
More or less the status-quo
|
||
|
||
We could officially say that Python characters are 16-bit and
|
||
require programmers to implement wide characters in their
|
||
application logic by combining surrogate pairs. This is a heavy
|
||
burden because emulating 32-bit characters is likely to be
|
||
very inefficient if it is coded entirely in Python. Plus these
|
||
abstracted pseudo-strings would not be legal as input to the
|
||
regular expression engine.
|
||
|
||
"Space-efficient Unicode" type
|
||
|
||
Another class of solution is to use some efficient storage
|
||
internally but present an abstraction of wide characters to
|
||
the programmer. Any of these would require a much more complex
|
||
implementation than the accepted solution. For instance consider
|
||
the impact on the regular expression engine. In theory, we could
|
||
move to this implementation in the future without breaking Python
|
||
code. A future Python could "emulate" wide Python semantics on
|
||
narrow Python. Guido is not willing to undertake the
|
||
implementation right now.
|
||
|
||
Two types
|
||
|
||
We could introduce a 32-bit Unicode type alongside the 16-bit
|
||
type. There is a lot of code that expects there to be only a
|
||
single Unicode type.
|
||
|
||
This PEP represents the least-effort solution. Over the next
|
||
several years, 32-bit Unicode characters will become more common
|
||
and that may either convince us that we need a more sophisticated
|
||
solution or (on the other hand) convince us that simply
|
||
mandating wide Unicode characters is an appropriate solution.
|
||
Right now the two options on the table are do nothing or do
|
||
this.
|
||
|
||
|
||
References
|
||
|
||
Unicode Glossary: http://www.unicode.org/glossary/
|
||
|
||
|
||
Copyright
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
End:
|