iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Some typo fixes in PEP 3138; also add variables footer.

This commit is contained in:
Georg Brandl 2012-09-30 08:55:27 +02:00
parent caf06e1290
commit 9c83bacdff
1 changed files with 146 additions and 127 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

273
pep-3138.txt
View File

@ -13,11 +13,11 @@ Post-History: 05-May-2008, 05-Jun-2008
Abstract Abstract
======== ========
This PEP proposes a new string representation form for Python 3000. In This PEP proposes a new string representation form for Python 3000.
Python prior to Python 3000, the repr() built-in function converted In Python prior to Python 3000, the repr() built-in function converted
arbitrary objects to printable ASCII strings for debugging and logging. arbitrary objects to printable ASCII strings for debugging and
For Python 3000, a wider range of characters, based on the Unicode logging. For Python 3000, a wider range of characters, based on the
standard, should be considered 'printable'. Unicode standard, should be considered 'printable'.
Motivation Motivation
@ -28,8 +28,8 @@ algorithm.
- Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'. - Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'.
- Convert other non-printable characters(0x00-0x1f, 0x7f) and non-ASCII - Convert other non-printable characters(0x00-0x1f, 0x7f) and
characters(>=0x80) to '\\xXX'. non-ASCII characters (>= 0x80) to '\\xXX'.
- Backslash-escape quote characters (apostrophe, ') and add the quote - Backslash-escape quote characters (apostrophe, ') and add the quote
character at the beginning and the end. character at the beginning and the end.
@ -39,177 +39,184 @@ For Unicode strings, the following additional conversions are done.
- Convert leading surrogate pair characters without trailing character - Convert leading surrogate pair characters without trailing character
(0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\\uXXXX'. (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\\uXXXX'.
- Convert 16-bit characters(>=0x100) to '\\uXXXX'. - Convert 16-bit characters (>= 0x100) to '\\uXXXX'.
- Convert 21-bit characters(>=0x10000) and surrogate pair characters to - Convert 21-bit characters (>= 0x10000) and surrogate pair characters
'\\U00xxxxxx'. to '\\U00xxxxxx'.
This algorithm converts any string to printable ASCII, and repr() is This algorithm converts any string to printable ASCII, and repr() is
used as a handy and safe way to print strings for debugging or for used as a handy and safe way to print strings for debugging or for
logging. Although all non-ASCII characters are escaped, this does not logging. Although all non-ASCII characters are escaped, this does not
matter when most of the string's characters are ASCII. But for other matter when most of the string's characters are ASCII. But for other
languages, such as Japanese where most characters in a string are not languages, such as Japanese where most characters in a string are not
ASCII, this is very inconvenient. ASCII, this is very inconvenient.
We can use ``print(aJapaneseString)`` to get a readable string, but we We can use ``print(aJapaneseString)`` to get a readable string, but we
don't have a similar workaround for printing strings from collections don't have a similar workaround for printing strings from collections
such as lists or tuples. ``print(listOfJapaneseStrings)`` uses repr() to such as lists or tuples. ``print(listOfJapaneseStrings)`` uses repr()
build the string to be printed, so the resulting strings are always to build the string to be printed, so the resulting strings are always
hex-escaped. Or when ``open(japaneseFilemame)`` raises an exception, the hex-escaped. Or when ``open(japaneseFilemame)`` raises an exception,
error message is something like ``IOError: [Errno 2] No such file or the error message is something like ``IOError: [Errno 2] No such file
directory: '\u65e5\u672c\u8a9e'``, which isn't helpful. or directory: '\u65e5\u672c\u8a9e'``, which isn't helpful.
Python 3000 has a lot of nice features for non-Latin users such as Python 3000 has a lot of nice features for non-Latin users such as
non-ASCII identifiers, so it would be helpful if Python could also progress non-ASCII identifiers, so it would be helpful if Python could also
in a similar way for printable output. progress in a similar way for printable output.
Some users might be concerned that such output will mess up their Some users might be concerned that such output will mess up their
console if they print binary data like images. But this is unlikely to console if they print binary data like images. But this is unlikely
happen in practice because bytes and strings are different types in to happen in practice because bytes and strings are different types in
Python 3000, so printing an image to the console won't mess it up. Python 3000, so printing an image to the console won't mess it up.
This issue was once discussed by Hye-Shik Chang [1]_ , but was rejected. This issue was once discussed by Hye-Shik Chang [1]_, but was rejected.
Specification Specification
============= =============
- Add a new function to the Python C API ``int Py_UNICODE_ISPRINTABLE - Add a new function to the Python C API ``int Py_UNICODE_ISPRINTABLE
(Py_UNICODE ch)``. This function returns 0 if repr() should escape the (Py_UNICODE ch)``. This function returns 0 if repr() should escape
Unicode character ``ch``; otherwise it returns 1. Characters that should the Unicode character ``ch``; otherwise it returns 1. Characters
be escaped are defined in the Unicode character database as: that should be escaped are defined in the Unicode character database
as:
* Cc (Other, Control) * Cc (Other, Control)
* Cf (Other, Format) * Cf (Other, Format)
* Cs (Other, Surrogate) * Cs (Other, Surrogate)
* Co (Other, Private Use) * Co (Other, Private Use)
* Cn (Other, Not Assigned) * Cn (Other, Not Assigned)
* Zl (Separator, Line), refers to LINE SEPARATOR ('\\u2028'). * Zl (Separator, Line), refers to LINE SEPARATOR ('\\u2028').
* Zp (Separator, Paragraph), refers to PARAGRAPH SEPARATOR ('\\u2029'). * Zp (Separator, Paragraph), refers to PARAGRAPH SEPARATOR
* Zs (Separator, Space) other than ASCII space('\\x20'). Characters in ('\\u2029').
this category should be escaped to avoid ambiguity. * Zs (Separator, Space) other than ASCII space ('\\x20'). Characters
in this category should be escaped to avoid ambiguity.
- The algorithm to build repr() strings should be changed to: - The algorithm to build repr() strings should be changed to:
* Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'. * Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'.
* Convert non-printable ASCII characters(0x00-0x1f, 0x7f) to '\\xXX'. * Convert non-printable ASCII characters (0x00-0x1f, 0x7f) to
'\\xXX'.
* Convert leading surrogate pair characters without trailing character * Convert leading surrogate pair characters without trailing
(0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\\uXXXX'. character (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to
'\\uXXXX'.
* Convert non-printable characters(Py_UNICODE_ISPRINTABLE() returns 0) * Convert non-printable characters (Py_UNICODE_ISPRINTABLE() returns
to 'xXX', '\\uXXXX' or '\\U00xxxxxx'. 0) to 'xXX', '\\uXXXX' or '\\U00xxxxxx'.
* Backslash-escape quote characters (apostrophe, 0x27) and add quote * Backslash-escape quote characters (apostrophe, 0x27) and add a
character at the beginning and the end. quote character at the beginning and the end.
- Set the Unicode error-handler for sys.stderr to 'backslashreplace' by - Set the Unicode error-handler for sys.stderr to 'backslashreplace'
default. by default.
- Add a new function to the Python C API ``PyObject *PyObject_ASCII - Add a new function to the Python C API ``PyObject *PyObject_ASCII
(PyObject *o)``. This function converts any python object to a string (PyObject *o)``. This function converts any python object to a
using PyObject_Repr() and then hex-escapes all non-ASCII characters. string using PyObject_Repr() and then hex-escapes all non-ASCII
``PyObject_ASCII()`` generates the same string as ``PyObject_Repr()`` characters. ``PyObject_ASCII()`` generates the same string as
in Python 2. ``PyObject_Repr()`` in Python 2.
- Add a new built-in function, ``ascii()``. This function converts any - Add a new built-in function, ``ascii()``. This function converts
python object to a string using repr() and then hex-escapes all non-ASCII any python object to a string using repr() and then hex-escapes all
characters. ``ascii()`` generates the same string as ``repr()`` in non-ASCII characters. ``ascii()`` generates the same string as
Python 2. ``repr()`` in Python 2.
- Add ``'%a'`` string format operator. ``'%a'`` converts any python - Add a ``'%a'`` string format operator. ``'%a'`` converts any python
object to a string using repr() and then hex-escapes all non-ASCII object to a string using repr() and then hex-escapes all non-ASCII
characters. The ``'%a'`` format operator generates the same string as characters. The ``'%a'`` format operator generates the same string
``'%r'`` in Python 2. Also, add ``'!a'`` conversion flags to the as ``'%r'`` in Python 2. Also, add ``'!a'`` conversion flags to the
``string.format()`` method and add ``'%A'`` operator to the ``string.format()`` method and add ``'%A'`` operator to the
PyUnicode_FromFormat(). They converts any object to an ASCII string PyUnicode_FromFormat(). They convert any object to an ASCII string
as ``'%a'`` string format operator. as ``'%a'`` string format operator.
- Add an ``isprintable()`` method to the string type. ``str.isprintable()`` - Add an ``isprintable()`` method to the string type.
returns False if repr() should escape any character in the string; ``str.isprintable()`` returns False if repr() would escape any
otherwise returns True. The ``isprintable()`` method calls the character in the string; otherwise returns True. The
``Py_UNICODE_ISPRINTABLE()`` function internally. ``isprintable()`` method calls the ``Py_UNICODE_ISPRINTABLE()``
function internally.
Rationale Rationale
========= =========
The repr() in Python 3000 should be Unicode not ASCII based, just like The repr() in Python 3000 should be Unicode, not ASCII based, just
Python 3000 strings. Also, conversion should not be affected by the like Python 3000 strings. Also, conversion should not be affected by
locale setting, because the locale is not necessarily the same as the the locale setting, because the locale is not necessarily the same as
output device's locale. For example, it is common for a daemon process the output device's locale. For example, it is common for a daemon
to be invoked in an ASCII setting, but writes UTF-8 to its log files. process to be invoked in an ASCII setting, but writes UTF-8 to its log
Also, web applications might want to report the error information in files. Also, web applications might want to report the error
more readable form based on the HTML page's encoding. information in more readable form based on the HTML page's encoding.
Characters not supported by the user's console could be hex-escaped on Characters not supported by the user's console could be hex-escaped on
printing, by the Unicode encoder's error-handler. If the error-handler printing, by the Unicode encoder's error-handler. If the
of the output file is 'backslashreplace', such characters are error-handler of the output file is 'backslashreplace', such
hex-escaped without raising UnicodeEncodeError. For example, if your default characters are hex-escaped without raising UnicodeEncodeError. For
encoding is ASCII, ``print('Hello ¢')`` will print 'Hello \\xa2'. If example, if the default encoding is ASCII, ``print('Hello ¢')`` will
your encoding is ISO-8859-1, 'Hello ¢' will be printed. print 'Hello \\xa2'. If the encoding is ISO-8859-1, 'Hello ¢' will be
printed.
The default error-handler for sys.stdout is 'strict'. Other applications The default error-handler for sys.stdout is 'strict'. Other
reading the output might not understand hex-escaped characters, so applications reading the output might not understand hex-escaped
unsupported characters should be trapped when writing. If you need to characters, so unsupported characters should be trapped when writing.
escape unsupported characters, you should explicitly change the If unsupported characters must be escaped, the error-handler should be
error-handler. Unlike sys.stdout, sys.stderr doesn't raise changed explicitly. Unlike sys.stdout, sys.stderr doesn't raise
UnicodeEncodingError by default, because the default error-handler is UnicodeEncodingError by default, because the default error-handler is
'backslashreplace'. So printing error messeges containing non-ASCII 'backslashreplace'. So printing error messages containing non-ASCII
characters to sys.stderr will not raise an exception. Also, information characters to sys.stderr will not raise an exception. Also,
about uncaught exceptions (exception object, traceback) are printed by information about uncaught exceptions (exception object, traceback) is
the interpreter without raising exceptions. printed by the interpreter without raising exceptions.
Alternate Solutions Alternate Solutions
------------------- -------------------
To help debugging in non-Latin languages without changing repr(), other To help debugging in non-Latin languages without changing repr(),
suggestions were made. other suggestions were made.
- Supply a tool to print lists or dicts. - Supply a tool to print lists or dicts.
Strings to be printed for debugging are not only contained by lists or Strings to be printed for debugging are not only contained by lists
dicts, but also in many other types of object. File objects contain a or dicts, but also in many other types of object. File objects
file name in Unicode, exception objects contain a message in Unicode, contain a file name in Unicode, exception objects contain a message
etc. These strings should be printed in readable form when repr()ed. in Unicode, etc. These strings should be printed in readable form
It is unlikely to be possible to implement a tool to print all when repr()ed. It is unlikely to be possible to implement a tool to
possible object types. print all possible object types.
- Use sys.displayhook and sys.excepthook. - Use sys.displayhook and sys.excepthook.
For interactive sessions, we can write hooks to restore hex escaped For interactive sessions, we can write hooks to restore hex escaped
characters to the original characters. But these hooks are called only characters to the original characters. But these hooks are called
when printing the result of evaluating an expression entered in an only when printing the result of evaluating an expression entered in
interactive Python session, and doesn't work for the ``print()`` function, an interactive Python session, and don't work for the ``print()``
for non-interactive sessions or for ``logging.debug("%r", ...)``, etc. function, for non-interactive sessions or for ``logging.debug("%r",
...)``, etc.
- Subclass sys.stdout and sys.stderr. - Subclass sys.stdout and sys.stderr.
It is difficult to implement a subclass to restore hex-escaped It is difficult to implement a subclass to restore hex-escaped
characters since there isn't enough information left by the time it's characters since there isn't enough information left by the time
a string to undo the escaping correctly in all cases. For example, it's a string to undo the escaping correctly in all cases. For
``print("\\"+"u0041")`` should be printed as '\\u0041', not 'A'. But example, ``print("\\"+"u0041")`` should be printed as '\\u0041', not
there is no chance to tell file objects apart. 'A'. But there is no chance to tell file objects apart.
- Make the encoding used by unicode_repr() adjustable, and make the - Make the encoding used by unicode_repr() adjustable, and make the
existing repr() the default. existing repr() the default.
With adjustable repr(), the result of using repr() is unpredictable With adjustable repr(), the result of using repr() is unpredictable
and would make it impossible to write correct code involving repr(). and would make it impossible to write correct code involving repr().
And if current repr() is the default, then the old convention remains And if current repr() is the default, then the old convention
intact and users may expect ASCII strings as the result of repr(). remains intact and users may expect ASCII strings as the result of
Third party applications or libraries could be confused when a custom repr(). Third party applications or libraries could be confused
repr() function is used. when a custom repr() function is used.
Backwards Compatibility Backwards Compatibility
======================= =======================
Changing repr() may break some existing code, especially testing code. Changing repr() may break some existing code, especially testing code.
Five of Python's regression tests fail with this modification. If you Five of Python's regression tests fail with this modification. If you
need repr() strings without non-ASCII character as Python 2, you can use need repr() strings without non-ASCII character as Python 2, you can
the following function. :: use the following function. ::
def repr_ascii(obj): def repr_ascii(obj):
return str(repr(obj).encode("ASCII", "backslashreplace"), "ASCII") return str(repr(obj).encode("ASCII", "backslashreplace"), "ASCII")
@ -221,25 +228,25 @@ UnicodeEncodeError. ::
log.write(repr(data)) # UnicodeEncodeError will be raised log.write(repr(data)) # UnicodeEncodeError will be raised
# if data contains unsupported characters. # if data contains unsupported characters.
To avoid exceptions being raised, you can explicitly specify the error- To avoid exceptions being raised, you can explicitly specify the
handler. :: error-handler. ::
log = open("logfile", "w", errors="backslashreplace") log = open("logfile", "w", errors="backslashreplace")
log.write(repr(data)) # Unsupported characters will be escaped. log.write(repr(data)) # Unsupported characters will be escaped.
For a console that uses a Unicode-based encoding, for example, en_US. For a console that uses a Unicode-based encoding, for example,
utf8 or de_DE.utf8, the backslashescape trick doesn't work and all en_US.utf8 or de_DE.utf8, the backslashreplace trick doesn't work and
printable characters are not escaped. This will cause a problem of all printable characters are not escaped. This will cause a problem
similarly drawing characters in Western, Greek and Cyrillic languages. of similarly drawing characters in Western, Greek and Cyrillic
These languages use similar (but different) alphabets (descended from a languages. These languages use similar (but different) alphabets
common ancestor) and contain letters that look similar but have (descended from a common ancestor) and contain letters that look
different character codes. For example, it is hard to distinguish Latin similar but have different character codes. For example, it is hard
'a', 'e' and 'o' from Cyrillic 'а', 'е' and 'о'. (The visual to distinguish Latin 'a', 'e' and 'o' from Cyrillic 'а', 'е' and 'о'.
representation, of course, very much depends on the fonts used but (The visual representation, of course, very much depends on the fonts
usually these letters are almost indistinguishable.) To avoid the used but usually these letters are almost indistinguishable.) To
problem, the user can adjust the terminal encoding to get a result avoid the problem, the user can adjust the terminal encoding to get a
suitable for their environment. result suitable for their environment.
Rejected Proposals Rejected Proposals
@ -252,20 +259,21 @@ Rejected Proposals
idea. [2]_ idea. [2]_
- Use character names to escape characters, instead of hex character - Use character names to escape characters, instead of hex character
codes. For example, ``repr('\u03b1')`` can be converted to ``"\N{GREEK codes. For example, ``repr('\u03b1')`` can be converted to
SMALL LETTER ALPHA}"``. ``"\N{GREEK SMALL LETTER ALPHA}"``.
Using character names can be very verbose compared to hex-escape. Using character names can be very verbose compared to hex-escape.
e.g., ``repr("\ufbf9")`` is converted to ``"\N{ARABIC LIGATURE UIGHUR e.g., ``repr("\ufbf9")`` is converted to ``"\N{ARABIC LIGATURE
KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA ISOLATED FORM}"``. UIGHUR KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA ISOLATED
FORM}"``.
- Default error-handler of sys.stdout should be 'backslashreplace'. - Default error-handler of sys.stdout should be 'backslashreplace'.
Stuff written to stdout might be consumed by another program that Stuff written to stdout might be consumed by another program that
might misinterpret the \ escapes. For interactive session, it is might misinterpret the \\ escapes. For interactive sessions, it is
possible to make 'backslashreplace' error-handler to default, but may possible to make the 'backslashreplace' error-handler the default,
add confusion of the kind "it works in interactive mode but not when but this may add confusion of the kind "it works in interactive mode
redirecting to a file". but not when redirecting to a file".
Implementation Implementation
@ -288,3 +296,14 @@ Copyright
========= =========
This document has been placed in the public domain. This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: