140 lines
5.4 KiB
Plaintext
140 lines
5.4 KiB
Plaintext
|
PEP: 278
|
|||
|
|
Title: Universal Newline Support
|
|||
|
|
Version: $Revision$
|
|||
|
|
Last-Modified: $Date$
|
|||
|
|
Author: jack@cwi.nl (Jack Jansen)
|
|||
|
|
Status: Draft
|
|||
|
|
Type: Standards Track
|
|||
|
|
Created: 14-Jan-2002
|
|||
|
|
Python-Version: 2.3
|
|||
|
|
Post-History:
|
|||
|
|
|
|||
|
|
|
|||
|
|
Abstract
|
|||
|
|
|
|||
|
|
This PEP discusses a way in which Python can support I/O on files
|
|||
|
|
which have a newline format that is not the native format on the
|
|||
|
|
platform, so that Python on each platform can read and import
|
|||
|
|
files with CR (Macintosh), LF (Unix) or CR LF (Windows) line
|
|||
|
|
endings.
|
|||
|
|
|
|||
|
|
It is more and more common to come across files that have an end
|
|||
|
|
of line that does not match the standard on the current platform:
|
|||
|
|
files downloaded over the net, remotely mounted filesystems on a
|
|||
|
|
different platform, Mac OS X with its double standard of Mac and
|
|||
|
|
Unix line endings, etc.
|
|||
|
|
|
|||
|
|
Many tools such as editors and compilers already handle this
|
|||
|
|
gracefully, it would be good if Python did so too.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Specification
|
|||
|
|
|
|||
|
|
Universal newline support needs to be enabled during the configure
|
|||
|
|
of Python.
|
|||
|
|
|
|||
|
|
In a Python with universal newline support the feature is
|
|||
|
|
automatically enabled for all import statements and source()
|
|||
|
|
calls.
|
|||
|
|
|
|||
|
|
In a Python with universal newline support open() the mode
|
|||
|
|
parameter can also be "t", meaning "open for input as a text file
|
|||
|
|
with universal newline interpretation". Mode "t" cannot be
|
|||
|
|
combined with other mode flags such as "+".
|
|||
|
|
|
|||
|
|
There is no special support for output to file with a different
|
|||
|
|
newline convention.
|
|||
|
|
|
|||
|
|
A file object that has been opened in universal newline mode gets
|
|||
|
|
a new attribute "newlines" which reflects the newline convention
|
|||
|
|
used in the file. The value for this attribute is one of None (no
|
|||
|
|
newline read yet), "\r", "\n", "\r\n" or "mixed" (multiple
|
|||
|
|
different types of newlines seen).
|
|||
|
|
|
|||
|
|
|
|||
|
|
Rationale
|
|||
|
|
|
|||
|
|
Universal newline support is implemented in C, not in Python.
|
|||
|
|
This is done because we want files with a foreign newline
|
|||
|
|
convention to be import-able, so a Python Lib directory can be
|
|||
|
|
shared over a remote file system connection, or between MacPython
|
|||
|
|
and Unix-Python on Mac OS X. For this to be feasible the
|
|||
|
|
universal newline convention needs to have a reasonably small
|
|||
|
|
impact on performance, which means a Python implementation is not
|
|||
|
|
an option as it would bog down all imports. And because of files
|
|||
|
|
with multiple newline conventions, which Visual C++ and other
|
|||
|
|
Windows tools will happily produce, doing a quick check for the
|
|||
|
|
newlines used in a file (handing off the import to C code if a
|
|||
|
|
platform-local newline is seen) will not work. Finally, a C
|
|||
|
|
implementation also allows tracebacks and such (which open the
|
|||
|
|
Python source module) to be handled easily.
|
|||
|
|
|
|||
|
|
Universal newline support is implemented (for this release) as a
|
|||
|
|
compile time option because there is a performance penalty, even
|
|||
|
|
though it should be a small one.
|
|||
|
|
|
|||
|
|
There is no output implementation of universal newlines, Python
|
|||
|
|
programs are expected to handle this by themselves or write files
|
|||
|
|
with platform-local convention otherwise. The reason for this is
|
|||
|
|
that input is the difficult case, outputting different newlines to
|
|||
|
|
a file is already easy enough in Python.
|
|||
|
|
|
|||
|
|
While universal newlines are automatically enabled for import they
|
|||
|
|
are not for opening, where you have to specifically say open(...,
|
|||
|
|
"t"). This is open to debate, but here are a few reasons for this
|
|||
|
|
design:
|
|||
|
|
|
|||
|
|
- Compatibility. Programs which already do their own
|
|||
|
|
interpretation of \r\n in text files would break. Programs
|
|||
|
|
which open binary files as text files on Unix would also break
|
|||
|
|
(but it could be argued they deserve it :-).
|
|||
|
|
|
|||
|
|
- Interface clarity. Universal newlines are only supported for
|
|||
|
|
input files, not for input/output files, as the semantics would
|
|||
|
|
become muddy. Would you write Mac newlines if all reads so far
|
|||
|
|
had encountered Mac newlines? But what if you then later read a
|
|||
|
|
Unix newline?
|
|||
|
|
|
|||
|
|
The newlines attribute is included so that programs that really
|
|||
|
|
care about the newline convention, such as text editors, can
|
|||
|
|
examine what was in a file. They can then save (a copy of) the
|
|||
|
|
file with the same newline convention (or, in case of a file with
|
|||
|
|
mixed newlines, ask the user what to do, or output in platform
|
|||
|
|
convention).
|
|||
|
|
|
|||
|
|
Feedback is explicitly solicited on one item in the reference
|
|||
|
|
implementation: whether or not the universal newlines routines
|
|||
|
|
should grab the global interpreter lock. Currently they do not,
|
|||
|
|
but this could be considered living dangerously, as they may
|
|||
|
|
modify fields in a FileObject. But as these routines are
|
|||
|
|
replacements for fgets() and fread() as well it may be difficult
|
|||
|
|
to decide whether or not the lock is held when the routine is
|
|||
|
|
called. Moreover, the only danger is that if two threads read the
|
|||
|
|
same FileObject at the same time an extraneous newline may be seen
|
|||
|
|
or the "newlines" attribute may inadvertently be set to mixed. I
|
|||
|
|
would argue that if you read the same FileObject in two threads
|
|||
|
|
simultaneously you are asking for trouble anyway.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Reference Implementation
|
|||
|
|
|
|||
|
|
A reference implementation is available in SourceForge patch #476814.
|
|||
|
|
|
|||
|
|
|
|||
|
|
References
|
|||
|
|
|
|||
|
|
None.
|
|||
|
|
|
|||
|
|
|
|||
|
|
Copyright
|
|||
|
|
|
|||
|
|
This document has been placed in the public domain.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Local Variables:
|
|||
|
|
mode: indented-text
|
|||
|
|
indent-tabs-mode: nil
|
|||
|
|
fill-column: 70
|
|||
|
|
End:
|