315 lines
12 KiB
Plaintext
315 lines
12 KiB
Plaintext
PEP: 3137
|
||
Title: Immutable Bytes and Mutable Buffer
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Guido van Rossum <guido@python.org>
|
||
Status: Final
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 26-Sep-2007
|
||
Python-Version: 3.0
|
||
Post-History: 26-Sep-2007, 30-Sep-2007
|
||
|
||
Introduction
|
||
============
|
||
|
||
After releasing Python 3.0a1 with a mutable bytes type, pressure
|
||
mounted to add a way to represent immutable bytes. Gregory P. Smith
|
||
proposed a patch that would allow making a bytes object temporarily
|
||
immutable by requesting that the data be locked using the new buffer
|
||
API from PEP 3118. This did not seem the right approach to me.
|
||
|
||
Jeffrey Yasskin, with the help of Adam Hupp, then prepared a patch to
|
||
make the bytes type immutable (by crudely removing all mutating APIs)
|
||
and fix the fall-out in the test suite. This showed that there aren't
|
||
all that many places that depend on the mutability of bytes, with the
|
||
exception of code that builds up a return value from small pieces.
|
||
|
||
Thinking through the consequences, and noticing that using the array
|
||
module as an ersatz mutable bytes type is far from ideal, and
|
||
recalling a proposal put forward earlier by Talin, I floated the
|
||
suggestion to have both a mutable and an immutable bytes type. (This
|
||
had been brought up before, but until seeing the evidence of Jeffrey's
|
||
patch I wasn't open to the suggestion.)
|
||
|
||
Moreover, a possible implementation strategy became clear: use the old
|
||
PyString implementation, stripped down to remove locale support and
|
||
implicit conversions to/from Unicode, for the immutable bytes type,
|
||
and keep the new PyBytes implementation as the mutable bytes type.
|
||
|
||
The ensuing discussion made it clear that the idea is welcome but
|
||
needs to be specified more precisely. Hence this PEP.
|
||
|
||
Advantages
|
||
==========
|
||
|
||
One advantage of having an immutable bytes type is that code objects
|
||
can use these. It also makes it possible to efficiently create hash
|
||
tables using bytes for keys; this may be useful when parsing protocols
|
||
like HTTP or SMTP which are based on bytes representing text.
|
||
|
||
Porting code that manipulates binary data (or encoded text) in Python
|
||
2.x will be easier using the new design than using the original 3.0
|
||
design with mutable bytes; simply replace ``str`` with ``bytes`` and
|
||
change '...' literals into b'...' literals.
|
||
|
||
Naming
|
||
======
|
||
|
||
I propose the following type names at the Python level:
|
||
|
||
- ``bytes`` is an immutable array of bytes (PyString)
|
||
|
||
- ``bytearray`` is a mutable array of bytes (PyBytes)
|
||
|
||
- ``memoryview`` is a bytes view on another object (PyMemory)
|
||
|
||
The old type named ``buffer`` is so similar to the new type
|
||
``memoryview``, introduce by PEP 3118, that it is redundant. The rest
|
||
of this PEP doesn't discuss the functionality of ``memoryview``; it is
|
||
just mentioned here to justify getting rid of the old ``buffer`` type.
|
||
(An earlier version of this PEP proposed ``buffer`` as the new name
|
||
for PyBytes; in the end this name was deemed to confusing given the
|
||
many other uses of the word buffer.)
|
||
|
||
While eventually it makes sense to change the C API names, this PEP
|
||
maintains the old C API names, which should be familiar to all.
|
||
|
||
Summary
|
||
-------
|
||
|
||
Here's a simple ASCII-art table summarizing the type names in various
|
||
Python versions::
|
||
|
||
+--------------+-------------+------------+--------------------------+
|
||
| C name | 2.x repr | 3.0a1 repr | 3.0a2 repr |
|
||
+--------------+-------------+------------+--------------------------+
|
||
| PyUnicode | unicode u'' | str '' | str '' |
|
||
| PyString | str '' | str8 s'' | bytes b'' |
|
||
| PyBytes | N/A | bytes b'' | bytearray bytearray(b'') |
|
||
| PyBuffer | buffer | buffer | N/A |
|
||
| PyMemoryView | N/A | memoryview | memoryview <...> |
|
||
+--------------+-------------+------------+--------------------------+
|
||
|
||
Literal Notations
|
||
=================
|
||
|
||
The b'...' notation introduced in Python 3.0a1 returns an immutable
|
||
bytes object, whatever variation is used. To create a mutable array
|
||
of bytes, use bytearray(b'...') or bytearray([...]). The latter form
|
||
takes a list of integers in range(256).
|
||
|
||
Functionality
|
||
=============
|
||
|
||
PEP 3118 Buffer API
|
||
-------------------
|
||
|
||
Both bytes and bytearray implement the PEP 3118 buffer API. The bytes
|
||
type only implements read-only requests; the bytearray type allows
|
||
writable and data-locked requests as well. The element data type is
|
||
always 'B' (i.e. unsigned byte).
|
||
|
||
Constructors
|
||
------------
|
||
|
||
There are four forms of constructors, applicable to both bytes and
|
||
bytearray:
|
||
|
||
- ``bytes(<bytes>)``, ``bytes(<bytearray>)``, ``bytearray(<bytes>)``,
|
||
``bytearray(<bytearray>)``: simple copying constructors, with the
|
||
note that ``bytes(<bytes>)`` might return its (immutable)
|
||
argument, but ``bytearray(<bytearray>)`` always makes a copy.
|
||
|
||
- ``bytes(<str>, <encoding>[, <errors>])``, ``bytearray(<str>,
|
||
<encoding>[, <errors>])``: encode a text string. Note that the
|
||
``str.encode()`` method returns an *immutable* bytes object. The
|
||
<encoding> argument is mandatory; <errors> is optional.
|
||
<encoding> and <errors>, if given, must be ``str`` instances.
|
||
|
||
- ``bytes(<memory view>)``, ``bytearray(<memory view>)``: construct
|
||
a bytes or bytearray object from anything that implements the PEP
|
||
3118 buffer API.
|
||
|
||
- ``bytes(<iterable of ints>)``, ``bytearray(<iterable of ints>)``:
|
||
construct a bytes or bytearray object from a stream of integers in
|
||
range(256).
|
||
|
||
- ``bytes(<int>)``, ``bytearray(<int>)``: construct a
|
||
zero-initialized bytes or bytearray object of a given length.
|
||
|
||
Comparisons
|
||
-----------
|
||
|
||
The bytes and bytearray types are comparable with each other and
|
||
orderable, so that e.g. b'abc' == bytearray(b'abc') < b'abd'.
|
||
|
||
Comparing either type to a str object for equality returns False
|
||
regardless of the contents of either operand. Ordering comparisons
|
||
with str raise TypeError. This is all conformant to the standard
|
||
rules for comparison and ordering between objects of incompatible
|
||
types.
|
||
|
||
(**Note:** in Python 3.0a1, comparing a bytes instance with a str
|
||
instance would raise TypeError, on the premise that this would catch
|
||
the occasional mistake quicker, especially in code ported from Python
|
||
2.x. However, a long discussion on the python-3000 list pointed out
|
||
so many problems with this that it is clearly a bad idea, to be rolled
|
||
back in 3.0a2 regardless of the fate of the rest of this PEP.)
|
||
|
||
Slicing
|
||
-------
|
||
|
||
Slicing a bytes object returns a bytes object. Slicing a bytearray
|
||
object returns a bytearray object.
|
||
|
||
Slice assignment to a bytearray object accepts anything that
|
||
implements the PEP 3118 buffer API, or an iterable of integers in
|
||
range(256).
|
||
|
||
Indexing
|
||
--------
|
||
|
||
Indexing bytes and bytearray returns small ints (like the bytes type in
|
||
3.0a1, and like lists or array.array('B')).
|
||
|
||
Assignment to an item of a bytearray object accepts an int in
|
||
range(256). (To assign from a bytes sequence, use a slice
|
||
assignment.)
|
||
|
||
Str() and Repr()
|
||
----------------
|
||
|
||
The str() and repr() functions return the same thing for these
|
||
objects. The repr() of a bytes object returns a b'...' style literal.
|
||
The repr() of a bytearray returns a string of the form "bytearray(b'...')".
|
||
|
||
Operators
|
||
---------
|
||
|
||
The following operators are implemented by the bytes and bytearray
|
||
types, except where mentioned:
|
||
|
||
- ``b1 + b2``: concatenation. With mixed bytes/bytearray operands,
|
||
the return type is that of the first argument (this seems arbitrary
|
||
until you consider how ``+=`` works).
|
||
|
||
- ``b1 += b2``: mutates b1 if it is a bytearray object.
|
||
|
||
- ``b * n``, ``n * b``: repetition; n must be an integer.
|
||
|
||
- ``b *= n``: mutates b if it is a bytearray object.
|
||
|
||
- ``b1 in b2``, ``b1 not in b2``: substring test; b1 can be any
|
||
object implementing the PEP 3118 buffer API.
|
||
|
||
- ``i in b``, ``i not in b``: single-byte membership test; i must
|
||
be an integer (if it is a length-1 bytes array, it is considered
|
||
to be a substring test, with the same outcome).
|
||
|
||
- ``len(b)``: the number of bytes.
|
||
|
||
- ``hash(b)``: the hash value; only implemented by the bytes type.
|
||
|
||
Note that the % operator is *not* implemented. It does not appear
|
||
worth the complexity.
|
||
|
||
Methods
|
||
-------
|
||
|
||
The following methods are implemented by bytes as well as bytearray, with
|
||
similar semantics. They accept anything that implements the PEP 3118
|
||
buffer API for bytes arguments, and return the same type as the object
|
||
whose method is called ("self")::
|
||
|
||
.capitalize(), .center(), .count(), .decode(), .endswith(),
|
||
.expandtabs(), .find(), .index(), .isalnum(), .isalpha(), .isdigit(),
|
||
.islower(), .isspace(), .istitle(), .isupper(), .join(), .ljust(),
|
||
.lower(), .lstrip(), .partition(), .replace(), .rfind(), .rindex(),
|
||
.rjust(), .rpartition(), .rsplit(), .rstrip(), .split(),
|
||
.splitlines(), .startswith(), .strip(), .swapcase(), .title(),
|
||
.translate(), .upper(), .zfill()
|
||
|
||
This is exactly the set of methods present on the str type in Python
|
||
2.x, with the exclusion of .encode(). The signatures and semantics
|
||
are the same too. However, whenever character classes like letter,
|
||
whitespace, lower case are used, the ASCII definitions of these
|
||
classes are used. (The Python 2.x str type uses the definitions from
|
||
the current locale, settable through the locale module.) The
|
||
.encode() method is left out because of the more strict definitions of
|
||
encoding and decoding in Python 3000: encoding always takes a Unicode
|
||
string and returns a bytes sequence, and decoding always takes a bytes
|
||
sequence and returns a Unicode string.
|
||
|
||
In addition, both types implement the class method ``.fromhex()``,
|
||
which constructs an object from a string containing hexadecimal values
|
||
(with or without spaces between the bytes).
|
||
|
||
The bytearray type implements these additional methods from the
|
||
MutableSequence ABC (see PEP 3119):
|
||
|
||
.extend(), .insert(), .append(), .reverse(), .pop(), .remove().
|
||
|
||
Bytes and the Str Type
|
||
----------------------
|
||
|
||
Like the bytes type in Python 3.0a1, and unlike the relationship
|
||
between str and unicode in Python 2.x, attempts to mix bytes (or
|
||
bytearray) objects and str objects without specifying an encoding will
|
||
raise a TypeError exception. (However, comparing bytes/bytearray and
|
||
str objects for equality will simply return False; see the section on
|
||
Comparisons above.)
|
||
|
||
Conversions between bytes or bytearray objects and str objects must
|
||
always be explicit, using an encoding. There are two equivalent APIs:
|
||
``str(b, <encoding>[, <errors>])`` is equivalent to
|
||
``b.decode(<encoding>[, <errors>])``, and
|
||
``bytes(s, <encoding>[, <errors>])`` is equivalent to
|
||
``s.encode(<encoding>[, <errors>])``.
|
||
|
||
There is one exception: we can convert from bytes (or bytearray) to str
|
||
without specifying an encoding by writing ``str(b)``. This produces
|
||
the same result as ``repr(b)``. This exception is necessary because
|
||
of the general promise that *any* object can be printed, and printing
|
||
is just a special case of conversion to str. There is however no
|
||
promise that printing a bytes object interprets the individual bytes
|
||
as characters (unlike in Python 2.x).
|
||
|
||
The str type currently implements the PEP 3118 buffer API. While this
|
||
is perhaps occasionally convenient, it is also potentially confusing,
|
||
because the bytes accessed via the buffer API represent a
|
||
platform-depending encoding: depending on the platform byte order and
|
||
a compile-time configuration option, the encoding could be UTF-16-BE,
|
||
UTF-16-LE, UTF-32-BE, or UTF-32-LE. Worse, a different implementation
|
||
of the str type might completely change the bytes representation,
|
||
e.g. to UTF-8, or even make it impossible to access the data as a
|
||
contiguous array of bytes at all. Therefore, the PEP 3118 buffer API
|
||
will be removed from the str type.
|
||
|
||
The ``basestring`` Type
|
||
-----------------------
|
||
|
||
The ``basestring`` type will be removed from the language. Code that
|
||
used to say ``isinstance(x, basestring)`` should be changed to use
|
||
``isinstance(x, str)`` instead.
|
||
|
||
Pickling
|
||
--------
|
||
|
||
Left as an exercise for the reader.
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|