python-peps/pep-0414.txt

PEP: 414
Title: Explicit Unicode Literal for Python 3.3
Version: $Revision$
Last-Modified: $Date$
Author: Armin Ronacher <armin.ronacher@active-4.com>
Status: Accepted
Type: Standards Track
Content-Type: text/x-rst
Created: 15-Feb-2012
Post-History: 28-Feb-2012


Abstract
========

This document proposes the reintegration of an explicit unicode literal
from Python 2.x to the Python 3.x language specification, in order to
enable side-by-side support of libraries for both Python 2 and Python 3
without the need for an explicit 2to3 run.


BDFL Pronouncement
==================

This PEP has been formally accepted for Python 3.3.


Rationale and Goals
===================

Python 3 is a major new revision of the language, and it was decided very
early on that breaking backwards compatibility was part of the design. The
migration from a Python 2.x to a Python 3 codebase is to be accomplished
with the aid of a separate translation tool that converts the Python 2.x
sourcecode to Python 3 syntax.  With more and more libraries supporting
Python 3, however, it has become clear that 2to3 as a tool is
insufficient, and people are now attempting to find ways to make the same
source work in both Python 2.x and Python 3.x, with varying levels of
success.

Python 2.6 and Python 2.7 support syntax features from Python 3 which for
the most part make a unified code base possible.  Many thought that the
``unicode_literals`` future import might make a common source possible,
but it turns out that it's doing more harm than good.

With the design of the updated WSGI specification a few new terms for
strings were loosely defined: unicode strings, byte strings and native
strings.  In Python 3 the native string type is unicode, in Python 2 the
native string type is a bytestring.  These native string types are used in
a couple of places.  The native string type can be interned and is
preferably used for identifier names, filenames, source code and a few
other low level interpreter operations such as the return value of a
``__repr__`` or exception messages.

In Python 2.7 these string types can be defined explicitly.  Without any
future imports ``b'foo'`` means bytestring, ``u'foo'`` declares a unicode
string and ``'foo'`` a native string which in Python 2.x means bytes.
With the ``unicode_literals`` import the native string type is no longer
available by syntax and has to be incorrectly labeled as bytestring.  If
such a codebase is then used in Python 3, the interpreter will start using
byte objects in places where they are no longer accepted (such as
identifiers).  This can be solved by a module that detects 2.x and 3.x and
provides wrapper functions that transcode literals at runtime (either by
having a ``u`` function that marks things as unicode without future
imports or the inverse by having a ``n`` function that marks strings as
native).  Unfortunately, this has the side effect of slowing down the
runtime performance of Python and makes for less beautiful code.
Considering that Python 2 and Python 3 support for most libraries will
have to continue side by side for several more years to come, this means
that such modules lose one of Python's key properties: easily readable and
understandable code.

Additionally, the vast majority of people who maintain Python 2.x
codebases are more familiar with Python 2.x semantics, and a per-file
difference in literal meanings will be very annoying for them in the long
run.  A quick poll on Twitter about the use of the division future import
supported my suspicions that people opt out of behaviour-changing future
imports because they are a maintenance burden.  Every time you review code
you have to check the top of the file to see if the behaviour was changed.
Obviously that was an unscientific informal poll, but it might be
something worth considering.

Proposed Solution
=================

The idea is to support (with Python 3.3) an explicit ``u`` and ``U``
prefix for native strings in addition to the prefix-less variants.  These
would stick around for the entirety of the Python 3 lifetime but might at
some point yield deprecation warnings if deemed appropriate.  This could
be something for pyflakes or other similar libraries to support.

Python 3.2 and earlier
======================

An argument against this proposal was made on the Python-Dev mailinglist,
mentioning that Ubuntu LTS will ship Python 3.2 and 2.7 for only 5 years.
The counterargument is that Python 2.7 is currently the Python version of
choice for users who want LTS support.  As it stands, when chosing between
2.7 and Python 3.2, Python 3 is currently not the best choice for certain
long-term investments, since the ecosystem is not yet properly developed,
and libraries are still fighting with their API decisions for Python 3.

A valid point is that this would encourage people to become dependent on
Python 3.3 for their ports.  Fortunately that is not a big problem since
that could be fixed at installation time similar to how many projects are
currently invoking 2to3 as part of their installation process.

For Python 3.1 and Python 3.2 (even 3.0 if necessary) a simple
on-installation hook could be provided that tokenizes all source files and
strips away the otherwise unnecessary ``u`` prefix at installation time.

Who Benefits?
=============

There are a couple of places where decisions have to be made for or
against unicode support almost arbitrarily.  This is mostly the case for
protocols that do not support unicode all the way down, or hide it behind
transport encodings that might or might not be unicode themselves.  HTTP,
Email and WSGI are good examples of that.  For certain ambiguous cases it
would be possible to apply the same logic for unicode that Python 3
applies to the Python 2 versions of the library as well but, if those
details were exposed to the user of the API, it would mean breaking
compatibility for existing users of the Python 2 API which is a no-go for
many situations.  The automatic upgrading of binary strings to unicode
strings that would be enabled by this proposal would make it much easier
to port such libraries over.

Not only the libraries but also the users of these APIs would benefit from
that.  For instance, the urllib module in Python 2 is using byte strings,
and the one in Python 3 is using unicode strings.  By leveraging a native
string, users can avoid having to adjust for that.

Problems with 2to3
==================

In practice 2to3 currently suffers from a few problems which make it
unnecessarily difficult and/or unpleasant to use:

-   Bad overall performance.  In many cases 2to3 runs 20 times slower than
    the testsuite for the library or application it's testing.  (This for
    instance is the case for the Jinja2 library).
-   Slightly different behaviour in 2to3 between different versions of
    Python cause different outcomes when paired with custom fixers.
-   Line numbers from error messages do not match up with the real source
    lines due to added/rewritten imports.
-   extending 2to3 with custom fixers is nontrivial without using
    distribute.  By default 2to3 works acceptably well for upgrading
    byte-based APIs to unicode based APIs but it fails to upgrade APIs
    which already support unicode to Python 3::

        --- test.py (original)
        +++ test.py (refactored)
        @@ -1,5 +1,5 @@
         class Foo(object):
             def __unicode__(self):
        -        return u'test'
        +        return 'test'
             def __str__(self):
        -        return unicode(self).encode('utf-8')
        +        return str(self).encode('utf-8')


APIs and Concepts Using Native Strings
======================================

The following is an incomplete list of APIs and general concepts that use
native strings and need implicit upgrading to unicode in Python 3, and
which would directly benefit from this support:

-   Python identifiers (dict keys, class names, module names, import
    paths)
-   URLs for the most part as well as HTTP headers in urllib/http servers
-   WSGI environment keys and CGI-inherited values
-   Python source code for dynamic compilation and AST hacks
-   Exception messages
-   ``__repr__`` return value
-   preferred filesystem paths
-   preferred OS environment


Modernizing Code
================

The 2to3 tool can be easily adjusted to generate code that runs on both
Python 2 and Python 3.  An experimental extension to 2to3 which only
modernizes Python code to the extent that it runs on Python 2.7 or later
with support for the ``six`` library is available as python-modernize
[1]_. For most cases the runtime impact of ``six`` can be neglected (like
a function that calls ``iteritems()`` on a passed dictionary under 2.x or
``items()`` under 3.x), but to make strings cheap for both 2.x and 3.x it
is nearly impossible.  The way it currently works is by abusing the
``unicode-escape`` codec on Python 2.x native strings.  This is especially
ugly if such a string literal is used in a tight loop.

This proposal would fix this.  The modernize module could easily be
adjusted to simply not translate unicode strings, and the runtime overhead
would disappear.

Possible Downsides
==================

The obvious downside for this is that potential Python 3 users would have
to be aware of the fact that ``u`` is an optional prefix for strings.
This is something that Python 3 in general tried to avoid.  The second
inequality comparison operator was removed, the ``L`` prefix for long
integers etc.  This PEP would propose a slight revert on that practice by
reintroducing redundant syntax.  On the other hand, Python already has
multiple literals for strings with mostly the same behavior (single
quoted, double quoted, single triple quoted, double triple quoted).

Runtime Overhead of Wrappers
============================

I did some basic timings on the performance of a ``u()`` wrapper function
as used by the ``six`` library.  The implementation of ``u()`` is as
follows::

    if sys.version_info >= (3, 0):
        def u(value):
            return value
    else:
        def u(value):
            return unicode(value, 'unicode-escape')

The intention is that ``u'foo'`` can be turned to ``u('foo')`` and that on
Python 2.x an implicit decoding happens.  In this case the wrapper will
have a decoding overhead for Python 2.x.  I did some basic timings [2]_ to
see how bad the performance loss would be.  The following examples measure
the execution time over 10000 iterations::

    u'\N{SNOWMAN}barbaz'            1000 loops, best of 3: 295 usec per loop
    u('\N{SNOWMAN}barbaz')          10 loops, best of 3: 18.5 msec per loop
    u'foobarbaz_%d' % x             100 loops, best of 3: 8.32 msec per loop
    u('foobarbaz_%d') % x           10 loops, best of 3: 25.6 msec per loop
    u'fööbarbaz'                    1000 loops, best of 3: 289 usec per loop
    u('fööbarbaz')                  100 loops, best of 3: 15.1 msec per loop
    u'foobarbaz'                    1000 loops, best of 3: 294 usec per loop
    u('foobarbaz')                  100 loops, best of 3: 14.3 msec per loop

The overhead of the wrapper function in Python 3 is the price of a
function call since the function only has to return the argument
unchanged.


References
==========

.. [1] Python-Modernize
   (http://github.com/mitsuhiko/python-modernize)
.. [2] Benchmark
   (https://github.com/mitsuhiko/unicode-literals-pep/blob/master/timing.py)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: