325 lines
9.9 KiB
ReStructuredText
325 lines
9.9 KiB
ReStructuredText
PEP: 624
|
|
Title: Remove Py_UNICODE encoder APIs
|
|
Author: Inada Naoki <songofacandy@gmail.com>
|
|
Status: Final
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 06-Jul-2020
|
|
Python-Version: 3.11
|
|
Post-History: 08-Jul-2020
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in Python 3.11:
|
|
|
|
* ``PyUnicode_Encode()``
|
|
* ``PyUnicode_EncodeASCII()``
|
|
* ``PyUnicode_EncodeLatin1()``
|
|
* ``PyUnicode_EncodeUTF7()``
|
|
* ``PyUnicode_EncodeUTF8()``
|
|
* ``PyUnicode_EncodeUTF16()``
|
|
* ``PyUnicode_EncodeUTF32()``
|
|
* ``PyUnicode_EncodeUnicodeEscape()``
|
|
* ``PyUnicode_EncodeRawUnicodeEscape()``
|
|
* ``PyUnicode_EncodeCharmap()``
|
|
* ``PyUnicode_TranslateCharmap()``
|
|
* ``PyUnicode_EncodeDecimal()``
|
|
* ``PyUnicode_TransformDecimalToASCII()``
|
|
|
|
.. note::
|
|
|
|
:pep:`623` propose to remove
|
|
Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP
|
|
is not relating to Unicode object. These PEPs are split because they have
|
|
different motivations and need different discussions.
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
In general, reducing the number of APIs that have been deprecated for
|
|
a long time and have few users is a good idea for not only it
|
|
improves the maintainability of CPython, but it also helps API users
|
|
and other Python implementations.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Deprecated since Python 3.3
|
|
---------------------------
|
|
|
|
``Py_UNICODE`` and APIs using it has been deprecated since Python 3.3.
|
|
|
|
|
|
Inefficient
|
|
-----------
|
|
|
|
All of these APIs are implemented using ``PyUnicode_FromWideChar``.
|
|
So these APIs are inefficient when user want to encode Unicode
|
|
object.
|
|
|
|
|
|
Not used widely
|
|
---------------
|
|
|
|
When searching from the top 4000 PyPI packages [1]_, only pyodbc use
|
|
these APIs.
|
|
|
|
* ``PyUnicode_EncodeUTF8()``
|
|
* ``PyUnicode_EncodeUTF16()``
|
|
|
|
pyodbc uses these APIs to encode Unicode object into bytes object.
|
|
So it is easy to fix it. [2]_
|
|
|
|
|
|
Alternative APIs
|
|
================
|
|
|
|
There are alternative APIs to accept ``PyObject *unicode`` instead of
|
|
``Py_UNICODE *``. Users can migrate to them.
|
|
|
|
|
|
========================================= ==========================================
|
|
Deprecated API Alternative APIs
|
|
========================================= ==========================================
|
|
``PyUnicode_Encode()`` ``PyUnicode_AsEncodedString()``
|
|
``PyUnicode_EncodeASCII()`` ``PyUnicode_AsASCIIString()`` \(1)
|
|
``PyUnicode_EncodeLatin1()`` ``PyUnicode_AsLatin1String()`` \(1)
|
|
``PyUnicode_EncodeUTF7()`` \(2)
|
|
``PyUnicode_EncodeUTF8()`` ``PyUnicode_AsUTF8String()`` \(1)
|
|
``PyUnicode_EncodeUTF16()`` ``PyUnicode_AsUTF16String()`` \(3)
|
|
``PyUnicode_EncodeUTF32()`` ``PyUnicode_AsUTF32String()`` \(3)
|
|
``PyUnicode_EncodeUnicodeEscape()`` ``PyUnicode_AsUnicodeEscapeString()``
|
|
``PyUnicode_EncodeRawUnicodeEscape()`` ``PyUnicode_AsRawUnicodeEscapeString()``
|
|
``PyUnicode_EncodeCharmap()`` ``PyUnicode_AsCharmapString()`` \(1)
|
|
``PyUnicode_TranslateCharmap()`` ``PyUnicode_Translate()``
|
|
``PyUnicode_EncodeDecimal()`` \(4)
|
|
``PyUnicode_TransformDecimalToASCII()`` \(4)
|
|
========================================= ==========================================
|
|
|
|
Notes:
|
|
|
|
(1)
|
|
``const char *errors`` parameter is missing.
|
|
|
|
(2)
|
|
There is no public alternative API. But user can use generic
|
|
``PyUnicode_AsEncodedString()`` instead.
|
|
|
|
(3)
|
|
``const char *errors, int byteorder`` parameters are missing.
|
|
|
|
(4)
|
|
There is no direct replacement. But ``Py_UNICODE_TODECIMAL``
|
|
can be used instead. CPython uses
|
|
``_PyUnicode_TransformDecimalAndSpaceToASCII`` for converting
|
|
from Unicode to numbers instead.
|
|
|
|
|
|
Plan
|
|
====
|
|
|
|
Remove these APIs in Python 3.11. They have been deprecated already.
|
|
|
|
* ``PyUnicode_Encode()``
|
|
* ``PyUnicode_EncodeASCII()``
|
|
* ``PyUnicode_EncodeLatin1()``
|
|
* ``PyUnicode_EncodeUTF7()``
|
|
* ``PyUnicode_EncodeUTF8()``
|
|
* ``PyUnicode_EncodeUTF16()``
|
|
* ``PyUnicode_EncodeUTF32()``
|
|
* ``PyUnicode_EncodeUnicodeEscape()``
|
|
* ``PyUnicode_EncodeRawUnicodeEscape()``
|
|
* ``PyUnicode_EncodeCharmap()``
|
|
* ``PyUnicode_TranslateCharmap()``
|
|
* ``PyUnicode_EncodeDecimal()``
|
|
* ``PyUnicode_TransformDecimalToASCII()``
|
|
|
|
|
|
Alternative Ideas
|
|
=================
|
|
|
|
Replace ``Py_UNICODE*`` with ``PyObject*``
|
|
------------------------------------------
|
|
|
|
As described in the "Alternative APIs" section, some APIs don't have
|
|
public alternative APIs accepting ``PyObject *unicode`` input.
|
|
And some public alternative APIs have restrictions like missing
|
|
``errors`` and ``byteorder`` parameters.
|
|
|
|
Instead of removing deprecated APIs, we can reuse their names for
|
|
alternative public APIs.
|
|
|
|
Since we have private alternative APIs already, it is just renaming
|
|
from private name to public and deprecated names.
|
|
|
|
============================= ================================
|
|
Rename to Rename from
|
|
============================= ================================
|
|
``PyUnicode_EncodeASCII()`` ``_PyUnicode_AsASCIIString()``
|
|
``PyUnicode_EncodeLatin1()`` ``_PyUnicode_AsLatin1String()``
|
|
``PyUnicode_EncodeUTF7()`` ``_PyUnicode_EncodeUTF7()``
|
|
``PyUnicode_EncodeUTF8()`` ``_PyUnicode_AsUTF8String()``
|
|
``PyUnicode_EncodeUTF16()`` ``_PyUnicode_EncodeUTF16()``
|
|
``PyUnicode_EncodeUTF32()`` ``_PyUnicode_EncodeUTF32()``
|
|
============================= ================================
|
|
|
|
Pros:
|
|
|
|
* We have a more consistent API set.
|
|
|
|
Cons:
|
|
|
|
* Backward incompatible.
|
|
* We have more public APIs to maintain for rare use cases.
|
|
* Existing public APIs are enough for most use cases, and
|
|
``PyUnicode_AsEncodedString()`` can be used in other cases.
|
|
|
|
|
|
Replace ``Py_UNICODE*`` with ``Py_UCS4*``
|
|
-----------------------------------------
|
|
|
|
We can replace ``Py_UNICODE`` with ``Py_UCS4`` and undeprecate
|
|
these APIs.
|
|
|
|
UTF-8, UTF-16, UTF-32 encoders support ``Py_UCS4`` internally.
|
|
So ``PyUnicode_EncodeUTF8()``, ``PyUnicode_EncodeUTF16()``, and
|
|
``PyUnicode_EncodeUTF32()`` can avoid to create a temporary Unicode
|
|
object.
|
|
|
|
|
|
Pros:
|
|
|
|
* We can avoid creating temporary Unicode object when encoding from
|
|
``Py_UCS4*`` into bytes object with UTF-8, UTF-16, UTF-32 codecs.
|
|
|
|
Cons:
|
|
|
|
* Backward incompatible.
|
|
* We have more public APIs to maintain for rare use cases.
|
|
* Other Python implementations that want to support Python/C API need
|
|
to support these APIs too.
|
|
* If we change the Unicode internal representation to UTF-8 in the
|
|
future, we need to keep UCS-4 support only for these APIs.
|
|
|
|
|
|
Replace ``Py_UNICODE*`` with ``wchar_t*``
|
|
-----------------------------------------
|
|
|
|
We can replace ``Py_UNICODE`` with ``wchar_t``. Since ``Py_UNICODE``
|
|
is typedef of ``wchar_t`` already, this is status quo.
|
|
|
|
On platforms where ``sizeof(wchar_t) == 4``, we can avoid to create a
|
|
temporary Unicode object when encoding from ``wchar_t*`` to bytes
|
|
objects using UTF-8, UTF-16, and UTF-32 codec, like the "Replace
|
|
``Py_UNICODE*`` with ``Py_UCS4*``" idea.
|
|
|
|
|
|
Pros:
|
|
|
|
* Backward compatible.
|
|
* We can avoid creating temporary Unicode object when encode from
|
|
``Py_UCS4*`` into bytes object with UTF-8, UTF-16, UTF-32 codecs
|
|
on platform where ``sizeof(wchar_t) == 4``.
|
|
|
|
Cons:
|
|
|
|
* Although Windows is the most major platform that uses ``wchar_t``
|
|
heavily, these APIs need to create a temporary Unicode object
|
|
always because ``sizeof(wchar_t) == 2`` on Windows.
|
|
* We have more public APIs to maintain for rare use cases.
|
|
* Other Python implementations that want to support Python/C API need
|
|
to support these APIs too.
|
|
* If we change the Unicode internal representation to UTF-8 in the
|
|
future, we need to keep UCS-4 support only for these APIs.
|
|
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
Emit runtime warning
|
|
--------------------
|
|
|
|
In addition to existing compiler warning, emitting runtime
|
|
``DeprecationWarning`` is suggested.
|
|
|
|
But these APIs doesn't release GIL for now. Emitting a warning from
|
|
such APIs is not safe. See this example.
|
|
|
|
.. code-block::
|
|
|
|
PyObject *u = PyList_GET_ITEM(list, i); // u is borrowed reference.
|
|
PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u),
|
|
PyUnicode_GET_SIZE(u), NULL);
|
|
// Assumes u is still living reference.
|
|
PyObject *t = PyTuple_Pack(2, u, b);
|
|
Py_DECREF(b);
|
|
return t;
|
|
|
|
If we emit Python warning from ``PyUnicode_EncodeUTF8()``, warning
|
|
filters and other threads may change the ``list`` and ``u`` can be
|
|
a dangling reference after ``PyUnicode_EncodeUTF8()`` returned.
|
|
|
|
|
|
Discussions
|
|
===========
|
|
|
|
* `[python-dev] Plan to remove Py_UNICODE APis except PEP 623
|
|
<https://mail.python.org/archives/list/python-dev@python.org/thread/S7KW2U6IGXZFBMGS6WSJB26NZIBW4OLE/#S7KW2U6IGXZFBMGS6WSJB26NZIBW4OLE>`_
|
|
* `bpo-41123: Remove Py_UNICODE APIs except PEP 623
|
|
<https://bugs.python.org/issue41123>`_
|
|
* `[python-dev] PEP 624: Remove Py_UNICODE encoder APIs
|
|
<https://mail.python.org/archives/list/python-dev@python.org/thread/THXVM7FZVT56B7CPEDIYKJG6VMAYIEK5/#QUGBVLQNBFVNX25AEIL77WSFOHQES6LJ>`_
|
|
|
|
|
|
Objections
|
|
----------
|
|
|
|
* Removing these APIs removes ability to use codec without temporary
|
|
Unicode.
|
|
|
|
* Codecs can not encode Unicode buffer directly without temporary
|
|
Unicode object since Python 3.3. All these APIs creates temporary
|
|
Unicode object for now. So removing them doesn't reduce any
|
|
abilities.
|
|
|
|
* Why not remove decoder APIs too?
|
|
|
|
* They are part of stable ABI.
|
|
|
|
* ``PyUnicode_DecodeASCII()`` and ``PyUnicode_DecodeUTF8()`` are
|
|
used very widely. Deprecating them is not worth enough.
|
|
|
|
* Decoder APIs can decode from byte buffer directly, without
|
|
creating temporary bytes object. On the other hand, encoder APIs
|
|
can not avoid temporary Unicode object.
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [1] Source package list chosen from top 4000 PyPI packages.
|
|
(https://github.com/methane/notes/blob/master/2020/wchar-cache/package_list.txt)
|
|
|
|
.. [2] pyodbc -- Don't use PyUnicode_Encode API #792
|
|
(https://github.com/mkleehammer/pyodbc/pull/792)
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|