206 lines
8.0 KiB
Plaintext
206 lines
8.0 KiB
Plaintext
PEP: 399
|
|
Title: Pure Python/C Accelerator Module Compatibility Requirements
|
|
Version: $Revision: 88219 $
|
|
Last-Modified: $Date: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $
|
|
Author: Brett Cannon <brett@python.org>
|
|
Status: Draft
|
|
Type: Informational
|
|
Content-Type: text/x-rst
|
|
Created: 04-Apr-2011
|
|
Python-Version: 3.3
|
|
Post-History:
|
|
|
|
Abstract
|
|
========
|
|
|
|
The Python standard library under CPython contains various instances
|
|
of modules implemented in both pure Python and C. This PEP requires
|
|
that in these instances that both the Python and C code *must* be
|
|
semantically identical (except in cases where implementation details
|
|
of a VM prevents it entirely). It is also required that new C-based
|
|
modules lacking a pure Python equivalent implementation get special
|
|
permissions to be added to the standard library.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Python has grown beyond the CPython virtual machine (VM). IronPython_,
|
|
Jython_, and PyPy_ all currently being viable alternatives to the
|
|
CPython VM. This VM ecosystem that has sprung up around the Python
|
|
programming language has led to Python being used in many different
|
|
areas where CPython cannot be used, e.g., Jython allowing Python to be
|
|
used in Java applications.
|
|
|
|
A problem all of the VMs other than CPython face is handling modules
|
|
from the standard library that are implemented in C. Since they do not
|
|
typically support the entire `C API of Python`_ they are unable to use
|
|
the code used to create the module. Often times this leads these other
|
|
VMs to either re-implement the modules in pure Python or in the
|
|
programming language used to implement the VM (e.g., in C# for
|
|
IronPython). This duplication of effort between CPython, PyPy, Jython,
|
|
and IronPython is extremely unfortunate as implementing a module *at
|
|
least* in pure Python would help mitigate this duplicate effort.
|
|
|
|
The purpose of this PEP is to minimize this duplicate effort by
|
|
mandating that all new modules added to Python's standard library
|
|
*must* have a pure Python implementation _unless_ special dispensation
|
|
is given. This makes sure that a module in the stdlib is available to
|
|
all VMs and not just to CPython.
|
|
|
|
Re-implementing parts (or all) of a module in C (in the case
|
|
of CPython) is still allowed for performance reasons, but any such
|
|
accelerated code must semantically match the pure Python equivalent to
|
|
prevent divergence. To accomplish this, the pure Python and C code must
|
|
be thoroughly tested with the *same* test suite to verify compliance.
|
|
This is to prevent users from accidentally relying
|
|
on semantics that are specific to the C code and are not reflected in
|
|
the pure Python implementation that other VMs rely upon, e.g., in
|
|
CPython 3.2.0, ``heapq.heappop()`` raises different exceptions
|
|
depending on whether the accelerated C code is used or not::
|
|
|
|
from test.support import import_fresh_module
|
|
|
|
c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
|
|
py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
|
|
|
|
|
|
class Spam:
|
|
"""Tester class which defines no other magic methods but
|
|
__len__()."""
|
|
def __len__(self):
|
|
return 0
|
|
|
|
|
|
try:
|
|
c_heapq.heappop(Spam())
|
|
except TypeError:
|
|
# "heap argument must be a list"
|
|
pass
|
|
|
|
try:
|
|
py_heapq.heappop(Spam())
|
|
except AttributeError:
|
|
# "'Foo' object has no attribute 'pop'"
|
|
pass
|
|
|
|
This kind of divergence is a problem for users as they unwittingly
|
|
write code that is CPython-specific. This is also an issue for other
|
|
VM teams as they have to deal with bug reports from users thinking
|
|
that they incorrectly implemented the module when in fact it was
|
|
caused by an untested case.
|
|
|
|
|
|
Details
|
|
=======
|
|
|
|
Starting in Python 3.3, any modules added to the standard library must
|
|
have a pure Python implementation. This rule can only be ignored if
|
|
the Python development team grants a special exemption for the module.
|
|
Typically the exemption would be granted only when a module wraps a
|
|
specific C-based library (e.g., sqlite3_). In granting an exemption it
|
|
will be recognized that the module will most likely be considered
|
|
exclusive to CPython and not part of Python's standard library that
|
|
other VMs are expected to support. Usage of ``ctypes`` to provide an
|
|
API for a C library will continue to be frowned upon as ``ctypes``
|
|
lacks compiler guarantees that C code typically relies upon to prevent
|
|
certain errors from occurring (e.g., API changes).
|
|
|
|
Even though a pure Python implementation is mandated by this PEP, it
|
|
does not preclude the use of a companion acceleration module. If an
|
|
acceleration module is provided it is to be named the same as the
|
|
module it is accelerating with an underscore attached as a prefix,
|
|
e.g., ``_warnings`` for ``warnings``. The common pattern to access
|
|
the accelerated code from the pure Python implementation is to import
|
|
it with an ``import *``, e.g., ``from _warnings import *``. This is
|
|
typically done at the end of the module to allow it to overwrite
|
|
specific Python objects with their accelerated equivalents. This kind
|
|
of import can also be done before the end of the module when needed,
|
|
e.g., an accelerated base class is provided but is then subclassed by
|
|
Python code. This PEP does not mandate that pre-existing modules in
|
|
the stdlib that lack a pure Python equivalent gain such a module. But
|
|
if people do volunteer to provide and maintain a pure Python
|
|
equivalent (e.g., the PyPy team volunteering their pure Python
|
|
implementation of the ``csv`` module and maintaining it) then such
|
|
code will be accepted.
|
|
|
|
Any accelerated code must be semantically identical to the pure Python
|
|
implementation. The only time any semantics are allowed to be
|
|
different are when technical details of the VM providing the
|
|
accelerated code prevent matching semantics from being possible, e.g.,
|
|
a class being a ``type`` when implemented in C. The semantics
|
|
equivalence requirement also dictates that no public API be provided
|
|
in accelerated code that does not exist in the pure Python code.
|
|
Without this requirement people could accidentally come to rely on a
|
|
detail in the acclerated code which is not made available to other VMs
|
|
that use the pure Python implementation. To help verify that the
|
|
contract of semantic equivalence is being met, a module must be tested
|
|
both with and without its accelerated code as thoroughly as possible.
|
|
|
|
As an example, to write tests which exercise both the pure Python and
|
|
C acclerated versions of a module, a basic idiom can be followed::
|
|
|
|
import collections.abc
|
|
from test.support import import_fresh_module, run_unittest
|
|
import unittest
|
|
|
|
c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
|
|
py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
|
|
|
|
|
|
class ExampleTest(unittest.TestCase):
|
|
|
|
def test_heappop_exc_for_non_MutableSequence(self):
|
|
# Raise TypeError when heap is not a
|
|
# collections.abc.MutableSequence.
|
|
class Spam:
|
|
"""Test class lacking many ABC-required methods
|
|
(e.g., pop())."""
|
|
def __len__(self):
|
|
return 0
|
|
|
|
heap = Spam()
|
|
self.assertFalse(isinstance(heap,
|
|
collections.abc.MutableSequence))
|
|
with self.assertRaises(TypeError):
|
|
self.heapq.heappop(heap)
|
|
|
|
|
|
class AcceleratedExampleTest(ExampleTest):
|
|
|
|
"""Test using the acclerated code."""
|
|
|
|
heapq = c_heapq
|
|
|
|
|
|
class PyExampleTest(ExampleTest):
|
|
|
|
"""Test with just the pure Python code."""
|
|
|
|
heapq = py_heapq
|
|
|
|
|
|
def test_main():
|
|
run_unittest(AcceleratedExampleTest, PyExampleTest)
|
|
|
|
|
|
if __name__ == '__main__':
|
|
test_main()
|
|
|
|
Thoroughness of the test can be verified using coverage measurements
|
|
with branching coverage on the pure Python code to verify that all
|
|
possible scenarios are tested using (or not using) accelerator code.
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
|
|
.. _IronPython: http://ironpython.net/
|
|
.. _Jython: http://www.jython.org/
|
|
.. _PyPy: http://pypy.org/
|
|
.. _C API of Python: http://docs.python.org/py3k/c-api/index.html
|
|
.. _sqlite3: http://docs.python.org/py3k/library/sqlite3.html
|