python-peps/pep-0436.txt

PEP: 436
Title: The Argument Clinic DSL
Version: $Revision$
Last-Modified: $Date$
Author: Larry Hastings <larry@hastings.org>
Discussions-To: Python-Dev <python-dev@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 22-Feb-2013


Abstract
========

This document proposes "Argument Clinic", a DSL designed to facilitate
argument processing for built-in functions in the implementation of
CPython.


Rationale and Goals
===================

The primary implementation of Python, "CPython", is written in a
mixture of Python and C.  One of the implementation details of CPython
is what are called "built-in" functions -- functions available to
Python programs but written in C.  When a Python program calls a
built-in function and passes in arguments, those arguments must be
translated from Python values into C values.  This process is called
"parsing arguments".

As of CPython 3.3, arguments to functions are primarily parsed with
one of two functions: the original ``PyArg_ParseTuple()``, [1]_ and
the more modern ``PyArg_ParseTupleAndKeywords()``. [2]_ The former
function only handles positional parameters; the latter also
accommodates keyword and keyword-only parameters, and is preferred for
new code.

``PyArg_ParseTuple()`` was a reasonable approach when it was first
conceived.  The programmer specified the translation for the arguments
in a "format string": [3]_ each parameter matched to a "format unit",
a one-or-two character sequence telling ``PyArg_ParseTuple()`` what
Python types to accept and how to translate them into the appropriate
C value for that parameter.  There were only a dozen or so of these
"format units", and each one was distinct and easy to understand.

Over the years the ``PyArg_Parse`` interface has been extended in
numerous ways.  The modern API is quite complex, to the point that it
is somewhat painful to use.  Consider:

  * There are now forty different "format units"; a few are even three
    characters long.  This makes it difficult to understand what the
    format string says without constantly cross-indexing it with the
    documentation.
  * There are also six meta-format units that may be buried in the
    format string.  (They are: ``"()|$:;"``.)
  * The more format units are added, the less likely it is the
    implementer can pick an easy-to-use mnemonic for the format unit,
    because the character of choice is probably already in use.  In
    other words, the more format units we have, the more obtuse the
    format units become.
  * Several format units are nearly identical to others, having only
    subtle differences.  This makes understanding the exact semantics
    of the format string even harder.
  * The docstring is specified as a static C string, which is mildly
    bothersome to read and edit.
  * When adding a new parameter to a function using
    ``PyArg_ParseTupleAndKeywords()``, it's necessary to touch six
    different places in the code: [4]_

      * Declaring the variable to store the argument.
      * Passing in a pointer to that variable in the correct spot in
        ``PyArg_ParseTupleAndKeywords()``, also passing in any
        "length" or "converter" arguments in the correct order.
      * Adding the name of the argument in the correct spot of the
        "keywords" array passed in to
        ``PyArg_ParseTupleAndKeywords()``.
      * Adding the format unit to the correct spot in the format
        string.
      * Adding the parameter to the prototype in the docstring.
      * Documenting the parameter in the docstring.

  * There is currently no mechanism for builtin functions to provide
    their "signature" information (see ``inspect.getfullargspec`` and
    ``inspect.Signature``).  Adding this information using a mechanism
    similar to the existing ``PyArg_Parse`` functions would require
    repeating ourselves yet again.

The goal of Argument Clinic is to replace this API with a mechanism
inheriting none of these downsides:

  * You need specify each parameter only once.
  * All information about a parameter is kept together in one place.
  * For each parameter, you specify its type in C; Argument Clinic
    handles the translation from Python value into C value for you.
  * Argument Clinic also allows for fine-tuning of argument processing
    behavior with highly-readable "flags", both per-parameter and
    applying across the whole function.
  * Docstrings are written in plain text.
  * From this, Argument Clinic generates for you all the mundane,
    repetitious code and data structures CPython needs internally.
    Once you've specified the interface, the next step is simply to
    write your implementation using native C types.  Every detail of
    argument parsing is handled for you.

Future goals of Argument Clinic include:

  * providing signature information for builtins, and
  * speed improvements to the generated code.


DSL Syntax Summary
==================

The Argument Clinic DSL is specified as a comment embedded in a C
file, as follows.  The "Example" column on the right shows you sample
input to the Argument Clinic DSL, and the "Section" column on the left
specifies what each line represents in turn.

::

 +-----------------------+-----------------------------------------------------+
 | Section               | Example                                             |
 +-----------------------+-----------------------------------------------------+
 | Clinic DSL start      | /*[clinic]                                          |
 | Function declaration  | module.function_name -> return_annotation           |
 | Function flags        | flag flag2 flag3=value                              |
 | Parameter declaration |       type name = default                           |
 | Parameter flags       |       flag flag2 flag3=value                        |
 | Parameter docstring   |           Lorem ipsum dolor sit amet, consectetur   |
 |                       |           adipisicing elit, sed do eiusmod tempor   |
 | Function docstring    | Lorem ipsum dolor sit amet, consectetur adipisicing |
 |                       | elit, sed do eiusmod tempor incididunt ut labore et |
 | Clinic DSL end        | [clinic]*/                                          |
 | Clinic output         | ...                                                 |
 | Clinic output end     | /*[clinic end output:<checksum>]*/                  |
 +-----------------------+-----------------------------------------------------+


General Behavior Of the Argument Clinic DSL
-------------------------------------------

All lines support ``#`` as a line comment delimiter *except*
docstrings.  Blank lines are always ignored.

Like Python itself, leading whitespace is significant in the Argument
Clinic DSL.  The first line of the "function" section is the
declaration; all subsequent lines at the same indent are function
flags.  Once you indent, the first line is a parameter declaration;
subsequent lines at that indent are parameter flags.  Indent one more
time for the lines of the parameter docstring.  Finally, dedent back
to the same level as the function declaration for the function
docstring.


Function Declaration
--------------------

The return annotation is optional.  If skipped, the arrow ("``->``")
must also be omitted.


Parameter Declaration
---------------------

The "type" is a C type.  If it's a pointer type, you must specify a
single space between the type and the "``*``", and zero spaces between
the "``*``" and the name.  (e.g. "``PyObject *foo``", not "``PyObject*
foo``")

The "name" must be a legal C identifier.

The "default" is a Python value.  Default values are optional; if not
specified you must omit the equals sign too.  Parameters which don't
have a default are implicitly required.  The default value is
dynamically assigned, "live" in the generated C code, and although
it's specified as a Python value, it's translated into a native C
value in the generated C code.

It's explicitly permitted to end the parameter declaration line with a
semicolon, though the semicolon is optional.  This is intended to
allow directly cutting and pasting in declarations from C code.
However, the preferred style is without the semicolon.


Flags
-----

"Flags" are like "``make -D``" arguments.  They're unordered.  Flags
lines are parsed much like the shell (specifically, using
``shlex.split()`` [5]_ ).  You can have as many flag lines as you
like.  Specifying a flag twice is currently an error.

Supported flags for functions:

``basename``
  The basename to use for the generated C functions.  By default this
  is the name of the function from the DSL, only with periods replaced
  by underscores.

``positional-only``
  This function only supports positional parameters, not keyword
  parameters.  See `Functions With Positional-Only Parameters`_ below.

Supported flags for parameters:

``bitwise``
  If the Python integer passed in is signed, copy the bits directly
  even if it is negative.  Only valid for unsigned integer types.

``converter``
  Backwards-compatibility support for parameter "converter"
  functions. [6]_ The value should be the name of the converter
  function in C.  Only valid when the type of the parameter is
  ``void *``.

``default``
  The Python value to use in place of the parameter's actual default
  in Python contexts.  Specifically, when specified, this value will
  be used for the parameter's default in the docstring, and in the
  ``Signature``.  (TBD: If the string is a valid Python expression
  which can be rendered into a Python value using ``eval()``, then the
  result of ``eval()`` on it will be used as the default in the
  ``Signature``.)  Ignored if there is no default.

``encoding``
  Encoding to use when encoding a Unicode string to a ``char *``.
  Only valid when the type of the parameter is ``char *``.

``group=``
  This parameter is part of a group of options that must either all be
  specified or none specified.  Parameters in the same "group" must be
  contiguous.  The value of the group flag is the name used for the
  group variable, and therefore must be legal as a C identifier.  Only
  valid for functions marked "``positional-only``"; see `Functions
  With Positional-Only Parameters`_ below.

``immutable``
  Only accept immutable values.

``keyword-only``
  This parameter (and all subsequent parameters) is keyword-only.
  Keyword-only parameters must also be optional parameters.  Not valid
  for positional-only functions.

``length``
  This is an iterable type, and we also want its length.  The DSL will
  generate a second ``Py_ssize_t`` variable; its name will be this
  parameter's name appended with "``_length``".

``nullable``
  ``None`` is a legal argument for this parameter.  If ``None`` is
  supplied on the Python side, the equivalent C argument will be
  ``NULL``.  Only valid for pointer types.

``required``
  Normally any parameter that has a default value is automatically
  optional.  A parameter that has "required" set will be considered
  required (non-optional) even if it has a default value.  The
  generated documentation will also not show any default value.

``types``
  Space-separated list of acceptable Python types for this object.
  There are also four special-case types which represent Python
  protocols:

    * buffer
    * mapping
    * number
    * sequence

``zeroes``
  This parameter is a string type, and its value should be allowed to
  have embedded zeroes.  Not valid for all varieties of string
  parameters.


Python Code
-----------

Argument Clinic also permits embedding Python code inside C files,
which is executed in-place when Argument Clinic processes the file.
Embedded code looks like this:

::

    /*[python]

    # this is python code!
    print("/" + "* Hello world! *" + "/")

    [python]*/

Any Python code is valid.  Python code sections in Argument Clinic can
also be used to modify Clinic's behavior at runtime; for example, see
`Extending Argument Clinic`_.


Output
======

Argument Clinic writes its output in-line in the C file, immediately
after the section of Clinic code.  For "python" sections, the output
is everything printed using ``builtins.print``.  For "clinic"
sections, the output is valid C code, including:

  * a ``#define`` providing the correct ``methoddef`` structure for the
    function
  * a prototype for the "impl" function -- this is what you'll write
    to implement this function
  * a function that handles all argument processing, which calls your
    "impl" function
  * the definition line of the "impl" function
  * and a comment indicating the end of output.

The intention is that you will write the body of your impl function
immediately after the output -- as in, you write a left-curly-brace
immediately after the end-of-output comment and write the
implementation of the builtin in the body there.  (It's a bit strange
at first, but oddly convenient.)

Argument Clinic will define the parameters of the impl function for
you.  The function will take the "self" parameter passed in
originally, all the parameters you define, and possibly some extra
generated parameters ("length" parameters; also "group" parameters,
see next section).

Argument Clinic also writes a checksum for the output section.  This
is a valuable safety feature: if you modify the output by hand, Clinic
will notice that the checksum doesn't match, and will refuse to
overwrite the file.  (You can force Clinic to overwrite with the
"``-f``" command-line argument; Clinic will also ignore the checksums
when using the "``-o``" command-line argument.)


Functions With Positional-Only Parameters
=========================================

A significant fraction of Python builtins implemented in C use the
older positional-only API for processing arguments
(``PyArg_ParseTuple()``).  In some instances, these builtins parse
their arguments differently based on how many arguments were passed
in.  This can provide some bewildering flexibility: there may be
groups of optional parameters, which must either all be specified or
none specified.  And occasionally these groups are on the *left!* (For
example: ``curses.window.addch()``.)

Argument Clinic supports these legacy use-cases with a special set of
flags.  First, set the flag "``positional-only``" on the entire
function.  Then, for every group of parameters that is collectively
optional, add a "``group=``" flag with a unique string to all the
parameters in that group.  Note that these groups are permitted on the
right *or left* of any required parameters!  However, all groups
(including the group of required parameters) must be contiguous.

The impl function generated by Clinic will add an extra parameter for
every group, "``int <group>_group``".  This argument will be nonzero
if the group was specified on this call, and zero if it was not.

Note that when operating in this mode, you cannot specify default
arguments.  You can simulate defaults by putting parameters in
individual groups and detecting whether or not they were specified;
generally speaking it's better to simply not use "positional-only"
where it isn't absolutely necessary.  (TBD: It might be possible to
relax this restriction.  But adding default arguments into the mix of
groups would seemingly make calculating which groups are active a good
deal harder.)

Also, note that it's possible to specify a set of groups to a function
such that there are several valid mappings from the number of
arguments to a valid set of groups.  If this happens, Clinic will exit
with an error message.  This should not be a problem, as
positional-only operation is only intended for legacy use cases, and
all the legacy functions using this quirky behavior should have
unambiguous mappings.


Current Status
==============

As of this writing, there is a working prototype implementation of
Argument Clinic available online. [7]_ The prototype implements the
syntax above, and generates code using the existing ``PyArg_Parse``
APIs.  It supports translating to all current format units except
``"w*"``.  Sample functions using Argument Clinic exercise all major
features, including positional-only argument parsing.


Extending Argument Clinic
-------------------------

The prototype also currently provides an experimental extension
mechanism, allowing adding support for new types on-the-fly.  See
``Modules/posixmodule.c`` in the prototype for an example of its use.


Notes / TBD
===========

* Guido proposed having the "function docstring" be hand-written inline,
  in the middle of the output, something like this:

  ::

   /*[clinic]
     ... prototype and parameters (including parameter docstrings) go here
   [clinic]*/
   ... some output ...
   /*[clinic docstring start]*/
   ... hand-edited function docstring goes here   <-- you edit this by hand!
   /*[clinic docstring end]*/
   ... more output
   /*[clinic output end]*/

  I tried it this way and don't like it -- I think it's clumsy.  I
  prefer that everything you write goes in one place, rather than
  having an island of hand-edited stuff in the middle of the DSL
  output.

* Do we need to support tuple unpacking?  (The "``(OOO)``" style
  format string.)  Boy I sure hope not.

* What about Python functions that take no arguments?  This syntax
  doesn't provide for that.  Perhaps a lone indented "None" should
  mean "no arguments"?

* This approach removes some dynamism / flexibility.  With the
  existing syntax one could theoretically pass in different encodings
  at runtime for the "``es``"/"``et``" format units.  AFAICT CPython
  doesn't do this itself, however it's possible external users might
  do this.  (Trivia: there are no uses of "``es``" exercised by
  regrtest, and all the uses of "``et``" exercised are in
  socketmodule.c, except for one in _ssl.c.  They're all static,
  specifying the encoding ``"idna"``.)

* Right now the "basename" flag on a function changes the ``#define
  methoddef`` name too.  Should it, or should the #define'd methoddef
  name always be ``{module_name}_{function_name}`` ?


References
==========

.. [1] ``PyArg_ParseTuple()``:
   http://docs.python.org/3/c-api/arg.html#PyArg_ParseTuple

.. [2] ``PyArg_ParseTupleAndKeywords()``:
   http://docs.python.org/3/c-api/arg.html#PyArg_ParseTupleAndKeywords

.. [3] ``PyArg_`` format units:
   http://docs.python.org/3/c-api/arg.html#strings-and-buffers

.. [4] Keyword parameters for extension functions:
   http://docs.python.org/3/extending/extending.html#keyword-parameters-for-extension-functions

.. [5] ``shlex.split()``:
   http://docs.python.org/3/library/shlex.html#shlex.split

.. [6] ``PyArg_`` "converter" functions, see ``"O&"`` in this section:
   http://docs.python.org/3/c-api/arg.html#other-objects

.. [7] Argument Clinic prototype:
   https://bitbucket.org/larry/python-clinic/


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: