python-peps/pep-0570.rst

PEP: 570
Title: Python Positional-Only Parameters
Version: $Revision$
Last-Modified: $Date$
Author: Larry Hastings <larry@hastings.org>,
        Pablo Galindo <pablogsal@gmail.com>,
        Mario Corchero <mariocj89@gmail.com>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Jan-2018


========
Overview
========

This PEP proposes a syntax for positional-only parameters in Python.
Positional-only parameters are parameters without an externally-usable
name; when a function accepting positional-only parameters is called,
positional arguments are mapped to these parameters based solely on
their position.

=========
Rationale
=========

Python has always supported positional-only parameters.
Early versions of Python lacked the concept of specifying
parameters by name, so naturally, all parameters were
positional-only.  This changed around Python 1.0 when
all parameters suddenly became positional-or-keyword.
This change allowed users to provide arguments to a function
either positionally or referencing the keyword used in the
function's definition. However, this is not always desirable,
and in fact even in current versions of Python many CPython
"builtin" functions still only accept positional-only
arguments.

Users might want to restrict their API to not allow for parameters
to be referenced via keywords, as that exposes the name of the
parameter as part of the API. If a user of said API starts using the
argument by keyword when calling it and then the parameter
gets renamed, it will be a breaking change. By using positional-only
parameters the developer can later change the name of any arguments or
transform them to ``*args`` without breaking the API.

Even if making arguments positional-only in a function can be achieved
by using ``*args`` parameters and extracting them one by one,
the solution is far from ideal and not as expressive as the one
proposed in this PEP, which targets providing syntax to specify
accepting a specific number of positional-only parameters. Also,
it makes the signature of the function ambiguous as users won't
know how many parameters the function takes by looking at ``help()``
or auto-generated documentation.

Additionally, this will bridge the gap we currently find between
builtin functions that can specify positional-only
parameters and pure Python implementations that lack the
syntax for it. The '/' syntax is already exposed in the
documentation of some builtins and interfaces generated by
the argument clinic. 

Making positional-only arguments a possibility in Python will make the
language more consistent and since it would be a normal feature of Python
rather than a feature exclusive to extension modules, it should reduce
surprise and confusion by users encountering functions with positional-only
arguments. Notably, major third-party packages are already using the "/"
notation in their interfaces [#numpy-ufuncs]_ [#scipy-gammaln]_.

Positional-only arguments may be useful in several situations. One of the more
extreme situations is in a function that can take any keyword parameter but
also can take a positional one. Well-known examples for this situation are
``Formatter.format`` and ``dict.update``. For instance, ``dict.update``
accepts a dictionary (positionally) and/or any set of keyword parameters to use
as key/value pairs. In this case, if the dictionary parameter were not
positional-only, the user could not use the name that the interface uses for
said parameter or, conversely, the function could not distinguish easily if
the parameter received is the dictionary or one key/value pair.

Another important scenario is when argument names do not have semantic meaning.
For example, let's say we want to create a function that converts from one type
to another::

    def as_my_type(x):
        ...

The name of the parameter provides no value whatsoever, and forces
the developer to maintain its name forever, as users might pass ``x`` as a
keyword.

Another good example is an API that wants make it clear that one of its
parameters is the "main" argument through positional-only arguments.
For example, see::

    def add_to_queue(item: QueueItem):
        ...

Again we get no value from using keyword arguments here, and it can limit
future evolution of the API. Say at a later time we want this function
to be able to take multiple items while preserving backwards compatibility::

    def add_to_queue(items: Union[QueueItem, List[QueueItem]]):
        ...

or to take them by using argument lists::

    def add_to_queue(*items: QueueItem):
        ...

we will be forced to always keep the original argument or we would
potentially break users. By being able to define positional-only arguments,
we can change the name of the parameters at will or even change them to
``*args`` as in the previous example. There are multiple interfaces in the
standard library that fall into this category, for example the "main"
argument of ``collections.defaultdict`` (called *default_factory* in its
documentation) can only be passed positionally. One special case of this
situation is the *self* parameter for class methods: it is undersired that
a user can bind by keyword to the name "self" when calling the method from
the class::

    io.FileIO.write(self=f, b=b"data")

Indeed, interfaces from the standard library implemented in C usually take
"self" as a positional-only argument::

    >>> help(io.FileIO.write)
    Help on method_descriptor:

    write(self, b, /)
        Write buffer b to file, return number of bytes written.

Another essential aspect to consider is PEP 399 [#PEP399]_, that mandates
that pure Python versions of modules in the standard library *must* have the
same interface and semantics that the accelerator modules implemented in C
().  For example, if ``collections.defaultdict`` were to have a pure Python
implementation it would need to make use of positional-only parameters to
match the interface of its C counterpart. A more detailed discussion about
this topic can be found in the Motivation_ section.

---------------------------------------------------
Positional-Only Parameter Semantics In Python Today
---------------------------------------------------

There are many, many examples of functions that only accept positional-only
parameters in the standard library. The resulting semantics are easily
experienced by the Python programmer -- just try calling one, specifying its
arguments by name::


    >>> help(pow)
    ...
    pow(x, y, z=None, /)
    ...
    >>> pow(x=5, y=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pow() takes no keyword arguments

``pow()`` clearly expresses that its arguments are only positional
via the ``/`` marker, but this at the moment is only a documentation convention,
Python developers cannot write such syntax.

Besides, there are some functions with particularly
interesting semantics:

* ``range()``, which accepts an optional parameter
  to the *left* of its required parameter. [#RANGE]_

* ``dict()``, whose mapping/iterator parameter is optional and semantically
  must be positional-only.  Any externally visible name for this parameter
  would occlude that name going into the ``**kwarg`` keyword variadic
  parameter dict! [#DICT]_

One can simulate any of these in pure Python code
by accepting ``(*args, **kwargs)`` and parsing the arguments
by hand.  However, this results in a disconnect between the
Python function signature and what the function accepts,
not to mention the work of implementing said argument parsing
and the lack of clarity in the resulting signature.

As mentioned before, this syntax is already being used outside the
CPython code base for similar use cases [#numpy-ufuncs]_ [#scipy-gammaln]_,
remarking that these scenarios are not exclusive to CPython and the
standard library.

Currently users are surprised when first encountering this notation, but this
is to be expected given that it has only recently been adequately documented
[#document-positional-only]_, and it is not possible to use it in Python code.
For these reasons, this notation is currently an oddity that appears only in
CPython's APIs developed in C. Documenting the notation and making it possible
to be used in Python code will certainly eliminate this problem.

==========
Motivation
==========

.. _Motivation:

The new syntax will allow developers to further control how their
API can be consumed. It will allow restricting certain arguments
to be positional-only, so they cannot be passed with a keyword.

A similar PEP with a broader scope (PEP 457) was proposed earlier
to define the syntax. This PEP builds partially on top of that,
to define and provide an implementation for the ``/`` syntax in
function signatures.

In addition to the API benefits outlined earlier in this document,
positional-only arguments are also faster, as demonstrated in this thread
about converting keyword arguments to positional:
[#thread-keyword-to-positional]_. In fact, because of these benefits there has
even been a recent trend towards moving builtins away from keyword arguments:
recently, backwards-incompatible changes were made to disallow keyword
arguments to ``bool``, ``float``, ``list``, ``int``, ``tuple``.

Providing a way to specify positional-only arguments in Python will make it
easier to maintain pure Python implementations of C modules and will allow
users to take advantage of these benefits even in code written only in Python.
It will also encourage users to start with positional-only arguments when they
believe that passing a keyword argument provides no clarity; unlike making a
keyword argument positional-only, allowing a positional argument to be passed
positionally is not a breaking change.

This is a well discussed, recurring topic on the Python mailing lists:

* September 2018: `Anders Hovmöller: [Python-ideas] Positional-only
  parameters
  <https://mail.python.org/pipermail/python-ideas/2018-September/053233.html>`_
* February 2017: `Victor Stinner: [Python-ideas] Positional-only
  parameters
  <https://mail.python.org/pipermail/python-ideas/2017-February/044879.html>`_,
  `discussion continued in March
  <https://mail.python.org/pipermail/python-ideas/2017-March/044956.html>`_
* February 2017: [#python-ideas-decorator-based]_
* March 2012: [#GUIDO]_
* May 2007: `George Sakkis: [Python-ideas] Positional only arguments
  <https://mail.python.org/pipermail/python-ideas/2007-May/000704.html>`_
* May 2006: `Benji York: [Python-Dev] Positional-only Arguments
  <https://mail.python.org/pipermail/python-dev/2006-May/064790.html>`_

Positional-only parameters have also the (minor) advantage of enforcing
some logical order when calling interfaces that make use of them. For
example, the ``range`` function takes all its parameters positionally and
this disallows forms like::

    range(stop=5, start=0, step=2)
    range(stop=5, step=2, start=0)
    range(step=2, start=0, stop=5)
    range(step=2, stop=5, start=0)

at the price of disallowing the use of keyword arguments for the (unique)
intended order::

    range(start=0, stop=5, step=2)

Another critical aspect that motivates positional-only arguments is
PEP 399 [#PEP399]_: Pure Python/C Accelerator Module Compatibility
Requirements.  This PEP states that :

    This PEP requires that in these instances that the C code must pass
    the test suite used for the pure Python code to act as much as
    a drop-in replacement as reasonably possible

It is clear that if the C code is implemented using the existing capabilities
to implement positional-only parameters using the argument clinic and
related machinery, it is not possible for the pure Python counterpart
to match the provided interface and requirements. This also creates a disparity
between the interfaces of some functions and classes in the CPython standard
library and other Python implementations. For example::

    $ python3 # CPython 3.7.2
    >>> import binascii; binascii.crc32(data=b'data')
    TypeError: crc32() takes no keyword arguments

    $ pypy3 # PyPy 6.0.0
    >>>> import binascii; binascii.crc32(data=b'data')
    2918445923

Other Python implementations can, of course, reproduce the CPython APIs
manually, but this goes against the spirit of PEP 399 [#PEP399]_ that
intends to avoid duplication of effort by mandating that all modules added
to Python's standard library **must** have a pure Python implementation
with the same interface and semantics.

A final argument in favor of positional-only arguments is that they allow
some new optimizations like the ones already present in the argument clinic
since said parameters must be passed in strict order. For instance, CPython's
internal *METH_FASTCALL* calling convention has been recently speciallized
for functions with positional-only parameters to eliminate the cost for
handling empty keywords. Similar performance improvements can be
applied when creating the evaluation frame of Python functions thanks to
positional-only parameters.

=================================================================
The Current State Of Documentation For Positional-Only Parameters
=================================================================

The documentation for positional-only parameters is incomplete
and inconsistent:

* Some functions denote optional groups of positional-only arguments
  by enclosing them in nested square brackets. [#BORDER]_

* Some functions denote optional groups of positional-only arguments
  by presenting multiple prototypes with varying numbers of
  arguments. [#SENDFILE]_

* Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_

One more important idea to consider: currently in the documentation
there is no way to tell whether a function takes positional-only
parameters.  ``open()`` accepts keyword arguments, ``ord()`` does
not, but there is no way of telling just by reading the
documentation.

====================
Syntax And Semantics
====================

From the "ten-thousand foot view", and ignoring ``*args`` and ``**kwargs``
for now, the grammar for a function definition currently looks like this::

    def name(positional_or_keyword_parameters, *, keyword_only_parameters):

Building on that perspective, the new syntax for functions would look
like this::

    def name(positional_only_parameters, /, positional_or_keyword_parameters,
             *, keyword_only_parameters):

All parameters before the ``/`` are positional-only.  If ``/`` is
not specified in a function signature, that function does not
accept any positional-only parameters.
The logic around optional values for positional-only arguments
remains the same as for positional-or-keyword arguments. Once
a positional-only argument is provided with a default,
the following positional-only and positional-or-keyword arguments
need to have defaults as well. Positional-only parameters that
do not have a default values are *required* positional-only parameters.
Therefore the following are valid signatures::

    def name(p1, p2, /, p_or_kw, *, kw):
    def name(p1, p2=None, /, p_or_kw=None, *, kw):
    def name(p1, p2=None, /, *, kw):
    def name(p1, p2=None, /):
    def name(p1, p2, /, p_or_kw):
    def name(p1, p2, /):

While the followings are not::

    def name(p1, p2=None, /, p_or_kw, *, kw):
    def name(p1=None, p2, /, p_or_kw=None, *, kw):
    def name(p1=None, p2, /):

--------------------------------
Origin of the "/" as a separator
--------------------------------

Using the "/" as a separator was initially proposed by Guido van Rossum
in 2012 [#GUIDO]_ :

    Alternative proposal: how about using '/' ? It's kind of the opposite
    of '*' which means "keyword argument", and '/' is not a new character.

==========================
Full grammar specification
==========================

A draft of the proposed grammar specification is::

    new_typedargslist:
      tfpdef ['=' test] (',' tfpdef ['=' test])* ',' '/' [',' [typedargslist]] | typedargslist

    new_varargslist:
      vfpdef ['=' test] (',' vfpdef ['=' test])* ',' '/' [',' [varargslist]] | varargslist

It will be added to the actual ``typedargslist`` and ``varargslist``, but for
more relaxed discussion it is presented as ``new_typedargslist`` and
``new_varargslist``. Also, notice that using a construction with two new rules
(new_varargslist and new_varargslist) is not possible with the current parser
as a rule is not LL(1). This is the reason the rule needs to be included in
the existing typedargslist and varargslist (in the same way keyword-only
arguments were introduced).


==============
Implementation
==============

An initial implementation that passes the CPython test suite is available
for evaluation [#posonly-impl]_.

The advantages of this implementation involve speed, consistency with the
implementation of keyword-only parameters as in PEP 3102 and a simpler
implementation of all the tools and modules that will be impacted by
this change.

==============
Rejected Ideas
==============

----------
Do Nothing
----------

Always an option, just not adding it. It was considered
though that the benefits of adding it is worth the complexity
it adds to the language.

---------------------
After marker proposal
---------------------

A complaint against the proposal is the fact that the modifier of
the signature impacts the tokens already passed.

This might make it confusing to users to read functions
with many arguments. Example::

  def really_bad_example_of_a_python_function(fist_long_argument, second_long_argument,
                                              third_long_argument, /):

It is not until reaching the end of the signature that the reader
realises the ``/``, and therefore the fact that the arguments are
position-only. This deviates from how the keyword-only marker works.

That said we could not find an implementation that would modify the
arguments after the marker, as that will force the one before the
marker to be position-only as well. Example::

  def (x, y, /, z):

If we define that ``/`` makes only z position-only, it will not be possible
to call x and y via keyword argument. Finding a way to work around it
will add confusion given that at the moment keyword arguments cannot be
followed by positional arguments. ``/`` will, therefore, make both the
preceding and following parameters position-only.

-------------------
Per-argument marker
-------------------

Using a per-argument marker might be an option as well. The approach adds a
token to each of the arguments that are position only and requires those to be
placed together. Example::

  def (.arg1, .arg2, arg3):

Note the dot on arg1 and arg2. Even if this approach might look easier
to read, it has been discarded as ``/`` goes further in line with the
keyword-only approach and is less error-prone.

Some libraries use leading underscore [#leading-underscore]_
to mark those arguments as positional-only.

----------------
Using decorators
----------------

It has been suggested on python-ideas [#python-ideas-decorator-based]_ to
provide a decorator written in Python as an implementation for this feature.
This approach has the advantage that keeps parameter declaration more easy to
read but also introduces an asymmetry on how parameter behaviour is declared.
Also, as the ``/`` syntax is already introduced for C functions, this
inconsistency will make it more difficult to implement all tools and modules
that deal with this syntax including but not limited to, the argument clinic,
the inspect module and the ast module. Another disadvantage of this approach
is that calling the decorated functions will be slower than the functions
generated if the feature was implemented directly in C.

======
Thanks
======

Credit for some of the content of this PEP is contained in Larry Hastings’s
PEP 457.

Credit for the use of '/' as the separator between positional-only and
positional-or-keyword parameters go to Guido van Rossum, in a proposal from
2012. [#GUIDO]_

Credit for discussion about the simplification of the grammar goes to
Braulio Valdivieso.


.. [#numpy-ufuncs]
   https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs

.. [#scipy-gammaln]
   https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaln.html

.. [#DICT]
    http://docs.python.org/3/library/stdtypes.html#dict

.. [#RANGE]
    http://docs.python.org/3/library/functions.html#func-range

.. [#BORDER]
    http://docs.python.org/3/library/curses.html#curses.window.border

.. [#SENDFILE]
    http://docs.python.org/3/library/os.html#os.sendfile

.. [#ADDCH]
    http://docs.python.org/3/library/curses.html#curses.window.addch

.. [#GUIDO]
   Guido van Rossum, posting to python-ideas, March 2012:
   https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
   and
   https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
   and
   https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

.. [#PEP399]
   https://www.python.org/dev/peps/pep-0399/

.. [#python-ideas-decorator-based]
   https://mail.python.org/pipermail/python-ideas/2017-February/044888.html

.. [#posonly-impl]
   https://github.com/pablogsal/cpython_positional_only

.. [#thread-keyword-to-positional]
   https://mail.python.org/pipermail/python-ideas/2016-January/037874.html

.. [#leading-underscore]
   https://mail.python.org/pipermail/python-ideas/2018-September/053319.html

.. [#document-positional-only]
   https://bugs.python.org/issue21314

=========
Copyright
=========

This document has been placed in the public domain.