Add PEP 519: Adding a file system path protocol
This commit is contained in:
Brett Cannon
parent
c856ebcc9e
commit
992c83274c
|
|
@ -0,0 +1,516 @@
|
|||
PEP: 519
|
||||
Title: Adding a file system path protocol
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Brett Cannon <brett@python.org>,
|
||||
Koos Zevenhoven <k7hoven@gmail.com>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 11-May-2016
|
||||
Post-History: 11-May-2016,
|
||||
12-May-2016,
|
||||
13-May-2016
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes a protocol for classes which represent a file system
|
||||
path to be able to provide a ``str`` or ``bytes`` representation.
|
||||
Changes to Python's standard library are also proposed to utilize this
|
||||
protocol where appropriate to facilitate the use of path objects where
|
||||
historically only ``str`` and/or ``bytes`` file system paths are
|
||||
accepted. The goal is to facilitate the migration of users towards
|
||||
rich path objects while providing an easy way to work with code
|
||||
expecting ``str`` or ``bytes``.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
Historically in Python, file system paths have been represented as
|
||||
strings or bytes. This choice of representation has stemmed from C's
|
||||
own decision to represent file system paths as
|
||||
``const char *`` [#libc-open]_. While that is a totally serviceable
|
||||
format to use for file system paths, it's not necessarily optimal. At
|
||||
issue is the fact that while all file system paths can be represented
|
||||
as strings or bytes, not all strings or bytes represent a file system
|
||||
path. This can lead to issues where any e.g. string duck-types to a
|
||||
file system path whether it actually represents a path or not.
|
||||
|
||||
To help elevate the representation of file system paths from their
|
||||
representation as strings and bytes to a richer object representation,
|
||||
the pathlib module [#pathlib]_ was provisionally introduced in
|
||||
Python 3.4 through PEP 428. While considered by some as an improvement
|
||||
over strings and bytes for file system paths, it has suffered from a
|
||||
lack of adoption. Typically the key issue listed for the low adoption
|
||||
rate has been the lack of support in the standard library. This lack
|
||||
of support required users of pathlib to manually convert path objects
|
||||
to strings by calling ``str(path)`` which many found error-prone.
|
||||
|
||||
One issue in converting path objects to strings comes from
|
||||
the fact that the only generic way to get a string representation of
|
||||
the path was to pass the object to ``str()``. This can pose a
|
||||
problem when done blindly as nearly all Python objects have some
|
||||
string representation whether they are a path or not, e.g.
|
||||
``str(None)`` will give a result that
|
||||
``builtins.open()`` [#builtins-open]_ will happily use to create a new
|
||||
file.
|
||||
|
||||
Exacerbating this whole situation is the
|
||||
``DirEntry`` object [#os-direntry]_. While path objects have a
|
||||
representation that can be extracted using ``str()``, ``DirEntry``
|
||||
objects expose a ``path`` attribute instead. Having no common
|
||||
interface between path objects, ``DirEntry``, and any other
|
||||
third-party path library has become an issue. A solution that allows
|
||||
any path-representing object to declare that it is a path and a way
|
||||
to extract a low-level representation that all path objects could
|
||||
support is desired.
|
||||
|
||||
This PEP then proposes to introduce a new protocol to be followed by
|
||||
objects which represent file system paths. Providing a protocol allows
|
||||
for explicit signaling of what objects represent file system paths as
|
||||
well as a way to extract a lower-level representation that can be used
|
||||
with older APIs which only support strings or bytes.
|
||||
|
||||
Discussions regarding path objects that led to this PEP can be found
|
||||
in multiple threads on the python-ideas mailing list archive
|
||||
[#python-ideas-archive]_ for the months of March and April 2016 and on
|
||||
the python-dev mailing list archives [#python-dev-archive]_ during
|
||||
April 2016.
|
||||
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
This proposal is split into two parts. One part is the proposal of a
|
||||
protocol for objects to declare and provide support for exposing a
|
||||
file system path representation. The other part deals with changes to
|
||||
Python's standard library to support the new protocol. These changes
|
||||
will also lead to the pathlib module dropping its provisional status.
|
||||
|
||||
Protocol
|
||||
--------
|
||||
|
||||
The following abstract base class defines the protocol for an object
|
||||
to be considered a path object::
|
||||
|
||||
import abc
|
||||
import typing as t
|
||||
|
||||
|
||||
class PathLike(abc.ABC):
|
||||
|
||||
"""Abstract base class for implementing the file system path protocol."""
|
||||
|
||||
@abc.abstractmethod
|
||||
def __fspath__(self) -> t.Union[str, bytes]:
|
||||
"""Return the file system path representation of the object."""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
Objects representing file system paths will implement the
|
||||
``__fspath__()`` method which will return the ``str`` or ``bytes``
|
||||
representation of the path. The ``str`` representation is the
|
||||
preferred low-level path representation as it is human-readable and
|
||||
what people historically represent paths as.
|
||||
|
||||
|
||||
Standard library changes
|
||||
------------------------
|
||||
|
||||
It is expected that most APIs in Python's standard library that
|
||||
currently accept a file system path will be updated appropriately to
|
||||
accept path objects (whether that requires code or simply an update
|
||||
to documentation will vary). The modules mentioned below, though,
|
||||
deserve specific details as they have either fundamental changes that
|
||||
empower the ability to use path objects, or entail additions/removal
|
||||
of APIs.
|
||||
|
||||
|
||||
builtins
|
||||
''''''''
|
||||
|
||||
``open()`` [#builtins-open]_ will be updated to accept path objects as
|
||||
well as continue to accept ``str`` and ``bytes``.
|
||||
|
||||
|
||||
os
|
||||
'''
|
||||
|
||||
The ``fspath()`` function will be added with the following semantics::
|
||||
|
||||
import typing as t
|
||||
|
||||
|
||||
def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
|
||||
"""Return the string representation of the path.
|
||||
|
||||
If str or bytes is passed in, it is returned unchanged.
|
||||
"""
|
||||
if isinstance(path, (str, bytes)):
|
||||
return path
|
||||
|
||||
# Work from the object's type to match method resolution of other magic
|
||||
# methods.
|
||||
path_type = type(path)
|
||||
try:
|
||||
return path_type.__fspath__(path)
|
||||
except AttributeError:
|
||||
if hasattr(path_type, '__fspath__'):
|
||||
raise
|
||||
|
||||
raise TypeError("expected str, bytes or os.PathLike object, not "
|
||||
+ path_type.__name__)
|
||||
|
||||
The ``os.fsencode()`` [#os-fsencode]_ and
|
||||
``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
|
||||
path objects. As both functions coerce their arguments to
|
||||
``bytes`` and ``str``, respectively, they will be updated to call
|
||||
``__fspath__()`` if present to convert the path object to a ``str`` or
|
||||
``bytes`` representation, and then perform their appropriate
|
||||
coercion operations as if the return value from ``__fspath__()`` had
|
||||
been the original argument to the coercion function in question.
|
||||
|
||||
The addition of ``os.fspath()``, the updates to
|
||||
``os.fsencode()``/``os.fsdecode()``, and the current semantics of
|
||||
``pathlib.PurePath`` provide the semantics necessary to
|
||||
get the path representation one prefers. For a path object,
|
||||
``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
|
||||
``bytes`` representation without any coersion, then ``os.fspath()``
|
||||
can be used. If a ``str`` is desired and the encoding of ``bytes``
|
||||
should be assumed to be the default file system encoding, then
|
||||
``os.fsdecode()`` should be used. If a ``bytes`` representation is
|
||||
desired and any strings should be encoded using the default file
|
||||
system encoding, then ``os.fsencode()`` is used. This PEP recommends
|
||||
using path objects when possible and falling back to string paths as
|
||||
necessary and using ``bytes`` as a last resort.
|
||||
|
||||
Another way to view this is as a hierarchy of file system path
|
||||
representations (highest- to lowest-level): path → str → bytes. The
|
||||
functions and classes under discussion can all accept objects on the
|
||||
same level of the hierarchy, but they vary in whether they promote or
|
||||
demote objects to another level. The ``pathlib.PurePath`` class can
|
||||
promote a ``str`` to a path object. The ``os.fspath()`` function can
|
||||
demote a path object to a ``str`` or ``bytes`` instance, depending
|
||||
on what ``__fspath__()`` returns.
|
||||
The ``os.fsdecode()`` function will demote a path object to
|
||||
a string or promote a ``bytes`` object to a ``str``. The
|
||||
``os.fsencode()`` function will demote a path or string object to
|
||||
``bytes``. There is no function that provides a way to demote a path
|
||||
object directly to ``bytes`` while bypassing string demotion.
|
||||
|
||||
The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
|
||||
method. It will return the same value as currently found on the
|
||||
``path`` attribute of ``DirEntry`` instances.
|
||||
|
||||
The Protocol_ ABC will be added to the ``os`` module under the name
|
||||
``os.PathLike``.
|
||||
|
||||
|
||||
os.path
|
||||
'''''''
|
||||
|
||||
The various path-manipulation functions of ``os.path`` [#os-path]_
|
||||
will be updated to accept path objects. For polymorphic functions that
|
||||
accept both bytes and strings, they will be updated to simply use
|
||||
``os.fspath()``.
|
||||
|
||||
During the discussions leading up to this PEP it was suggested that
|
||||
``os.path`` not be updated using an "explicit is better than implicit"
|
||||
argument. The thinking was that since ``__fspath__()`` is polymorphic
|
||||
itself it may be better to have code working with ``os.path`` extract
|
||||
the path representation from path objects explicitly. There is also
|
||||
the consideration that adding support this deep into the low-level OS
|
||||
APIs will lead to code magically supporting path objects without
|
||||
requiring any documentation updated, leading to potential complaints
|
||||
when it doesn't work, unbeknownst to the project author.
|
||||
|
||||
But it is the view of this PEP that "practicality beats purity" in
|
||||
this instance. To help facilitate the transition to supporting path
|
||||
objects, it is better to make the transition as easy as possible than
|
||||
to worry about unexpected/undocumented duck typing support for
|
||||
path objects by projects.
|
||||
|
||||
There has also been the suggestion that ``os.path`` functions could be
|
||||
used in a tight loop and the overhead of checking or calling
|
||||
``__fspath__()`` would be too costly. In this scenario only
|
||||
path-consuming APIs would be directly updated and path-manipulating
|
||||
APIs like the ones in ``os.path`` would go unmodified. This would
|
||||
require library authors to update their code to support path objects
|
||||
if they performed any path manipulations, but if the library code
|
||||
passed the path straight through then the library wouldn't need to be
|
||||
updated. It is the view of this PEP and Guido, though, that this is an
|
||||
unnecessary worry and that performance will still be acceptable.
|
||||
|
||||
|
||||
pathlib
|
||||
'''''''
|
||||
|
||||
The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
|
||||
updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
|
||||
will continue to not accept ``bytes`` path representations, and so if
|
||||
``__fspath__()`` returns ``bytes`` it will raise an exception.
|
||||
|
||||
The ``path`` attribute will be removed as this PEP makes it
|
||||
redundant (it has not been included in any released version of Python
|
||||
and so is not a backwards-compatibility concern).
|
||||
|
||||
|
||||
C API
|
||||
'''''
|
||||
|
||||
The C API will gain an equivalent function to ``os.fspath()``::
|
||||
|
||||
/*
|
||||
Return the file system path representation of the object.
|
||||
|
||||
If the object is str or bytes, then allow it to pass through with
|
||||
an incremented refcount. If the object defines __fspath__(), then
|
||||
return the result of that method. All other types raise a TypeError.
|
||||
*/
|
||||
PyObject *
|
||||
PyOS_FSPath(PyObject *path)
|
||||
{
|
||||
_Py_IDENTIFIER(__fspath__);
|
||||
PyObject *func = NULL;
|
||||
PyObject *path_repr = NULL;
|
||||
|
||||
if (PyUnicode_Check(path) || PyBytes_Check(path)) {
|
||||
Py_INCREF(path);
|
||||
return path;
|
||||
}
|
||||
|
||||
func = _PyObject_LookupSpecial(path, &PyId___fspath__);
|
||||
if (NULL == func) {
|
||||
return PyErr_Format(PyExc_TypeError,
|
||||
"expected str, bytes or os.PathLike object, "
|
||||
"not %S",
|
||||
path->ob_type);
|
||||
}
|
||||
|
||||
path_repr = PyObject_CallFunctionObjArgs(func, NULL);
|
||||
Py_DECREF(func);
|
||||
return path_repr;
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
Backwards compatibility
|
||||
=======================
|
||||
|
||||
There are no explicit backwards-compatibility concerns. Unless an
|
||||
object incidentally already defines a ``__fspath__()`` method there is
|
||||
no reason to expect the pre-existing code to break or expect to have
|
||||
its semantics implicitly changed.
|
||||
|
||||
Libraries wishing to support path objects and a version of Python
|
||||
prior to Python 3.6 and the existence of ``os.fspath()`` can use the
|
||||
idiom of
|
||||
``path.__fspath__() if hasattr(path, "__fspath__") else path``.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
This is the task list for what this PEP proposes:
|
||||
|
||||
#. Remove the ``path`` attribute from pathlib
|
||||
#. Remove the provisional status of pathlib
|
||||
#. Add ``os.PathLike``
|
||||
#. Add ``os.fspath()``
|
||||
#. Add ``PyOS_FSPath()``
|
||||
#. Update ``os.fsencode()``
|
||||
#. Update ``os.fsdecode()``
|
||||
#. Update ``pathlib.PurePath`` and ``pathlib.Path``
|
||||
#. Update ``builtins.open()``
|
||||
#. Update ``os.DirEntry``
|
||||
#. Update ``os.path``
|
||||
#. Add a glossary entry for "path-like"
|
||||
|
||||
|
||||
Rejected Ideas
|
||||
==============
|
||||
|
||||
Other names for the protocol's method
|
||||
-------------------------------------
|
||||
|
||||
Various names were proposed during discussions leading to this PEP,
|
||||
including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
|
||||
the end people seemed to gravitate towards ``__fspath__`` for being
|
||||
unambiguous without being unnecessarily long.
|
||||
|
||||
|
||||
Separate str/bytes methods
|
||||
--------------------------
|
||||
|
||||
At one point it was suggested that ``__fspath__()`` only return
|
||||
strings and another method named ``__fspathb__()`` be introduced to
|
||||
return bytes. The thinking is that by making ``__fspath__()`` not be
|
||||
polymorphic it could make dealing with the potential string or bytes
|
||||
representations easier. But the general consensus was that returning
|
||||
bytes will more than likely be rare and that the various functions in
|
||||
the os module are the better abstraction to promote over direct
|
||||
calls to ``__fspath__()``.
|
||||
|
||||
|
||||
Providing a ``path`` attribute
|
||||
------------------------------
|
||||
|
||||
To help deal with the issue of ``pathlib.PurePath`` not inheriting
|
||||
from ``str``, originally it was proposed to introduce a ``path``
|
||||
attribute to mirror what ``os.DirEntry`` provides. In the end,
|
||||
though, it was determined that a protocol would provide the same
|
||||
result while not directly exposing an API that most people will never
|
||||
need to interact with directly.
|
||||
|
||||
|
||||
Have ``__fspath__()`` only return strings
|
||||
------------------------------------------
|
||||
|
||||
Much of the discussion that led to this PEP revolved around whether
|
||||
``__fspath__()`` should be polymorphic and return ``bytes`` as well as
|
||||
``str`` or only return ``str``. The general sentiment for this view
|
||||
was that ``bytes`` are difficult to work with due to their
|
||||
inherent lack of information about their encoding and PEP 383 makes
|
||||
it possible to represent all file system paths using ``str`` with the
|
||||
``surrogateescape`` handler. Thus, it would be better to forcibly
|
||||
promote the use of ``str`` as the low-level path representation for
|
||||
high-level path objects.
|
||||
|
||||
In the end, it was decided that using ``bytes`` to represent paths is
|
||||
simply not going to go away and thus they should be supported to some
|
||||
degree. The hope is that people will gravitate towards path objects
|
||||
like pathlib and that will move people away from operating directly
|
||||
with ``bytes``.
|
||||
|
||||
|
||||
A generic string encoding mechanism
|
||||
-----------------------------------
|
||||
|
||||
At one point there was a discussion of developing a generic mechanism
|
||||
to extract a string representation of an object that had semantic
|
||||
meaning (``__str__()`` does not necessarily return anything of
|
||||
semantic significance beyond what may be helpful for debugging). In
|
||||
the end, it was deemed to lack a motivating need beyond the one this
|
||||
PEP is trying to solve in a specific fashion.
|
||||
|
||||
|
||||
Have __fspath__ be an attribute
|
||||
-------------------------------
|
||||
|
||||
It was briefly considered to have ``__fspath__`` be an attribute
|
||||
instead of a method. This was rejected for two reasons. One,
|
||||
historically protocols have been implemented as "magic methods" and
|
||||
not "magic methods and attributes". Two, there is no guarantee that
|
||||
the lower-level representation of a path object will be pre-computed,
|
||||
potentially misleading users that there was no expensive computation
|
||||
behind the scenes in case the attribute was implemented as a property.
|
||||
|
||||
This also indirectly ties into the idea of introducing a ``path``
|
||||
attribute to accomplish the same thing. This idea has an added issue,
|
||||
though, of accidentally having any object with a ``path`` attribute
|
||||
meet the protocol's duck typing. Introducing a new magic method for
|
||||
the protocol helpfully avoids any accidental opting into the protocol.
|
||||
|
||||
|
||||
Provide specific type hinting support
|
||||
-------------------------------------
|
||||
|
||||
There was some consideration to provdinga generic ``typing.PathLike``
|
||||
class which would allow for e.g. ``typing.PathLike[str]`` to specify
|
||||
a type hint for a path object which returned a string representation.
|
||||
While potentially beneficial, the usefulness was deemed too small to
|
||||
bother adding the type hint class.
|
||||
|
||||
This also removed any desire to have a class in the ``typing`` module
|
||||
which represented the union of all acceptable path-representing types
|
||||
as that can be represented with
|
||||
``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
|
||||
is users will slowly gravitate to path objects only.
|
||||
|
||||
|
||||
Provide ``os.fspathb()``
|
||||
------------------------
|
||||
|
||||
It was suggested that to mirror the structure of e.g.
|
||||
``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
|
||||
``str`` and that another function named ``os.fspathb()`` be
|
||||
introduced that only returned ``bytes``. This was rejected as the
|
||||
purposes of the ``*b()`` functions are tied to querying the file
|
||||
system where there is a need to get the raw bytes back. As this PEP
|
||||
does not work directly with data on a file system (but which *may*
|
||||
be), the view was taken this distinction is unnecessary. It's also
|
||||
believed that the need for only bytes will not be common enough to
|
||||
need to support in such a specific manner as ``os.fsencode()`` will
|
||||
provide similar functionality.
|
||||
|
||||
|
||||
Call ``__fspath__()`` off of the instance
|
||||
-----------------------------------------
|
||||
|
||||
An earlier draft of this PEP had ``os.fspath()`` calling
|
||||
``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
|
||||
changed to be consistent with how other magic methods in Python are
|
||||
resolved.
|
||||
|
||||
|
||||
Acknowledgements
|
||||
================
|
||||
|
||||
Thanks to everyone who participated in the various discussions related
|
||||
to this PEP that spanned both python-ideas and python-dev. Special
|
||||
thanks to Stephen Turnbull for direct feedback on early drafts of this
|
||||
PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
|
||||
only feedback on early drafts of this PEP but also helping to drive
|
||||
the overall discussion on this topic across the two mailing lists.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [#python-ideas-archive] The python-ideas mailing list archive
|
||||
(https://mail.python.org/pipermail/python-ideas/)
|
||||
|
||||
.. [#python-dev-archive] The python-dev mailing list archive
|
||||
(https://mail.python.org/pipermail/python-dev/)
|
||||
|
||||
.. [#libc-open] ``open()`` documention for the C standard library
|
||||
(http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html)
|
||||
|
||||
.. [#pathlib] The ``pathlib`` module
|
||||
(https://docs.python.org/3/library/pathlib.html#module-pathlib)
|
||||
|
||||
.. [#builtins-open] The ``builtins.open()`` function
|
||||
(https://docs.python.org/3/library/functions.html#open)
|
||||
|
||||
.. [#os-fsencode] The ``os.fsencode()`` function
|
||||
(https://docs.python.org/3/library/os.html#os.fsencode)
|
||||
|
||||
.. [#os-fsdecode] The ``os.fsdecode()`` function
|
||||
(https://docs.python.org/3/library/os.html#os.fsdecode)
|
||||
|
||||
.. [#os-direntry] The ``os.DirEntry`` class
|
||||
(https://docs.python.org/3/library/os.html#os.DirEntry)
|
||||
|
||||
.. [#os-path] The ``os.path`` module
|
||||
(https://docs.python.org/3/library/os.path.html#module-os.path)
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue