python-peps/pep-0562.rst

PEP: 562
Title: Module __getattr__ and __dir__
Author: Ivan Levkivskyi <levkivskyi@gmail.com>
Status: Accepted
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017
Resolution: https://mail.python.org/pipermail/python-dev/2017-December/151033.html


Abstract
========

It is proposed to support ``__getattr__`` and ``__dir__`` function defined
on modules to provide basic customization of module attribute access.



Rationale
=========

It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning ``__class__`` of a module object
to a custom subclass of ``types.ModuleType`` or replacing the ``sys.modules``
item with a custom wrapper instance. It would be convenient to simplify this
procedure by recognizing ``__getattr__`` defined directly in a module that
would act like a normal ``__getattr__`` method, except that it will be defined
on module *instances*. For example::

  # lib.py

  from warnings import warn

  deprecated_names = ["old_function", ...]

  def _deprecated_old_function(arg, other):
      ...

  def __getattr__(name):
      if name in deprecated_names:
          warn(f"{name} is deprecated", DeprecationWarning)
          return globals()[f"_deprecated_{name}"]
      raise AttributeError(f"module {__name__} has no attribute {name}")

  # main.py

  from lib import old_function  # Works, but emits the warning

Another widespread use case for ``__getattr__`` would be lazy submodule
imports. Consider a simple example::

  # lib/__init__.py

  import importlib

  __all__ = ['submod', ...]

  def __getattr__(name):
      if name in __all__:
          return importlib.import_module("." + name, __name__)
      raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

  # lib/submod.py

  print("Submodule loaded")
  class HeavyClass:
      ...

  # main.py

  import lib
  lib.submodule.HeavyClass  # prints "Submodule loaded"

There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP proposes
a faster and simpler mechanism, but provides more basic customization.
An additional motivation for this proposal is that PEP 484 already defines
the use of module ``__getattr__`` for this purpose in Python stub files,
see [1]_.

In addition, to allow modifying result of a ``dir()`` call on a module
to show deprecated and other dynamically generated attributes, it is
proposed to support module level ``__dir__`` function. For example::

  # lib.py

  deprecated_names = ["old_function", ...]
  __all__ = ["new_function_one", "new_function_two", ...]

  def new_function_one(arg, other):
     ...
  def new_function_two(arg, other):
      ...

  def __dir__():
      return sorted(__all__ + deprecated_names)

  # main.py

  import lib

  dir(lib)  # prints ["new_function_one", "new_function_two", "old_function", ...]


Specification
=============

The ``__getattr__`` function at the module level should accept one argument
which is the name of an attribute and return the computed value or raise
an ``AttributeError``::

  def __getattr__(name: str) -> Any: ...

If an attribute is not found on a module object through the normal lookup
(i.e. ``object.__getattribute__``), then ``__getattr__`` is searched in
the module ``__dict__`` before raising an ``AttributeError``. If found, it is
called with the attribute name and the result is returned. Looking up a name
as a module global will bypass module ``__getattr__``. This is intentional,
otherwise calling ``__getattr__`` for builtins will significantly harm
performance.

The ``__dir__`` function should accept no arguments, and return
a list of strings that represents the names accessible on module::

  def __dir__() -> List[str]: ...

If present, this function overrides the standard ``dir()`` search on
a module.

The reference implementation for this PEP can be found in [2]_.


Backwards compatibility and impact on performance
=================================================

This PEP may break code that uses module level (global) names ``__getattr__``
and ``__dir__``.  (But the language reference explicitly reserves *all*
undocumented dunder names, and allows "breakage without warning"; see [3]_.)
The performance implications of this PEP are minimal, since ``__getattr__``
is called only for missing attributes.

Some tools that perform module attributes discovery might not expect
``__getattr__``. This problem is not new however, since it is already possible
to replace a module with a module subclass with overridden ``__getattr__`` and
``__dir__``, but with this PEP such problems can occur more often.


Discussion
==========

Note that the use of module ``__getattr__`` requires care to keep the referred
objects pickleable. For example, the ``__name__`` attribute of a function
should correspond to the name with which it is accessible via
``__getattr__``::

  def keep_pickleable(func):
      func.__name__ = func.__name__.replace('_deprecated_', '')
      func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
      return func

  @keep_pickleable
  def _deprecated_old_function(arg, other):
      ...

One should be also careful to avoid recursion as one would do with
a class level ``__getattr__``.

To use a module global with triggering ``__getattr__`` (for example if one
wants to use a lazy loaded submodule) one can access it as::

    sys.modules[__name__].some_global

or as::

    from . import some_global

Note that the latter sets the module attribute, thus ``__getattr__`` will be
called only once.


References
==========

.. [1] PEP 484 section about ``__getattr__`` in stub files
   (https://www.python.org/dev/peps/pep-0484/#stub-files)

.. [2] The reference implementation
   (https://github.com/ilevkivskyi/cpython/pull/3/files)

.. [3] Reserved classes of identifiers
   (https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: