369 lines
13 KiB
ReStructuredText
369 lines
13 KiB
ReStructuredText
PEP: 561
|
||
Title: Distributing and Packaging Type Information
|
||
Author: Ethan Smith <ethan@ethanhs.me>
|
||
Status: Accepted
|
||
Type: Standards Track
|
||
Topic: Packaging, Typing
|
||
Content-Type: text/x-rst
|
||
Created: 09-Sep-2017
|
||
Python-Version: 3.7
|
||
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017, 12-Apr-2018
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
:pep:`484` introduced type hinting to Python, with goals of making typing
|
||
gradual and easy to adopt. Currently, typing information must be distributed
|
||
manually. This PEP provides a standardized means to leverage existing tooling
|
||
to package and distribute type information with minimal work and an ordering
|
||
for type checkers to resolve modules and collect this information for type
|
||
checking.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Currently, package authors wish to distribute code that has inline type
|
||
information. Additionally, maintainers would like to distribute stub files
|
||
to keep Python 2 compatibility while using newer annotation syntax. However,
|
||
there is no standard method to distribute packages with type information.
|
||
Also, if one wished to ship stub files privately the only method available
|
||
would be via setting ``MYPYPATH`` or the equivalent to manually point to
|
||
stubs. If the package can be released publicly, it can be added to
|
||
typeshed [1]_. However, this does not scale and becomes a burden on the
|
||
maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
|
||
of the tool using typeshed.
|
||
|
||
:pep:`484` has a brief section on distributing typing information. In this
|
||
:pep:`section <484#storing-and-distributing-stub-files>`
|
||
the PEP recommends using ``shared/typehints/pythonX.Y/`` for
|
||
shipping stub files. However, manually adding a path to stub files for each
|
||
third party library does not scale. The simplest approach people have taken
|
||
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
|
||
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
|
||
and Django).
|
||
|
||
|
||
Definition of Terms
|
||
===================
|
||
|
||
The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
|
||
to be interpreted as described in :rfc:`2119`.
|
||
|
||
"inline" - the types are part of the runtime code using :pep:`526` and
|
||
:pep:`3107` syntax (the filename ends in ``.py``).
|
||
|
||
"stubs" - files containing only type information, empty of runtime code
|
||
(the filename ends in ``.pyi``).
|
||
|
||
"Distributions" are the packaged files which are used to publish and distribute
|
||
a release. (:pep:`426`)
|
||
|
||
"Module" a file containing Python runtime code or stubbed type information.
|
||
|
||
"Package" a directory or directories that namespace Python modules.
|
||
(Note the distinction between packages and distributions. While most
|
||
distributions are named after the one package they install, some
|
||
distributions install multiple packages.)
|
||
|
||
|
||
Specification
|
||
=============
|
||
|
||
There are several motivations and methods of supporting typing in a package.
|
||
This PEP recognizes three types of packages that users of typing wish to
|
||
create:
|
||
|
||
1. The package maintainer would like to add type information inline.
|
||
|
||
2. The package maintainer would like to add type information via stubs.
|
||
|
||
3. A third party or package maintainer would like to share stub files for
|
||
a package, but the maintainer does not want to include them in the source
|
||
of the package.
|
||
|
||
This PEP aims to support all three scenarios and make them simple to add to
|
||
packaging and deployment.
|
||
|
||
The two major parts of this specification are the packaging specifications
|
||
and the resolution order for resolving module type information. The type
|
||
checking spec is meant to replace the ``shared/typehints/pythonX.Y/``
|
||
:pep:`spec of PEP 484 <484#storing-and-distributing-stub-files>`.
|
||
|
||
New third party stub libraries SHOULD distribute stubs via the third party
|
||
packaging methods proposed in this PEP in place of being added to typeshed.
|
||
Typeshed will remain in use, but if maintainers are found, third party stubs
|
||
in typeshed MAY be split into their own package.
|
||
|
||
|
||
Packaging Type Information
|
||
--------------------------
|
||
|
||
In order to make packaging and distributing type information as simple and
|
||
easy as possible, packaging and distribution is done through existing
|
||
frameworks.
|
||
|
||
Package maintainers who wish to support type checking of their code MUST add
|
||
a marker file named ``py.typed`` to their package supporting typing. This marker applies
|
||
recursively: if a top-level package includes it, all its sub-packages MUST support
|
||
type checking as well. To have this file installed with the package,
|
||
maintainers can use existing packaging options such as ``package_data`` in
|
||
distutils, shown below.
|
||
|
||
Distutils option example::
|
||
|
||
setup(
|
||
...,
|
||
package_data = {
|
||
'foopkg': ['py.typed'],
|
||
},
|
||
...,
|
||
)
|
||
|
||
For namespace packages (see :pep:`420`), the ``py.typed`` file should be in the
|
||
submodules of the namespace, to avoid conflicts and for clarity.
|
||
|
||
This PEP does not support distributing typing information as part of
|
||
module-only distributions or single-file modules within namespace packages.
|
||
|
||
The single-file module should be refactored into a package
|
||
and indicate that the package supports typing as described
|
||
above.
|
||
|
||
Stub-only Packages
|
||
''''''''''''''''''
|
||
|
||
For package maintainers wishing to ship stub files containing all of their
|
||
type information, it is preferred that the ``*.pyi`` stubs are alongside the
|
||
corresponding ``*.py`` files. However, the stubs can also be put in a separate
|
||
package and distributed separately. Third parties can also find this method
|
||
useful if they wish to distribute stub files. The name of the stub package
|
||
MUST follow the scheme ``foopkg-stubs`` for type stubs for the package named
|
||
``foopkg``. Note that for stub-only packages adding a ``py.typed`` marker is not
|
||
needed since the name ``*-stubs`` is enough to indicate it is a source of typing
|
||
information.
|
||
|
||
Third parties seeking to distribute stub files are encouraged to contact the
|
||
maintainer of the package about distribution alongside the package. If the
|
||
maintainer does not wish to maintain or package stub files or type information
|
||
inline, then a third party stub-only package can be created.
|
||
|
||
In addition, stub-only distributions SHOULD indicate which version(s)
|
||
of the runtime package are supported by indicating the runtime distribution's
|
||
version(s) through normal dependency data. For example, the
|
||
stub package ``flyingcircus-stubs`` can indicate the versions of the
|
||
runtime ``flyingcircus`` distribution it supports through ``install_requires``
|
||
in distutils-based tools, or the equivalent in other packaging tools. Note that
|
||
in pip 9.0, if you update ``flyingcircus-stubs``, it will update
|
||
``flyingcircus``. In pip 9.0, you can use the
|
||
``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default
|
||
behavior.
|
||
|
||
For namespace packages (see :pep:`420`), stub-only packages should
|
||
use the ``-stubs`` suffix on only the root namespace package.
|
||
All stub-only namespace packages should omit ``__init__.pyi`` files. ``py.typed``
|
||
marker files are not necessary for stub-only packages, but similarly
|
||
to packages with inline types, if used, they should be in submodules of the namespace to
|
||
avoid conflicts and for clarity.
|
||
|
||
For example, if the ``pentagon`` and ``hexagon`` are separate distributions
|
||
installing within the namespace package ``shapes.polygons``
|
||
The corresponding types-only distributions should produce packages
|
||
laid out as follows::
|
||
|
||
shapes-stubs
|
||
└── polygons
|
||
└── pentagon
|
||
└── __init__.pyi
|
||
|
||
shapes-stubs
|
||
└── polygons
|
||
└── hexagon
|
||
└── __init__.pyi
|
||
|
||
|
||
Type Checker Module Resolution Order
|
||
------------------------------------
|
||
|
||
The following is the order in which type checkers supporting this PEP SHOULD
|
||
resolve modules containing type information:
|
||
|
||
|
||
1. Stubs or Python source manually put in the beginning of the path. Type
|
||
checkers SHOULD provide this to allow the user complete control of which
|
||
stubs to use, and to patch broken stubs/inline types from packages.
|
||
In mypy the ``$MYPYPATH`` environment variable can be used for this.
|
||
|
||
2. User code - the files the type checker is running on.
|
||
|
||
3. Stub packages - these packages SHOULD supersede any installed inline
|
||
package. They can be found at ``foopkg-stubs`` for package ``foopkg``.
|
||
|
||
4. Inline packages - if there is nothing overriding the installed
|
||
package, *and* it opts into type checking, inline types SHOULD be used.
|
||
|
||
5. Typeshed (if used) - Provides the stdlib types and several third party
|
||
libraries.
|
||
|
||
If typecheckers identify a stub-only namespace package without the desired module
|
||
in step 3, they should continue to step 4/5. Typecheckers should identify namespace packages
|
||
by the absence of ``__init__.pyi``. This allows different subpackages to
|
||
independently opt for inline vs stub-only.
|
||
|
||
Type checkers that check a different Python version than the version they run
|
||
on MUST find the type information in the ``site-packages``/``dist-packages``
|
||
of that Python version. This can be queried e.g.
|
||
``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended
|
||
that the type checker allow for the user to point to a particular Python
|
||
binary, in case it is not in the path.
|
||
|
||
|
||
Partial Stub Packages
|
||
---------------------
|
||
|
||
Many stub packages will only have part of the type interface for libraries
|
||
completed, especially initially. For the benefit of type checking and code
|
||
editors, packages can be "partial". This means modules not found in the stub
|
||
package SHOULD be searched for in parts four and five of the module resolution
|
||
order above, namely inline packages and typeshed.
|
||
|
||
Type checkers should merge the stub package and runtime package or typeshed
|
||
directories. This can be thought of as the functional equivalent of copying the
|
||
stub package into the same directory as the corresponding runtime package or
|
||
typeshed folder and type checking the combined directory structure. Thus type
|
||
checkers MUST maintain the normal resolution order of checking ``*.pyi`` before
|
||
``*.py`` files.
|
||
|
||
If a stub package distribution is partial it MUST include ``partial\n`` in a
|
||
``py.typed`` file. For stub-packages distributing within a namespace
|
||
package (see :pep:`420`), the ``py.typed`` file should be in the
|
||
submodules of the namespace.
|
||
|
||
Type checkers should treat namespace packages within stub-packages as
|
||
incomplete since multiple distributions may populate them.
|
||
Regular packages within namespace packages in stub-package distributions
|
||
are considered complete unless a ``py.typed`` with ``partial\n`` is included.
|
||
|
||
|
||
Implementation
|
||
==============
|
||
|
||
The proposed scheme of indicating support for typing is completely backwards
|
||
compatible, and requires no modification to package tooling. A sample package
|
||
with inline types is available [typed_package]_, as well as a [stub_package]_. A
|
||
sample package checker [pkg_checker]_ which reads the metadata of installed
|
||
packages and reports on their status as either not typed, inline typed, or a
|
||
stub package.
|
||
|
||
The mypy type checker has an implementation of :pep:`561` searching which can be
|
||
read about in the mypy docs [4]_.
|
||
|
||
[numpy-stubs]_ is an example of a real stub-only package for the numpy
|
||
distribution.
|
||
|
||
|
||
Acknowledgements
|
||
================
|
||
|
||
This PEP would not have been possible without the ideas, feedback, and support
|
||
of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Andrey
|
||
Vlasovskikh, Nathaniel Smith, and Guido van Rossum.
|
||
|
||
|
||
Version History
|
||
===============
|
||
|
||
* 2021-09-20
|
||
|
||
* Clarify expectations and typechecker behavior for stub-only namespace packages
|
||
* Clarify handling of single-file modules within namespace packages.
|
||
|
||
* 2018-07-09
|
||
|
||
* Add links to sample stub-only packages
|
||
|
||
* 2018-06-19
|
||
|
||
* Partial stub packages can look at typeshed as well as runtime packages
|
||
|
||
* 2018-05-15
|
||
|
||
* Add partial stub package spec.
|
||
|
||
* 2018-04-09
|
||
|
||
* Add reference to mypy implementation
|
||
* Clarify stub package priority.
|
||
|
||
* 2018-02-02
|
||
|
||
* Change stub-only package suffix to be -stubs not _stubs.
|
||
* Note that py.typed is not needed for stub-only packages.
|
||
* Add note about pip and upgrading stub packages.
|
||
|
||
* 2017-11-12
|
||
|
||
* Rewritten to use existing tooling only
|
||
* No need to indicate kind of type information in metadata
|
||
* Name of marker file changed from ``.typeinfo`` to ``py.typed``
|
||
|
||
* 2017-11-10
|
||
|
||
* Specification re-written to use package metadata instead of distribution
|
||
metadata.
|
||
* Removed stub-only packages and merged into third party packages spec.
|
||
* Removed suggestion for typecheckers to consider checking runtime versions
|
||
* Implementations updated to reflect PEP changes.
|
||
|
||
* 2017-10-26
|
||
|
||
* Added implementation references.
|
||
* Added acknowledgements and version history.
|
||
|
||
* 2017-10-06
|
||
|
||
* Rewritten to use .distinfo/METADATA over a distutils specific command.
|
||
* Clarify versioning of third party stub packages.
|
||
|
||
* 2017-09-11
|
||
|
||
* Added information about current solutions and typeshed.
|
||
* Clarify rationale.
|
||
|
||
|
||
References
|
||
==========
|
||
.. [1] Typeshed (https://github.com/python/typeshed)
|
||
|
||
.. [4] Example implementation in a type checker
|
||
(https://mypy.readthedocs.io/en/latest/installed_packages.html)
|
||
|
||
.. [stub_package] A stub-only package
|
||
(https://github.com/ethanhs/stub-package)
|
||
|
||
.. [typed_package] Sample typed package
|
||
(https://github.com/ethanhs/sample-typed-package)
|
||
|
||
.. [numpy-stubs] Stubs for numpy
|
||
(https://github.com/numpy/numpy-stubs)
|
||
|
||
.. [pkg_checker] Sample package checker
|
||
(https://github.com/ethanhs/check_typedpkg)
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|