Update PEP 561 to use per-package marker file (#461)
Typing information relates to import packages rather than distribution packages, so it needs to be inside the individual packages, rather than in the distribution metadata. As an added benefit, this means running directly from source control will still do the right thing.
This commit is contained in:
Ethan Smith
Nick Coghlan
parent
9a3d13f690
commit
1bf93be451
194
pep-0561.rst
194
pep-0561.rst
|
|
@ -6,32 +6,33 @@ Type: Standards Track
|
|||
Content-Type: text/x-rst
|
||||
Created: 09-Sep-2017
|
||||
Python-Version: 3.7
|
||||
Post-History:
|
||||
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
PEP 484 introduced type hinting to Python, with goals of making typing
|
||||
gradual and easy to adopt. Currently, typing information must be distributed
|
||||
manually. This PEP provides a standardized means to package and distribute
|
||||
type information and an ordering for type checkers to resolve modules and
|
||||
collect this information for type checking using existing packaging
|
||||
architecture.
|
||||
gradual and easy to adopt. Currently, typing information must be distributed
|
||||
manually. This PEP provides a standardized means to leverage existing tooling
|
||||
to package and distribute type information with minimal work and an ordering
|
||||
for type checkers to resolve modules and collect this information for type
|
||||
checking.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
Currently, package authors wish to distribute code that has
|
||||
inline type information. However, there is no standard method to distribute
|
||||
packages with inline type annotations or syntax that can simultaneously
|
||||
be used at runtime and in type checking. Additionally, if one wished to
|
||||
ship typing information privately the only method would be via setting
|
||||
``MYPYPATH`` or the equivalent to manually point to stubs. If the package
|
||||
can be released publicly, it can be added to typeshed [1]_. However, this
|
||||
does not scale and becomes a burden on the maintainers of typeshed.
|
||||
Additionally, it ties bugfixes to releases of the tool using typeshed.
|
||||
Currently, package authors wish to distribute code that has inline type
|
||||
information. Additionally, maintainers would like to distribute stub files
|
||||
to keep Python 2 compatibility while using newer annotation syntax. However,
|
||||
there is no standard method to distribute packages with type information.
|
||||
Also, if one wished to ship stub files privately the only method available
|
||||
would be via setting ``MYPYPATH`` or the equivalent to manually point to
|
||||
stubs. If the package can be released publicly, it can be added to
|
||||
typeshed [1]_. However, this does not scale and becomes a burden on the
|
||||
maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
|
||||
of the tool using typeshed.
|
||||
|
||||
PEP 484 has a brief section on distributing typing information. In this
|
||||
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
|
||||
|
|
@ -42,139 +43,168 @@ checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
|
|||
and Django).
|
||||
|
||||
|
||||
Definition of Terms
|
||||
===================
|
||||
|
||||
The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
|
||||
to be interpreted as described in RFC 2119.
|
||||
|
||||
"inline" - the types are part of the runtime code using PEP 526 and 3107
|
||||
syntax.
|
||||
|
||||
"stubs" - files containing only type information, empty of runtime code.
|
||||
|
||||
"Distributions" are the packaged files which are used to publish and distribute
|
||||
a release. [3]_
|
||||
|
||||
"Module" a file containing Python runtime code or stubbed type information.
|
||||
|
||||
"Package" a directory or directories that namespace Python modules.
|
||||
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
There are several motivations and methods of supporting typing in a package.
|
||||
This PEP recognizes three (3) types of packages that may be created:
|
||||
This PEP recognizes three (3) types of packages that users of typing wish to
|
||||
create:
|
||||
|
||||
1. The package maintainer would like to add type information inline.
|
||||
|
||||
2. The package maintainer would like to add type information via stubs.
|
||||
|
||||
3. A third party would like to share stub files for a package, but the
|
||||
maintainer does not want to include them in the source of the package.
|
||||
3. A third party or package maintainer would like to share stub files for
|
||||
a package, but the maintainer does not want to include them in the source
|
||||
of the package.
|
||||
|
||||
This PEP aims to support these scenarios and make them simple to add to
|
||||
packaging and deployment.
|
||||
|
||||
The two major parts of this specification are the packaging specifications
|
||||
and the resolution order for resolving module type information. The packaging
|
||||
spec is based on and extends PEP 345 metadata. The type checking spec is
|
||||
meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_.
|
||||
and the resolution order for resolving module type information. The type
|
||||
checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
|
||||
of PEP 484 [2]_.
|
||||
|
||||
New third party stub libraries SHOULD distribute stubs via the third party
|
||||
packaging methods proposed in this PEP in place of being added to typeshed.
|
||||
Typeshed will remain in use, but if maintainers are found, third party stubs
|
||||
in typeshed MAY be split into their own package.
|
||||
|
||||
New third party stub libraries are encouraged to distribute stubs via the
|
||||
third party packaging proposed in this PEP in place of being added to
|
||||
typeshed. Typeshed will remain in use, but if maintainers are found, third
|
||||
party stubs in typeshed are encouraged to be split into their own package.
|
||||
|
||||
Packaging Type Information
|
||||
--------------------------
|
||||
|
||||
In order to make packaging and distributing type information as simple and
|
||||
easy as possible, the distribution of type information, and typed Python code
|
||||
is done through existing packaging frameworks. This PEP adds a new item to the
|
||||
``*.distinfo/METADATA`` file to contain metadata about a package's support for
|
||||
typing. The new item is optional, but must have a name of ``Typed`` and have a
|
||||
value of either ``inline`` or ``stubs``, if present.
|
||||
easy as possible, packaging and distribution is done through existing
|
||||
frameworks.
|
||||
|
||||
Metadata Examples::
|
||||
Package maintainers who wish to support type checking of their code MUST add
|
||||
a ``py.typed`` file to their package supporting typing. This marker is
|
||||
recursive, if a top-level package includes it, all sub-packages MUST support
|
||||
type checking as well. To have this file installed with the package,
|
||||
maintainers can use existing packaging options such as ``package_data`` in
|
||||
distutils, shown below.
|
||||
|
||||
Typed: inline
|
||||
Typed: stubs
|
||||
Distutils option example::
|
||||
|
||||
...
|
||||
package_data = {
|
||||
'pkg': ['py.typed'],
|
||||
},
|
||||
...
|
||||
|
||||
For namespace packages, the ``py.typed`` file should be in the submodules of
|
||||
the namespace, to avoid conflicts and for clarity.
|
||||
|
||||
Stub Only Packages
|
||||
''''''''''''''''''
|
||||
|
||||
For package maintainers wishing to ship stub files containing all of their
|
||||
type information, it is prefered that the ``*.pyi`` stubs are alongside the
|
||||
corresponding ``*.py`` files. However, the stubs may be put in a sub-folder
|
||||
of the Python sources, with the same name the ``*.py`` files are in. For
|
||||
example, the ``flyingcircus`` package would have its stubs in the folder
|
||||
``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
|
||||
not found in ``flyingcircus/`` the type checker may treat the subdirectory as
|
||||
a normal package. The normal resolution order of checking ``*.pyi`` before
|
||||
``*.py`` will be maintained.
|
||||
|
||||
Third Party Stub Packages
|
||||
'''''''''''''''''''''''''
|
||||
type information, it is preferred that the ``*.pyi`` stubs are alongside the
|
||||
corresponding ``*.py`` files. However, the stubs can also be put in a separate
|
||||
package and distributed separately. Third parties can also find this method
|
||||
useful if they wish to distribute stub files. The name of the stub package
|
||||
MUST follow the scheme ``pkg_stubs`` for type stubs for the package named
|
||||
``pkg``. The normal resolution order of checking ``*.pyi`` before ``*.py``
|
||||
will be maintained.
|
||||
|
||||
Third parties seeking to distribute stub files are encouraged to contact the
|
||||
maintainer of the package about distribution alongside the package. If the
|
||||
maintainer does not wish to maintain or package stub files or type information
|
||||
inline, then a "third party stub package" should be created. The structure is
|
||||
similar, but slightly different from that of stub only packages. If the stubs
|
||||
are for the library ``flyingcircus`` then the package should be named
|
||||
``flyingcircus-stubs`` and the stub files should be put in a sub-directory
|
||||
named ``flyingcircus``. This allows the stubs to be checked as if they were in
|
||||
a regular package.
|
||||
inline, then a third party stub only package can be created.
|
||||
|
||||
In addition, the third party stub package should indicate which version(s)
|
||||
of the runtime package are supported by indicating the runtime package's
|
||||
version(s) through the normal dependency data. For example, if there was a
|
||||
stub package ``flyingcircus-stubs``, it can indicate the versions of the
|
||||
runtime ``flyingcircus`` package supported through ``install_requires``
|
||||
In addition, stub only distributions SHOULD indicate which version(s)
|
||||
of the runtime package are supported by indicating the runtime distribution's
|
||||
version(s) through normal dependency data. For example, if there was a
|
||||
stub package ``flyingcircus_stubs``, it can indicate the versions of the
|
||||
runtime ``Flyingcircus`` distribution supported through ``install_requires``
|
||||
in distutils based tools, or the equivalent in other packaging tools.
|
||||
|
||||
|
||||
Type Checker Module Resolution Order
|
||||
------------------------------------
|
||||
|
||||
The following is the order that type checkers supporting this PEP should
|
||||
The following is the order that type checkers supporting this PEP SHOULD
|
||||
resolve modules containing type information:
|
||||
|
||||
1. User code - the files the type checker is running on.
|
||||
|
||||
2. Stubs or Python source manually put in the beginning of the path. Type
|
||||
checkers should provide this to allow the user complete control of which
|
||||
checkers SHOULD provide this to allow the user complete control of which
|
||||
stubs to use, and patch broken stubs/inline types from packages.
|
||||
|
||||
3. Third party stub packages - these packages can supersede the installed
|
||||
untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``,
|
||||
however it is encouraged to check the package's metadata using packaging
|
||||
query APIs such as ``pkg_resources`` to assure that the package is meant
|
||||
for type checking, and is compatible with the installed version.
|
||||
3. Stub packages - these packages can supersede the installed packages.
|
||||
They can be found at ``pkg_stubs`` for package ``pkg``.
|
||||
|
||||
4. Inline packages - finally, if there is nothing overriding the installed
|
||||
package, and it opts into type checking.
|
||||
4. Inline packages - if there is nothing overriding the installed
|
||||
package, and it opts into type checking, inline types SHOULD be used.
|
||||
|
||||
5. Typeshed (if used) - Provides the stdlib types and several third party
|
||||
libraries
|
||||
libraries.
|
||||
|
||||
Type checkers that check a different Python version than the version they run
|
||||
on must find the type information in the ``site-packages``/``dist-packages``
|
||||
on MUST find the type information in the ``site-packages``/``dist-packages``
|
||||
of that Python version. This can be queried e.g.
|
||||
``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended
|
||||
that the type checker allow for the user to point to a particular Python
|
||||
binary, in case it is not in the path.
|
||||
|
||||
To check if a package has opted into type checking, type checkers are
|
||||
recommended to use the ``pkg_resources`` module to query the package
|
||||
metadata. If the ``typed`` package metadata has ``None`` as its value, the
|
||||
package has not opted into type checking, and the type checker should skip
|
||||
that package.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
A CPython branch with a modified distutils supporting the ``typed`` setup
|
||||
keyword lives here: [impl]_. In addition, a sample package with inline types is
|
||||
available [typed_pkg]_, as well as a sample package [pkg_checker]_ which reads the metadata
|
||||
of installed packages and reports on their status as either not typed, inline
|
||||
typed, or a stub package.
|
||||
The proposed scheme of indicating support for typing is completely backwards
|
||||
compatible, and requires no modification to tooling. A sample package with
|
||||
inline types is available [typed_pkg]_, as well as a sample package checker
|
||||
[pkg_checker]_ which reads the metadata of installed packages and reports on
|
||||
their status as either not typed, inline typed, or a stub package.
|
||||
|
||||
|
||||
Acknowledgements
|
||||
================
|
||||
|
||||
This PEP would not have been possible without the ideas, feedback, and support
|
||||
of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, and
|
||||
Guido van Rossum.
|
||||
of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Nathaniel
|
||||
Smith, and Guido van Rossum.
|
||||
|
||||
|
||||
Version History
|
||||
===============
|
||||
|
||||
* 2017-11-12
|
||||
|
||||
* Rewritten to use existing tooling only
|
||||
* No need to indicate kind of type information in metadata
|
||||
* Name of marker file changed from ``.typeinfo`` to ``py.typed``
|
||||
|
||||
* 2017-11-10
|
||||
|
||||
* Specification re-written to use package metadata instead of distribution
|
||||
metadata.
|
||||
* Removed stub only packages and merged into third party packages spec.
|
||||
* Removed suggestion for typecheckers to consider checking runtime versions
|
||||
* Implementations updated to reflect PEP changes.
|
||||
|
||||
* 2017-10-26
|
||||
|
||||
* Added implementation references.
|
||||
|
|
@ -197,9 +227,9 @@ References
|
|||
|
||||
.. [2] PEP 484, Storing and Distributing Stub Files
|
||||
(https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)
|
||||
|
||||
.. [impl] CPython sample implementation
|
||||
(https://github.com/ethanhs/cpython/tree/typeddist)
|
||||
|
||||
.. [3] PEP 426 definitions
|
||||
(https://www.python.org/dev/peps/pep-0426/)
|
||||
|
||||
.. [typed_pkg] Sample typed package
|
||||
(https://github.com/ethanhs/sample-typed-package)
|
||||
|
|
|
|||
Loading…
Reference in New Issue