PEP 720: add "Cross-compiling Python packages" PEP draft (#3192)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>
This commit is contained in:
parent
c1335476ff
commit
3782a8213c
|
@ -597,7 +597,7 @@ pep-0713.rst @ambv
|
|||
pep-0714.rst @dstufft
|
||||
pep-0715.rst @dstufft
|
||||
pep-0719.rst @Yhg1s
|
||||
# pep-0720.rst (reserved for https://github.com/python/peps/pull/3192)
|
||||
pep-0720.rst @FFY00
|
||||
pep-0721.rst @encukou
|
||||
# ...
|
||||
# pep-0754.txt
|
||||
|
|
|
@ -0,0 +1,943 @@
|
|||
PEP: 720
|
||||
Title: Cross-compiling Python packages
|
||||
Author: Filipe Laíns <lains@riseup.net>
|
||||
PEP-Delegate:
|
||||
Status: Draft
|
||||
Type: Informational
|
||||
Content-Type: text/x-rst
|
||||
Created: 01-Jul-2023
|
||||
Python-Version: 3.12
|
||||
Resolution:
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP attempts to document the status of cross-compilation of downstream
|
||||
projects.
|
||||
|
||||
It should give an overview of the approaches currently used by distributors
|
||||
(Linux distros, WASM environment providers, etc.) to cross-compile downstream
|
||||
projects (3rd party extensions, etc.).
|
||||
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
We write this PEP to express the challenges in cross-compilation and act as a
|
||||
supporting document in future improvement proposals.
|
||||
|
||||
|
||||
Analysis
|
||||
========
|
||||
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
There are a couple different approaches being used to tackle this, with
|
||||
different levels of interaction required from the user, but they all require a
|
||||
significant amount of effort. This is due to the lack of standardized
|
||||
cross-compilation infrastructure on the Python packaging ecosystem, which itself
|
||||
stems from the complexity of cross-builds, making it a huge undertaking.
|
||||
|
||||
|
||||
Upstream support
|
||||
----------------
|
||||
|
||||
Some major projects like CPython, setuptools, etc. provide some support to help
|
||||
with cross-compilation, but it's unofficial and at a best-effort basis. For
|
||||
example, the ``sysconfig`` module allows overwriting the data module name via
|
||||
the ``_PYTHON_SYSCONFIGDATA_NAME`` environment variable, something that is
|
||||
required for cross-builds, and setuptools `accepts patches`__ [1]_ to tweak/fix
|
||||
its logic to be compatible with popular
|
||||
:ref:`"environment faking" <approach-target-environment>` workflows [2]_.
|
||||
|
||||
The lack of first-party support in upstream projects leads to cross-compilation
|
||||
being fragile and requiring a significant effort from users, but at the same
|
||||
time, the lack of standardization makes it harder for upstreams to improve
|
||||
support as there's no clarity on how this feature should be provided.
|
||||
|
||||
.. [1] At the time of writing (Jun 2023), setuptools' compiler interface code,
|
||||
the component that most of affects cross-compilation, is developed on the
|
||||
`pypa/distutils`__ repository, which gets periodically synced to the
|
||||
setuptools repository.
|
||||
|
||||
.. [2] We specifically mention *popular* workflows, because this is not
|
||||
standardized. Though, many of the most popular implementations
|
||||
(crossenv_, conda-forge_'s build system, etc.) work similarly, and this
|
||||
is what we are referring to here. For clarity, the implementations we are
|
||||
referring to here could be described as *crossenv-style*.
|
||||
|
||||
.. __: https://github.com/pypa/distutils/pulls?q=cross
|
||||
.. __: https://github.com/pypa/distutils
|
||||
|
||||
Projects with decent cross-build support
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
It seems relevant to point out that there are a few modern Python package
|
||||
build-backends with, at least, decent cross-compilation support, those being
|
||||
scikit-build__ and meson-python__. Both these projects integrate external mature
|
||||
build-systems into Python packaging — CMake__ and Meson__, respectively — so
|
||||
cross-build support is inherited from them.
|
||||
|
||||
.. __: https://github.com/scikit-build/scikit-build
|
||||
.. __: https://github.com/mesonbuild/meson-python
|
||||
.. __: https://cmake.org/
|
||||
.. __: https://mesonbuild.com/
|
||||
|
||||
|
||||
Downstream approaches
|
||||
---------------------
|
||||
|
||||
Cross-compilation approaches fall in a spectrum that goes from, by design,
|
||||
requiring extensive user interaction to (ideally) almost none. Usually, they'll
|
||||
be based on one of two main strategies, using a
|
||||
:ref:`cross-build environment <approach-cross-environment>`, or
|
||||
:ref:`faking the target environment <approach-target-environment>`.
|
||||
|
||||
.. _approach-cross-environment:
|
||||
|
||||
Cross-build environment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This consists of running the Python interpreter normally and utilizing the
|
||||
cross-build provided by the projects' build-system. However, as we saw above,
|
||||
upstream support is lacking, so this approach only works for a small-ish set of
|
||||
projects. When this fails, the usual strategy is to patch the build-system code
|
||||
to build use the correct toolchain, system details, etc. [3]_.
|
||||
|
||||
Since this approach often requires package-specific patching, it requires a lot
|
||||
of user interaction.
|
||||
|
||||
.. admonition:: Examples
|
||||
:class: note
|
||||
|
||||
:ref:`python-for-android`, :ref:`kivy-ios`, etc.
|
||||
|
||||
.. [3] The scope of the build-system patching varies between users and usually
|
||||
depends on the their goal — some (eg. Linux distributions) may patch the
|
||||
build-system to support cross-builds, while others might hardcode
|
||||
compiler paths and system information in the build-system, to simply make
|
||||
the build work.
|
||||
|
||||
.. _approach-target-environment:
|
||||
|
||||
Faking the target environment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Aiming to drop the requirement for user input, a popular approach is trying to
|
||||
fake the target environment. It generally consists of monkeypatching the Python
|
||||
interpreter to get it to mimic the interpreter on the target system, which
|
||||
constitutes of changing many of the ``sys`` module attributes, the ``sysconfig``
|
||||
data, etc. Using this strategy, build-backends do not need to have any
|
||||
cross-build support, and should just work without any code changes.
|
||||
|
||||
Unfortunately, though, it isn't possible to truly fake the target environment.
|
||||
There are many reasons for this, one of the main ones being that it breaks code
|
||||
that actually needs to introspect the running interpreter. As a result,
|
||||
monkeypatching Python to look like target is very tricky — to achieve the less
|
||||
amount of breakage, we can only patch certain aspects of the interpreter.
|
||||
Consequently, build-backends may need some code changes, but these are generally
|
||||
much smaller than the previous approach. This is an inherent limitation of the
|
||||
technique, meaning this strategy still requires some user interaction.
|
||||
|
||||
Nonetheless, this strategy still works out-of-the-box with significantly more
|
||||
projects than the approach above, and requires much less effort in these cases.
|
||||
It is successful in decreasing the amount of user interaction needed, even
|
||||
though it doesn't succeed in being generic.
|
||||
|
||||
.. admonition:: Examples
|
||||
:class: note
|
||||
|
||||
:ref:`crossenv`, :ref:`conda-forge`, etc.
|
||||
|
||||
|
||||
Environment introspection
|
||||
-------------------------
|
||||
|
||||
As explained above, most build system code is written with the assumption that
|
||||
the target system is the same as where the build is occurring, so introspection
|
||||
is usually used to guide the build.
|
||||
|
||||
In this section, we try to document most of the ways this is accomplished. It
|
||||
should give a decent overview of of environment details that are required by
|
||||
build systems.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Snippet
|
||||
- Description
|
||||
- Variance
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> importlib.machinery.EXTENSION_SUFFIXES
|
||||
[
|
||||
'.cpython-311-x86_64-linux-gnu.so',
|
||||
'.abi3.so',
|
||||
'.so',
|
||||
]
|
||||
|
||||
- Extension (native module) suffixes supported by this interpreter.
|
||||
- This is implementation-defined, but it **usually** differs based on the
|
||||
implementation, system architecture, build configuration, Python
|
||||
language version, and implementation version — if one exists.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> importlib.machinery.SOURCE_SUFFIXES
|
||||
['.py']
|
||||
|
||||
- Source (pure-Python) suffixes supported by this interpreter.
|
||||
- This is implementation-defined, but it **usually** doesn't differ
|
||||
(outside exotic implementations or systems).
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> importlib.machinery.all_suffixes()
|
||||
[
|
||||
'.py',
|
||||
'.pyc',
|
||||
'.cpython-311-x86_64-linux-gnu.so',
|
||||
'.abi3.so',
|
||||
'.so',
|
||||
]
|
||||
|
||||
- All module file suffixes supported by this interpreter. It *should* be the
|
||||
union of all ``importlib.machinery.*_SUFFIXES`` attributes.
|
||||
- This is implementation-defined, but it **usually** differs based on the
|
||||
implementation, system architecture, build configuration, Python
|
||||
language version, and implementation version — if one exists. See the
|
||||
entries above for more information.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.abiflags
|
||||
''
|
||||
|
||||
- ABI flags, as specified in :pep:`3149`.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.api_version
|
||||
1013
|
||||
|
||||
- C API version.
|
||||
- Differs based on the Python installation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.base_prefix
|
||||
/usr
|
||||
|
||||
- Prefix of the installation-wide directories where platform independent
|
||||
files are installed.
|
||||
- Differs based on the platform, and installation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.base_exec_prefix
|
||||
/usr
|
||||
|
||||
- Prefix of the installation-wide directories where platform dependent
|
||||
files are installed.
|
||||
- Differs based on the platform, and installation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.byteorder
|
||||
'little'
|
||||
|
||||
- Native byte order.
|
||||
- Differs based on the platform.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.builtin_module_names
|
||||
('_abc', '_ast', '_codecs', ...)
|
||||
|
||||
- Names of all modules that are compiled into the Python interpreter.
|
||||
- Differs based on the platform, system architecture, and build
|
||||
configuration.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.exec_prefix
|
||||
/usr
|
||||
|
||||
- Prefix of the site-specific directories where platform independent files
|
||||
are installed. Because it concerns the site-specific directories, in
|
||||
standard virtual environment implementation, it will be a
|
||||
virtual-environment-specific path.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.executable
|
||||
'/usr/bin/python'
|
||||
|
||||
- Path of the Python interpreter being used.
|
||||
- Differs based on the installation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> with open(sys.executable, 'rb') as f:
|
||||
... header = f.read(4)
|
||||
... if is_elf := (header == b'\x7fELF'):
|
||||
... elf_class = int(f.read(1))
|
||||
... size = {1: 52, 2: 64}.get(elf_class)
|
||||
... elf_header = f.read(size - 5)
|
||||
|
||||
- Whether the Python interpreter is an ELF file, and the ELF header. This
|
||||
approach is something used to identify the target architecture of the
|
||||
installation (example__).
|
||||
- Differs based on the installation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.float_info
|
||||
sys.float_info(
|
||||
max=1.7976931348623157e+308,
|
||||
max_exp=1024,
|
||||
max_10_exp=308,
|
||||
min=2.2250738585072014e-308,
|
||||
min_exp=-1021,
|
||||
min_10_exp=-307,
|
||||
dig=15,
|
||||
mant_dig=53,
|
||||
epsilon=2.220446049250313e-16,
|
||||
radix=2,
|
||||
rounds=1,
|
||||
)
|
||||
|
||||
- Low level information about the float type, as defined by ``float.h``.
|
||||
- Differs based on the architecture, and platform.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.getandroidapilevel()
|
||||
21
|
||||
|
||||
- Integer representing the Android API level.
|
||||
- Differs based on the platform.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.getwindowsversion()
|
||||
sys.getwindowsversion(
|
||||
major=10,
|
||||
minor=0,
|
||||
build=19045,
|
||||
platform=2,
|
||||
service_pack='',
|
||||
)
|
||||
|
||||
- Windows version of the system.
|
||||
- Differs based on the platform.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.hexversion
|
||||
0x30b03f0
|
||||
|
||||
- Python version encoded as an integer.
|
||||
- Differs based on the Python language version.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.implementation
|
||||
namespace(
|
||||
name='cpython',
|
||||
cache_tag='cpython-311',
|
||||
version=sys.version_info(
|
||||
major=3,
|
||||
minor=11,
|
||||
micro=3,
|
||||
releaselevel='final',
|
||||
serial=0,
|
||||
),
|
||||
hexversion=0x30b03f0,
|
||||
_multiarch='x86_64-linux-gnu',
|
||||
)
|
||||
|
||||
- Interpreter implementation details.
|
||||
- Differs based on the interpreter implementation, Python language
|
||||
version, and implementation version — if one exists. It may also include
|
||||
architecture-dependent information, so it may also differ based on the
|
||||
system architecture.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.int_info
|
||||
sys.int_info(
|
||||
bits_per_digit=30,
|
||||
sizeof_digit=4,
|
||||
default_max_str_digits=4300,
|
||||
str_digits_check_threshold=640,
|
||||
)
|
||||
|
||||
- Low level information about Python's internal integer representation.
|
||||
- Differs based on the architecture, platform, implementation, build, and
|
||||
runtime flags.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.maxsize
|
||||
0x7fffffffffffffff
|
||||
|
||||
- Maximum value a variable of type ``Py_ssize_t`` can take.
|
||||
- Differs based on the architecture, platform, and implementation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.maxunicode
|
||||
0x10ffff
|
||||
|
||||
- Value of the largest Unicode code point.
|
||||
- Differs based on the implementation, and on Python versions older than
|
||||
3.3, the build.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.platform
|
||||
linux
|
||||
|
||||
- Platform identifier.
|
||||
- Differs based on the platform.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.prefix
|
||||
/usr
|
||||
|
||||
- Prefix of the site-specific directories where platform dependent files
|
||||
are installed. Because it concerns the site-specific directories, in
|
||||
standard virtual environment implementation, it will be a
|
||||
virtual-environment-specific path.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.platlibdir
|
||||
lib
|
||||
|
||||
- Platform-specific library directory.
|
||||
- Differs based on the platform, and vendor.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.version_info
|
||||
sys.version_info(
|
||||
major=3,
|
||||
minor=11,
|
||||
micro=3,
|
||||
releaselevel='final',
|
||||
serial=0,
|
||||
)
|
||||
|
||||
- Python language version implemented by the interpreter.
|
||||
- Differs if the target Python version is not the same [4]_.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.thread_info
|
||||
sys.thread_info(
|
||||
name='pthread',
|
||||
lock='semaphore',
|
||||
version='NPTL 2.37',
|
||||
)
|
||||
|
||||
- Information about the thread implementation.
|
||||
- Differs based on the platform, and implementation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sys.winver
|
||||
3.8-32
|
||||
|
||||
- Version number used to form Windows registry keys.
|
||||
- Differs based on the platform, and implementation.
|
||||
|
||||
* - .. code-block:: python
|
||||
|
||||
>>> sysconfig.get_config_vars()
|
||||
{ ... }
|
||||
>>> sysconfig.get_config_var(...)
|
||||
...
|
||||
|
||||
- Python distribution configuration variables. It includes a set of
|
||||
variables [5]_ — like ``prefix``, ``exec_prefix``, etc. — based on the
|
||||
running context [6]_, and may include some extra variables based on the
|
||||
Python implementation and system.
|
||||
|
||||
In CPython and most other implementations that use the same
|
||||
build-system, the "extra" variables mention above are: on POSIX, all
|
||||
variables from the ``Makefile`` used to build the interpreter, and on
|
||||
Windows, it usually only includes a small subset of the those [7]_ —
|
||||
like ``EXT_SUFFIX``, ``BINDIR``, etc.
|
||||
|
||||
- This is implementation-defined, but it **usually** differs between
|
||||
non-identical builds. Please refer to the
|
||||
:ref:`env-sysconfig-config-vars` table for a overview of the different
|
||||
configuration variable that are usually present.
|
||||
|
||||
.. [4] Ideally, you want to perform cross-builds with the same Python version
|
||||
and implementation, however, this is often not the case. It should not
|
||||
be very problematic as long as the major and minor versions don't
|
||||
change.
|
||||
|
||||
.. [5] The set of config variables that will always be present mostly consists
|
||||
of variables needed to calculate the installation scheme paths.
|
||||
|
||||
.. [6] The context we refer here consists of the "path initialization", which is
|
||||
a process that happens in the interpreter startup and is responsible for
|
||||
figuring out which environment it is being run — eg. global environment,
|
||||
virtual environment, etc. — and setting ``sys.prefix`` and other
|
||||
attributes accordingly.
|
||||
|
||||
.. [7] This is because Windows builds may not use the ``Makefile``, and instead
|
||||
`use the Visual Studio build system`__. A subset of the most relevant
|
||||
``Makefile`` variables is provided to make user code that uses them
|
||||
simpler.
|
||||
|
||||
.. __: https://github.com/pypa/packaging/blob/2f80de7fd2a8bc199dadf5cf3f5f302a17084792/src/packaging/_manylinux.py#L43-L50
|
||||
.. __: https://docs.python.org/3/using/configure.html#debug-build
|
||||
|
||||
|
||||
CPython (and similar)
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
||||
.. _env-sysconfig-config-vars:
|
||||
|
||||
.. list-table:: ``sysconfig`` configuration variables
|
||||
:header-rows: 1
|
||||
:widths: 20 20 30 30
|
||||
|
||||
* - Name
|
||||
- Example Value
|
||||
- Description
|
||||
- Variance
|
||||
|
||||
* - ``SOABI``
|
||||
- ``cpython-311-x86_64-linux-gnu``
|
||||
- ABI string — defined by :pep:`3149`.
|
||||
- Differs based on the implementation, system architecture, Python
|
||||
language version, and implementation version — if one exists.
|
||||
|
||||
* - ``SHLIB_SUFFIX``
|
||||
- ``.so``
|
||||
- Shared library suffix.
|
||||
- Differs based on the platform.
|
||||
|
||||
* - ``EXT_SUFFIX``
|
||||
- ``.cpython-311-x86_64-linux-gnu.so``
|
||||
- Interpreter-specific Python extension (native module) suffix — generally
|
||||
defined as ``.{SOABI}.{SHLIB_SUFFIX}``.
|
||||
- Differs based on the implementation, system architecture, Python
|
||||
language version, and implementation version — if one exists.
|
||||
|
||||
* - ``LDLIBRARY``
|
||||
- ``libpython3.11.so``
|
||||
- Shared ``libpython`` library name — if available. If unavailable [8]_,
|
||||
the variable will be empty, if available, the library should be located
|
||||
in ``LIBDIR``.
|
||||
- Differs based on the implementation, system architecture, build
|
||||
configuration, Python language version, and implementation version — if
|
||||
one exists.
|
||||
|
||||
* - ``PY3LIBRARY``
|
||||
- ``libpython3.so``
|
||||
- Shared Python 3 only (major version bound only) [9]_ ``libpython``
|
||||
library name — if available. If unavailable [8]_, the variable will be
|
||||
empty, if available, the library should be located in ``LIBDIR``.
|
||||
- Differs based on the implementation, system architecture, build
|
||||
configuration, Python language version, and implementation version — if
|
||||
one exists.
|
||||
|
||||
* - ``LIBRARY``
|
||||
- ``libpython3.11.a``
|
||||
- Static ``libpython`` library name — if available. If unavailable [8]_,
|
||||
the variable will be empty, if available, the library should be located
|
||||
in ``LIBDIR``.
|
||||
- Differs based on the implementation, system architecture, build
|
||||
configuration, Python language version, and implementation version — if
|
||||
one exists.
|
||||
|
||||
* - ``Py_DEBUG``
|
||||
- ``0``
|
||||
- Whether this is a `debug build`__.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``WITH_PYMALLOC``
|
||||
- ``1``
|
||||
- Whether this build has pymalloc__ support.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``Py_TRACE_REFS``
|
||||
- ``0``
|
||||
- Whether reference tracing (debug build only) is enabled.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``Py_UNICODE_SIZE``
|
||||
-
|
||||
- Size of the ``Py_UNICODE`` object, in bytes. This variable is only
|
||||
present in CPython versions older than 3.3, and was commonly used to
|
||||
detect if the build uses UCS2 or UCS4 for unicode objects — before
|
||||
:pep:`393`.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``Py_ENABLE_SHARED``
|
||||
- ``1``
|
||||
- Whether a shared ``libpython`` is available.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``PY_ENABLE_SHARED``
|
||||
- ``1``
|
||||
- Whether a shared ``libpython`` is available.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``CC``
|
||||
- ``gcc``
|
||||
- The C compiler used to build the Python distribution.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``CXX``
|
||||
- ``g++``
|
||||
- The C compiler used to build the Python distribution.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``CFLAGS``
|
||||
- ``-DNDEBUG -g -fwrapv ...``
|
||||
- The C compiler flags used to build the Python distribution.
|
||||
- Differs based on the build configuration.
|
||||
|
||||
* - ``py_version``
|
||||
- ``3.11.3``
|
||||
- Full form of the Python version.
|
||||
- Differs based on the Python language version.
|
||||
|
||||
* - ``py_version_short``
|
||||
- ``3.11``
|
||||
- Custom form of the Python version, containing only the major and minor
|
||||
numbers.
|
||||
- Differs based on the Python language version.
|
||||
|
||||
* - ``py_version_nodot``
|
||||
- ``311``
|
||||
- Custom form of the Python version, containing only the major and minor
|
||||
numbers, and no dots.
|
||||
- Differs based on the Python language version.
|
||||
|
||||
* - ``prefix``
|
||||
- ``/usr``
|
||||
- Same as ``sys.prefix``, please refer to the entry in table above.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - ``base``
|
||||
- ``/usr``
|
||||
- Same as ``sys.prefix``, please refer to the entry in table above.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - ``exec_prefix``
|
||||
- ``/usr``
|
||||
- Same as ``sys.exec_prefix``, please refer to the entry in table above.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - ``platbase``
|
||||
- ``/usr``
|
||||
- Same as ``sys.exec_prefix``, please refer to the entry in table above.
|
||||
- Differs based on the platform, installation, and environment.
|
||||
|
||||
* - ``installed_base``
|
||||
- ``/usr``
|
||||
- Same as ``sys.base_prefix``, please refer to the entry in table above.
|
||||
- Differs based on the platform, and installation.
|
||||
|
||||
* - ``installed_platbase``
|
||||
- ``/usr``
|
||||
- Same as ``sys.base_exec_prefix``, please refer to the entry in table
|
||||
above.
|
||||
- Differs based on the platform, and installation.
|
||||
|
||||
* - ``platlibdir``
|
||||
- ``lib``
|
||||
- Same as ``sys.platlibdir``, please refer to the entry in table above.
|
||||
- Differs based on the platform, and vendor.
|
||||
|
||||
* - ``SIZEOF_*``
|
||||
- ``4``
|
||||
- Size of a certain C type (``double``, ``float``, etc.).
|
||||
- Differs based on the system architecture, and build details.
|
||||
|
||||
|
||||
.. [8] Due to Python bring compiled without shared or static ``libpython``
|
||||
support, respectively.
|
||||
|
||||
.. [9] This is the ``libpython`` library that users of the `stable ABI`__ should
|
||||
link against, if they need to link against ``libpython``.
|
||||
|
||||
.. __: https://docs.python.org/3/c-api/intro.html#debugging-builds
|
||||
.. __: https://docs.python.org/3/c-api/memory.html#pymalloc
|
||||
.. __: https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface
|
||||
|
||||
|
||||
Relevant Information
|
||||
--------------------
|
||||
|
||||
There are some bits of information required by build systems — eg. platform
|
||||
particularities — scattered across many places, and it often is difficult to
|
||||
identify code with assumptions based on them. In this section, we try to
|
||||
document the most relevant cases.
|
||||
|
||||
|
||||
When should extensions be linked against ``libpython``?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Short answer
|
||||
Yes, on Windows. No on POSIX platforms, except Android, Cygwin, and other
|
||||
Windows-based POSIX-like platforms.
|
||||
|
||||
When building extensions for dynamic loading, depending on the target platform,
|
||||
they may need to be linked against ``libpython``.
|
||||
|
||||
On Windows, extensions need to link against ``libpython``, because all symbols
|
||||
must be resolvable at link time. POSIX-like platforms based on Windows — like
|
||||
Cygwin, MinGW, or MSYS — will also require linking against ``libpython``.
|
||||
|
||||
On most POSIX platforms, it is not necessary to link against ``libpython``, as
|
||||
the symbols will already be available in the due to the interpreter — or, when
|
||||
embedding, the executable/library in question — already linking to
|
||||
``libpython``. Not linking an extension module against ``libpython`` will allow
|
||||
it to be loaded by static Python builds, so when possible, it is desirable to do
|
||||
so (see GH-65735__).
|
||||
|
||||
This might not be the case on all POSIX platforms, so make sure you check. One
|
||||
example is Android, where only the main executable and ``LD_PRELOAD`` entries
|
||||
are considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``)
|
||||
[10]_, which causes the ``libpython`` symbols be unavailable when loading the
|
||||
extension.
|
||||
|
||||
.. [10] Refer to `dlopen's man page`__ for more information.
|
||||
|
||||
.. __: https://github.com/python/cpython/issues/65735
|
||||
.. __: https://man.archlinux.org/man/dlopen.3
|
||||
|
||||
|
||||
What are ``prefix``, ``exec_prefix``, ``base_prefix``, and ``base_exec_prefix``?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These are ``sys`` attributes `set in the Python initialization`__ that describe
|
||||
the running environment. They refer to the prefix of directories where
|
||||
installation/environment files are installed, according to the table below.
|
||||
|
||||
==================== ====================================== =================
|
||||
Name Target files Environment Scope
|
||||
==================== ====================================== =================
|
||||
``prefix`` platform independent (eg. pure Python) site-specific
|
||||
``exec_prefix`` platform dependent (eg. native code) site-specific
|
||||
``base_prefix`` platform independent (eg. pure Python) installation-wide
|
||||
``base_exec_prefix`` platform dependent (eg. native code) installation-wide
|
||||
==================== ====================================== =================
|
||||
|
||||
Because the site-specific prefixes will be different inside virtual
|
||||
environments, checking ``sys.prexix != sys.base_prefix`` is commonly used to
|
||||
check if we are in a virtual environment.
|
||||
|
||||
.. __: https://github.com/python/cpython/blob/6a70edf24ca217c5ed4a556d0df5748fc775c762/Modules/getpath.py
|
||||
|
||||
Case studies
|
||||
============
|
||||
|
||||
|
||||
.. _crossenv:
|
||||
|
||||
crossenv
|
||||
--------
|
||||
|
||||
:Description: Virtual Environments for Cross-Compiling Python Extension Modules.
|
||||
:URL: https://github.com/benfogle/crossenv
|
||||
|
||||
``crossenv`` is a tool to create a virtual environment with a monkeypatched
|
||||
Python installation that tries to emulate the target machine in certain
|
||||
scenarios. More about this approach can be found in the
|
||||
:ref:`approach-target-environment` section.
|
||||
|
||||
|
||||
.. _conda-forge:
|
||||
|
||||
conda-forge
|
||||
-----------
|
||||
|
||||
:Description: A community-led collection of recipes, build infrastructure and distributions for the conda package manager.
|
||||
:URL: https://conda-forge.org/
|
||||
|
||||
XXX: Jaime will write a quick summary once the PEP draft is public.
|
||||
|
||||
XXX
|
||||
Uses a modified crossenv.
|
||||
|
||||
|
||||
Yocto Project
|
||||
-------------
|
||||
|
||||
:Description: The Yocto Project is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture.
|
||||
:URL: https://www.yoctoproject.org/
|
||||
|
||||
XXX: Sent email to the mailing list.
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
Buildroot
|
||||
---------
|
||||
|
||||
:Description: Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation.
|
||||
:URL: https://buildroot.org/
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
Pyodide
|
||||
-------
|
||||
|
||||
:Description: Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.
|
||||
:URL: https://pyodide.org/en/stable/
|
||||
|
||||
XXX: Hood should review/expand this section.
|
||||
|
||||
``Pyodide`` is a provides a Python distribution compiled to WebAssembly__
|
||||
using the Emscripten__ toolchain.
|
||||
|
||||
It patches several aspects of the CPython installation and some external
|
||||
components. A custom package manager — micropip__ — supporting both Pure and
|
||||
wasm32/Emscripten wheels, is also provided as a part of the distribution. On top
|
||||
of this, a repo with a `selected set of 3rd party packages`__ is also provided
|
||||
and enabled by default.
|
||||
|
||||
.. __: https://webassembly.org/
|
||||
.. __: https://emscripten.org/
|
||||
.. __: https://micropip.pyodide.org/
|
||||
.. __: https://pyodide.org/en/stable/usage/packages-in-pyodide.html
|
||||
|
||||
|
||||
Beeware
|
||||
-------
|
||||
|
||||
:Description: BeeWare allows you to write your app in Python and release it on multiple platforms.
|
||||
:URL: https://beeware.org/
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
.. _python-for-android:
|
||||
|
||||
python-for-android
|
||||
------------------
|
||||
|
||||
:Description: Turn your Python application into an Android APK.
|
||||
:URL: https://github.com/kivy/python-for-android
|
||||
|
||||
resource https://github.com/Android-for-Python/Android-for-Python-Users
|
||||
|
||||
``python-for-android`` is a tool to package Python apps on Android. It creates a
|
||||
Python distribution with your app and its dependencies.
|
||||
|
||||
Pure-Python dependencies are handled automatically and in a generic way, but
|
||||
native dependencies need recipes__. A set of recipes for
|
||||
`popular dependencies`__ is provided, but users need to provide their own
|
||||
recipes for any other native dependencies.
|
||||
|
||||
.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
|
||||
.. __: https://github.com/kivy/python-for-android/tree/develop/pythonforandroid/recipes
|
||||
|
||||
|
||||
.. _kivy-ios:
|
||||
|
||||
kivy-ios
|
||||
--------
|
||||
|
||||
:Description: Toolchain for compiling Python / Kivy / other libraries for iOS.
|
||||
:URL: https://github.com/kivy/kivy-ios
|
||||
|
||||
``kivy-ios`` is a tool to package Python apps on iOS. It provides a toolchain to
|
||||
build a Python distribution with your app and its dependencies, as well as a CLI
|
||||
to create and manage Xcode projects that integrate with the toolchain.
|
||||
|
||||
It uses the same approach as :ref:`python-for-android` (also maintained by the
|
||||
`Kivy project`__) for app dependencies — pure-Python dependencies are handled
|
||||
automatically, but native dependencies need recipes__, and the project provides
|
||||
recipes for `popular dependencies`__.
|
||||
|
||||
.. __: https://kivy.org
|
||||
.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
|
||||
.. __: https://github.com/kivy/kivy-ios/tree/master/kivy_ios/recipes
|
||||
|
||||
|
||||
AidLearning
|
||||
-----------
|
||||
|
||||
:Description: AI, Android, Linux, ARM: AI application development platform based on Android+Linux integrated ecology.
|
||||
:URL: https://github.com/aidlearning/AidLearning-FrameWork
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
QPython
|
||||
-------
|
||||
|
||||
:Description: QPython is the Python engine for android.
|
||||
:URL: https://github.com/qpython-android/qpython
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
pyqtdeploy
|
||||
----------
|
||||
|
||||
:Description: pyqtdeploy is a tool for deploying PyQt applications.
|
||||
:URL: https://www.riverbankcomputing.com/software/pyqtdeploy/
|
||||
|
||||
contact https://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.html
|
||||
contacted Phil, the maintainer
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
Chaquopy
|
||||
--------
|
||||
|
||||
:Description: Chaquopy provides everything you need to include Python components in an Android app.
|
||||
:URL: https://chaquo.com/chaquopy/
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
EDK II
|
||||
------
|
||||
|
||||
:Description: EDK II is a modern, feature-rich, cross-platform firmware development environment for the UEFI and PI specifications.
|
||||
:URL: https://github.com/tianocore/edk2-libc/tree/master/AppPkg/Applications/Python
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
ActivePython
|
||||
------------
|
||||
|
||||
:Description: Commercial-grade, quality-assured Python distribution focusing on easy installation and cross-platform compatibility on Windows, Linux, Mac OS X, Solaris, HP-UX and AIX.
|
||||
:URL: https://www.activestate.com/products/python/
|
||||
|
||||
TODO
|
||||
|
||||
|
||||
Termux
|
||||
------
|
||||
|
||||
:Description: Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required.
|
||||
:URL: https://termux.dev/en/
|
||||
|
||||
TODO
|
Loading…
Reference in New Issue