931 lines
31 KiB
ReStructuredText
931 lines
31 KiB
ReStructuredText
PEP: 720
|
|
Title: Cross-compiling Python packages
|
|
Author: Filipe Laíns <lains@riseup.net>
|
|
PEP-Delegate:
|
|
Status: Draft
|
|
Type: Informational
|
|
Content-Type: text/x-rst
|
|
Created: 01-Jul-2023
|
|
Python-Version: 3.12
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP attempts to document the status of cross-compilation of downstream
|
|
projects.
|
|
|
|
It should give an overview of the approaches currently used by distributors
|
|
(Linux distros, WASM environment providers, etc.) to cross-compile downstream
|
|
projects (3rd party extensions, etc.).
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
We write this PEP to express the challenges in cross-compilation and act as a
|
|
supporting document in future improvement proposals.
|
|
|
|
|
|
Analysis
|
|
========
|
|
|
|
|
|
Introduction
|
|
------------
|
|
|
|
There are a couple different approaches being used to tackle this, with
|
|
different levels of interaction required from the user, but they all require a
|
|
significant amount of effort. This is due to the lack of standardized
|
|
cross-compilation infrastructure on the Python packaging ecosystem, which itself
|
|
stems from the complexity of cross-builds, making it a huge undertaking.
|
|
|
|
|
|
Upstream support
|
|
----------------
|
|
|
|
Some major projects like CPython, setuptools, etc. provide some support to help
|
|
with cross-compilation, but it's unofficial and at a best-effort basis. For
|
|
example, the ``sysconfig`` module allows overwriting the data module name via
|
|
the ``_PYTHON_SYSCONFIGDATA_NAME`` environment variable, something that is
|
|
required for cross-builds, and setuptools `accepts patches`__ [1]_ to tweak/fix
|
|
its logic to be compatible with popular `"environment faking"
|
|
<faking the target environment_>`_ workflows [2]_.
|
|
|
|
The lack of first-party support in upstream projects leads to cross-compilation
|
|
being fragile and requiring a significant effort from users, but at the same
|
|
time, the lack of standardization makes it harder for upstreams to improve
|
|
support as there's no clarity on how this feature should be provided.
|
|
|
|
.. [1] At the time of writing (Jun 2023), setuptools' compiler interface code,
|
|
the component that most of affects cross-compilation, is developed on the
|
|
`pypa/distutils`__ repository, which gets periodically synced to the
|
|
setuptools repository.
|
|
|
|
.. [2] We specifically mention *popular* workflows, because this is not
|
|
standardized. Though, many of the most popular implementations
|
|
(crossenv_, conda-forge_'s build system, etc.) work similarly, and this
|
|
is what we are referring to here. For clarity, the implementations we are
|
|
referring to here could be described as *crossenv-style*.
|
|
|
|
.. __: https://github.com/pypa/distutils/pulls?q=cross
|
|
.. __: https://github.com/pypa/distutils
|
|
|
|
Projects with decent cross-build support
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It seems relevant to point out that there are a few modern Python package
|
|
build-backends with, at least, decent cross-compilation support, those being
|
|
scikit-build__ and meson-python__. Both these projects integrate external mature
|
|
build-systems into Python packaging — CMake__ and Meson__, respectively — so
|
|
cross-build support is inherited from them.
|
|
|
|
.. __: https://github.com/scikit-build/scikit-build
|
|
.. __: https://github.com/mesonbuild/meson-python
|
|
.. __: https://cmake.org/
|
|
.. __: https://mesonbuild.com/
|
|
|
|
|
|
Downstream approaches
|
|
---------------------
|
|
|
|
Cross-compilation approaches fall in a spectrum that goes from, by design,
|
|
requiring extensive user interaction to (ideally) almost none. Usually, they'll
|
|
be based on one of two main strategies, using a `cross-build environment`_,
|
|
or `faking the target environment`_.
|
|
|
|
.. _approach-cross-environment:
|
|
|
|
Cross-build environment
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
This consists of running the Python interpreter normally and utilizing the
|
|
cross-build provided by the projects' build-system. However, as we saw above,
|
|
upstream support is lacking, so this approach only works for a small-ish set of
|
|
projects. When this fails, the usual strategy is to patch the build-system code
|
|
to build use the correct toolchain, system details, etc. [3]_.
|
|
|
|
Since this approach often requires package-specific patching, it requires a lot
|
|
of user interaction.
|
|
|
|
.. admonition:: Examples
|
|
:class: note
|
|
|
|
`python-for-android`_, `kivy-ios`_, etc.
|
|
|
|
.. [3] The scope of the build-system patching varies between users and usually
|
|
depends on the their goal — some (eg. Linux distributions) may patch the
|
|
build-system to support cross-builds, while others might hardcode
|
|
compiler paths and system information in the build-system, to simply make
|
|
the build work.
|
|
|
|
Faking the target environment
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Aiming to drop the requirement for user input, a popular approach is trying to
|
|
fake the target environment. It generally consists of monkeypatching the Python
|
|
interpreter to get it to mimic the interpreter on the target system, which
|
|
constitutes of changing many of the ``sys`` module attributes, the ``sysconfig``
|
|
data, etc. Using this strategy, build-backends do not need to have any
|
|
cross-build support, and should just work without any code changes.
|
|
|
|
Unfortunately, though, it isn't possible to truly fake the target environment.
|
|
There are many reasons for this, one of the main ones being that it breaks code
|
|
that actually needs to introspect the running interpreter. As a result,
|
|
monkeypatching Python to look like target is very tricky — to achieve the less
|
|
amount of breakage, we can only patch certain aspects of the interpreter.
|
|
Consequently, build-backends may need some code changes, but these are generally
|
|
much smaller than the previous approach. This is an inherent limitation of the
|
|
technique, meaning this strategy still requires some user interaction.
|
|
|
|
Nonetheless, this strategy still works out-of-the-box with significantly more
|
|
projects than the approach above, and requires much less effort in these cases.
|
|
It is successful in decreasing the amount of user interaction needed, even
|
|
though it doesn't succeed in being generic.
|
|
|
|
.. admonition:: Examples
|
|
:class: note
|
|
|
|
`crossenv`_, `conda-forge`_, etc.
|
|
|
|
|
|
Environment introspection
|
|
-------------------------
|
|
|
|
As explained above, most build system code is written with the assumption that
|
|
the target system is the same as where the build is occurring, so introspection
|
|
is usually used to guide the build.
|
|
|
|
In this section, we try to document most of the ways this is accomplished. It
|
|
should give a decent overview of of environment details that are required by
|
|
build systems.
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
* - Snippet
|
|
- Description
|
|
- Variance
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> importlib.machinery.EXTENSION_SUFFIXES
|
|
[
|
|
'.cpython-311-x86_64-linux-gnu.so',
|
|
'.abi3.so',
|
|
'.so',
|
|
]
|
|
|
|
- Extension (native module) suffixes supported by this interpreter.
|
|
- This is implementation-defined, but it **usually** differs based on the
|
|
implementation, system architecture, build configuration, Python
|
|
language version, and implementation version — if one exists.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> importlib.machinery.SOURCE_SUFFIXES
|
|
['.py']
|
|
|
|
- Source (pure-Python) suffixes supported by this interpreter.
|
|
- This is implementation-defined, but it **usually** doesn't differ
|
|
(outside exotic implementations or systems).
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> importlib.machinery.all_suffixes()
|
|
[
|
|
'.py',
|
|
'.pyc',
|
|
'.cpython-311-x86_64-linux-gnu.so',
|
|
'.abi3.so',
|
|
'.so',
|
|
]
|
|
|
|
- All module file suffixes supported by this interpreter. It *should* be the
|
|
union of all ``importlib.machinery.*_SUFFIXES`` attributes.
|
|
- This is implementation-defined, but it **usually** differs based on the
|
|
implementation, system architecture, build configuration, Python
|
|
language version, and implementation version — if one exists. See the
|
|
entries above for more information.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.abiflags
|
|
''
|
|
|
|
- ABI flags, as specified in :pep:`3149`.
|
|
- Differs based on the build configuration.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.api_version
|
|
1013
|
|
|
|
- C API version.
|
|
- Differs based on the Python installation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.base_prefix
|
|
/usr
|
|
|
|
- Prefix of the installation-wide directories where platform independent
|
|
files are installed.
|
|
- Differs based on the platform, and installation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.base_exec_prefix
|
|
/usr
|
|
|
|
- Prefix of the installation-wide directories where platform dependent
|
|
files are installed.
|
|
- Differs based on the platform, and installation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.byteorder
|
|
'little'
|
|
|
|
- Native byte order.
|
|
- Differs based on the platform.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.builtin_module_names
|
|
('_abc', '_ast', '_codecs', ...)
|
|
|
|
- Names of all modules that are compiled into the Python interpreter.
|
|
- Differs based on the platform, system architecture, and build
|
|
configuration.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.exec_prefix
|
|
/usr
|
|
|
|
- Prefix of the site-specific directories where platform independent files
|
|
are installed. Because it concerns the site-specific directories, in
|
|
standard virtual environment implementation, it will be a
|
|
virtual-environment-specific path.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.executable
|
|
'/usr/bin/python'
|
|
|
|
- Path of the Python interpreter being used.
|
|
- Differs based on the installation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> with open(sys.executable, 'rb') as f:
|
|
... header = f.read(4)
|
|
... if is_elf := (header == b'\x7fELF'):
|
|
... elf_class = int(f.read(1))
|
|
... size = {1: 52, 2: 64}.get(elf_class)
|
|
... elf_header = f.read(size - 5)
|
|
|
|
- Whether the Python interpreter is an ELF file, and the ELF header. This
|
|
approach is something used to identify the target architecture of the
|
|
installation (example__).
|
|
- Differs based on the installation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.float_info
|
|
sys.float_info(
|
|
max=1.7976931348623157e+308,
|
|
max_exp=1024,
|
|
max_10_exp=308,
|
|
min=2.2250738585072014e-308,
|
|
min_exp=-1021,
|
|
min_10_exp=-307,
|
|
dig=15,
|
|
mant_dig=53,
|
|
epsilon=2.220446049250313e-16,
|
|
radix=2,
|
|
rounds=1,
|
|
)
|
|
|
|
- Low level information about the float type, as defined by ``float.h``.
|
|
- Differs based on the architecture, and platform.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.getandroidapilevel()
|
|
21
|
|
|
|
- Integer representing the Android API level.
|
|
- Differs based on the platform.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.getwindowsversion()
|
|
sys.getwindowsversion(
|
|
major=10,
|
|
minor=0,
|
|
build=19045,
|
|
platform=2,
|
|
service_pack='',
|
|
)
|
|
|
|
- Windows version of the system.
|
|
- Differs based on the platform.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.hexversion
|
|
0x30b03f0
|
|
|
|
- Python version encoded as an integer.
|
|
- Differs based on the Python language version.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.implementation
|
|
namespace(
|
|
name='cpython',
|
|
cache_tag='cpython-311',
|
|
version=sys.version_info(
|
|
major=3,
|
|
minor=11,
|
|
micro=3,
|
|
releaselevel='final',
|
|
serial=0,
|
|
),
|
|
hexversion=0x30b03f0,
|
|
_multiarch='x86_64-linux-gnu',
|
|
)
|
|
|
|
- Interpreter implementation details.
|
|
- Differs based on the interpreter implementation, Python language
|
|
version, and implementation version — if one exists. It may also include
|
|
architecture-dependent information, so it may also differ based on the
|
|
system architecture.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.int_info
|
|
sys.int_info(
|
|
bits_per_digit=30,
|
|
sizeof_digit=4,
|
|
default_max_str_digits=4300,
|
|
str_digits_check_threshold=640,
|
|
)
|
|
|
|
- Low level information about Python's internal integer representation.
|
|
- Differs based on the architecture, platform, implementation, build, and
|
|
runtime flags.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.maxsize
|
|
0x7fffffffffffffff
|
|
|
|
- Maximum value a variable of type ``Py_ssize_t`` can take.
|
|
- Differs based on the architecture, platform, and implementation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.maxunicode
|
|
0x10ffff
|
|
|
|
- Value of the largest Unicode code point.
|
|
- Differs based on the implementation, and on Python versions older than
|
|
3.3, the build.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.platform
|
|
linux
|
|
|
|
- Platform identifier.
|
|
- Differs based on the platform.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.prefix
|
|
/usr
|
|
|
|
- Prefix of the site-specific directories where platform dependent files
|
|
are installed. Because it concerns the site-specific directories, in
|
|
standard virtual environment implementation, it will be a
|
|
virtual-environment-specific path.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.platlibdir
|
|
lib
|
|
|
|
- Platform-specific library directory.
|
|
- Differs based on the platform, and vendor.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.version_info
|
|
sys.version_info(
|
|
major=3,
|
|
minor=11,
|
|
micro=3,
|
|
releaselevel='final',
|
|
serial=0,
|
|
)
|
|
|
|
- Python language version implemented by the interpreter.
|
|
- Differs if the target Python version is not the same [4]_.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.thread_info
|
|
sys.thread_info(
|
|
name='pthread',
|
|
lock='semaphore',
|
|
version='NPTL 2.37',
|
|
)
|
|
|
|
- Information about the thread implementation.
|
|
- Differs based on the platform, and implementation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sys.winver
|
|
3.8-32
|
|
|
|
- Version number used to form Windows registry keys.
|
|
- Differs based on the platform, and implementation.
|
|
|
|
* - .. code-block:: python
|
|
|
|
>>> sysconfig.get_config_vars()
|
|
{ ... }
|
|
>>> sysconfig.get_config_var(...)
|
|
...
|
|
|
|
- Python distribution configuration variables. It includes a set of
|
|
variables [5]_ — like ``prefix``, ``exec_prefix``, etc. — based on the
|
|
running context [6]_, and may include some extra variables based on the
|
|
Python implementation and system.
|
|
|
|
In CPython and most other implementations that use the same
|
|
build-system, the "extra" variables mention above are: on POSIX, all
|
|
variables from the ``Makefile`` used to build the interpreter, and on
|
|
Windows, it usually only includes a small subset of the those [7]_ —
|
|
like ``EXT_SUFFIX``, ``BINDIR``, etc.
|
|
|
|
- This is implementation-defined, but it **usually** differs between
|
|
non-identical builds. Please refer to the
|
|
`sysconfig configuration variables`_ table for a overview of the different
|
|
configuration variable that are usually present.
|
|
|
|
.. [4] Ideally, you want to perform cross-builds with the same Python version
|
|
and implementation, however, this is often not the case. It should not
|
|
be very problematic as long as the major and minor versions don't
|
|
change.
|
|
|
|
.. [5] The set of config variables that will always be present mostly consists
|
|
of variables needed to calculate the installation scheme paths.
|
|
|
|
.. [6] The context we refer here consists of the "path initialization", which is
|
|
a process that happens in the interpreter startup and is responsible for
|
|
figuring out which environment it is being run — eg. global environment,
|
|
virtual environment, etc. — and setting ``sys.prefix`` and other
|
|
attributes accordingly.
|
|
|
|
.. [7] This is because Windows builds may not use the ``Makefile``, and instead
|
|
`use the Visual Studio build system`__. A subset of the most relevant
|
|
``Makefile`` variables is provided to make user code that uses them
|
|
simpler.
|
|
|
|
.. __: https://github.com/pypa/packaging/blob/2f80de7fd2a8bc199dadf5cf3f5f302a17084792/src/packaging/_manylinux.py#L43-L50
|
|
.. __: https://docs.python.org/3/using/configure.html#debug-build
|
|
|
|
|
|
CPython (and similar)
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
.. list-table:: ``sysconfig`` configuration variables
|
|
:name: sysconfig configuration variables
|
|
:header-rows: 1
|
|
:widths: 20 20 30 30
|
|
|
|
* - Name
|
|
- Example Value
|
|
- Description
|
|
- Variance
|
|
|
|
* - ``SOABI``
|
|
- ``cpython-311-x86_64-linux-gnu``
|
|
- ABI string — defined by :pep:`3149`.
|
|
- Differs based on the implementation, system architecture, Python
|
|
language version, and implementation version — if one exists.
|
|
|
|
* - ``SHLIB_SUFFIX``
|
|
- ``.so``
|
|
- Shared library suffix.
|
|
- Differs based on the platform.
|
|
|
|
* - ``EXT_SUFFIX``
|
|
- ``.cpython-311-x86_64-linux-gnu.so``
|
|
- Interpreter-specific Python extension (native module) suffix — generally
|
|
defined as ``.{SOABI}.{SHLIB_SUFFIX}``.
|
|
- Differs based on the implementation, system architecture, Python
|
|
language version, and implementation version — if one exists.
|
|
|
|
* - ``LDLIBRARY``
|
|
- ``libpython3.11.so``
|
|
- Shared ``libpython`` library name — if available. If unavailable [8]_,
|
|
the variable will be empty, if available, the library should be located
|
|
in ``LIBDIR``.
|
|
- Differs based on the implementation, system architecture, build
|
|
configuration, Python language version, and implementation version — if
|
|
one exists.
|
|
|
|
* - ``PY3LIBRARY``
|
|
- ``libpython3.so``
|
|
- Shared Python 3 only (major version bound only) [9]_ ``libpython``
|
|
library name — if available. If unavailable [8]_, the variable will be
|
|
empty, if available, the library should be located in ``LIBDIR``.
|
|
- Differs based on the implementation, system architecture, build
|
|
configuration, Python language version, and implementation version — if
|
|
one exists.
|
|
|
|
* - ``LIBRARY``
|
|
- ``libpython3.11.a``
|
|
- Static ``libpython`` library name — if available. If unavailable [8]_,
|
|
the variable will be empty, if available, the library should be located
|
|
in ``LIBDIR``.
|
|
- Differs based on the implementation, system architecture, build
|
|
configuration, Python language version, and implementation version — if
|
|
one exists.
|
|
|
|
* - ``Py_DEBUG``
|
|
- ``0``
|
|
- Whether this is a `debug build`__.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``WITH_PYMALLOC``
|
|
- ``1``
|
|
- Whether this build has pymalloc__ support.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``Py_TRACE_REFS``
|
|
- ``0``
|
|
- Whether reference tracing (debug build only) is enabled.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``Py_UNICODE_SIZE``
|
|
-
|
|
- Size of the ``Py_UNICODE`` object, in bytes. This variable is only
|
|
present in CPython versions older than 3.3, and was commonly used to
|
|
detect if the build uses UCS2 or UCS4 for unicode objects — before
|
|
:pep:`393`.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``Py_ENABLE_SHARED``
|
|
- ``1``
|
|
- Whether a shared ``libpython`` is available.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``PY_ENABLE_SHARED``
|
|
- ``1``
|
|
- Whether a shared ``libpython`` is available.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``CC``
|
|
- ``gcc``
|
|
- The C compiler used to build the Python distribution.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``CXX``
|
|
- ``g++``
|
|
- The C compiler used to build the Python distribution.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``CFLAGS``
|
|
- ``-DNDEBUG -g -fwrapv ...``
|
|
- The C compiler flags used to build the Python distribution.
|
|
- Differs based on the build configuration.
|
|
|
|
* - ``py_version``
|
|
- ``3.11.3``
|
|
- Full form of the Python version.
|
|
- Differs based on the Python language version.
|
|
|
|
* - ``py_version_short``
|
|
- ``3.11``
|
|
- Custom form of the Python version, containing only the major and minor
|
|
numbers.
|
|
- Differs based on the Python language version.
|
|
|
|
* - ``py_version_nodot``
|
|
- ``311``
|
|
- Custom form of the Python version, containing only the major and minor
|
|
numbers, and no dots.
|
|
- Differs based on the Python language version.
|
|
|
|
* - ``prefix``
|
|
- ``/usr``
|
|
- Same as ``sys.prefix``, please refer to the entry in table above.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - ``base``
|
|
- ``/usr``
|
|
- Same as ``sys.prefix``, please refer to the entry in table above.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - ``exec_prefix``
|
|
- ``/usr``
|
|
- Same as ``sys.exec_prefix``, please refer to the entry in table above.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - ``platbase``
|
|
- ``/usr``
|
|
- Same as ``sys.exec_prefix``, please refer to the entry in table above.
|
|
- Differs based on the platform, installation, and environment.
|
|
|
|
* - ``installed_base``
|
|
- ``/usr``
|
|
- Same as ``sys.base_prefix``, please refer to the entry in table above.
|
|
- Differs based on the platform, and installation.
|
|
|
|
* - ``installed_platbase``
|
|
- ``/usr``
|
|
- Same as ``sys.base_exec_prefix``, please refer to the entry in table
|
|
above.
|
|
- Differs based on the platform, and installation.
|
|
|
|
* - ``platlibdir``
|
|
- ``lib``
|
|
- Same as ``sys.platlibdir``, please refer to the entry in table above.
|
|
- Differs based on the platform, and vendor.
|
|
|
|
* - ``SIZEOF_*``
|
|
- ``4``
|
|
- Size of a certain C type (``double``, ``float``, etc.).
|
|
- Differs based on the system architecture, and build details.
|
|
|
|
|
|
.. [8] Due to Python bring compiled without shared or static ``libpython``
|
|
support, respectively.
|
|
|
|
.. [9] This is the ``libpython`` library that users of the `stable ABI`__ should
|
|
link against, if they need to link against ``libpython``.
|
|
|
|
.. __: https://docs.python.org/3/c-api/intro.html#debugging-builds
|
|
.. __: https://docs.python.org/3/c-api/memory.html#pymalloc
|
|
.. __: https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface
|
|
|
|
|
|
Relevant Information
|
|
--------------------
|
|
|
|
There are some bits of information required by build systems — eg. platform
|
|
particularities — scattered across many places, and it often is difficult to
|
|
identify code with assumptions based on them. In this section, we try to
|
|
document the most relevant cases.
|
|
|
|
|
|
When should extensions be linked against ``libpython``?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Short answer
|
|
Yes, on Windows. No on POSIX platforms, except Android, Cygwin, and other
|
|
Windows-based POSIX-like platforms.
|
|
|
|
When building extensions for dynamic loading, depending on the target platform,
|
|
they may need to be linked against ``libpython``.
|
|
|
|
On Windows, extensions need to link against ``libpython``, because all symbols
|
|
must be resolvable at link time. POSIX-like platforms based on Windows — like
|
|
Cygwin, MinGW, or MSYS — will also require linking against ``libpython``.
|
|
|
|
On most POSIX platforms, it is not necessary to link against ``libpython``, as
|
|
the symbols will already be available in the due to the interpreter — or, when
|
|
embedding, the executable/library in question — already linking to
|
|
``libpython``. Not linking an extension module against ``libpython`` will allow
|
|
it to be loaded by static Python builds, so when possible, it is desirable to do
|
|
so (see GH-65735__).
|
|
|
|
This might not be the case on all POSIX platforms, so make sure you check. One
|
|
example is Android, where only the main executable and ``LD_PRELOAD`` entries
|
|
are considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``)
|
|
[10]_, which causes the ``libpython`` symbols be unavailable when loading the
|
|
extension.
|
|
|
|
.. [10] Refer to `dlopen's man page`__ for more information.
|
|
|
|
.. __: https://github.com/python/cpython/issues/65735
|
|
.. __: https://man.archlinux.org/man/dlopen.3
|
|
|
|
|
|
What are ``prefix``, ``exec_prefix``, ``base_prefix``, and ``base_exec_prefix``?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
These are ``sys`` attributes `set in the Python initialization`__ that describe
|
|
the running environment. They refer to the prefix of directories where
|
|
installation/environment files are installed, according to the table below.
|
|
|
|
==================== ====================================== =================
|
|
Name Target files Environment Scope
|
|
==================== ====================================== =================
|
|
``prefix`` platform independent (eg. pure Python) site-specific
|
|
``exec_prefix`` platform dependent (eg. native code) site-specific
|
|
``base_prefix`` platform independent (eg. pure Python) installation-wide
|
|
``base_exec_prefix`` platform dependent (eg. native code) installation-wide
|
|
==================== ====================================== =================
|
|
|
|
Because the site-specific prefixes will be different inside virtual
|
|
environments, checking ``sys.prexix != sys.base_prefix`` is commonly used to
|
|
check if we are in a virtual environment.
|
|
|
|
.. __: https://github.com/python/cpython/blob/6a70edf24ca217c5ed4a556d0df5748fc775c762/Modules/getpath.py
|
|
|
|
Case studies
|
|
============
|
|
|
|
|
|
crossenv
|
|
--------
|
|
|
|
:Description: Virtual Environments for Cross-Compiling Python Extension Modules.
|
|
:URL: https://github.com/benfogle/crossenv
|
|
|
|
``crossenv`` is a tool to create a virtual environment with a monkeypatched
|
|
Python installation that tries to emulate the target machine in certain
|
|
scenarios. More about this approach can be found in the
|
|
`Faking the target environment`_ section.
|
|
|
|
|
|
conda-forge
|
|
-----------
|
|
|
|
:Description: A community-led collection of recipes, build infrastructure and distributions for the conda package manager.
|
|
:URL: https://conda-forge.org/
|
|
|
|
XXX: Jaime will write a quick summary once the PEP draft is public.
|
|
|
|
XXX
|
|
Uses a modified crossenv.
|
|
|
|
|
|
Yocto Project
|
|
-------------
|
|
|
|
:Description: The Yocto Project is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture.
|
|
:URL: https://www.yoctoproject.org/
|
|
|
|
XXX: Sent email to the mailing list.
|
|
|
|
TODO
|
|
|
|
|
|
Buildroot
|
|
---------
|
|
|
|
:Description: Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation.
|
|
:URL: https://buildroot.org/
|
|
|
|
TODO
|
|
|
|
|
|
Pyodide
|
|
-------
|
|
|
|
:Description: Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.
|
|
:URL: https://pyodide.org/en/stable/
|
|
|
|
XXX: Hood should review/expand this section.
|
|
|
|
``Pyodide`` is a provides a Python distribution compiled to WebAssembly__
|
|
using the Emscripten__ toolchain.
|
|
|
|
It patches several aspects of the CPython installation and some external
|
|
components. A custom package manager — micropip__ — supporting both Pure and
|
|
wasm32/Emscripten wheels, is also provided as a part of the distribution. On top
|
|
of this, a repo with a `selected set of 3rd party packages`__ is also provided
|
|
and enabled by default.
|
|
|
|
.. __: https://webassembly.org/
|
|
.. __: https://emscripten.org/
|
|
.. __: https://micropip.pyodide.org/
|
|
.. __: https://pyodide.org/en/stable/usage/packages-in-pyodide.html
|
|
|
|
|
|
Beeware
|
|
-------
|
|
|
|
:Description: BeeWare allows you to write your app in Python and release it on multiple platforms.
|
|
:URL: https://beeware.org/
|
|
|
|
TODO
|
|
|
|
|
|
python-for-android
|
|
------------------
|
|
|
|
:Description: Turn your Python application into an Android APK.
|
|
:URL: https://github.com/kivy/python-for-android
|
|
|
|
resource https://github.com/Android-for-Python/Android-for-Python-Users
|
|
|
|
``python-for-android`` is a tool to package Python apps on Android. It creates a
|
|
Python distribution with your app and its dependencies.
|
|
|
|
Pure-Python dependencies are handled automatically and in a generic way, but
|
|
native dependencies need recipes__. A set of recipes for
|
|
`popular dependencies`__ is provided, but users need to provide their own
|
|
recipes for any other native dependencies.
|
|
|
|
.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
|
|
.. __: https://github.com/kivy/python-for-android/tree/develop/pythonforandroid/recipes
|
|
|
|
|
|
kivy-ios
|
|
--------
|
|
|
|
:Description: Toolchain for compiling Python / Kivy / other libraries for iOS.
|
|
:URL: https://github.com/kivy/kivy-ios
|
|
|
|
``kivy-ios`` is a tool to package Python apps on iOS. It provides a toolchain to
|
|
build a Python distribution with your app and its dependencies, as well as a CLI
|
|
to create and manage Xcode projects that integrate with the toolchain.
|
|
|
|
It uses the same approach as `python-for-android`_ (also maintained by the
|
|
`Kivy project`__) for app dependencies — pure-Python dependencies are handled
|
|
automatically, but native dependencies need recipes__, and the project provides
|
|
recipes for `popular dependencies`__.
|
|
|
|
.. __: https://kivy.org
|
|
.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
|
|
.. __: https://github.com/kivy/kivy-ios/tree/master/kivy_ios/recipes
|
|
|
|
|
|
AidLearning
|
|
-----------
|
|
|
|
:Description: AI, Android, Linux, ARM: AI application development platform based on Android+Linux integrated ecology.
|
|
:URL: https://github.com/aidlearning/AidLearning-FrameWork
|
|
|
|
TODO
|
|
|
|
|
|
QPython
|
|
-------
|
|
|
|
:Description: QPython is the Python engine for android.
|
|
:URL: https://github.com/qpython-android/qpython
|
|
|
|
TODO
|
|
|
|
|
|
pyqtdeploy
|
|
----------
|
|
|
|
:Description: pyqtdeploy is a tool for deploying PyQt applications.
|
|
:URL: https://www.riverbankcomputing.com/software/pyqtdeploy/
|
|
|
|
contact https://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.html
|
|
contacted Phil, the maintainer
|
|
|
|
TODO
|
|
|
|
|
|
Chaquopy
|
|
--------
|
|
|
|
:Description: Chaquopy provides everything you need to include Python components in an Android app.
|
|
:URL: https://chaquo.com/chaquopy/
|
|
|
|
TODO
|
|
|
|
|
|
EDK II
|
|
------
|
|
|
|
:Description: EDK II is a modern, feature-rich, cross-platform firmware development environment for the UEFI and PI specifications.
|
|
:URL: https://github.com/tianocore/edk2-libc/tree/master/AppPkg/Applications/Python
|
|
|
|
TODO
|
|
|
|
|
|
ActivePython
|
|
------------
|
|
|
|
:Description: Commercial-grade, quality-assured Python distribution focusing on easy installation and cross-platform compatibility on Windows, Linux, Mac OS X, Solaris, HP-UX and AIX.
|
|
:URL: https://www.activestate.com/products/python/
|
|
|
|
TODO
|
|
|
|
|
|
Termux
|
|
------
|
|
|
|
:Description: Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required.
|
|
:URL: https://termux.dev/en/
|
|
|
|
TODO
|