From 2cbb99c5776a18a518ab90518174be7291a9228b Mon Sep 17 00:00:00 2001 From: Malcolm Smith Date: Fri, 12 Jan 2024 09:39:19 +0000 Subject: [PATCH] PEP 738 (new): Adding Android as a supported platform (GH-3586) Co-authored-by: Hugo van Kemenade Co-authored-by: Carol Willing --- .github/CODEOWNERS | 1 + peps/pep-0738.rst | 410 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 411 insertions(+) create mode 100644 peps/pep-0738.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f760c8188..78c34156c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -615,6 +615,7 @@ peps/pep-0734.rst @ericsnowcurrently peps/pep-0735.rst @brettcannon peps/pep-0736.rst @gvanrossum @Rosuav peps/pep-0737.rst @vstinner +peps/pep-0738.rst @encukou # ... # peps/pep-0754.rst # ... diff --git a/peps/pep-0738.rst b/peps/pep-0738.rst new file mode 100644 index 000000000..03e452111 --- /dev/null +++ b/peps/pep-0738.rst @@ -0,0 +1,410 @@ +PEP: 738 +Title: Adding Android as a supported platform +Author: Malcolm Smith +Sponsor: Petr Viktorin +Discussions-To: https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975 +Status: Draft +Type: Standards Track +Created: 12-Dec-2023 +Python-Version: 3.13 + + +Abstract +======== + +This PEP proposes adding Android as a supported platform in CPython. The initial +goal is for Android to achieve Tier 3 support in Python 3.13. + +This PEP is based on :pep:`730` – "Adding iOS as a supported platform" by +Russell Keith-Magee, and covers many of the same issues. Notable differences +between the two platforms can be found by searching for the word "iOS". + + +Motivation +========== + +Over the last 15 years, mobile platforms have become increasingly important +parts of the computing landscape. Android is the operating system that runs on +about `70% of these devices +`__. However, there +is no official support for Android in CPython. + +The `Chaquopy `__, `BeeWare +`__ and `Kivy `__ projects +have all supported Android for many years, and they have all been used to +generate applications that have been accepted for publication in the Google Play +Store. This demonstrates the technical feasibility of Android support. + +It is important for the future of Python as a language that it is able to be +used on any platform that has widespread adoption. Otherwise, potential users +will choose other languages that *do* provide support for these platforms. This +is especially true in education, where the next generation of developers is in +many cases already spending more time using mobile platforms than desktop ones. + + +Rationale +========= + +General +------- + +Android is broadly a POSIX platform, based on a Linux kernel and the +ELF binary format. It does not use glibc, instead providing its own C +library implementation called Bionic. As a result, it is generally not +binary-compatible with any other Linux distribution, even if the architecture +matches. + +However, Android's source-compatibility with Linux is quite good. In its early years, +the C library was very incomplete, but most of the gaps were filled by around +2014. Since then, any C code that compiles for Linux can usually be compiled for +Android, unless it involves direct access to hardware devices or operating +system services. + +This is also true of CPython. Although it has never officially supported +Android, recent versions (since 3.6) can already be compiled for Android with +minimal patching. + + +OS versions +----------- + +Each Android version can be identified in three ways: + +* A conventional dotted version number (though recent versions have all used + whole numbers) +* A sequential integer "API level" (the most common form in developer + documentation) +* An alphabetic confectionery-themed code name (no longer used for marketing, + but still appears in developer documentation) + +There is no consistent pattern to link one of these to another; they must be +looked up in `a table `__. + +A new major Android version is released each year, but the updates available to +each device are entirely under the control of its manufacturer. Unfortunately +many manufacturers stop sending updates to devices long before their users are +ready to dispose of them. For example, as of October 2023, the oldest Android +version still receiving security updates was API level 30, but according to +`Google's own statistics +`__, only 60% +of devices were on that version or newer. + +For Python 3.13 we therefore propose the minimum Android version to be 6.0 +(API level 23). This would support 98% of active devices, and would +allow us to rely on a number of `dynamic linker improvements +`__ +which simplify the use of dynamic libraries. + + +Development tools +----------------- + +The Android development tools are equally supported on Linux (x86_64), Windows +(x86_64) and macOS (x86_64 and ARM64). For CPython, the most important tools +are: + +* The NDK (native development kit) contains a C and C++ compiler (clang), + linker (lld), and headers for all the system libraries. + + Binary compatibility between libraries compiled with different versions of the + NDK is generally very good, but for reproducbility it would be best for each + Python version to stick with one NDK version throughout its life. For Python + 3.13, this would be the current NDK long-term support version, r26. + + Each NDK version can be set to target any of a wide range of Android versions. + For example, NDK r26 supports API levels 21 to 34. However, binaries compiled + for an older Android version will usually keep on working indefinitely on + newer versions; exceptions to this rule are only made for security reasons. + +* Gradle is the tool used to build complete, deployable apps. + +* The emulator, based on QEMU, is a simulated Android device running on a + development machine. Unlike on iOS, an emulator uses the same ABI as a real + device of the same architecture, and can run the same binaries. + +These tools may all be used either from the command line, or through the Android +Studio IDE, which is based on IntelliJ IDEA. + + +Architectures +------------- + +Android currently supports 4 architectures. Their names as used by the Android +tools are: + +* ``armeabi-v7a`` +* ``arm64-v8a`` +* ``x86`` +* ``x86_64`` + +Virtually all current physical devices use one of the ARM architectures. ``x86`` +and ``x86_64`` are supported for use in the emulator. + +For Python 3.13 we propose that Tier 3 support will only cover the 64-bit platforms +(``arm64-v8a`` and ``x86_64``): + +* ``x86`` has not been supported as a development platform since 2020, and no + new emulator images have been released since then. + +* ``armeabi-v7a``'s proportion of active devices is now + `less than 10% and steadily falling + `__. + + It would also be more difficult to cover with an automated test, since there + are no native hosts available for the emulator (ARM64 Macs don't have hardware + support for ARM32 code). Although cross-architecture emulation is possible, it + has much worse performance and stability, which is why the ``armeabi-v7a`` + emulator images have not been updated since 2016. + + However, it continues to be used for watches and ultra-low-cost phones. If + this persists, we may need to consider adding it in a future Python version. + +Even if 32-bit architectures are not officially supported, no changes should be +made which would impede any downstream projects which still wish to build them. + + +App lifecycle +------------- + +The primary programming language in Android apps is Java, or its modern descendant +Kotlin. As such, an app does not provide its own executable file. Instead, all +apps start off as a Java virtual machine running an executable provided by the +operating system. The app's Java code can then add native code to the process by +loading dynamic libraries and calling them through JNI. + +Unlike iOS, creating subprocesses *is* supported on Android. However apps may +only run executables in `certain locations +`__, none of which +are writable at runtime. Long-running subprocesses are `officially discouraged +`__, and are not +guaranteed to be supported in future Android versions. + +Android does provide a command-line shell, but this is intended only for use by +developers, and is not available to the typical end user. + +For these reasons, the primary way of running Python on Android will be by +loading ``libpython3.x.so`` into the main app process. Although there will also +be a ``python3.x`` executable linked against ``libpython3.x.so``, this is only +for debugging, not production use. + + +Specification +============= + +Scope of work +------------- + +The focus of this work will be to produce an Android equivalent to the existing +`Windows embeddable package +`__, +i.e. a set of compiled libraries which developers +can add to their apps. No installer will be required. + +Adding Android as a Tier 3 platform only requires adding support for compiling +an Android-compatible build from the unpatched CPython source code. It does not +necessarily require there to be any officially distributed Android artifacts on +python.org, although these could be added in the future. + +A Gradle project will be provided for the purpose of running the CPython test +suite. Tooling will be provided to automate the process of building the test +suite app, starting the emulator, installing the test suite, and executing +it. + + +Linkage +------- + +For the reasons discussed in `App lifecycle`_, Python will be included in the +app as a dynamic ``libpython3.x.so`` library. All Android extension modules should +be linked against this library. This allows using the +``-Wl,--no-undefined`` option to detect missing symbols at build time, which can +be a significant time-saver. + +An extension module linked against ``libpython3.x.so`` cannot be loaded by an +executable that has been statically linked against ``libpython3.x.a``. +Therefore, a static ``libpython3.x.a`` library will not be supported on Android. +This is the same pattern used by CPython on Windows. + +Unlike iOS, Android allows dynamic libraries to be loaded from any location, so +a directory tree containing co-located .py, .pyc and .so files can be handled by +Python's standard importer. + + +Standard library +---------------- + +A number of standard library modules will not be supported on Android because +the underlying C APIs are not available: + +* ``curses`` and ``readline`` +* ``dbm.gnu`` and ``dbm.ndbm`` +* ``grp`` and ``spwd`` +* ``multiprocessing`` – although subprocesses in general are allowed (see `App + lifecycle`_), Android does not support any part of the `System V IPC API + `__. +* ``tkinter`` and ``turtle`` – these would require an Android build of Tk + itself, which is not officially supported. + + +Platform identification +----------------------- + +``sys`` +''''''' + +``sys.platform`` will return ``"android"``. Although Android is based on Linux, +it differs in enough significant ways that a separate name is justified. + +``platform`` +'''''''''''' + +Most of the values returned by the ``platform`` module will match those returned +by ``os.uname()``, with the exception of: + +* ``platform.system()`` - ``"Android"``, instead of the default ``"Linux"`` + +* ``platform.release()`` - Android version number, as a string (e.g. ``"14"``), + instead of the Linux kernel version + +In addition, a ``platform.android_ver()`` method will be added, which returns a +namedtuple containing the following: + +* ``release`` - Android version, as a string (e.g. ``"14"``) +* ``api_level`` - Android API level, as an integer (e.g. ``34``) +* ``min_api_level`` - Minimum API level this build of Python can run on, as + an integer (e.g. ``23``). This is the same as ``sys.getandroidapilevel``. +* ``model`` - the model name of the device, as a string (e.g. ``"Pixel 7"``). + +``os`` +'''''' + +``os.uname()`` will return the raw result of a POSIX ``uname()`` call. This will +result in the following values: + +* ``sysname`` - ``"Linux"`` + +* ``release`` - The Linux kernel version (e.g. + ``"5.10.157-android13-4-00003-gdfb1120f912b-ab10994928"``) + +This approach treats the ``os`` module as a "raw" interface to system APIs, and +``platform`` as a higher-level API providing more generally useful values. + + +CI resources +------------ + +Since Android emulators and physical devices use the same ABI, and come with +identical or very similar operating system binaries, testing on emulators will +be adequate. x86_64 emulators can be run on Linux, macOS or Windows, but ARM64 +emulators are only supported on ARM64 Macs. + +GitHub Actions is able to host Android emulators on their Linux and macOS +runners. The free tier currently only provides x86_64 machines; however ARM64 +macOS runners `recently became available on paid plans `__. + +If necessary, `Anaconda `__ has also offered to provide +Android CI resources. + + +Packaging +--------- + +Android wheels will use tags in the format ``android__``. For +example: + +* ``android_23_arm64_v8a`` +* ``android_23_x86_64`` + +For the meaning of ````, see `OS versions`_. In the context of +the wheel tag, it indicates the minimum Android version that was selected when +the wheel was compiled. Installation tools such as pip should interpret this in +a similar way to the existing macOS tags, i.e. an app with a minimum API level +of N can incorporate wheels tagged with API level N or older. + +This format originates from the Chaquopy project, which currently maintains a +`wheel repository `__ with tags varying between +API levels 16 and 21. + +However, relying on a small group of Android enthusiasts to build the whole +Python ecosystem is not a scalable solution. Until prominent libraries routinely +release their own Android wheels, the ability of the community to adopt +Python on Android will be limited. + +Therefore, it will be necessary to clearly document how projects can add Android +builds to their CI and release tooling. Adding Android support to tools like +`crossenv `__ and `cibuildwheel +`__ may be one way to achieve this. + +The Android wheel tag format should also be added to the list of tags accepted +by PyPI. + + +PEP 11 Update +------------- + +:pep:`11` will be updated to include the two supported Android ABIs. Autoconf +already identifies them with the following triplets: + +* ``aarch64-linux-android`` +* ``x86_64-linux-android`` + +Petr Viktorin will serve as the initial core team contact for these ABIs. + + +Backwards Compatibility +======================= + +Adding a new platform does not introduce any backwards compatibility concerns to +CPython itself. However, there may be some backwards compatibility implications +on the projects that have historically provided CPython support (i.e., BeeWare +and Kivy) if the final form of any CPython patches don't align with the patches +they have historically used. + + +Security Implications +===================== + +Adding a new platform does not add any new security implications. + + +How to Teach This +================= + +The education needs related to this PEP relate to two groups of developers. + +First, developers of *apps* need to know how to build Python into an Android +app, along with their own Python code and any supporting packages, and how to +use them all at runtime. The documentation will cover this in a similar form to +the existing `Windows embeddable package +`__. +However, it will recommend most developers to use higher-level tools such as +`Briefcase `__, +`Chaquopy `__ and `Buildozer +`__, all of which already have +comprehensive documentation. + +Second, developers of *packages* with binary components need to know how to +build and release them for Android (see `Packaging`_). + + +Reference Implementation +======================== + +The `Chaquopy repository +`__ contains a reference +patch and build scripts. These will have to be decoupled from the other +components of Chaquopy before they can be upstreamed. + +`Briefcase `__ provides a reference +implementation of code to execute test suites on Android devices and emulators. +The `Toga Testbed `__ is an +example of a test suite that is executed on the Android emulator using GitHub +Actions. + + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive.