PEP 513: latest update from Nathaniel

pep-0513.txt

@@ -9,7 +9,7 @@ Status: Draft
 Type: Informational
 Content-Type: text/x-rst
 Created: 19-Jan-2016
-Post-History: 19-Jan-2016, 25-Jan-2016
+Post-History: 19-Jan-2016, 25-Jan-2016, 29-Jan-2016
 
 
 Abstract
@@ -240,7 +240,7 @@ well as the latest releases of Python and ``pip``.
 Auditwheel
 ----------
 
-The second tools is a command line executable called ``auditwheel`` [10]_ that
+The second tool is a command line executable called ``auditwheel`` [10]_ that
 may aid in package maintainers in dealing with third-party external
 dependencies.
 
@@ -347,31 +347,69 @@ think otherwise.
 We know of three main sources of potential incompatibility that are likely to
 arise in practice:
 
+* Eventually, in the future, there may exist distributions that break
+  compatibility with this profile (e.g., if one of the libraries in
+  the profile changes its ABI in a backwards-incompatible way)
 * A linux distribution that is too old (e.g. RHEL 4)
 * A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is
   based on musl ``libc``, or Android)
-* Eventually, in the future, there may exist distributions that break
-  compatibility with this profile
 
-To handle the first two cases, we propose the following simple and reliable
+Therefore, we propose a two-pronged approach. To catch the first
-check: ::
+case, we standardize a mechanism for a Python distributor to signal
+that a particular Python install definitely is or is not compatible
+with ``manylinux1``: this is done by installing a module named
+``_manylinux``, and setting its ``manylinux1_compatible``
+attribute. We do not propose adding any such module to the standard
+library -- this is merely a well-known name by which distributors and
+installation tools can rendezvous. However, if a distributor does add
+this module, *they should add it to the standard library* rather than
+to a ``site-packages/`` directory, because the standard library is
+inherited by virtualenvs (which we want), and ``site-packages/`` in
+general is not.
 
-    def have_glibc_version(major, minimum_minor):
+Then, to handle the latter two cases for existing Python
+distributions, we suggest a simple and reliable method to check for
+the presence and version of ``glibc`` (basically using it as a "clock"
+for the overall age of the distribution).
+
+Specifically, the algorithm we propose is::
+
+    def is_manylinux1_compatible():
+        # Only Linux, and only x86-64 / i386
+        from distutils.util import get_platform
+        if get_platform() not in ["linux_x86_64", "linux_i386"]:
+            return False
+
+        # Check for presence of _manylinux module
+        try:
+            import _manylinux
+            return bool(_manylinux.manylinux1_compatible)
+        except (ImportError, AttributeError):
+            # Fall through to heuristic check below
+            pass
+
+        # Check glibc version. CentOS 5 uses glibc 2.5.
+        return have_compatible_glibc(2, 5)
+
+    def have_compatible_glibc(major, minimum_minor):
         import ctypes
 
         process_namespace = ctypes.CDLL(None)
         try:
             gnu_get_libc_version = process_namespace.gnu_get_libc_version
         except AttributeError:
-            # We are not linked to glibc.
+            # Symbol doesn't exist -> therefore, we are not linked to
+            # glibc.
             return False
 
+        # Call gnu_get_libc_version, which returns a string like "2.5".
         gnu_get_libc_version.restype = ctypes.c_char_p
         version_str = gnu_get_libc_version()
         # py2 / py3 compatibility:
         if not isinstance(version_str, str):
             version_str = version_str.decode("ascii")
 
+        # Parse string and check against requested version.
         version = [int(piece) for piece in version_str.split(".")]
         assert len(version) == 2
         if major != version[0]:
@@ -380,30 +418,40 @@ check: ::
             return False
         return True
 
-    # CentOS 5 uses glibc 2.5.
+**Rejected alternatives:** We also considered using a configuration
-    is_manylinux1_compatible = have_glibc_version(2, 5)
+file, e.g. ``/etc/python/compatibility.cfg``. The problem with this is
+that a single filesystem might contain many different interpreter
+environments, each with their own ABI profile -- the ``manylinux1``
+compatibility of a system-installed x86_64 CPython might not tell us
+much about the ``manylinux1`` compatibility of a user-installed i386
+PyPy. Locating this configuration information within the Python
+environment itself ensures that it remains attached to the correct
+binary, and dramatically simplifies lookup code.
 
-To handle the third case, we propose the creation of a file
+We also considered using a more elaborate structure, like a list of
-``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
+all platform tags that should be considered compatible, together with
-contents: ::
+their preference ordering, for example: ``_binary_compat.compatible =
+["manylinux1_x86_64", "centos5_x86_64", "linux_x86_64"]``. However,
+this introduces several complications. For example, we want to be able
+to distinguish between the state of "doesn't support ``manylinux1``"
+(or eventually ``manylinux2``, etc.) versus "doesn't specify either
+way whether it supports ``manylinux1``", which is not entirely obvious
+in the above representation; and, it's not at all clear what features
+are really needed vis a vis preference ordering given that right now
+the only possible platform tags are ``manylinux1`` and ``linux``.  So
+we're deferring a more complete solution here for a separate PEP, when
+/ if Linux gets more platform tags.
 
-   [manylinux1]
+For the library compatibility check, we also considered much more
-   compatible = true
+elaborate checks (e.g. checking the kernel version, searching for and
+checking the versions of all the individual libraries listed in the
+``manylinux1`` profile, etc.), but ultimately decided that this would
+be more likely to introduce confusing bugs than actually help the
+user. (For example: different distributions vary in where they
+actually put these libraries, and if our checking code failed to use
+the correct path search then it could easily return incorrect
+answers.)
 
-where the supported values for the ``manylinux1.compatible`` entry are the
-same as those supported by the ConfigParser ``getboolean`` method.
-
-The proposed logic for ``pip`` or related tools, then, is:
-
-0) If ``distutils.util.get_platform()`` does not start with the string
-   ``"linux"``, then assume the current system is not ``manylinux1``
-   compatible.
-1) If ``/etc/python/compatibility.conf`` exists and contains a ``manylinux1``
-   key, then trust that.
-2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
-   current system can handle ``manylinux1`` wheels.
-3) Otherwise, assume that the current system cannot handle ``manylinux1``
-   wheels.
 
 
 PyPI Support