Additional rationale in PEP 453

I realised after the last round of reviews that a *big* chunk of the
rationale for this PEP was assumed knowledge on distutils-sig, and
this lead directly to the high rate of replies on python-dev that
just didn't get why the requested exemption to include the new
feature in Python 2.7.6 is so important.

The PEP now attempts to do a better job of bridging that gap, as
well as further expanding on how having the ``ensurepip`` module
fully public in older releases should actually make things *less*
confusing, rather than more, since it gives us a location to
explain the alternative fallback bootstrapping methods.

pep-0453.txt

@@ -26,12 +26,14 @@ of that recommendation.
 Rationale
 =========
 
-This change is considered a necessary step in the evolution of the Python
+There are two related, but distinct rationales for the proposal in this
-packaging ecosystem, as the community has embraced the Python Package Index
+PEP. The first relates to the experience of new users, while the second
-as a mechanism for distributing and installing Python software, but the
+relates to better enabling the evolution of the broader Python packaging
-different concerns of language evolution and secure software distribution
+ecosystem.
-mean that a faster feature release cycle is needed to properly support the
+
-latter.
+
+Improving the new user experience
+---------------------------------
 
 Currently, on systems without a platform package manager and repository,
 installing a third-party Python package into a freshly installed Python
@@ -130,6 +132,114 @@ provides a reasonably secure, cross platform mechanism to get access to
 these utilities.
 
 
+Enabling the evolution of the broader Python packaging ecosystem
+----------------------------------------------------------------
+
+As no new packaging standard can achieve widespread adoption without a
+transition strategy that covers the versions of Python that are in
+widespread *current* use (rather than merely future versions, like most
+language features), the change proposed in this PEP is considered a
+necessary step in the evolution of the Python packaging ecosystem
+
+The broader community has embraced the Python Package Index as a mechanism
+for distributing and installing Python software, but the different concerns
+of language evolution and secure software distribution mean that a faster
+feature release cycle that encompasses older versions is needed to properly
+support the latter.
+
+In addition, the core CPython development team have the luxury of
+dropping support for earlier Python versions well before the rest of the
+community, as downstream commercial redistributors pick up the task of
+providing support for those versions to users that still need it, while
+many third party libraries maintain compatibility with those versions as
+long as they remain in widespread use.
+
+This means that the current ``setup.py install`` based model for package
+installation poses serious difficulties for the development and adoption
+of new packaging standards, as, depending on how a project writes their
+``setup.py`` file, the installation command (along with other operations)
+may end up invoking the standard library's ``distutils`` package.
+
+As an indicator of how this may cause problems for the broader ecosystem,
+consider that the feature set of ``distutils`` in Python 2.6 was frozen
+in June 2008 (with the release of Python 2.6b1), while the feature set of
+``distutils`` in Python 2.7 was frozen in April 2010 (with the release of
+Python 2.7b1).
+
+By contrast, using a separate installer application like ``pip`` (which
+ensures that even ``setup.py`` files that invoke ``distutils`` directly
+still support the new packaging standards) makes it possible to support
+new packaging standards in older versions of Python, just by upgrading
+``pip``. The situation on older versions of Python is further improved by
+making it easier for end users to install and upgrade newer build systems
+like ``setuptools`` or improved PyPI upload utilities like ``twine``.
+
+It is not coincidental that this proposed model of using a separate installer
+program with more metadata heavy and less active distribution formats matches
+that used by most operating systems (including Windows since the introduction
+of the installer service and the MSI file format), as well as many other
+language specific installers.
+
+For Python 2.6, this compatibility issue is largely limited to various
+enterprise Linux distributions (and their downstream derivatives). These
+distributions often have even slower update cycles than CPython, so they
+offer full support for versions of Python that are considered "security
+fix only" versions upstream (and sometimes may even be to the point where
+the core development team no longer support them at all - you can still get
+commercial support for Python 2.3 if you really need it!).
+
+In practice, the fact that tools like ``wget`` and ``curl`` are readily
+available on Linux systems, that most users of Python on Linux are
+already familiar with the command line, and that most Linux distributions
+ship with a default configuration that makes running Python scripts easy,
+means that the existing ``pip`` bootstrapping instructions for any \*nix
+system are already quite straightforward. Even if ``pip`` isn't provided by
+the system package manager, then using ``wget`` or ``curl`` to retrieve the
+bootstrap script from www.pip-installer.org and then running it is just a
+couple of shell commands that can easily be copied and pasted as necessary.
+
+Accordingly, for any version of Python on any \*nix system, the need to
+bootstrap ``pip`` in older versions isn't considered a major barrier to
+adoption of new packaging standards, since it's just one more small
+speedbump encountered by users of these long term stable releases. For
+\*nix systems, this PEP's formal endorsement of ``pip`` as the preferred
+default packaging tool is seen as more important than the underlying
+technical details involved in making ``pip`` available by default, since
+it shifts the nature of the conversation between the developers of ``pip``
+and downstream repackagers of both ``pip`` and CPython.
+
+For Python 2.7, on the other hand, the compatibility issue for adopting new
+metadata standards is far more widespread, as it affects the python.org
+binary installers for Windows and Mac OS X, as well as even relatively
+fast moving \*nix platforms.
+
+Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported
+upstream version, and will remain so until the release of Python 2.7.9
+(currently scheduled for May 2015), at which time it is expected to enter
+the usual "security fix only" mode. That means there are at least another
+19 months where Python 2.7 is a deployment target for Python applications
+that enjoys full upstream support. Even after the core development team
+switches 2.7 to security release only mode in 2015, Python 2.7 will likely
+remain a commercially supported legacy target out beyond 2020.
+
+While Python 3 already presents a compelling alternative over Python 2 for
+*new* Python applications and deployments without an existing investment
+in Python 2 and without a dependency on specific Python 2 only third party
+modules (a set which is getting ever smaller over time), it is going to take
+longer to create compelling business cases to update existing Python 2.7
+based infrastructure to Python 3, especially in situations where the culture
+of automated testing is weak (or nonexistent), making it difficult to
+effectively use the available migration utilities.
+
+It is quite likely that it is this difference in perspective regarding what
+it means for a version of Python to be "supported" which lies at the heart
+of the long history of conflicts between the developers of Python packaging
+tools and the core development team for CPython. A key goal of this PEP is
+thus to better enable the two groups to collaborate more effectively, by
+using the ``ensurepip`` module as the technical bridge between the two
+distinct software lifecycles and deployment models.
+
+
 Why pip?
 --------
 
@@ -170,7 +280,7 @@ utility).
 To support that end, this PEP proposes the inclusion of an ``ensurepip``
 bootstrapping module in Python 3.4 (along with the upcoming maintenance
 releases of Python 2.7 and 3.3), as well as changes to the way Python
-installed scripts are handled on Windows.
+installed scripts are handled on Windows (for Python 3.4 only).
 
 To clearly demarcate development responsibilities, and to avoid
 inadvertently downgrading ``pip`` when updating CPython, the proposed
@@ -179,10 +289,10 @@ mechanism in the standard library that is invoked automatically by the
 CPython installers provided on python.org.
 
 To ensure the smoothest possible experience for new users of Python, this
-PEP also proposes that the installer changes and the ``ensurepip`` module
+PEP also proposes that the ``ensurepip`` module and the option to install
-be backported to Python 2.7 and 3.3. It does *not* propose backporting any
+``pip`` when installing CPython be backported to Python 2.7 and 3.3. It
-changes to ``pyvenv`` (in Python 3.3) or to Windows script handling (in
+does *not* propose backporting any changes to ``pyvenv`` (in Python 3.3)
-either version).
+or to Windows script handling (in either version).
 
 Finally, the PEP also strongly recommends that CPython redistributors and
 other Python implementations ensure that ``pip`` is available by default, or
@@ -436,7 +546,26 @@ updated to reference the revised module installation guide.
 
 In Python 2.7 and 3.3, the documentation will make clear that the feature
 was added in a maintenance release and users may need to upgrade in order
-to take advantage of it.
+to take advantage of it. Specifically, it is proposed to include the
+following warning as a note in the documentation for the ``ensurepip``
+module in these versions (adjust version numbers for 3.3 as appropriate):
+
+    This is an optional module, which may not be available in all
+    installations of Python 2.7. It is provided solely to simplify the
+    process of bootstrapping ``pip`` onto end user's systems. If it is
+    not available, please investigate the following alternatives:
+
+    * This module was first added in Python 2.7.6. If using an earlier
+      maintenance release, it will not be available. If upgrading to
+      a more recent maintenance release is not an option, consider
+      the alternative bootstrapping mechanisms below.
+    * Some platforms provide alternative mechanisms to obtain ``pip``.
+      In such cases, the platform documentation should provide
+      appropriate details.
+    * If upgrading to the latest maintenance release is not feasible, and
+      no platform specific instructions are provided, then refer to the
+      upstream `pip bootstrapping instructions
+      <http://www.pip-installer.org/en/latest/installing.html>`__.
 
 The existing content of the module installation guide will be retained in
 all versions, but under a new "Invoking distutils directly" subsection.
@@ -523,8 +652,8 @@ maintenance releases". The rationale for doing so in this case is slightly
 different for each of the two versions.
 
 
-Rationale for the policy
+Rationale for this policy on maintenance releases
-------------------------
+-------------------------------------------------
 
 Python's strict "no new features in maintenance releases" was instituted
 following the introduction of a number of new features over the course of
@@ -574,12 +703,20 @@ obtaining ``pip`` for users of Python 2.7 and 3.3 on Windows and Mac OS X
 systems, all of the existing ``pip`` bootstrapping mechanisms will still
 work in cases where upgrading Python isn't a practical alternative.
 
+As described in the documentation update proposal, the ``ensurepip``
+documentation in older releases will include the text explaining how to
+obtain ``pip`` if updating to the latest maintenance release isn't an
+option, or if the module has been removed by a redistributor. This contrasts
+significantly with the changes made during the Python 2.2 series, where they
+were normal additions with no alternatives except to update to a sufficiently
+recent version of Python if a library or application depended on them.
+
 
 Potential consequences of permitting this exemption
 ---------------------------------------------------
 
-There is some legitimate concern that approving an exemption to the
+The concern has been expressed that approving an exemption to the "no new
-"no new features in maintenance releases" policy in this case will open the
+features in maintenance releases" policy in this case will open the
 flood gates to requests for more such exemptions in the future. It is the
 perspective of the PEP authors that the specific nature of this proposal
 should itself serve to allay those fears.
@@ -615,9 +752,10 @@ libraries in the common Python 2/3 subset means there are (as of
 September 2013) still more Python 2 only libraries than there are Python 3
 only libraries.
 
-Since the primary purpose of this PEP is to aid new Python users, it is
+Since one of the primary aims of this PEP is to aid new Python users, it is
 contrary to its spirit to target *only* Python 3.4, when so many users in
-at least the next 12-18 months are still going to be introduced to Python
+at least the next 12-18 months (where Python 2.7 is still fully supported
+by the core development team) are still going to be introduced to Python
 2 before being introduced to Python 3.
 
 Users first installing Python 2.7 on Windows and Mac OS X following
@@ -628,9 +766,22 @@ beginning to explore the PyPI ecosystem, the bootstrapping instructions
 can be simplified to "just install the latest maintenance release of
 CPython".
 
-Making ``pip`` readily available also serves to improve the usability of
+Making ``pip`` readily available also serves to ease the migration path
-Python 2.7 itself, as a number of the standard library additions in Python
+from Python 2 to Python 3, as a number of the standard library additions
-3 are also available from PyPI for Python 2.
+in Python 3 are also available from PyPI for Python 2. Lowering the barrier
+to adoption for these backports makes it easier for current Python 2 users
+to selectively adopt backporting Python 3 versions, reducing the number of
+updates needed in any eventual Python 3 migration.
+
+Finally, this PEP solves a serious problem for the ``distutils-sig``
+community, as it means we will finally have a standard mechanism decoupled
+from the standard library's development lifecycle that we can reasonably
+assume to be present on end user's systems (or at least readily
+available) that allows us to support new packaging standards in older
+versions of Python. A tentative, half-hearted endorsement from the
+CPython core development team that tries to hide the existence of the
+pip boostrapping support from end users is unlikely to provide quite the
+same benefits.
 
 
 Rationale for permitting the exemption in Python 3.3
@@ -788,9 +939,8 @@ downstream distributors:
 * Migrate build systems to utilize `pip`_ and `Wheel`_ wherever feasible
   and avoid directly invoking ``setup.py``.
 
-  * This isn't strictly required by this PEP, but will help ensure a
+  * This will help ensure a smoother and more timely migration to improved
-    smoother and more timely adoption of improved metadata formats as the
+    metadata formats as the Python packaging ecosystem continues to evolve.
-    Python packaging ecosystem continues to evolve.
 
 In the event that a Python redistributor chooses *not* to follow these
 recommendations, we request that they explicitly document this fact and
@@ -908,8 +1058,8 @@ Naming the module `_ensurepip`` in Python 2.7 and 3.3 was considered as
 another means of skirting the "no new features in maintenance releases"
 policy. However, similar to the proposal to only include the new
 feature in the installers rather than the standard library, this feels like
-relying on a technicality to "comply" with the policy, while still breaking
+relying on a technicality to nominally "comply" with the policy, while still
-it in spirit.
+breaking it in spirit.
 
 It is the considered opinion of the PEP authors that attempting to hide
 the addition of the ``ensurepip`` module in earlier versions will only
@@ -917,6 +1067,11 @@ serve to increase confusion rather than to reduce it, so the proposal
 remains to be up front about the fact that the policy is being broken in
 this case, and clearly documenting the rationale for doing so in this PEP.
 
+As noted in the section describing the proposed documentation updates,
+having ``ensurepip`` as a public module in these earlier versions also
+provides a convenient home for the fallback bootstrapping instructions in
+those cases where it *isn't* available.
+
 
 Automatically contacting PyPI when bootstrapping pip
 ----------------------------------------------------