PEP 466: narrow scope, add legacy SSL branch

pep-0466.txt

@@ -33,17 +33,24 @@ internet. In particular, the exception will apply to:
 * the ``ssl`` module
 * the ``hashlib`` module
 * the ``hmac`` module
-* the ``sha`` module (Python 2 only)
+* the components of the ``random`` and ``os`` modules that the above
-* the components of other networking modules that make use of these modules
+  modules rely on for cryptographic randomness
-* the components of the ``random`` and ``os`` modules that are relevant to
+* the version of OpenSSL bundled with the binary installers for Windows
-  cryptographic applications
+  and Mac OS X
-* the version of OpenSSL bundled with the binary installers
 
-Proposed backports for these modules will still need to undergo normal
+Changes to these modules will still need to undergo normal backwards
-backwards compatibility assessments, but new features will be permitted where
+compatibility assessments, but otherwise they will be kept entirely aligned
-appropriate, making it easier to implement secure networked software in
+with the latest feature release of their Python 3 counterparts. This is
-Python, even for software that needs to remain compatible with older feature
+designed to make it easier to implement secure networked software in
-releases of Python.
+Python, even for software that currently needs to remain compatible with
+the Python 2 series (which includes a lot of network infrastructure
+software).
+
+The development branches will be arranged in such a way that even though
+any further Python 2.7 releases published by the core development team
+provide updated network security infrastructure, it will remain possible
+for downstream redistributors to publish "Python 2.7 with legacy SSL
+infrastructure" if they choose to do so.
 
 While this PEP does not make any changes to the core development team's
 handling of security-fix-only branches that are no longer in active
@@ -66,7 +73,6 @@ their counterparts in the latest feature release of Python 3:
 * the ``ssl`` module
 * the ``hashlib`` module
 * the ``hmac`` module
-* the ``sha`` module (Python 2 only)
 
 This exemption applies to *all* proposals to backport backwards compatible
 changes in these modules to Python 2.7 maintenance releases. This choice is
@@ -80,21 +86,22 @@ As part of this policy, permission is also granted to upgrade to newer
 feature releases of OpenSSL when preparing the binary installers
 for new maintenance releases of CPython.
 
+Note that the ``sha`` and ``md5`` modules are not covered by this policy,
+but have been deprecated in favour of ``hashlib`` since Python 2.5 and have
+been removed entirely in the Python 3 series.
+
 In addition to the above blanket exemption, a conditional exemption is
 granted for these modules that may include some network security related
 features:
 
 * the ``os`` module (specifically ``os.urandom``)
 * the ``random`` module
-* networking related modules that depend on one or more of the network
-  security related modules listed above
 
 This more limited exemption for these modules requires that the *specific*
 enhancement being proposed for backporting needs to be justified as being
-network security related. If the enhancement under discussion is designed
+network security related. This is generally restricted to cases where the
-to take advantage of a new feature in one of the network security related
+feature in question is needed by an update to one of the modules covered
-modules, then that will be taken as implying that the enhancement is
+by the blanket exemption.
-security related.
 
 
 Backwards Compatibility Considerations
@@ -117,8 +124,8 @@ the Python version.
 It is then up to library and framework code to provide an appropriate warning
 and fallback behaviour if a desired feature is found to be missing. While
 some especially security sensitive software MAY fail outright if a desired
-security feature is unavailable, most software SHOULD instead continue
+security feature is unavailable, most software SHOULD instead emit a warning
-operating using a slightly degraded security configuration.
+and continue operating using a slightly degraded security configuration.
 
 Affected APIs SHOULD be designed to allow library and application code to
 perform the following actions after detecting the presence of a relevant
@@ -132,13 +139,15 @@ network security related feature:
   Python version checks to determine the Python feature release, but MUST
   NOT require checking for a specific maintenance release)
 
-Security related changes to other modules (such as data format processing
+Security related changes to other modules (such as higher level networking
-libraries) will continue to be made available as backports and new modules
+libraries and data format processing libraries) will continue to be made
-on the Python Package Index, as independent distribution remains the
+available as backports and new modules on the Python Package Index, as
-preferred approach to handling software that needs to evolve faster than
+independent distribution remains the preferred approach to handling
-the standard library. Refer to the `Motivation and Rationale`_ section for
+software that must continue to evolve to handle changing development
-a review of the characteristics that make the secure networking
+requirements independently of the Python 2 standard library. Refer to
-infrastructure worthy of special consideration.
+the `Motivation and Rationale`_ section for a review of the characteristics
+that make the secure networking infrastructure worthy of special
+consideration.
 
 
 Other Considerations
@@ -151,23 +160,24 @@ This policy does NOT represent a commitment by volunteer contributors to
 actually backport network security related changes from the Python 3 series
 to the Python 2 series. Rather, it is intended to send a clear signal to
 potential corporate contributors that the core development team are willing
-to review and merge corporate contributions that put this policy into
+to accept offers of corporate assistance in putting this policy into
-effect.
+effect and handling the resulting increase in the Python 2 maintenance
+load.
 
 Backporting security related fixes and enhancements to earlier versions is
 a common service for commercial redistributors to offer to their customers.
 This policy represents an explicit invitation to contribute some of those
 changes back to the upstream community in cases where they are likely to
 have a broad impact that helps improve the security of the internet as a
-whole.
+whole, rather than sitting back and waiting for unpaid volunteers to do it
+for them.
 
 
 Documentation
 -------------
 
-All modules that take advantage of this policy to backport network
+All modules covered by this policy MUST include a "Security Considerations"
-security related enhancements to earlier Python versions MUST include
+section in their documentation.
-a "Security Considerations" section in their documentation.
 
 In addition to any other module specific contents, this section MUST
 enumerate key security enhancements and fixes (with CVE identifiers where
@@ -183,23 +193,49 @@ releases - those will continue to be source only releases that
 include only critical security fixes.
 
 However, the recommendations for library and application developers are
-deliberately designed to accommodate commercial redistributors applying
+deliberately designed to accommodate commercial redistributors that choose
-this policy to any Python release series that is either in security
+to apply this policy to additional Python release series that are either in
-fix only mode, or has been declared "end of life" by the core development
+security fix only mode, or have been declared "end of life" by the core
-team.
+development team.
 
 Whether or not redistributors choose to exercise that option will be up
-to the redistributor.
+to the individual redistributor.
 
 
 Integration testing
 -------------------
 
-Third party integration testing services would likely need to start
+Third party integration testing services should offer users the ability
-offering users a choice of multiple Python 2.7.x versions to test against,
+to test against both the latest Python 2.7 maintenance release and the
-to ensure that the application is correctly degrading gracefully if it
+latest "Python 2.7 with legacy SSL infrastructure" release, to ensure that
-attempts to use newer networking features on maintenance releases that
+libraries, frameworks and applications handle the legacy security
-are too old to provide them.
+infrastructure correctly (either failing or degrading gracefully, depending
+on the security sensitivity of the software).
+
+
+Handling lower security environments with low risk tolerance
+------------------------------------------------------------
+
+For better or for worse (mostly worse), there are some environments where
+the risk of latent security defects is more tolerated than the risk of
+regressions in maintenance releases. This policy largely excludes these
+environments from consideration where the modules covered by the exemption
+are concerned.
+
+However, one concession is made for the benefit of such environments: while
+the main ``2.7`` branch in Mercurial will include the updated network
+security infrastructure, the development process will be updated to include
+a ``2.7-legacy-ssl`` branch with the more limited feature set of the
+original 2.7 network security infrastructure, allowing downstream
+redistributors to continue to provide Python 2.7 with the legacy SSL
+infrastructure if they choose to do so. This branch will be tested
+on the CPython continuous integration infrastructure, but no releases will
+be made from it by the core development team.
+
+As noted above, corporate redistributors and users are expected to provide
+the additional development effort needed to make this practical. It is not
+acceptable to expect volunteer contributors to resolve a problem created
+largely by conservative corporate development practices.
 
 
 Evolution of this Policy
@@ -212,8 +248,9 @@ in Python and the system that application is running on. Thus the focus on
 network security protocols and related cryptographic infrastructure - Python
 is a popular choice for the development of web services and clients, and
 thus the capabilities of widely used Python versions have implications for
-the security design of other services that may be using newer versions of
+the security design of other services that may themselves be using newer
-Python or other development languages.
+versions of Python or other development languages, but need to interoperate
+with clients or servers written using older versions of Python.
 
 The intent behind this requirement is to minimise any impact that the
 introduction of this policy may have on the stability and compatibility of
@@ -282,7 +319,7 @@ on security sensitive comparisons than the Python 3 series, as it lacks a
 standard library equivalent to the timing attack resistant
 ``hmac.compare_digest()`` function. While appropriate secure comparison
 functions can be implemented in third party extensions, may users don't
-even consider the problem and use ordinary equality comparisons instead
+even consider the issue and use ordinary equality comparisons instead
 - while a standard library solution doesn't automatically fix that problem,
 it *does* make the barrier to resolution much lower once the problem is
 pointed out.
@@ -307,37 +344,57 @@ Alternative: advise developers of networked software to migrate to Python 3
 This alternative represents the status quo. Unfortunately, it has proven
 to be unworkable in practice, as the backwards compatibility implications
 mean that this is a non-trivial migration process for large applications
-and integration projects.
+and integration projects. While the tools for migration have evolved to
+a point where it is possible to migrate even large applications
+opportunistically and incrementally (rather than all at once) by updating
+code to run in the large common subset of Python 2 and Python 3, using the
+most recent technology often isn't a priority in commercial environments.
 
-Now that we're fully aware of the impact the limitations in Python 2 may be
+Previously, this was considered an acceptable harm, as while it was an
-having on the evolution of internet security standards, I no longer believe
+unfortunate problem for the affected developers to have to face, it was
-that it is reasonable to expect platform and application developers to
+seen as an issue between them and their management chain to make the case
-resolve all of the latent defects in an application's Unicode correctness
+for infrastructure modernisation, and this case would become naturally
-solely in order to gain access to the network security enhancements
+more compelling as the Python 3 series evolved.
-available in Python 3.
 
-While (as far as I am aware) Ubuntu has successfully switched to Python 3.4
+However, now that we're fully aware of the impact the limitations of the
-as its main Python interpreter for its 14.04 LTS release, Fedora still
+Python 2 standard library may be having on the evolution of internet
-has a lot of work to do to migrate, and it will take a non-trivial amount
+security standards, I no longer believe that it is reasonable to expect
-of time to migrate the relevant infrastructure components. While Red Hat
+platform and application developers to resolve all of the latent defects
-are also actively working to make it easier for users to use more recent
+in an application's Unicode correctness solely in order to gain access to
-versions of Python on our stable platforms, it's going to take time for
+the network security enhancements already available in Python 3.
-those efforts to start having an impact on end users' choice of version,
+
-and those changes won't affect the core tools regardless.
+While Ubuntu (and to some extent Debian as well) are committed to porting all
+default system services and scripts to Python 3, and to removing Python 2
+from its default distribution images (but not from its archives), this is
+a mammoth task and won't be completed for the Ubuntu 14.04 LTS release
+(at least for the desktop image - it may be achieved for the mobile and
+server images).
+
+Fedora has even more work to do to migrate, and it will take a non-trivial
+amount of time to migrate the relevant infrastructure components. While
+Red Hat are also actively working to make it easier for users to use more
+recent versions of Python on our stable platforms, it's going to take time
+for those efforts to start having an impact on end users' choice of version,
+and any such changes also don't benefit the core platform infrastructure
+that runs in the integrated system Python by necessity.
 
 The OpenStack migration to Python 3 is also still in its infancy, and even
 though that's a project with an extensive and relatively robust automated
-test suite, it's large enough that it is going to take quite some time
+test suite, it's still large enough that it is going to take quite some time
 to migrate.
 
 And that's just three of the highest profile open source projects that
 make heavy use of Python. Given the likely existence of large amounts of
 legacy code that lacks the kind of automated regression test suite needed
-to help support a migration from Python 2 to Python 3. The key point of
+to help support a migration from Python 2 to Python 3, there are likely to
-this PEP is that those situations affect more people than just the
+be many cases where reimplementation (perhaps even in Python 3) proves
-developers and users of the affected application: their existence becomes
+easier than migration. The key point of this PEP is that those situations
-something that developers of secure networked services need to take into
+affect more people than just the developers and users of the affected
-account as part of their security design.
+application: the existence of clients and servers with outdated network
+security infrastructure becomes something that developers of secure
+networked services need to take into account as part of their security
+design, and that's a problem that inhibits the adoption of better security
+standards.
 
 As Terry Reedy noted, if we try to persist with the status quo, the likely
 outcome is that commercial redistributors will attempt to do something
@@ -366,45 +423,36 @@ would likely just skip entirely.
 
 Attempting to create a Python 2.8 release would also bring in suggestions
 to backport many additional features from Python 3 (such as ``tracemalloc``
-and the improved coroutine support).
+and the improved coroutine support), making the migration from Python 2.7
+to this hypothetical 2.8 release even riskier and more disruptive.
 
 This is not a recommended approach, as it would involve substantial
-additional work for a result that is actually less effective as a solution
+additional work for a result that is actually less effective in achieving
-to the original problem (the widespread use of the aging network security
+the original aim (which is to eliminate the current widespread use of the
-infrastructure in Python 2).
+aging network security infrastructure in the Python 2 series).
 
 
 Alternative: distribute the security enhancements via PyPI
 ----------------------------------------------------------
 
-While it initially appears to be an attractive and easier to manage
+While this initially appears to be an attractive and easier to manage
-approach, there are actually several significant problems with this
+approach, it actually suffers from several significant problems.
-idea.
 
-Firstly, this PEP encompasses a non-trivial portion of the standard library.
+Firstly, this is complex, low level, cross-platform code that integrates
-It's not just the underlying SSL support, but also the libraries for other
-network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the
-SSL infrastructure to provide secure links, and that's just the protocols
-in the standard library.  Even if an API compatible ``ssl2`` module was
-made available, it would need to be imported and injected
-into ``sys.modules`` as ``ssl`` before importing any other module that
-needed it.
-
-Secondly, this is complex, low level, cross-platform code that integrates
 with the underlying operating system across a variety of POSIX platforms
 (including Mac OS X) and Windows. The CPython BuildBot fleet is already set
 up to handle continuous integration in that context, but most of the
 freely available continuous integration services just offer Linux, and
 perhaps paid access to Windows. Those services work reasonably well for
 software that largely runs on the abstraction layers offered by Python and
-other dynamic languages, but won't suffice for the kind of code involved
+other dynamic languages, as well as the more comprehensive abstraction
-here.
+offered by the JVM, but won't suffice for the kind of code involved here.
 
 The OpenSSL dependency for the network security support also qualifies as
 the kind of "complex binary dependency" that isn't yet handled well by the
-``pip`` based software distribution ecosystem. Relying on a binary
+``pip`` based software distribution ecosystem. Relying on a third party
-dependency also creates potential compatibility problems for ``pip`` when
+binary dependency also creates potential compatibility problems for ``pip``
-running on other interpreters like ``PyPy``.
+when running on other interpreters like ``PyPy``.
 
 Another practical problem with the idea is the fact that ``pip`` itself
 relies on the ``ssl`` support in the standard library (with some additional
@@ -412,8 +460,7 @@ support from a bundled copy of ``requests``, which in turn bundles
 ``backport.ssl_match_hostname``), and hence would require any replacement
 module to also be bundled within ``pip``. This wouldn't pose any
 insurmountable difficulties (it's just another dependency to vendor), but
-it *would* mean yet another copy of OpenSSL to keep up to
+it *would* mean yet another copy of OpenSSL to keep up to date.
-date.
 
 This approach also has the same flaw as all other "improve security by
 renaming things" approaches: they completely miss the users who most need
@@ -422,17 +469,20 @@ to do the right thing when their infrastructure supports it (since
 "use this other module" is a much higher impact change than "turn on this
 higher security setting"). Deprecating the aging SSL infrastructure in the
 standard library in favour of an external module would be even more user
-hostile than taking the risk of trying to upgrade it in place.
+hostile than accepting the slightly increased risk of regressions associated
+with upgrading it in place.
 
 Last, but certainly not least, this approach suffers from the same problem
 as the idea of doing a Python 2.8 release: likely not solving the actual
 problem. Commercial redistributors of Python are set up to redistribute
 *Python*, and a pre-existing set of additional packages. Getting new
-packages added to the pre-existing set *can* be done, but means going
+packages added to the pre-existing set *can* be done, but means approaching
-around to each and every redistributor and asking them to update their
+each and every redistributor and asking them to update their
 repackaging process accordingly. By contrast, the approach described in
-this PEP would require redistributors to *opt out* of the security
+this PEP would require redistributors to deliberately *opt out* of the
-enhancements, which most of them are unlikely to do.
+security enhancements (by switching to redistributing directly from the
+``2.7-legacy-ssl`` branch rather than the main ``2.7`` branch), which most
+of them are unlikely to do.
 
 
 Open Questions
@@ -445,9 +495,6 @@ Open Questions
   feature release. In particular, is it possible Windows C extensions may
   be linking against the Python provided OpenSSL module?
 
-* Are there any other security relevant modules that should be covered
-  by either a blanket or conditional exemption?
-
 
 Disclosure of Interest
 ======================
@@ -473,13 +520,14 @@ of the web as a whole.
 Christian and Donald Stufft also provided valuable feedback on a preliminary
 draft of this proposal.
 
-Thanks also to participants in the python-dev mailing list thread [1]_
+Thanks also to participants in the python-dev mailing list threads [1,2]_
 
 
 References
 ==========
 
 .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html
+.. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html
 
 
 Copyright