Update PEP 466 based on first round of feedback

pep-0466.txt

@@ -1,5 +1,5 @@
 PEP: 466
-Title: Network Security Enhancement Exception for All Branches
+Title: Network Security Enhancement Exception for Python 2.7
 Version: $Revision$
 Last-Modified: $Date$
 Author: Nick Coghlan <ncoghlan@gmail.com>,
@@ -18,10 +18,17 @@ proposed enhancements. Most patches to fix behavioural errors are
 applied to all active maintenance branches.  Enhancement patches are
 restricted to the default branch that becomes the next Python version.
 
-This PEP relaxes the latter restriction allowing enhancements to be applied
+This cadence works reasonably well during Python's normal 18-24 month
-to maintenance branches for standard library components that have
+feature release cycle, which is still applicable to the Python 3 series.
-implications for the overall security of the internet. In particular, the
+However, the age of the standard library in Python 2 has now reached a point
-exception will apply to:
+where it is sufficiently far behind the state of the art in network security
+protocols for it to be causing real problems in commercial use cases
+where upgrading to Python 3 in the near term may not be practical.
+
+Accordingly, this PEP relaxes the normal restrictions by allowing
+enhancements to be applied in Python 2.7 maintenance releases for standard
+library components that have implications for the overall security of the
+internet. In particular, the exception will apply to:
 
 * the ``ssl`` module
 * the ``hashlib`` module
@@ -32,19 +39,29 @@ exception will apply to:
   cryptographic applications
 * the version of OpenSSL bundled with the binary installers
 
-Changes to these modules will still need to undergo normal backwards
+Proposed backports for these modules will still need to undergo normal
-compatibility assessments, but new features will be permitted where
+backwards compatibility assessments, but new features will be permitted where
 appropriate, making it easier to implement secure networked software in
 Python, even for software that needs to remain compatible with older feature
 releases of Python.
 
+While this PEP does not make any changes to the core development team's
+handling of security-fix-only branches that are no longer in active
+maintenance, it *does* recommend that commercial redistributors providing
+extended support periods for the Python standard library either adopt a
+similar approach to ensuring that the secure networking infrastructure
+keeps pace with the evolution of the internet, or else disclaim support
+for the use of older versions in roles that involving connecting
+directly to the public internet.
+
 
 Exemption Policy
 ================
 
 Under this policy, the following network security related modules are
 granted a blanket exemption to the restriction against adding new features
-in maintenance releases:
+in maintenance releases, for the purpose of keeping their APIs aligned with
+their counterparts in the latest feature release of Python 3:
 
 * the ``ssl`` module
 * the ``hashlib`` module
@@ -52,7 +69,7 @@ in maintenance releases:
 * the ``sha`` module (Python 2 only)
 
 This exemption applies to *all* proposals to backport backwards compatible
-features in these modules to active maintenance branches. This choice is
+changes in these modules to Python 2.7 maintenance releases. This choice is
 made deliberately to ensure that the "feature or fix?" argument isn't simply
 replaced by a "security related or not?" argument. These particular modules
 are inherently security related, and all enhancements to them improve
@@ -67,7 +84,7 @@ In addition to the above blanket exemption, a conditional exemption is
 granted for these modules that may include some network security related
 features:
 
-* the ``os`` module (primarily ``os.urandom``)
+* the ``os`` module (specifically ``os.urandom``)
 * the ``random`` module
 * networking related modules that depend on one or more of the network
   security related modules listed above
@@ -84,10 +101,12 @@ Backwards Compatibility Considerations
 ======================================
 
 This PEP does *not* grant any general exemptions to the usual backwards
-compatibility policy for maintenance releases. Instead, it is designed
+compatibility policy for maintenance releases. Instead, by explicitly
-to make it easier to provide more "secure by default" behaviour in future
+encouraging the use of feature based checks and explicitly opting in to
-feature releases, while still limiting the risk of breaking currently
+less secure configurations, it is designed to make it easier to provide
-working software when upgrading to a new maintenance release.
+more "secure by default" behaviour in future feature releases, while still
+limiting the risk of breaking currently working software when upgrading to
+a new maintenance release.
 
 In *all* cases where this policy is applied to backport enhancements to
 maintenance releases, it MUST be possible to write cross-version compatible
@@ -95,24 +114,56 @@ code that operates by "feature detection" (for example, checking for
 particular attributes in the module), without needing to explicitly check
 the Python version.
 
-It is then up to library and framework code to provide an appropriate error
+It is then up to library and framework code to provide an appropriate warning
-message or fallback behaviour if a desired feature is found to be missing.
+and fallback behaviour if a desired feature is found to be missing. While
+some especially security sensitive software MAY fail outright if a desired
+security feature is unavailable, most software SHOULD instead continue
+operating using a slightly degraded security configuration.
 
 Affected APIs SHOULD be designed to allow library and application code to
 perform the following actions after detecting the presence of a relevant
 network security related feature:
 
 * explicitly opt in to more secure settings (to allow the use of enhanced
-  security features in older versions of Python)
+  security features in older maintenance releases of Python)
 * explicitly opt in to less secure settings (to allow the use of newer Python
-  versions in lower security environments)
+  feature releases in lower security environments)
 * determine the default setting for the feature (this MAY require explicit
   Python version checks to determine the Python feature release, but MUST
-  NOT depend on the specific maintenance release)
+  NOT require checking for a specific maintenance release)
+
+Security related changes to other modules (such as data format processing
+libraries) will continue to be made available as backports and new modules
+on the Python Package Index, as independent distribution remains the
+preferred approach to handling software that needs to evolve faster than
+the standard library. Refer to the `Motivation and Rationale`_ section for
+a review of the characteristics that make the secure networking
+infrastructure worthy of special consideration.
 
 
-Documentation Requirements
+Other Considerations
-==========================
+====================
+
+Maintainability
+---------------
+
+This policy does NOT represent a commitment by volunteer contributors to
+actually backport network security related changes from the Python 3 series
+to the Python 2 series. Rather, it is intended to send a clear signal to
+potential corporate contributors that the core development team are willing
+to review and merge corporate contributions that put this policy into
+effect.
+
+Backporting security related fixes and enhancements to earlier versions is
+a common service for commercial redistributors to offer to their customers.
+This policy represents an explicit invitation to contribute some of those
+changes back to the upstream community in cases where they are likely to
+have a broad impact that helps improve the security of the internet as a
+whole.
+
+
+Documentation
+-------------
 
 All modules that take advantage of this policy to backport network
 security related enhancements to earlier Python versions MUST include
@@ -124,6 +175,33 @@ applicable), along with the feature and maintenance releases that first
 included them.
 
 
+Security releases
+-----------------
+
+This PEP does not propose any changes to the handling of security
+releases - those will continue to be source only releases that
+include only critical security fixes.
+
+However, the recommendations for library and application developers are
+deliberately designed to accommodate commercial redistributors applying
+this policy to any Python release series that is either in security
+fix only mode, or has been declared "end of life" by the core development
+team.
+
+Whether or not redistributors choose to exercise that option will be up
+to the redistributor.
+
+
+Integration testing
+-------------------
+
+Third party integration testing services would likely need to start
+offering users a choice of multiple Python 2.7.x versions to test against,
+to ensure that the application is correctly degrading gracefully if it
+attempts to use newer networking features on maintenance releases that
+are too old to provide them.
+
+
 Evolution of this Policy
 ========================
 
@@ -152,6 +230,10 @@ library release cycle" proposals discussed in PEP 407 and PEP 413,
 focusing specifically on those areas which have implications beyond the
 Python community.
 
+
+Background
+----------
+
 The creation of this PEP was prompted primarily by the aging SSL support in
 the Python 2 series. As of March 2014, the Python 2.7 SSL module is
 approaching four years of age, and the SSL support in the still popular
@@ -162,16 +244,22 @@ in good conscience for secure networking software that operates over the
 public internet, especially in an era where it is becoming quite clearly
 evident that advanced persistent security threats are even more widespread
 and more indiscriminate in their targeting than had previously been
-understood.
+understood. While they represented reasonable security infrastructure in
+their time, the state of the art has moved on, and we need to investigate
+mechanisms for effectively providing more up to date network security
+infrastructure for users that, for whatever reason, are not currently in
+a position to migrate to Python 3.
 
 While the use of the system OpenSSL installation addresses many of these
 concerns on Linux platforms, it doesn't address all of them, and in the
 case of the binary installers for Windows and Mac OS X that are published
 on python.org, the version of OpenSSL used is entirely within the control
-of the Python core development team.
+of the Python core development team, and currently limited to OpenSSL
+maintenance releases for the version initially shipped with the corresponding
+Python feature release.
 
 With increased popularity comes increased responsibility, and this policy
-is about recognising the fact that Python's popularity and adoption has now
+aims to acknowledge the fact that Python's popularity and adoption has now
 reached a level where some of our design and policy decisions have
 significant implications beyond the Python development community.
 
@@ -207,35 +295,144 @@ enhancement policy, that position is difficult to justify when it comes to
 software that operates over the public internet. Just as many developers
 consider it too difficult to develop truly secure modern networked software
 in C/C++ (largely due to the challenges associated with manual
-memory management), I consider it too difficult to develop truly secure
+memory management), I anticipate that in the not too distant future, it
-modern networked software using the Python 2 series without introducing this
+will be considered too difficult to develop truly secure modern networked
-network security enhancement policy, as doing so would mean reimplementing
+software using the Python 2 series (some developers would argue that we
-substantial portions of the standard library as third party modules to gain
+have already reached that point).
-access to the newer underlying network security protocols and mechanisms,
-and then injecting those into the module namespace to override their
-standard library counterparts.
 
-Requiring that latent defects in an application's Unicode correctness be
-addressed in order to migrate to Python 3 is not a reasonable alternative
-recommendation, especially given the likely existence of legacy code that
-lacks the kind of automated regression test suite needed to help support
-a migration from Python 2 to Python 3. The key point of this PEP is that
-those situations affect more people than just the developers and users of
-the affected application: their existence becomes something that developers
-of secure networked services need to take into account as part of their
-security design. By making it more feasible to enhance the security of the
-Python 2 series, we can help contribute to the evolution of a more secure
-internet for all concerned.
 
-For the Python 2 series, this proposal will most obviously impact the
+Alternative: advise developers of networked software to migrate to Python 3
-remaining maintenance releases of Python 2.7. However, commercial
+---------------------------------------------------------------------------
-redistributors that continue to offer full support for Python 2.6, and will
+
-also do so for Python 2.7 after upstream maintenance ends, may choose
+This alternative represents the status quo. Unfortunately, it has proven
-to take this policy into account when deciding what changes to backport to
+to be unworkable in practice, as the backwards compatibility implications
-their own maintenance updates. Providing commercial redistributors such
+mean that this is a non-trivial migration process for large applications
-as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and
+and integration projects.
-Red Hat that option is one of the benefits offered by requiring feature
+
-detection based forward compatibility for the modules covered by this policy.
+Now that we're fully aware of the impact the limitations in Python 2 may be
+having on the evolution of internet security standards, I no longer believe
+that it is reasonable to expect platform and application developers to
+resolve all of the latent defects in an application's Unicode correctness
+solely in order to gain access to the network security enhancements
+available in Python 3.
+
+While (as far as I am aware) Ubuntu has successfully switched to Python 3.4
+as its main Python interpreter for its 14.04 LTS release, Fedora still
+has a lot of work to do to migrate, and it will take a non-trivial amount
+of time to migrate the relevant infrastructure components. While Red Hat
+are also actively working to make it easier for users to use more recent
+versions of Python on our stable platforms, it's going to take time for
+those efforts to start having an impact on end users' choice of version,
+and those changes won't affect the core tools regardless.
+
+The OpenStack migration to Python 3 is also still in its infancy, and even
+though that's a project with an extensive and relatively robust automated
+test suite, it's large enough that it is going to take quite some time
+to migrate.
+
+And that's just three of the highest profile open source projects that
+make heavy use of Python. Given the likely existence of large amounts of
+legacy code that lacks the kind of automated regression test suite needed
+to help support a migration from Python 2 to Python 3. The key point of
+this PEP is that those situations affect more people than just the
+developers and users of the affected application: their existence becomes
+something that developers of secure networked services need to take into
+account as part of their security design.
+
+As Terry Reedy noted, if we try to persist with the status quo, the likely
+outcome is that commercial redistributors will attempt to do something
+like this on behalf of their customers *anyway*, but in a potentially
+inconsistent and ad hoc manner. By drawing the scope definition process
+into the upstream project we are in a better position to influence the
+approach taken to address the situation and to help ensure some consistency
+across redistributors.
+
+The problem is real, so *something* needs to change, and this PEP describes
+my currently preferred approach to addressing the situation.
+
+
+Alternative: create and release Python 2.8
+------------------------------------------
+
+With sufficient corporate support, it likely *would* be possible to create
+and release Python 2.8 (it's highly unlikely such a project would garner
+enough interest to be achievable with only volunteers). However, this
+wouldn't actually solve the problem, as the aim is to provide a *relatively
+low impact* way to incorporate enhanced security features into integrated
+products and deployments that make use of Python 2. Upgrading to a new
+Python feature release would mean both more work for the core development
+team, as well as a more disruptive update that most potential end users
+would likely just skip entirely.
+
+Attempting to create a Python 2.8 release would also bring in suggestions
+to backport many additional features from Python 3 (such as ``tracemalloc``
+and the improved coroutine support).
+
+This is not a recommended approach, as it would involve substantial
+additional work for a result that is actually less effective as a solution
+to the original problem (the widespread use of the aging network security
+infrastructure in Python 2).
+
+
+Alternative: distribute the security enhancements via PyPI
+----------------------------------------------------------
+
+While it initially appears to be an attractive and easier to manage
+approach, there are actually several significant problems with this
+idea.
+
+Firstly, this PEP encompasses a non-trivial portion of the standard library.
+It's not just the underlying SSL support, but also the libraries for other
+network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the
+SSL infrastructure to provide secure links, and that's just the protocols
+in the standard library.  Even if an API compatible ``ssl2`` module was
+made available, it would need to be imported and injected
+into ``sys.modules`` as ``ssl`` before importing any other module that
+needed it.
+
+Secondly, this is complex, low level, cross-platform code that integrates
+with the underlying operating system across a variety of POSIX platforms
+(including Mac OS X) and Windows. The CPython BuildBot fleet is already set
+up to handle continuous integration in that context, but most of the
+freely available continuous integration services just offer Linux, and
+perhaps paid access to Windows. Those services work reasonably well for
+software that largely runs on the abstraction layers offered by Python and
+other dynamic languages, but won't suffice for the kind of code involved
+here.
+
+The OpenSSL dependency for the network security support also qualifies as
+the kind of "complex binary dependency" that isn't yet handled well by the
+``pip`` based software distribution ecosystem. Relying on a binary
+dependency also creates potential compatibility problems for ``pip`` when
+running on other interpreters like ``PyPy``.
+
+Another practical problem with the idea is the fact that ``pip`` itself
+relies on the ``ssl`` support in the standard library (with some additional
+support from a bundled copy of ``requests``, which in turn bundles
+``backport.ssl_match_hostname``), and hence would require any replacement
+module to also be bundled within ``pip``. This wouldn't pose any
+insurmountable difficulties (it's just another dependency to vendor), but
+it *would* mean yet another copy of OpenSSL to keep up to
+date.
+
+This approach also has the same flaw as all other "improve security by
+renaming things" approaches: they completely miss the users who most need
+help, and raise significant barriers against being able to encourage users
+to do the right thing when their infrastructure supports it (since
+"use this other module" is a much higher impact change than "turn on this
+higher security setting"). Deprecating the aging SSL infrastructure in the
+standard library in favour of an external module would be even more user
+hostile than taking the risk of trying to upgrade it in place.
+
+Last, but certainly not least, this approach suffers from the same problem
+as the idea of doing a Python 2.8 release: likely not solving the actual
+problem. Commercial redistributors of Python are set up to redistribute
+*Python*, and a pre-existing set of additional packages. Getting new
+packages added to the pre-existing set *can* be done, but means going
+around to each and every redistributor and asking them to update their
+repackaging process accordingly. By contrast, the approach described in
+this PEP would require redistributors to *opt out* of the security
+enhancements, which most of them are unlikely to do.
 
 
 Open Questions
@@ -243,17 +440,27 @@ Open Questions
 
 * What are the risks associated with allowing OpenSSL to be updated to
   new feature versions in the Windows and Mac OS X binary installers for
-  maintenance releases?
+  maintenance releases? Currently we just upgrade to the appropriate
+  OpenSSL maintenance releases, rather than switching to the latest
+  feature release. In particular, is it possible Windows C extensions may
+  be linking against the Python provided OpenSSL module?
 
 * Are there any other security relevant modules that should be covered
   by either a blanket or conditional exemption?
 
-* Should we enumerate a specific list of "other networking modules" rather
+
-  than leaving it implicit?
+Disclosure of Interest
+======================
+
+The author of this PEP currently works for Red Hat on test automation tools.
+If this proposal is accepted, I will be strongly encouraging Red Hat to take
+advantage of the resulting opportunity to help improve the overall security
+of the Python ecosystem. However, I do not speak for Red Hat in this matter,
+and cannot make any commitments on Red Hat's behalf.
 
 
-Acknowledgement
+Acknowledgements
-===============
+================
 
 Thanks to Christian Heimes for his recent efforts on greatly improving
 Python's SSL support in the Python 3 series, and a variety of members of
@@ -266,6 +473,14 @@ of the web as a whole.
 Christian and Donald Stufft also provided valuable feedback on a preliminary
 draft of this proposal.
 
+Thanks also to participants in the python-dev mailing list thread [1]_
+
+
+References
+==========
+
+.. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html
+
 
 Copyright
 =========