python-peps/pep-0493.txt

PEP: 493
Title: HTTPS verification recommendations for Python 2.7 redistributors
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 10-May-2015

Abstract
========

PEP 476 updated Python's default handling of HTTPS certificates to be
appropriate for communication over the public internet. The Python 2.7 long
term maintenance series was judged to be in scope for this change, with the
new behaviour introduced in the Python 2.7.9 maintenance release.

This PEP provides recommendations to downstream redistributors wishing to
provide a smoother migration experience when helping their users to manage
this change in Python's default behaviour.

*Note that this PEP is not currently accepted, so it is a *proposed*
recommendation, rather than an active one.*

Rationale
=========

PEP 476 changed Python's default behaviour to better match the needs and
expectations of developers operating over the public internet, a category
which appears to include most new Python developers. It is the position of
the authors of this PEP that this was a correct decision.

However, it is also the case that this change *does* cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new default
certificate verification settings.

The long term answer for such environments is to update their internal
certificate management to at least match the standards set by the public
internet, but in the meantime, it is desirable to offer these administrators
a way to continue receiving maintenance updates to the Python 2.7 series,
without having to gate that on upgrades to their certificate management
infrastructure.

PEP 476 did attempt to address this question, by covering how to revert the
new settings process wide by monkeypatching the ``ssl`` module to restore the
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case of interest to the
authors of this PEP is the one where a Linux distributor aims to provide
their users with a
`smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.

Rather than allowing a plethora of mutually incompatibile migration techniques
to bloom, this PEP proposes two alternative approaches that redistributors
may take when addressing these problems. Redistributors may choose to implement
one, both, or neither of these approaches based on their assessment of the
needs of their particular userbase.

These designs are being proposed as a recommendation for redistributors, rather
than as new upstream features, as they are needed purely to support legacy
environments migrating from older versions of Python 2.7. Neither approach
is being proposed as an upstream Python 2.7 feature, nor as a feature in any
version of Python 3 (whether published directly by the Python Software
Foundation or by a redistributor).

Recommendation for backporting to earlier Python versions
=========================================================

Some redistributors, most notably Linux distributions, may choose to backport
the PEP 476 HTTPS verification changes to modified Python versions based on
earlier Python 2 maintenance releases. In these cases, a configuration
mechanism is needed that provides:

* an opt-in model that allows the decision to enable HTTPS certificate
  verification to be made independently of the decision to upgrade to the
  Python version where the feature was first backported
* the ability for system administrators to set the default behaviour of Python
  applications and scripts run directly in the system Python installation
* the ability for system administrators to set the default behaviour of Python
  applications and scripts run in a ``virtualenv`` created virtual environment
* the ability for the redistributor to consider changing the default behaviour
  of *new* installations at some point in the future without impacting existing
  installations that have been explicitly configured to skip verifying HTTPS
  certificates by default

The recommended solution for this scenario should also avoid introducing any
new attack vectors that don't already allow direct attacks against the system
certificate store.

This approach should not be used for any Python installation that advertises
itself as providing Python 2.7.9 or later, as most Python users will have the
reasonable expectation that all such environments will validate HTTPS
certificates by default.

Recommended modifications to the Python standard library
--------------------------------------------------------

The recommended approach to backporting the PEP 476 modifications to an earlier
point release is to implement the following changes relative to the default
PEP 476 behaviour implemented in Python 2.7.9+:

* modify the ``ssl`` module to read a system wide configuration file when the
  module is first imported into a Python process
* default to verifying HTTPS certificates if this configuration file is not
  present (this preserves the upstream default behaviour in the absence of the
  configuration file)
* if the ``sys.real_prefix`` attribute is defined, read the configuration
  setting for virtual environments, otherwise read the configuration setting
  for the system Python
* support selection between the following three modes of operation:

  * ensure HTTPS certificate verification is enabled
  * ensure HTTPS certificate verification is disabled
  * delegate the decision to the redistributor providing this Python version

* set the ``ssl._create_default_https_context`` function to be an alias for
  either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
  based on the given configuration setting.

Recommended file location
-------------------------

This approach is currently only defined for \*nix system Python installations.

The recommended configuration file name is
(to be confirmed, currently ``/etc/python/cert-verification.cfg``).

The ``.cfg`` filename extension is recommended for consistency with the
``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.

Recommended file format
-----------------------

The configuration file should use a ConfigParser ini-style format with a
single section named ``[https]`` containing one required setting ``verify``,
and one optional setting ``verify_in_virtualenv``.

Permitted values for ``verify`` and ``verify_in_virtualenv`` are:

* ``enable``: ensure HTTPS certificate verification is enabled by default
* ``disable``: ensure HTTPS certificate verification is disabled by default
* ``platform_default``: delegate the decision to the redistributor providing
  this particular Python version

If the ``[https]`` section or the ``verify`` setting are missing, or if the
``verify`` setting is set to an unknown value, it should be treated as if the
configuration file is not present.

If ``sys.real_prefix`` is set, and the ``verify_in_virtualenv`` setting is
present and set to one of the known options, then it should be used in
preference to the ``verify`` setting.

Example implementation
----------------------

::

    def _get_https_context_factory():
        config_file = '/etc/python/cert-verification.conf'
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        # Check for a system-wide override of the default behaviour
        import configparser
        config = configparser.RawConfigParser()
        config.read(config_file)
        section = 'https'
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'enable'
        else:
            # Check if there's a different default for a virtual environment
            if hasattr(sys, "real_prefix"):
                try:
                    verify_mode = config.get('https', 'verify_in_virtualenv')
                except (ConfigParser.NoOptionError):
                    pass
        return context_factories.get(verify_mode, create_default_context)

    _create_default_https_context = _get_https_context_factory()


Security Considerations
-----------------------

The specific recommendations for the backporting case are designed to work for
privileged, security sensitive processes, even those being run in the following
locked down configuration:

* run from a locked down administrator controlled directory rather than a normal
  user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
  privilege escalation attacks)
* run using the ``-s`` switch (preventing user site directory based privilege
  escalation attacks)
* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
  escalation attacks)

The intent is that the *only* reason HTTPS verification should be getting
turned off system wide when using this approach is because:

* an end user is running a redistributor supported version of CPython rather
  than running upstream CPython directly
* that redistributor has decided to provide a smoother migration path to
  verifying HTTPS certificates by default than that being provided by the
  upstream project
* either the redistributor or the local infrastructure administrator has
  determined that it is appropriate to override the default upstream behaviour
  (at least for the time being)

Using an administrator controlled configuration file rather than an environment
variable not only provides compatibility with the ``-E`` switch, but also
ensures that in any situation where an attacker gains sufficient access to allow
them to modify the configuration file, they're likely already in a position to
attack the system certificate store directly.

Recommendation for an environment variable based security downgrade
===================================================================

Some redistributors may wish to provide a per-application option to disable
certificate verification in selected applications that run on or embed CPython
without needing to modify the application itself.

In these cases, a configuration mechanism is needed that provides:

* an opt-out model that allows certificate verification to be selectively
  turned off for particular applications after upgrading to a version of
  Python that verifies certificates by default
* the ability for all users to configure this setting on a per-application
  basis, rather than on a per-system, or per-Python-installation basis

This recommendation is not considered appropriate for system wide Python
installations, but may be suitable for user level Python installations and
versions of Python embedded in or bundled with particular applications.

This approach may be used for Python installations that advertises
themselves as providing Python 2.7.9 or later.

Recommended modifications to the Python standard library
--------------------------------------------------------

The recommended approach to providing a per-application configuration setting
for HTTPS certificate verification that doesn't require modifications to the
application itself is to:

* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
  variable when the module is first imported into a Python process
* set the ``ssl._create_default_https_context`` function to be an alias for
  ``ssl._create_unverified_context`` if this environment variable is present
  and set to '0'
* otherwise, set the ``ssl._create_default_https_context`` function to be an
  alias for ``ssl.create_default_context`` as usual

Example implementation
----------------------

::

    def _get_https_context_factory():
        config_setting = os.environ.get('PYTHONHTTPSVERIFY')
        if config_setting == '0':
            return _create_unverified_context
        return create_default_context

    _create_default_https_context = _get_https_context_factory()


Security Considerations
-----------------------

Relative to an unmodified version of CPython 2.7.9 or later, this approach
does introduce a new downgrade attack against the default security settings
that potentially allows a sufficiently determined attacker to revert Python
to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
Such an attack requires the ability to modify the execution environment of
a Python process prior to the import of the ``ssl`` module.

Redistributors should balance this marginal increase in risk against the
ability to offer a smoother migration path to their users when deciding whether
or not it is appropriate for them to implement this per-application "opt out"
model.


Copyright
=========

This document has been placed into the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8