580 lines
24 KiB
Plaintext
580 lines
24 KiB
Plaintext
PEP: 493
|
|
Title: HTTPS verification migration tools for Python 2.7
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Nick Coghlan <ncoghlan@gmail.com>,
|
|
Robert Kuska <rkuska@redhat.com>,
|
|
Marc-André Lemburg <mal@lemburg.com>
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 10-May-2015
|
|
Python-Version: 2.7.12
|
|
Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
PEP 476 updated Python's default handling of HTTPS certificates in client
|
|
modules to align with certificate handling in web browsers, by validating
|
|
that the certificates received belonged to the server the client was attempting
|
|
to contact. The Python 2.7 long term maintenance series was judged to be in
|
|
scope for this change, with the new behaviour introduced in the Python 2.7.9
|
|
maintenance release.
|
|
|
|
This has created a non-trivial barrier to adoption for affected Python 2.7
|
|
maintenance releases, so this PEP proposes additional Python 2.7 specific
|
|
features that allow system administrators and other users to more easily
|
|
decouple the decision to verify server certificates in HTTPS client modules
|
|
from the decision to update to newer Python 2.7 maintenance releases.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
PEP 476 changed Python's default behaviour to align with expectations
|
|
established by web browsers in regards to the semantics of HTTPS URLs:
|
|
starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library
|
|
validate server certificates by default.
|
|
|
|
However, it is also the case that this change *does* cause problems for
|
|
infrastructure administrators operating private intranets that rely on
|
|
self-signed certificates, or otherwise encounter problems with the new default
|
|
certificate verification settings.
|
|
|
|
To manage these kinds of situations, web browsers provide users with "click
|
|
through" warnings that allow the user to add the server's certificate to the
|
|
browser's certificate store. Network client tools like ``curl`` and ``wget``
|
|
offer options to switch off certificate checking entirely (by way of
|
|
``curl --insecure`` and ``wget --no-check-certificate``, respectively).
|
|
|
|
At a different layer of the technology stack, Linux security modules like
|
|
`SELinux` and `AppArmor`, while enabled by default by distribution vendors,
|
|
offer relatively straightforward mechanisms for turning them off.
|
|
|
|
At the moment, no such convenient mechanisms exist to disable Python's
|
|
default certificate checking for a whole process.
|
|
|
|
PEP 476 did attempt to address this question, by covering how to revert to the
|
|
old settings process wide by monkeypatching the ``ssl`` module to restore the
|
|
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
|
|
to allow system administrators to disable the feature by default in their
|
|
Standard Operating Environment definition has been determined to be
|
|
insufficient in at least some cases. The specific case that led to the
|
|
initial creation of this PEP is the one where a Linux distributor aims to
|
|
provide their users with a
|
|
`smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
|
|
than the standard one provided by consuming upstream CPython 2.7 releases
|
|
directly, but other potential challenges have also been pointed out with
|
|
updating embedded Python runtimes and other user level installations of Python.
|
|
|
|
Rather than allowing a plethora of mutually incompatibile migration techniques
|
|
to bloom, this PEP proposes an additional feature to be added to Python 2.7.12
|
|
to make it easier to revert a process to the past behaviour of skipping
|
|
certificate validation in HTTPS client modules. It also provides additional
|
|
recommendations to redistributors backporting these features to versions of
|
|
Python prior to Python 2.7.9.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
In the absence of clear upstream guidance and recommendations, commercial
|
|
redistributors will still make their own design decisions in the interests of
|
|
their customers. The main approaches available are:
|
|
|
|
* Continuing to rebase on new Python 2.7.x releases, while providing no
|
|
additional assistance beyond the mechanisms defined in PEP 476 in migrating
|
|
from unchecked to checked hostnames in standard library HTTPS clients
|
|
* Gating availability of the changes in default handling of HTTPS connections
|
|
on upgrading from Python 2 to Python 3
|
|
* For Linux distribution vendors, gating availability of the changes in default
|
|
handling of HTTPS connections on upgrading to a new operating system version
|
|
* Implementing one or both of the backport suggestions described in this PEP,
|
|
regardless of the formal status of the PEP
|
|
|
|
|
|
Scope Limitations
|
|
=================
|
|
|
|
These changes are being proposed purely as tools for helping to manage the
|
|
transition to the new default certificate handling behaviour in the context
|
|
of Python 2.7. They are not being proposed as new features for Python 3, as
|
|
it is expected that the vast majority of client applications affected by this
|
|
problem without the ability to update the application itself will be Python 2
|
|
applications.
|
|
|
|
It would likely be desirable for a future version of Python 3 to allow the
|
|
default certificate handling for secure protocols to be configurable on a
|
|
per-protocol basis, but that question is beyond the scope of this PEP.
|
|
|
|
|
|
Requirements for capability detection
|
|
=====================================
|
|
|
|
As the proposals in this PEP aim to facilitate backports to earlier Python
|
|
versions, the Python version number cannot be used as a reliable means for
|
|
detecting them. Instead, they are designed to allow the presence
|
|
or absence of the feature to be determined using the following technique::
|
|
|
|
python -c "import ssl; ssl._relevant_attribute"
|
|
|
|
This will fail with `AttributeError` (and hence a non-zero return code) if the
|
|
relevant capability is not available.
|
|
|
|
The marker attributes are prefixed with an underscore to indicate the
|
|
implementation dependent nature of these capabilities - not all Python
|
|
distributions will offer them, only those that are providing a multi-stage
|
|
migration process from the original Python 2.7 HTTPS handling to the new
|
|
default behaviour.
|
|
|
|
|
|
Feature: Configuration API
|
|
==========================
|
|
|
|
This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
|
|
releases. It consists of a new ``ssl._https_verify_certificates()`` to specify
|
|
the default handling of HTTPS certificates in standard library client libraries.
|
|
|
|
It is not proposed to forward port this change to Python 3, so Python 3
|
|
applications that need to support skipping certificate verification will still
|
|
need to define their own suitable security context.
|
|
|
|
Feature detection
|
|
-----------------
|
|
|
|
The marker attribute on the ``ssl`` module related to this feature is the
|
|
``ssl._https_verify_certificates`` function itself.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
The ``ssl._https_verify_certificates`` function will work as follows::
|
|
|
|
def _https_verify_certificates(enable=True):
|
|
"""Verify server HTTPS certificates by default?"""
|
|
global _create_default_https_context
|
|
if enable:
|
|
_create_default_https_context = create_default_context
|
|
else:
|
|
_create_default_https_context = _create_unverified_context
|
|
|
|
If called without arguments, or with ``enable`` set to a true value, then
|
|
standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification.
|
|
|
|
If called with ``enable`` set to a false value, then standard library client
|
|
modules will subsequently skip verifying HTTPS certificates by default.
|
|
|
|
Security Considerations
|
|
-----------------------
|
|
|
|
The inclusion of this feature will allow security sensitive applications to
|
|
include the following forward-compatible snippet in their code::
|
|
|
|
if hasattr(ssl, "_https_verify_certificates"):
|
|
ssl._https_verify_certificates()
|
|
|
|
Some developers may also choose to opt out of certificate checking using
|
|
``ssl._https_verify_certificates(enable=False)``. This doesn't introduce any
|
|
major new security concerns, as monkeypatching the affected internal APIs was
|
|
already possible.
|
|
|
|
|
|
Feature: environment based configuration
|
|
========================================
|
|
|
|
This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
|
|
releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that
|
|
allows the default verification to be disabled without modifying the
|
|
application source code (which may not even be available in cases of
|
|
bytecode-only application distribution)
|
|
|
|
It is not proposed to forward port this change to Python 3, so Python 3
|
|
applications that need to support skipping certificate verification will still
|
|
need to define their own suitable security context.
|
|
|
|
Feature detection
|
|
-----------------
|
|
|
|
The marker attribute on the ``ssl`` module related to this feature is:
|
|
|
|
* the ``ssl._https_verify_envvar`` attribute, giving the name of environment
|
|
variable affecting the default behaviour
|
|
|
|
This not only makes it straightforward to detect the presence (or absence) of
|
|
the capability, it also makes it possible to programmatically determine the
|
|
relevant environment variable name.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
Rather than always defaulting to the use of ``ssl.create_default_context``,
|
|
the ``ssl`` module will be modified to:
|
|
|
|
* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first
|
|
imported into a Python process
|
|
* set the ``ssl._create_default_https_context`` function to be an alias for
|
|
``ssl._create_unverified_context`` if this environment variable is present
|
|
and set to ``'0'``
|
|
* otherwise, set the ``ssl._create_default_https_context`` function to be an
|
|
alias for ``ssl.create_default_context`` as usual
|
|
|
|
Example implementation
|
|
----------------------
|
|
|
|
::
|
|
|
|
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
|
|
|
|
def _get_https_context_factory():
|
|
if not sys.flags.ignore_environment:
|
|
config_setting = os.environ.get(_https_verify_envvar)
|
|
if config_setting == '0':
|
|
return _create_unverified_context
|
|
return create_default_context
|
|
|
|
_create_default_https_context = _get_https_context_factory()
|
|
|
|
Security Considerations
|
|
-----------------------
|
|
|
|
Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
|
|
approach does introduce a new downgrade attack against the default security
|
|
settings that potentially allows a sufficiently determined attacker to revert
|
|
Python to the default behaviour used in CPython 2.7.8 and earlier releases.
|
|
However, such an attack requires the ability to modify the execution
|
|
environment of a Python process prior to the import of the ``ssl`` module,
|
|
and any attacker with such access would already be able to modify the
|
|
behaviour of the underlying OpenSSL implementation.
|
|
|
|
Interaction with Python virtual environments
|
|
--------------------------------------------
|
|
|
|
The default setting is read directly from the process environment, and hence
|
|
works the same way regardless of whether or not the interpreter is being run
|
|
inside an activated Python virtual environment.
|
|
|
|
|
|
Backporting this PEP to earlier Python versions
|
|
===============================================
|
|
|
|
If this PEP is accepted, then commercial Python redistributors may choose to
|
|
backport the per-process configuration mechanisms defined in this PEP to base
|
|
versions older than Python 2.7.9, *without* also backporting PEP 476's change
|
|
to the default behaviour of the overall Python installation.
|
|
|
|
Such a backport would differ from the mechanism proposed in this PEP solely in
|
|
the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would
|
|
continue to default to skipping certificate validation.
|
|
|
|
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
|
|
set to anything *other* than ``'0'``, then HTTPS certificate verification
|
|
should be enabled.
|
|
|
|
Feature detection
|
|
-----------------
|
|
|
|
There's no specific attribute indicating that this situation applies. Rather,
|
|
it is indicated by the ``ssl._https_verify_certificates`` and
|
|
``ssl._https_verify_envvar`` attributes being present in a Python version that
|
|
is nominally older than Python 2.7.9.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
Implementing this backport involves backporting the changes in PEP 466, 476 and
|
|
this PEP, with the following change to the handling of the
|
|
``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module:
|
|
|
|
* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first
|
|
imported into a Python process
|
|
* set the ``ssl._create_default_https_context`` function to be an alias for
|
|
``ssl.create_default_context`` if this environment variable is present
|
|
and set to any value other than ``'0'``
|
|
* otherwise, set the ``ssl._create_default_https_context`` function to be an
|
|
alias for ``ssl._create_unverified_context``
|
|
|
|
Example implementation
|
|
----------------------
|
|
|
|
::
|
|
|
|
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
|
|
|
|
def _get_https_context_factory():
|
|
if not sys.flags.ignore_environment:
|
|
config_setting = os.environ.get(_https_verify_envvar)
|
|
if config_setting != '0':
|
|
return create_default_context
|
|
return _create_unverified_context
|
|
|
|
_create_default_https_context = _get_https_context_factory()
|
|
|
|
def _disable_https_default_verification():
|
|
"""Skip verification of HTTPS certificates by default"""
|
|
global _create_default_https_context
|
|
_create_default_https_context = _create_unverified_context
|
|
|
|
Security Considerations
|
|
-----------------------
|
|
|
|
This change would be a strict security upgrade for any Python version that
|
|
currently defaults to skipping certificate validation in standard library
|
|
HTTPS clients. The technical trade-offs to be taken into account relate largely
|
|
to the magnitude of the PEP 466 backport also required rather than to anything
|
|
security related.
|
|
|
|
Interaction with Python virtual environments
|
|
--------------------------------------------
|
|
|
|
The default setting is read directly from the process environment, and hence
|
|
works the same way regardless of whether or not the interpreter is being run
|
|
inside an activated Python virtual environment.
|
|
|
|
|
|
Backporting PEP 476 to earlier Python versions
|
|
==============================================
|
|
|
|
The backporting approach described above leaves the default HTTPS certificate
|
|
verification behaviour of a Python 2.7 installation unmodified: verifying
|
|
certificates still needs to be opted into on a per-connection or per-process
|
|
basis.
|
|
|
|
To allow the default behaviour of the entire installation to be modified
|
|
without breaking backwards compatibility, Red Hat designed a configuration
|
|
mechanism for the system Python 2.7 installation in Red Hat Enterprise Linux
|
|
7.2+ that provides:
|
|
|
|
* an opt-in model that allows the decision to enable HTTPS certificate
|
|
verification to be made independently of the decision to upgrade to the
|
|
operating system version where the feature was first backported
|
|
* the ability for system administrators to set the default behaviour of Python
|
|
applications and scripts run directly in the system Python installation
|
|
* the ability for the redistributor to consider changing the default behaviour
|
|
of *new* installations at some point in the future without impacting existing
|
|
installations that have been explicitly configured to skip verifying HTTPS
|
|
certificates by default
|
|
|
|
As it only affects backports to earlier releases of Python 2.7, this change is
|
|
not proposed for inclusion in upstream CPython, but rather is offered as
|
|
a recommendation to other redistributors that choose to offer a similar feature
|
|
to their users.
|
|
|
|
This PEP doesn't take a position on whether or not this particular change is a
|
|
good idea - rather, it suggests that *if* a redistributor chooses to go down
|
|
the path of making the default behaviour configurable in a version of Python
|
|
older than Python 2.7.9, then maintaining a consistent approach across
|
|
redistributors would be beneficial for users.
|
|
|
|
However, this approach SHOULD NOT be used for any Python installation that
|
|
advertises itself as providing Python 2.7.9 or later, as most Python users
|
|
will have the reasonable expectation that all such environments will verify
|
|
HTTPS certificates by default.
|
|
|
|
|
|
Feature detection
|
|
-----------------
|
|
|
|
The marker attribute on the ``ssl`` module related to this feature is::
|
|
|
|
_cert_verification_config = '<path to configuration file>'
|
|
|
|
This not only makes it straightforward to detect the presence (or absence) of
|
|
the capability, it also makes it possible to programmatically determine the
|
|
relevant configuration file name.
|
|
|
|
|
|
Recommended modifications to the Python standard library
|
|
--------------------------------------------------------
|
|
|
|
The recommended approach to backporting the PEP 476 modifications to an earlier
|
|
point release is to implement the following changes relative to the default
|
|
PEP 476 behaviour implemented in Python 2.7.9+:
|
|
|
|
* modify the ``ssl`` module to read a system wide configuration file when the
|
|
module is first imported into a Python process
|
|
* define a platform default behaviour (either verifying or not verifying HTTPS
|
|
certificates) to be used if this configuration file is not present
|
|
* support selection between the following three modes of operation:
|
|
|
|
* ensure HTTPS certificate verification is enabled
|
|
* ensure HTTPS certificate verification is disabled
|
|
* delegate the decision to the redistributor providing this Python version
|
|
|
|
* set the ``ssl._create_default_https_context`` function to be an alias for
|
|
either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
|
|
based on the given configuration setting.
|
|
|
|
|
|
Recommended file location
|
|
-------------------------
|
|
|
|
As the PEP authors are not aware of any vendors providing long-term support
|
|
releases targeting Windows, Mac OS X or \*BSD systems, this approach is
|
|
currently only specifically defined for Linux system Python installations.
|
|
|
|
The recommended configuration file name on Linux systems is
|
|
``/etc/python/cert-verification.cfg``.
|
|
|
|
The ``.cfg`` filename extension is recommended for consistency with the
|
|
``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
|
|
|
|
|
|
Recommended file format
|
|
-----------------------
|
|
|
|
The configuration file should use a ConfigParser ini-style format with a
|
|
single section named ``[https]`` containing one required setting ``verify``.
|
|
|
|
The suggested section name is taken from the "https" URL schema passed to
|
|
affected client APIs.
|
|
|
|
Permitted values for ``verify`` are:
|
|
|
|
* ``enable``: ensure HTTPS certificate verification is enabled by default
|
|
* ``disable``: ensure HTTPS certificate verification is disabled by default
|
|
* ``platform_default``: delegate the decision to the redistributor providing
|
|
this particular Python version
|
|
|
|
If the ``[https]`` section or the ``verify`` setting are missing, or if the
|
|
``verify`` setting is set to an unknown value, it should be treated as if the
|
|
configuration file is not present.
|
|
|
|
|
|
Example implementation
|
|
----------------------
|
|
|
|
::
|
|
|
|
_cert_verification_config = '/etc/python/cert-verification.cfg'
|
|
|
|
def _get_https_context_factory():
|
|
# Check for a system-wide override of the default behaviour
|
|
context_factories = {
|
|
'enable': create_default_context,
|
|
'disable': _create_unverified_context,
|
|
'platform_default': _create_unverified_context, # For now :)
|
|
}
|
|
import ConfigParser
|
|
config = ConfigParser.RawConfigParser()
|
|
config.read(_cert_verification_config)
|
|
try:
|
|
verify_mode = config.get('https', 'verify')
|
|
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
|
|
verify_mode = 'platform_default'
|
|
default_factory = context_factories.get('platform_default')
|
|
return context_factories.get(verify_mode, default_factory)
|
|
|
|
_create_default_https_context = _get_https_context_factory()
|
|
|
|
|
|
Security Considerations
|
|
-----------------------
|
|
|
|
The specific recommendations for this backporting case are designed to work for
|
|
privileged, security sensitive processes, even those being run in the following
|
|
locked down configuration:
|
|
|
|
* run from a locked down administrator controlled directory rather than a normal
|
|
user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
|
|
* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
|
|
privilege escalation attacks)
|
|
* run using the ``-s`` switch (preventing user site directory based privilege
|
|
escalation attacks)
|
|
* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
|
|
escalation attacks)
|
|
|
|
The intent is that the *only* reason HTTPS verification should be getting
|
|
turned off installation wide when using this approach is because:
|
|
|
|
* an end user is running a redistributor provided version of CPython rather
|
|
than running upstream CPython directly
|
|
* that redistributor has decided to provide a smoother migration path to
|
|
verifying HTTPS certificates by default than that being provided by the
|
|
upstream project
|
|
* either the redistributor or the local infrastructure administrator has
|
|
determined that it is appropriate to retaing the default pre-2.7.9 behaviour
|
|
(at least for the time being)
|
|
|
|
Using an administrator controlled configuration file rather than an environment
|
|
variable has the essential feature of providing a smoother migration path, even
|
|
for applications being run with the ``-E`` switch.
|
|
|
|
Interaction with Python virtual environments
|
|
--------------------------------------------
|
|
|
|
This setting is scoped by the interpreter installation and affects all Python
|
|
processes using that interpreter, regardless of whether or not the interpreter
|
|
is being run inside an activated Python virtual environment.
|
|
|
|
Origins of this recommendation
|
|
------------------------------
|
|
|
|
This recommendation is based on the backporting approach adopted for Red Hat
|
|
Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP
|
|
and described in detail in `this KnowledgeBase article
|
|
<https://access.redhat.com/articles/2039753>`__. Red Hat's patches implementing
|
|
this backport for Python 2.7.5 can be found in the `CentOS git repository
|
|
<https://git.centos.org/commit/rpms!python.git/refs!heads!c7>`__.
|
|
|
|
|
|
Recommendation for combined feature backports
|
|
=============================================
|
|
|
|
If a redistributor chooses to backport the environment variable based
|
|
configuration setting from this PEP to a modified Python version that also
|
|
implements the configuration file based PEP 476 backport, then the environment
|
|
variable should take precedence over the system-wide configuration setting.
|
|
This allows the setting to be changed for a given user or application,
|
|
regardless of the installation-wide default behaviour.
|
|
|
|
Example implementation
|
|
----------------------
|
|
|
|
::
|
|
|
|
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
|
|
_cert_verification_config = '/etc/python/cert-verification.cfg'
|
|
|
|
def _get_https_context_factory():
|
|
# Check for an environmental override of the default behaviour
|
|
if not sys.flags.ignore_environment:
|
|
config_setting = os.environ.get(_https_verify_envvar)
|
|
if config_setting is not None:
|
|
if config_setting == '0':
|
|
return _create_unverified_context
|
|
return create_default_context
|
|
|
|
# Check for a system-wide override of the default behaviour
|
|
context_factories = {
|
|
'enable': create_default_context,
|
|
'disable': _create_unverified_context,
|
|
'platform_default': _create_unverified_context, # For now :)
|
|
}
|
|
import ConfigParser
|
|
config = ConfigParser.RawConfigParser()
|
|
config.read(_cert_verification_config)
|
|
try:
|
|
verify_mode = config.get('https', 'verify')
|
|
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
|
|
verify_mode = 'platform_default'
|
|
default_factory = context_factories.get('platform_default')
|
|
return context_factories.get(verify_mode, default_factory)
|
|
|
|
_create_default_https_context = _get_https_context_factory()
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed into the public domain.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|