python-peps/pep-0476.txt

PEP: 476
Title: Enabling certificate verification by default for stdlib http clients
Version: $Revision$
Last-Modified: $Date$
Author: Alex Gaynor <alex.gaynor@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-August-2014

Abstract
========

Currently when a standard library http client (the ``urllib`` and ``http``
modules) encounters an ``https://`` URL it will wrap the network HTTP traffic
in a TLS stream, as is necessary to communicate with such a server. However,
during the TLS handshake it will not actually check that the server has an X509
certificate is signed by a CA in any trust root, nor will it verify that the
Common Name (or Subject Alternate Name) on the presented certificate matches
the requested host.

The failure to do these checks means that anyone with a privileged network
position is able to trivially execute a man in the middle attack against a
Python application using either of these HTTP clients, and change traffic at
will.

This PEP proposes to enable verification of X509 certificate signatures, as
well as hostname verification for Python's HTTP clients by default, subject to
opt-out on a per-call basis.

Rationale
=========

The "S" in "HTTPS" stands for secure. When Python's users type "HTTPS" they are
expecting a secure connection, and Python should adhere to a reasonable
standard of care in delivering this. Currently we are failing at this, and in
doing so, APIs which appear simple are misleading users.

When asked, many Python users state that they were not aware that Python failed
to perform these validations, and are shocked.

The popularity of ``requests`` (which enables these checks by default)
demonstrates that these checks are not overly burdensome in any way, and the
fact that it is widely recommended as a major security improvement over the
standard library clients demonstrates that many expect a higher standard for
"security by default" from their tools.

The failure of various applications to note Python's negligence in this matter
is a source of *regular* CVE assignment [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_
[#]_ [#]_ [#]_.

.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444

Technical Details
=================

Python would use the system provided certificate database on all platforms.
Failure to locate such a database would be an error, and users would need to
explicitly specify a location to fix it.

This can be achieved by simply replacing the use of
``ssl._create_stdlib_context`` with ``ssl.create_default_context`` in
``http.client``.

Trust database
--------------

This PEP proposes using the system-provided certificate database. Previous
discussions have suggested bundling Mozilla's certificate database and using
that by default. This was decided against for several reasons:

* Using the platform trust database imposes a lower maintenance burden on the
  Python developers -- shipping our own trust database would require doing a
  release every time a certificate was revoked.
* Linux vendors, and other downstreams, would unbundle the Mozilla
  certificates, resulting in a more fragmented set of behaviors.
* Using the platform stores makes it easier to handle situations such as
  corporate internal CAs.

OpenSSL also has a pair of environment variables, ``SSL_CERT_DIR`` and
``SSL_CERT_FILE`` which can be used to point Python at a different certificate
database.

Backwards compatibility
-----------------------

This change will have the appearance of causing some HTTPS connections to
"break", because they will now raise an Exception during handshake.

This is misleading however, in fact these connections are presently failing
silently, an HTTPS URL indicates an expectation of confidentiality and
authentication. The fact that Python does not actually verify that the user's
request has been made is a bug, further: "Errors should never pass silently."

Nevertheless, users who have a need to access servers with self-signed or
incorrect certificates would be able to do so by providing a context with
custom trust roots or which disables validation (documentation should strongly
recommend the former where possible). Users will also be able to add necessary
certificates to system trust stores in order to trust them globally.

Twisted's 14.0 release made this same change, and it has been met with almost
no opposition.

In order to make this transition as smooth as possible, the next 3.4.x release
following this PEP will be modified to emit a warning in cases that would raise
an Exception in Python 3.5.

Warnings
--------

To support this warning, in 3.4.next a new ``verify_mode`` is introduced
``CERT_WARN``, which is equivilant to ``CERT_NONE``, except in cases that would
fail as ``CERT_REQUIRED`` or fail the hostname check emits a warning. In
3.4.next the ``httplib`` module will set this as the ``verify_mode`` if the
default context is used.

Other protocols
===============

This PEP only proposes requiring this level of validation for HTTP clients, not
for other protocols such as SMTP.

This is because while a high percentage of HTTPS servers have correct
certificates, as a result of the validation performed by browsers, for other
protocols self-signed or otherwise incorrect certificates are far more common.
Note that for SMTP at least, this appears to be changing and should be reviewed
for a potential similar PEP in the future:

* https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223
* https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526

Python Versions
===============

This PEP proposes making these changes to ``default`` (Python 3) branch. I
strongly believe these changes also belong in Python 2, but doing them in a
patch-release isn't reasonable, and there is strong opposition to doing a 2.8
release.

Copyright
=========

This document has been placed into the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8