Merge pull request #13 from ncoghlan/security-sensitive-rng
PEP 522: BlockingIOError in security sensitive APIs on Linux
This commit is contained in:
ncoghlan
GitHub
commit
69c76773ee
|
|
@ -0,0 +1,244 @@
|
|||
PEP: 522
|
||||
Title: Raise BlockingIOError in security sensitive APIs on Linux
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Nick Coghlan <ncoghlan@gmail.com>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 16 June 2016
|
||||
Python-Version: 3.6
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
On Linux systems, the documentation for ``os.urandom`` currently makes the
|
||||
following contradictory promises:
|
||||
|
||||
* to provide random numbers that are suitable for security sensitive
|
||||
operations (such as client authentication and cryptography)
|
||||
* to provide access to the best available randomness source provided by
|
||||
the underlying operating system
|
||||
* to present a relatively thin wrapper around the system ``/dev/urandom``
|
||||
device
|
||||
|
||||
This PEP proposes that in Python 3.6+ the 3rd guarantee be dropped in order to
|
||||
preserve the first two: on Linux systems that provide the ``getrandom()``
|
||||
syscall, ``os.urandom()`` would become a wrapper around that API, and raise
|
||||
``BlockingIOError`` in cases where directly accessing ``/dev/urandom/`` would
|
||||
instead return random data that may not be adequately unpredictable for use in
|
||||
security sensitive operations.
|
||||
|
||||
As higher level abstractions over the lower level ``os.urandom()`` API, both
|
||||
``random.SystemRandom()`` and the ``secrets`` would also be documented as
|
||||
potentially raising ``BlockingIOError``.
|
||||
|
||||
In all cases, as soon as a call to ``os.urandom()`` succeeds, all future
|
||||
calls to ``os.urandom()`` in that process will succeed (once the operating
|
||||
system random number generator is ready after system boot, it remains ready).
|
||||
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call
|
||||
the new Linux ``getrandom()``` syscall in non-blocking mode if available and
|
||||
raise ``BlockingIOError: system random number generator is not ready`` if
|
||||
the kernel reports that the call would block.
|
||||
|
||||
No changes are proposed for Windows or Mac OS X systems, as neither of those
|
||||
platforms provides any mechanism to run Python code before the operating
|
||||
system random number generator has been initialised. Mac OS X goes so far as
|
||||
to kernel panic and abort the boot process if it can't properly initialise the
|
||||
random number generator (although Apple's restrictions on the supported
|
||||
hardware platforms make that exceedingly unlikely in practice).
|
||||
|
||||
Other \*nix systems that offer a non-blocking API for requesting random numbers
|
||||
suitable for use in security sensitive applications could potentially receive
|
||||
a similar update, but such changes are out of scope for this particular
|
||||
proposal.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
For several years now, the security community's guidance has been to use
|
||||
``os.urandom()`` (or the ``random.SystemRandom()`` wrapper) when implementing
|
||||
security sensitive operations in Python.
|
||||
|
||||
To help improve API discoverability and make it clearer that secrecy and
|
||||
simulation are not the same problem (even though they both involve
|
||||
random numbers), PEP 506 collected several of the one line recipes based
|
||||
on the lower level ``os.urandom()`` API into a new ``secrets`` module.
|
||||
|
||||
However, this guidance has also come with a longstanding caveat: developers
|
||||
writing security sensitive software at least for Linux, and potentially for
|
||||
some other \*BSD systems, may need to wait until the operating system's
|
||||
random number generator is ready before relying on it for security sensitive
|
||||
operations.
|
||||
|
||||
Unfortunately, there's currently no clear indicator to developers that their
|
||||
software may not be working as expected when run early in the Linux boot
|
||||
process, or on hardware without good sources of entropy to seed the operating
|
||||
system's random number generator: due to the behaviour of the underlying
|
||||
``/dev/urandom`` device, ``os.urandom()`` on Linux returns a result either way,
|
||||
and it takes extensive statistical analysis to show that a security
|
||||
vulnerability exists.
|
||||
|
||||
By contrast, if ``BlockingIOError`` is raised in those situations, then
|
||||
developers can easily choose their desired behaviour:
|
||||
|
||||
1. Loop until the call succeeds (security sensitive)
|
||||
2. Switch to using the random module (non-security sensitive)
|
||||
3. Switch to reading ``/dev/urandom`` directly (non-security sensitive)
|
||||
|
||||
|
||||
Why now?
|
||||
--------
|
||||
|
||||
The main reason is because the 3.5 SipHash initialisation bug causing a deadlock
|
||||
when attempting to run Python scripts during the Linux init process resulted in
|
||||
a rash of proposals to add *new* APIs like ``getrandom()``, ``urandom_block()``,
|
||||
``pseudorandom()`` and ``cryptorandom()`` to the ``os`` module and to start
|
||||
trying to educate users on when they should call those APIs instead of
|
||||
``os.urandom()``.
|
||||
|
||||
This is a *really* obscure problem, and we definitely shouldn't clutter up the
|
||||
standard library with new APIs without a compelling reason, especially with the
|
||||
``secrets`` module already being added as the "use this and don't worry about
|
||||
the low level details" for developers that don't need to worry about versions
|
||||
prior to Python 3.6.
|
||||
|
||||
However, it's also the case that low cost ARM devices are becoming increasingly
|
||||
prevalent, with a lot of them running Linux, and a lot of folks writing
|
||||
Python applications that run on those devices. That creates an opportunity to
|
||||
take an obscure security problem that requires a lot of knowledge about
|
||||
Linux boot processes and secure random number generation and turn it into a
|
||||
relatively mundane and easy-to-find-in-an-internet-search runtime exception.
|
||||
|
||||
|
||||
Background
|
||||
==========
|
||||
|
||||
On operating systems other than Linux, ``os.urandom()`` may already block
|
||||
waiting for the operating system's random number generator to be ready.
|
||||
|
||||
On Linux, even when the operating system's random number generator doesn't
|
||||
consider itself ready for use in security sensitive operations, it will return
|
||||
random values based on the entropy it as available.
|
||||
|
||||
This behaviour is potentially problematic, so Linux 3.17 added a new
|
||||
``getrandom()`` syscall that (amongst other benefits) allows callers to
|
||||
either block waiting for the random number generator to be ready, or
|
||||
else request an error return if the random number generator is not ready.
|
||||
Notably, the new API does *not* support the old behaviour of returning
|
||||
data that is not suitable for security sensitive use cases.
|
||||
|
||||
Versions of Python prior up to and including Python 3.4 access the
|
||||
Linux ``/dev/urandom`` device directly.
|
||||
|
||||
Python 3.5.0 and 3.5.1 called ``getrandom()`` in blocking mode in order to
|
||||
avoid the use of a file descriptor to access ``/dev/urandom``. While there
|
||||
were no specific problems reported due to ``os.urandom()`` blocking in user
|
||||
code, there *were* problems due to CPython implicitly invoking the blocking
|
||||
behaviour during interpreter startup.
|
||||
|
||||
Rather than trying to decouple SipHash initialisation from the
|
||||
``os.urandom()`` implementation, Python 3.5.2 switched to calling
|
||||
``getrandom()`` in non-blocking mode, and falling back to reading from
|
||||
``/dev/urandom`` if the syscall indicates it will block.
|
||||
|
||||
|
||||
Backwards Compatibility Impact Assessment
|
||||
=========================================
|
||||
|
||||
Similar to PEP 476, this is a proposal to turn a previously silent security
|
||||
failure into a noisy exception that requires the application developer to
|
||||
make an explicit decision regarding the behaviour they desire.
|
||||
|
||||
As no changes are proposed for operating systems other than Linux,
|
||||
``os.urandom()`` retains its existing behaviour as a nominally blocking API
|
||||
that is non-blocking in practice due to the difficulty of scheduling Python
|
||||
code to run before the operating system random number generator is ready. We
|
||||
believe it may be possible on \*BSD, but nobody has explicitly demonstrated
|
||||
that. On Mac OS X and Windows, it appears to be straight up impossible to
|
||||
even try to run a Python interpreter that early in the boot process.
|
||||
|
||||
On Linux, ``os.urandom()`` retains its status as a guaranteed non-blocking API.
|
||||
However, the means of achieving that status changes in the specific case of
|
||||
the operating system random number generator not being ready for use in security
|
||||
sensitive operations: historically it would return potentially predictable
|
||||
random data, with this PEP it would change to raise ``BlockingIOError``.
|
||||
|
||||
Developers of affected applications would then be required to make one of the
|
||||
following changes to forward compatibility with Python 3.6, based on the kind
|
||||
of application they're developing.
|
||||
|
||||
|
||||
Unaffected Applications
|
||||
-----------------------
|
||||
|
||||
The following kinds of applications would be entirely unaffected by the change,
|
||||
regardless of whether or not they perform security sensitive operations:
|
||||
|
||||
- applications that don't support Linux
|
||||
- applications that are only run on desktops or conventional servers
|
||||
- applications that are only run after the system RNG is ready
|
||||
|
||||
|
||||
Affected security sensitive applications
|
||||
----------------------------------------
|
||||
|
||||
Security sensitive applications would need to either change their system
|
||||
configuration so the application is only started after the operating system
|
||||
random number generator is ready for security sensitive operations, or else
|
||||
change their code to busy loop until the operating system is ready::
|
||||
|
||||
def blocking_urandom(num_bytes):
|
||||
while True:
|
||||
try:
|
||||
return os.urandom(num_bytes)
|
||||
except BlockingIOError:
|
||||
pass
|
||||
|
||||
|
||||
Affected Linux specific non-security sensitive applications
|
||||
-----------------------------------------------------------
|
||||
|
||||
Non-security sensitive applications that don't need to worry about cross
|
||||
platform compatibility can be updated to access ``/dev/urandom`` directly::
|
||||
|
||||
def dev_urandom(num_bytes):
|
||||
with open("/dev/urandom", "rb") as f:
|
||||
return f.read(num_bytes)
|
||||
|
||||
|
||||
Affected portable non-security sensitive applications
|
||||
-----------------------------------------------------
|
||||
|
||||
Non-security sensitive applications that don't want to assume access to
|
||||
``/dev/urandom`` can be updated to use the ``random`` module instead::
|
||||
|
||||
def pseudorandom(num_bytes):
|
||||
random.getrandbits(num_bytes*8).to_bytes(num_bytes, "little")
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* Victor's summary: http://haypo-notes.readthedocs.io/pep_random.html
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed into the public domain.
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
Loading…
Reference in New Issue