iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 522: Make secrets API block implicitly 

- the PEP now proposes that the secrets module block
  instead of raising BlockingIOError
- this makes secrets.token_bytes a blocking counterpart
  to os.urandom that's a dropin replacement
- drop discussions of behaviour of interpreter startup
  and random module import as already addressed
- note os.getrandom() is useful regardless of outcome
  of this PEP
This commit is contained in:
Nick Coghlan 2016-08-06 20:01:41 +10:00
parent a919235364
commit 5392cf9fb8
1 changed files with 145 additions and 152 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

297
pep-0522.txt
View File

@ -24,12 +24,19 @@ permitting reads from ``/dev/urandom`` before the system random number
generator is fully initialized, whereas most other operating systems will generator is fully initialized, whereas most other operating systems will
implicitly block on such reads until the random number generator is ready. implicitly block on such reads until the random number generator is ready.
This PEP proposes changing such failures in Python 3.6 from the current silent, For the lower level ``os.urandom`` and ``random.SystemRandom`` APIs, this PEP
proposes changing such failures in Python 3.6 from the current silent,
hard to detect, and hard to debug, errors to easily detected and debugged errors hard to detect, and hard to debug, errors to easily detected and debugged errors
by raising ``BlockingIOError`` with a suitable error message, allowing by raising ``BlockingIOError`` with a suitable error message, allowing
developers the opportunity to unambiguously specify their preferred approach developers the opportunity to unambiguously specify their preferred approach
for handling the situation. for handling the situation.
For the new high level ``secrets`` API, it proposes to block implicitly if
needed whenever random number is generated by that module, as well as to
expose a new ``secrets.wait_for_system_rng()`` function to allow code otherwise
using the low level APIs to explicitly wait for the system random number
generator to be available.
This change will impact any operating system that offers the ``getrandom()`` This change will impact any operating system that offers the ``getrandom()``
system call, regardless of whether the default behaviour of the system call, regardless of whether the default behaviour of the
``/dev/urandom`` device is to return potentially predictable results when the ``/dev/urandom`` device is to return potentially predictable results when the
@ -39,19 +46,8 @@ userspace code prior to the initialization of the system random number
generator, or do not offer the ``getrandom()`` syscall, will be entirely generator, or do not offer the ``getrandom()`` syscall, will be entirely
unaffected by the proposed change (e.g. Windows, Mac OS X, OpenBSD). unaffected by the proposed change (e.g. Windows, Mac OS X, OpenBSD).
The APIs affected by this change would be: The new exception or the blocking behaviour in the ``secrets`` module would
potentially be encountered in the following situations:
* ``os.urandom``
* ``random.SystemRandom``
* the new ``secrets`` module added by PEP 506
A new ``secrets.wait_for_system_rng()`` API would be added to allow affected
applications, frameworks and scripts that encounter the new exception to
readily opt-in to blocking behaviour if they so choose, either by modifying
the affected application directly, or by running a preceding
``python3 -c "import secrets; secrets.wait_for_system_rng()"`` command.
The new exception would potentially be encountered in the following situations:
* Python code calling these APIs during Linux system initialization * Python code calling these APIs during Linux system initialization
* Python code running on improperly initialized Linux systems (e.g. embedded * Python code running on improperly initialized Linux systems (e.g. embedded
@ -59,19 +55,28 @@ The new exception would potentially be encountered in the following situations:
generator, or Linux VMs that aren't configured to accept entropy from the generator, or Linux VMs that aren't configured to accept entropy from the
VM host) VM host)
CPython interpreter initialization and ``random`` module initialization would
also be updated to gracefully fall back to alternative seeding options if the
system random number generator is not ready.
Relationship with other PEPs Relationship with other PEPs
============================ ============================
This PEP depends on the Accepted PEP 506, which adds the ``secrets`` module. This PEP depends on the Accepted PEP 506, which adds the ``secrets`` module.
This PEP competes with Victor Stinner's `currently unnumbered proposal This PEP competes with Victor Stinner's PEP 524, which proposes to make
<http://haypo-notes.readthedocs.io/pep_random.html>`_ to make ``os.urandom`` itself implicitly block when the system RNG is not ready.
``os.urandom`` implicitly block when the system RNG is not ready.
Changes independent of this PEP
===============================
CPython interpreter initialization and ``random`` module initialization have
already been updated to gracefully fall back to alternative seeding options if
the system random number generator is not ready.
This PEP does not compete with the proposal in PEP 524 to add an
``os.getrandom()`` API to expose the ``getrandom`` syscall on platforms that
offer it. There is sufficient motive for adding that API in the ``os`` module's
role as a thin wrapper around potentially platform dependent operating system
features that it can be added regardless of what happens to the default
behaviour of ``os.urandom()`` on these systems.
Proposal Proposal
@ -81,18 +86,22 @@ Changing ``os.urandom()`` on platforms with the getrandom() system call
----------------------------------------------------------------------- -----------------------------------------------------------------------
This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call
the ``getrandom()`` syscall in non-blocking mode if available and the ``getrandom()`` syscall in non-blocking mode if available and raise
raise ``BlockingIOError: system random number generator is not ready`` if ``BlockingIOError: system random number generator is not ready; see 'secrets.token_bytes()'``
the kernel reports that the call would block. if the kernel reports that the call would block.
This behaviour will then propagate through to higher level standard library This behaviour will then propagate through to the existing
APIs that depend on ``os.urandom`` (specifically ``random.SystemRandom`` and ``random.SystemRandom``, which provides a relatively thin wrapper around
the new ``secrets`` module introduced by PEP 506). ``os.urandom()`` that matches the ``random.Random()`` API.
However, the new ``secrets`` module introduced by PEP 506 will be updated to
catch the new exception and implicitly wait for the system random number
generator if the exception is ever encountered.
In all cases, as soon as a call to one of these security sensitive APIs In all cases, as soon as a call to one of these security sensitive APIs
succeeds, all future calls to these APIs in that process will succeed (once succeeds, all future calls to these APIs in that process will succeed
the operating system random number generator is ready after system boot, it without blocking (once the operating system random number generator is ready
remains ready). after system boot, it remains ready).
On Linux and NetBSD, this will replace the previous behaviour of returning On Linux and NetBSD, this will replace the previous behaviour of returning
potentially predictable results read from ``/dev/urandom``. potentially predictable results read from ``/dev/urandom``.
@ -102,9 +111,9 @@ implicitly blocking until the system random number generator is ready. However,
it is not clear if these operating systems actually allow userspace code (and it is not clear if these operating systems actually allow userspace code (and
hence Python) to run before the system random number generator is ready. hence Python) to run before the system random number generator is ready.
Note that in all cases, if calling the ``getrandom()`` API reports ``ENOSYS`` Note that in all cases, if calling the underlying ``getrandom()`` API reports
rather than returning a successful response or reporting ``EAGAIN``, CPython ``ENOSYS`` rather than returning a successful response or reporting ``EAGAIN``,
will continue to fall back to reading from ``/dev/urandom`` directly. CPython will continue to fall back to reading from ``/dev/urandom`` directly.
Adding ``secrets.wait_for_system_rng()`` Adding ``secrets.wait_for_system_rng()``
@ -113,7 +122,8 @@ Adding ``secrets.wait_for_system_rng()``
A new exception shouldn't be added without a straightforward recommendation A new exception shouldn't be added without a straightforward recommendation
for how to resolve that error when encountered (however rare encountering for how to resolve that error when encountered (however rare encountering
the new error is expected to be in practice). For security sensitive code that the new error is expected to be in practice). For security sensitive code that
actually does need to use the system random number generator, and does receive actually does need to use the lower level interfaces to the system random
number generator (rather than the new ``secrets`` module), and does receive
live bug reports indicating this is a real problem for the userbase of that live bug reports indicating this is a real problem for the userbase of that
particular application rather than a theoretical one, this PEP's recommendation particular application rather than a theoretical one, this PEP's recommendation
will be to add the following snippet (directly or indirectly) to the will be to add the following snippet (directly or indirectly) to the
@ -131,10 +141,29 @@ Or, if compatibility with versions prior to Python 3.6 is needed::
else: else:
secrets.wait_for_system_rng() secrets.wait_for_system_rng()
Within the ``secrets`` module itself, this will then be used in
``token_bytes()`` to block implicitly if the new exception is encountered::
def token_bytes(nbytes=None):
if nbytes is None:
nbytes = DEFAULT_ENTROPY
try:
result = os.urandom(nbytes)
except BlockingIOError:
wait_for_system_rng()
result = os.urandom(nbytes)
return result
Other parts of the module will then be updated to use ``token_bytes()`` as
their basic random number generation building block, rather than calling
``os.urandom()`` directly.
Application frameworks covering use cases where access to the system random Application frameworks covering use cases where access to the system random
number generator is almost certain to be needed (e.g. web frameworks) may number generator is almost certain to be needed (e.g. web frameworks) may
choose to incorporate this step implicitly into the commands that start the choose to incorporate a call to ``secrets.wait_for_system_rng()`` implicitly
application. into the commands that start the application such that existing calls to
``os.urandom()`` will be guaranteed to never raise the new exception when using
those frameworks.
For cases where the error is encountered for an application which cannot be For cases where the error is encountered for an application which cannot be
modified directly, then the following command can be used to wait for the modified directly, then the following command can be used to wait for the
@ -147,62 +176,50 @@ For example, this snippet could be added to a shell script or a systemd
system random number generator to be ready, even if the subsequent command system random number generator to be ready, even if the subsequent command
is not itself an application running under Python 3.6) is not itself an application running under Python 3.6)
Given the changes proposed to ``os.urandom()`` above, the suggested Given the changes proposed to ``os.urandom()`` above, and the inclusion of
an ``os.getrandom()`` API on systems that support it, the suggested
implementation of this function would be:: implementation of this function would be::
def wait_for_system_rng(): if hasattr(os, "getrandom"):
"""Block waiting for system random number generator to be ready""" # os.getrandom() always blocks waiting for the system RNG by default
# If the system RNG is already seeded, don't wait at all def wait_for_system_rng():
try: """Block waiting for system random number generator to be ready"""
os.urandom(1) os.getrandom(1)
return return
except BlockingIOError: else:
pass # As far as we know, other platforms will never get BlockingIOError
# Avoid the below busy loop if possible # below but the implementation makes pessimistic assumptions
try: def wait_for_system_rng():
block_on_system_rng = open("/dev/random", "rb") """Block waiting for system random number generator to be ready"""
except FileNotFoundError: # If the system RNG is already seeded, don't wait at all
pass
else:
with block_on_system_rng:
block_on_system_rng.read(1)
# Busy loop until the system RNG is ready
while True:
try: try:
os.urandom(1) os.urandom(1)
break return
except BlockingIOError: except BlockingIOError:
# Only check once per millisecond pass
time.sleep(0.001) # Avoid the below busy loop if possible
try:
block_on_system_rng = open("/dev/random", "rb")
except FileNotFoundError:
pass
else:
with block_on_system_rng:
block_on_system_rng.read(1)
# Busy loop until the system RNG is ready
while True:
try:
os.urandom(1)
break
except BlockingIOError:
# Only check once per millisecond
time.sleep(0.001)
On systems where it is possible to wait for the system RNG to be ready, this On systems where it is possible to wait for the system RNG to be ready, this
function will do so without a busy loop if either ``os.urandom()`` function will do so without a busy loop if ``os.getrandom()`` is defined,
itself implicitly blocks, or the ``/dev/random`` device is available. If the ``os.urandom()`` itself implicitly blocks, or the ``/dev/random`` device is
system random number generator is ready, this call is guaranteed to never available. If the system random number generator is ready, this call is
block, even if the system's ``/dev/random`` device uses a design that permits guaranteed to never block, even if the system's ``/dev/random`` device uses
it to block intermittently during normal system operation. a design that permits it to block intermittently during normal system operation.
Related changes
---------------
Currently, SipHash initialization and ``random`` module initialization
both gather random bytes using the same code that underlies
``os.urandom``. This PEP proposes to modify these so that in situations where
``os.urandom`` would raise a ``BlockingIOError``, they automatically
fall back on potentially more predictable sources of randomness.
In the SipHash case, this will also print a warning message to ``stderr``
indicating that that particular Python process should not be used to process
untrusted data: "Python reverted to potentially predictable hash
initialization. Avoid handling untrusted data in this process.". This
warning would NOT be displayed when hash randomization is explicitly disabled
or set to a known value via ``PYTHONHASHSEED``.
To transparently accommodate a potential future where Linux adopts the same
"potentially blocking during system initialization" ``/dev/urandom`` behaviour
used by other \*nix systems, this fallback source of randomness will *not* be
the ``/dev/urandom`` device.
Limitations on scope Limitations on scope
@ -232,6 +249,21 @@ the new ``getrandom()`` syscall will also remain unchanged.
Rationale Rationale
========= =========
Ensuring the ``secrets`` module implicitly blocks when needed
-------------------------------------------------------------
This is done to help encourage the meme that arises for folks that want the
simplest possible answer to the right way to generate security sensitive random
numbers to be "Use the secrets module when available or your application might
crash unexpectedly", rather than the more boilerplate heavy "Always call
secrets.wait_for_system_rng() when available or your application might crash
unexpectedly".
It's also done due to the BDFL having a higher tolerance for APIs that might
block unexpectedly than he does for APIs that might throw an unexpected
exception [11]_.
Raising ``BlockingIOError`` in ``os.urandom()`` on Linux Raising ``BlockingIOError`` in ``os.urandom()`` on Linux
-------------------------------------------------------- --------------------------------------------------------
@ -278,8 +310,8 @@ developers using Python 3.6+ can easily choose their desired behaviour:
2. Switch to using the random module (non-security sensitive) 2. Switch to using the random module (non-security sensitive)
Adding ``secrets.wait_for_system_rng()`` Making ``secrets.wait_for_system_rng()`` public
---------------------------------------- -----------------------------------------------
Earlier versions of this PEP proposed a number of recipes for wrapping Earlier versions of this PEP proposed a number of recipes for wrapping
``os.urandom()`` to make it suitable for use in security sensitive use cases. ``os.urandom()`` to make it suitable for use in security sensitive use cases.
@ -305,58 +337,6 @@ developers could continue to call ``os.urandom()`` without worrying that it
might unexpectedly start blocking waiting for the system RNG to be available. might unexpectedly start blocking waiting for the system RNG to be available.
Issuing a warning for potentially predictable internal hash initialization
--------------------------------------------------------------------------
The challenge for internal hash initialization is that it might be very
important to initialize SipHash with a reliably unpredictable random seed
(for processes that are exposed to potentially hostile input) or it might be
totally unimportant (for processes that never have to deal with untrusted data).
The Python runtime has no way to know which case a given invocation involves,
which means that if we allow SipHash initialization to block or error out,
then our intended security enhancement may break code that is already safe
and working fine, which is unacceptable -- especially since we are reasonably
confident that most Python invocations that might run during Linux system
initialization fall into this category (exposure to untrusted input tends to
involve network access, which typically isn't brought up until after the system
random number generator is initialized).
However, at the same time, since Python has no way to know whether any given
invocation needs to handle untrusted data, when the default SipHash
initialization fails this *might* indicate a genuine security problem, which
should not be allowed to pass silently.
Accordingly, if internal hash initialization needs to fall back to a potentially
predictable seed due to the system random number generator not being ready, it
will also emit a warning message on ``stderr`` to say that the system random
number generator is not available and that processing potentially hostile
untrusted data should be avoided.
Allowing potentially predictable ``random`` module initialization
-----------------------------------------------------------------
Other than for ``random.SystemRandom`` (which is a relatively thin
wrapper around ``os.urandom``), the ``random`` module has long documented
that the numbers it generates are not suitable for use in security sensitive
operations. Instead, the use of the system random number generator to seed the
default Mersenne Twister instance is primarily aimed at ensuring Python isn't
biased towards any particular starting states for simulation use cases.
However, this seeding approach has also turned out to be beneficial as a harm
mitigation measure for code that is using the ``random`` module inappropriately.
Since a single call to ``os.urandom()`` is cheap once the system random
number generator has been initialized it makes sense to retain that as the
default behaviour, but there's no need to issue a warning when falling back to
a potentially more predictable alternative when necessary (in such cases,
a warning will typically already have been issued as part of interpreter
startup, as the only way for the call when importing the random module to
fail without the implicit call during interpreter startup also failing if for
the latter to have been skipped by entirely disabling the hash randomization
mechanism).
Backwards Compatibility Impact Assessment Backwards Compatibility Impact Assessment
========================================= =========================================
@ -394,7 +374,9 @@ regardless of whether or not they perform security sensitive operations:
- applications that don't support Linux - applications that don't support Linux
- applications that are only run on desktops or conventional servers - applications that are only run on desktops or conventional servers
- applications that are only run after the system RNG is ready - applications that are only run after the system RNG is ready (including
those where an application framework calls ``secrets.wait_for_system_rng()``
on their behalf)
Applications in this category simply won't encounter the new exception, so it Applications in this category simply won't encounter the new exception, so it
will be reasonable for developers to wait and see if they receive will be reasonable for developers to wait and see if they receive
@ -407,14 +389,23 @@ Affected security sensitive applications
Security sensitive applications would need to either change their system Security sensitive applications would need to either change their system
configuration so the application is only started after the operating system configuration so the application is only started after the operating system
random number generator is ready for security sensitive operations, or else random number generator is ready for security sensitive operations, change the
change the application startup code to invoke ``secrets.wait_for_system_rng()`` application startup code to invoke ``secrets.wait_for_system_rng()``, or
else switch to using the new ``secrets.token_bytes()`` API.
As an example for components started via a systemd unit file, the following As an example for components started via a systemd unit file, the following
snippet would delay activation until the system RNG was ready: snippet would delay activation until the system RNG was ready:
ExecStartPre=python3 -c "import secrets; secrets.wait_for_system_rng()" ExecStartPre=python3 -c "import secrets; secrets.wait_for_system_rng()"
Alternatively, the following snippet will use ``secrets.token_bytes()`` if
available, and fall back to ``os.urandom()`` otherwise:
try:
import secrets.token_bytes as _get_random_bytes
except ImportError:
import os.urandom as _get_random_bytes
Affected non-security sensitive applications Affected non-security sensitive applications
-------------------------------------------- --------------------------------------------
@ -671,8 +662,8 @@ decision has to be made:
* the higher level ``random.SystemRandom`` public API * the higher level ``random.SystemRandom`` public API
* the new ``secrets`` module public API added by PEP 506 * the new ``secrets`` module public API added by PEP 506
Currently, these five places all use the same underlying code, and Previously, these five places all used the same underlying code, and
thus make this decision in the same way. thus made this decision in the same way.
This whole problem was first noticed because 3.5.0 switched that This whole problem was first noticed because 3.5.0 switched that
underlying code to the ``generate_unpredictable_bytes_or_block`` behavior, underlying code to the ``generate_unpredictable_bytes_or_block`` behavior,
@ -705,15 +696,14 @@ We have not received any specific complaints regarding direct calls to
only problem reports due to the implicit blocking on interpreter startup and only problem reports due to the implicit blocking on interpreter startup and
as a side-effect of importing the random module. as a side-effect of importing the random module.
Accordingly, this PEP proposes providing consistent shared behaviour for the Independently of this PEP, the first two cases have already been updated to
latter three cases (ensuring that their behaviour is unequivocally suitable for never block, regardless of the behaviour of ``os.urandom()``.
all security sensitive operations), while updating the first two cases to
account for that behavioural change.
This approach should mean that the vast majority of Python users never need to Where PEP 524 proposes to make all 3 of the latter cases block implicitly,
even be aware that this change was made, while those few whom it affects will this PEP proposes that approach only for the last case (the ``secrets``)
receive an exception at runtime that they can look up online and find suitable module, with ``os.urandom()`` and ``random.SystemRandom()`` instead raising
guidance on addressing. an exception when they detect that the underlying operating system call
would block.
References References
@ -751,6 +741,9 @@ References
.. [10] Does the HAVE_GETRANDOM_SYSCALL config setting make sense? .. [10] Does the HAVE_GETRANDOM_SYSCALL config setting make sense?
(https://mail.python.org/pipermail/security-sig/2016-June/000060.html) (https://mail.python.org/pipermail/security-sig/2016-June/000060.html)
.. [11] Take a decision for os.urandom() in Python 3.6
(https://mail.python.org/pipermail/security-sig/2016-August/000084.htm)
For additional background details beyond those captured in this PEP and Victor's For additional background details beyond those captured in this PEP and Victor's
competing PEP, also see Victor's prior collection of relevant information and competing PEP, also see Victor's prior collection of relevant information and