Transparent Encryption in HDFS

Overview

For transparent encryption, we introduce a new abstraction to HDFS: the encryption zone. An encryption zone is a special directory whose contents will be transparently encrypted upon write and transparently decrypted upon read. Each encryption zone is associated with a single encryption zone key which is specified when the zone is created. Each file within an encryption zone has its own unique data encryption key (DEK). DEKs are never handled directly by HDFS. Instead, HDFS only ever handles an encrypted data encryption key (EDEK). Clients decrypt an EDEK, and then use the subsequent DEK to read and write data. HDFS datanodes simply see a stream of encrypted bytes.

A very important use case of encryption is to “switch it on” and ensure all files across the entire filesystem are encrypted. To support this strong guarantee without losing the flexibility of using different encryption zone keys in different parts of the filesystem, HDFS allows nested encryption zones. After an encryption zone is created (e.g. on the root directory /), a user can create more encryption zones on its descendant directories (e.g. /home/alice) with different keys. The EDEK of a file will be generated using the encryption zone key from the closest ancestor encryption zone.

A new cluster service is required to manage encryption keys: the Hadoop Key Management Server (KMS). In the context of HDFS encryption, the KMS performs three basic responsibilities:

1. Providing access to stored encryption zone keys

2. Generating new encrypted data encryption keys for storage on the NameNode

3. Decrypting encrypted data encryption keys for use by HDFS clients

The KMS will be described in more detail below.

Accessing data within an encryption zone

When creating a new file in an encryption zone, the NameNode asks the KMS to generate a new EDEK encrypted with the encryption zone’s key. The EDEK is then stored persistently as part of the file’s metadata on the NameNode.

When reading a file within an encryption zone, the NameNode provides the client with the file’s EDEK and the encryption zone key version used to encrypt the EDEK. The client then asks the KMS to decrypt the EDEK, which involves checking that the client has permission to access the encryption zone key version. Assuming that is successful, the client uses the DEK to decrypt the file’s contents.

All of the above steps for the read and write path happen automatically through interactions between the DFSClient, the NameNode, and the KMS.

Access to encrypted file data and metadata is controlled by normal HDFS filesystem permissions. This means that if HDFS is compromised (for example, by gaining unauthorized access to an HDFS superuser account), a malicious user only gains access to ciphertext and encrypted keys. However, since access to encryption zone keys is controlled by a separate set of permissions on the KMS and key store, this does not pose a security threat.

Key Management Server, KeyProvider, EDEKs

The KMS is a proxy that interfaces with a backing key store on behalf of HDFS daemons and clients. Both the backing key store and the KMS implement the Hadoop KeyProvider API. See the KMS documentation for more information.

In the KeyProvider API, each encryption key has a unique key name. Because keys can be rolled, a key can have multiple key versions, where each key version has its own key material (the actual secret bytes used during encryption and decryption). An encryption key can be fetched by either its key name, returning the latest version of the key, or by a specific key version.

The KMS implements additional functionality which enables creation and decryption of encrypted encryption keys (EEKs). Creation and decryption of EEKs happens entirely on the KMS. Importantly, the client requesting creation or decryption of an EEK never handles the EEK’s encryption key. To create a new EEK, the KMS generates a new random key, encrypts it with the specified key, and returns the EEK to the client. To decrypt an EEK, the KMS checks that the user has access to the encryption key, uses it to decrypt the EEK, and returns the decrypted encryption key.

In the context of HDFS encryption, EEKs are encrypted data encryption keys (EDEKs), where a data encryption key (DEK) is what is used to encrypt and decrypt file data. Typically, the key store is configured to only allow end users access to the keys used to encrypt DEKs. This means that EDEKs can be safely stored and handled by HDFS, since the HDFS user will not have access to unencrypted encryption keys.

Configuring the cluster KeyProvider

Selecting an encryption algorithm and codec

Namenode configuration

createZone

Usage: [-createZone -keyName <keyName> -path <path>]

Create a new encryption zone.

listZones

Usage: [-listZones]

List all encryption zones. Requires superuser permissions.

provisionTrash

Usage: [-provisionTrash -path <path>]

Provision a trash directory for an encryption zone.

getFileEncryptionInfo

Usage: [-getFileEncryptionInfo -path <path>]

Get encryption information from a file. This can be used to find out whether a file is being encrypted, and the key name / key version used to encrypt it.

reencryptZone

Usage: [-reencryptZone <action> -path <zone>]

Re-encrypts an encryption zone, by iterating through the encryption zone, and calling the KeyProvider’s reencryptEncryptedKeys interface to batch-re-encrypt all files’ EDEKs with the latest version encryption zone key in the key provider. Requires superuser permissions.

Note that re-encryption does not apply to snapshots, due to snapshots’ immutable nature.

Re-encryption is a NameNode-only operation in HDFS, so could potentially put intensive load to the NameNode. The following configurations can be changed to control the stress on the NameNode, depending on the acceptable throughput impact to the cluster.

listReencryptionStatus

Usage: [-listReencryptionStatus]

List re-encryption information for all encryption zones. Requires superuser permissions.

Running as the superuser

One common usecase for distcp is to replicate data between clusters for backup and disaster recovery purposes. This is typically performed by the cluster administrator, who is an HDFS superuser.

To enable this same workflow when using HDFS encryption, we introduced a new virtual path prefix, /.reserved/raw/, that gives superusers direct access to the underlying block data in the filesystem. This allows superusers to distcp data without needing having access to encryption keys, and also avoids the overhead of decrypting and re-encrypting data. It also means the source and destination data will be byte-for-byte identical, which would not be true if the data was being re-encrypted with a new EDEK.

When using /.reserved/raw to distcp encrypted data, it’s important to preserve extended attributes with the -px flag. This is because encrypted file attributes (such as the EDEK) are exposed through extended attributes within /.reserved/raw, and must be preserved to be able to decrypt the file. This means that if the distcp is initiated at or above the encryption zone root, it will automatically create an encryption zone at the destination if it does not already exist. However, it’s still recommended that the admin first create identical encryption zones on the destination cluster to avoid any potential mishaps.

Copying into encrypted locations

By default, distcp compares checksums provided by the filesystem to verify that the data was successfully copied to the destination. When copying from unencrypted or encrypted location into an encrypted location, the filesystem checksums will not match since the underlying block data is different because a new EDEK will be used to encrypt at destination. In this case, specify the -skipcrccheck and -update distcp flags to avoid verifying checksums.

Hardware access exploits

These exploits assume that attacker has gained physical access to hard drives from cluster machines, i.e. datanodes and namenodes.

1. Access to swap files of processes containing data encryption keys.

   • By itself, this does not expose cleartext, as it also requires access to encrypted block files.

   • This can be mitigated by disabling swap, using encrypted swap, or using mlock to prevent keys from being swapped out.

2. Access to encrypted block files.

   • By itself, this does not expose cleartext, as it also requires access to DEKs.

Root access exploits

These exploits assume that attacker has gained root shell access to cluster machines, i.e. datanodes and namenodes. Many of these exploits cannot be addressed in HDFS, since a malicious root user has access to the in-memory state of processes holding encryption keys and cleartext. For these exploits, the only mitigation technique is carefully restricting and monitoring root shell access.

1. Access to encrypted block files.

   • By itself, this does not expose cleartext, as it also requires access to encryption keys.

2. Dump memory of client processes to obtain DEKs, delegation tokens, cleartext.

   • No mitigation.

3. Recording network traffic to sniff encryption keys and encrypted data in transit.

   • By itself, insufficient to read cleartext without the EDEK encryption key.

4. Dump memory of datanode process to obtain encrypted block data.

   • By itself, insufficient to read cleartext without the DEK.

5. Dump memory of namenode process to obtain encrypted data encryption keys.

   • By itself, insufficient to read cleartext without the EDEK’s encryption key and encrypted block files.

HDFS admin exploits

These exploits assume that the attacker has compromised HDFS, but does not have root or hdfs user shell access.

1. Access to encrypted block files.

   • By itself, insufficient to read cleartext without the EDEK and EDEK encryption key.

2. Access to encryption zone and encrypted file metadata (including encrypted data encryption keys), via -fetchImage.

   • By itself, insufficient to read cleartext without EDEK encryption keys.

Rogue user exploits

A rogue user can collect keys of files they have access to, and use them later to decrypt the encrypted data of those files. As the user had access to those files, they already had access to the file contents. This can be mitigated through periodic key rolling policies. The reencryptZone command is usually required after key rolling, to make sure the EDEKs on existing files use the new version key.

Manual steps to a complete key rolling and re-encryption are listed below. These instructions assume that you are running as the key admin or HDFS superuser as is appropriate.

# As the key admin, roll the key to a new version
hadoop key roll exposedKey

# As the super user, re-encrypt the encryption zone. Possibly list zones first.
hdfs crypto -listZones
hdfs crypto -reencryptZone -start -path /zone

# As the super user, periodically check the status of re-encryption
hdfs crypto -listReencryptionStatus

# As the super user, get encryption information from the file and double check it's encryption key version
hdfs crypto -getFileEncryptionInfo -path /zone/helloWorld
# console output: {cipherSuite: {name: AES/CTR/NoPadding, algorithmBlockSize: 16}, cryptoProtocolVersion: CryptoProtocolVersion{description='Encryption zones', version=2, unknownValue=null}, edek: 2010d301afbd43b58f10737ce4e93b39, iv: ade2293db2bab1a2e337f91361304cb3, keyName: exposedKey, ezKeyVersionName: exposedKey@1}