mirror of
https://github.com/apache/nifi.git
synced 2025-03-03 16:09:19 +00:00
NIFI-10506 Moved Repository Encryption to Admin Guide
- Removed Repository Encryption section from User Guide - Added Configuration Change Considerations section Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #6615.
This commit is contained in:
exceptionfactory
Nathan Gough
parent
4e4f6437ee
commit
f3cd416816
@ -1599,11 +1599,156 @@ Being added to both the view and modify policies for the process group, User2 ca
|
||||
|
||||
image:user2-edit-connection.png["User2 Edit Connection"]
|
||||
|
||||
[[repository-encryption]]
|
||||
== Repository Encryption
|
||||
|
||||
NiFi supports encryption of local repositories using a configurable Key Provider to enable protection of information
|
||||
on the filesystem. Repository encryption configuration uses a version number to indicate the cipher algorithms, metadata
|
||||
format, and repository implementation classes. This approach provides a generalized method for configuration without the
|
||||
need to customize each repository implementation class.
|
||||
|
||||
Repository encryption incurs a performance cost due to the overhead of cipher operations. Filesystem encryption at the
|
||||
operating system level provides an alternative solution, with different performance characteristics. For deployments
|
||||
where filesystem encryption is not configured, repository encryption provides an enhanced level of data protection.
|
||||
Due to increased performance requirements, more computing resources may be necessary to achieve sufficient throughput
|
||||
when enabling repository encryption.
|
||||
|
||||
The security of repository encryption depends on a combination of the cipher algorithms and the protection of encryption
|
||||
keys. Key protection and key rotation are important parts of securing an encrypted repository configuration.
|
||||
Key protection involves limiting access to the Key Provider and key rotation requires manual updates to generate and
|
||||
specify a new encryption key.
|
||||
|
||||
=== Configuration Change Considerations
|
||||
|
||||
Disabling repository encryption on existing installations requires removing existing repository contents, and
|
||||
restarting the system after making configuration changes. For this reason, flow administrators should confirm that the
|
||||
system has processed all available FlowFiles to avoid losing information when disabling repository encryption.
|
||||
|
||||
=== Repository Encryption Protocol Version 1
|
||||
|
||||
The first version of support for repository encryption includes the following cipher algorithms:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Cipher Algorithm
|
||||
|Content Repository |AES/CTR/NoPadding
|
||||
|FlowFile Repository |AES/GCM/NoPadding
|
||||
|FlowFile Swap Manager |AES/GCM/NoPadding
|
||||
|Provenance Repository |AES/GCM/NoPadding
|
||||
|=======================
|
||||
|
||||
The following classes provide the direct repository encryption implementation, extending standard classes:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Class
|
||||
|Content Repository |org.apache.nifi.content.EncryptedFileSystemRepository
|
||||
|FlowFile Repository |org.apache.nifi.wali.EncryptedSequentialAccessWriteAheadLog
|
||||
|FlowFile Swap Manager |org.apache.nifi.controller.EncryptedFileSystemSwapManager
|
||||
|Provenance Repository |org.apache.nifi.provenance.EncryptedWriteAheadProvenanceRepository
|
||||
|=======================
|
||||
|
||||
==== Encryption Metadata Serialization
|
||||
|
||||
Each repository implementation class leverages standard cipher operations to perform encryption and decryption. In order
|
||||
to support AES, the encryption process writes metadata associated with each encryption operation. Encryption protocol
|
||||
version 1 uses Java Object serialization to write objects containing the encryption Key Identifier, the cipher
|
||||
Initialization Vector, and other required properties. Serialized objects include the following required properties:
|
||||
|
||||
[options="header"]
|
||||
|=====================
|
||||
|Property Name |Property Type |Description
|
||||
|keyId |String |Encryption key identifier
|
||||
|ivBytes |byte[] |Cipher initialization vector
|
||||
|algorithm |String |Cipher algorithm
|
||||
|version |String |Encryption protocol version
|
||||
|cipherByteLength |int |Length of enciphered record
|
||||
|=====================
|
||||
|
||||
Metadata serialization uses the standard `java.io.ObjectOutputStream.writeObject()` method to write objects to a stream
|
||||
that can be converted to a byte array. The deserialization process uses a custom extension of the
|
||||
`java.io.ObjectInputStream` to read objects regardless of the original class name associated with the record. This
|
||||
approach requires the presence of the standard metadata properties, but provides a compatibility layer that avoids
|
||||
linking the implementation to a specific Java class.
|
||||
|
||||
The initial implementation of encrypted repositories used different byte array markers when writing metadata. Each
|
||||
repository implementation uses the following byte array markers before writing a serialized metadata record:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Byte Array
|
||||
|Content Repository |byte[]{0x00, 0x00}
|
||||
|FlowFile Repository |byte[]{}
|
||||
|Provenance Repository |byte[]{0x01}
|
||||
|=======================
|
||||
|
||||
=== Repository Encryption Configuration
|
||||
|
||||
Configuring repository encryption requires specifying the encryption protocol version and the associated Key Provider
|
||||
properties. Repository encryption can be configured on new or existing installations using standard properties. Records
|
||||
in existing repositories should be readable using standard capabilities, and the encrypted repository will write new
|
||||
records using the specified configuration.
|
||||
|
||||
==== Protocol Version Configuration
|
||||
|
||||
Setting the following protocol version property enables encryption for all repositories:
|
||||
|
||||
....
|
||||
nifi.repository.encryption.protocol.version=1
|
||||
....
|
||||
|
||||
==== Key Provider Configuration
|
||||
|
||||
All encrypted repositories require a Key Provider to perform encryption and decryption operations. NiFi supports
|
||||
configuring the Key Provider implementation as well as the Key Identifier that will be used for new encryption
|
||||
operations. Key Provider implementations can hold multiple keys to support using a new key while maintaining access to
|
||||
information encrypted using the previous key.
|
||||
|
||||
Repository encryption supports access to secret keys using standard `java.security.KeyStore` files.
|
||||
See <<secret-key-generation-and-storage-using-keytool>> for details on supported KeyStore types, as well as examples of
|
||||
generating secret keys.
|
||||
|
||||
The following configuration properties provide an example using a PKCS12 KeyStore file named `repository.p12` containing
|
||||
a secret key labeled with an alias of `primary-key`:
|
||||
|
||||
....
|
||||
nifi.repository.encryption.key.id=primary-key
|
||||
nifi.repository.encryption.key.provider=KEYSTORE
|
||||
nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
|
||||
nifi.repository.encryption.key.provider.keystore.password=2fRKmwDyMYmT7P5L
|
||||
....
|
||||
|
||||
[[secret-key-generation-and-storage-using-keytool]]
|
||||
==== Secret Key Generation and Storage using Keytool
|
||||
|
||||
The `KeyStoreKeyProvider` supports reading from a `java.security.KeyStore` using a configured password to load AES Secret Key entries.
|
||||
The `KeyStoreKeyProvider` can be configured with any of the encrypted repository implementations.
|
||||
|
||||
The provider supports the following KeyStore Types:
|
||||
|
||||
* BCFKS
|
||||
* PKCS12
|
||||
|
||||
The keystore filename extension must be either `.p12` indicating PKCS12 or `.bcfks` indicating BCFKS.
|
||||
|
||||
The `keytool` command can be used to generate an AES-256 Secret Key stored in a PKCS12 file for repository encryption:
|
||||
|
||||
keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12
|
||||
|
||||
The `keytool` command requires additional arguments specifying the BouncyCastle Security Provider to store
|
||||
Secret Keys using BCFKS. The arguments must include a reference to the BouncyCastle Security Provider library, which
|
||||
is available in the `lib/bootstrap` directory under the NiFi installation.
|
||||
|
||||
The following command can be used to generate an AES-256 Secret Key stored using BCFKS:
|
||||
|
||||
keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.bcfks -storetype BCFKS -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath lib/bootstrap/bcprov-jdk15on-*.jar
|
||||
|
||||
Enter a keystore password when prompted. The same value must be used for both the keystore password and key password.
|
||||
The keystore password will be used in the provider configuration properties.
|
||||
|
||||
[[encryption]]
|
||||
== Encryption Configuration
|
||||
|
||||
This section provides an overview of the capabilities of NiFi to encrypt and decrypt data.
|
||||
|
||||
The EncryptContent processor allows for the encryption and decryption of data, both internal to NiFi and integrated with external systems, such as `openssl` and other data sources and consumers.
|
||||
|
||||
[[key-derivation-functions]]
|
||||
|
||||
@ -2970,150 +2970,6 @@ Once these property changes have been made, restart NiFi.
|
||||
|
||||
**Note:** Detailed descriptions for each of these properties can be found in <<administration-guide.adoc#system_properties,System Properties>>.
|
||||
|
||||
== Repository Encryption
|
||||
|
||||
NiFi supports encryption of local repositories using a configurable Key Provider to enable protection of information
|
||||
on the filesystem. Repository encryption configuration uses a version number to indicate the cipher algorithms, metadata
|
||||
format, and repository implementation classes. This approach provides a generalized method for configuration without the
|
||||
need to customize each repository implementation class.
|
||||
|
||||
Repository encryption incurs a performance cost due to the overhead of cipher operations. Filesystem encryption at the
|
||||
operating system level provides an alternative solution, with different performance characteristics. For deployments
|
||||
where filesystem encryption is not configured, repository encryption provides an enhanced level of data protection.
|
||||
Due to increased performance requirements, more computing resources may be necessary to achieve sufficient throughput
|
||||
when enabling repository encryption.
|
||||
|
||||
The security of repository encryption depends on a combination of the cipher algorithms and the protection of encryption
|
||||
keys. Key protection and key rotation are important parts of securing an encrypted repository configuration.
|
||||
Key protection involves limiting access to the Key Provider and key rotation requires manual updates to generate and
|
||||
specify a new encryption key.
|
||||
|
||||
=== Repository Encryption Protocol Version 1
|
||||
|
||||
The first version of support for repository encryption includes the following cipher algorithms:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Cipher Algorithm
|
||||
|Content Repository |AES/CTR/NoPadding
|
||||
|FlowFile Repository |AES/GCM/NoPadding
|
||||
|FlowFile Swap Manager |AES/GCM/NoPadding
|
||||
|Provenance Repository |AES/GCM/NoPadding
|
||||
|=======================
|
||||
|
||||
The following classes provide the direct repository encryption implementation, extending standard classes:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Class
|
||||
|Content Repository |org.apache.nifi.content.EncryptedFileSystemRepository
|
||||
|FlowFile Repository |org.apache.nifi.wali.EncryptedSequentialAccessWriteAheadLog
|
||||
|FlowFile Swap Manager |org.apache.nifi.controller.EncryptedFileSystemSwapManager
|
||||
|Provenance Repository |org.apache.nifi.provenance.EncryptedWriteAheadProvenanceRepository
|
||||
|=======================
|
||||
|
||||
==== Encryption Metadata Serialization
|
||||
|
||||
Each repository implementation class leverages standard cipher operations to perform encryption and decryption. In order
|
||||
to support AES, the encryption process writes metadata associated with each encryption operation. Encryption protocol
|
||||
version 1 uses Java Object serialization to write objects containing the encryption Key Identifier, the cipher
|
||||
Initialization Vector, and other required properties. Serialized objects include the following required properties:
|
||||
|
||||
[options="header"]
|
||||
|=====================
|
||||
|Property Name |Property Type |Description
|
||||
|keyId |String |Encryption key identifier
|
||||
|ivBytes |byte[] |Cipher initialization vector
|
||||
|algorithm |String |Cipher algorithm
|
||||
|version |String |Encryption protocol version
|
||||
|cipherByteLength |int |Length of enciphered record
|
||||
|=====================
|
||||
|
||||
Metadata serialization uses the standard `java.io.ObjectOutputStream.writeObject()` method to write objects to a stream
|
||||
that can be converted to a byte array. The deserialization process uses a custom extension of the
|
||||
`java.io.ObjectInputStream` to read objects regardless of the original class name associated with the record. This
|
||||
approach requires the presence of the standard metadata properties, but provides a compatibility layer that avoids
|
||||
linking the implementation to a specific Java class.
|
||||
|
||||
The initial implementation of encrypted repositories used different byte array markers when writing metadata. Each
|
||||
repository implementation uses the following byte array markers before writing a serialized metadata record:
|
||||
|
||||
[options="header"]
|
||||
|=======================
|
||||
|Repository Type |Byte Array
|
||||
|Content Repository |byte[]{0x00, 0x00}
|
||||
|FlowFile Repository |byte[]{}
|
||||
|Provenance Repository |byte[]{0x01}
|
||||
|=======================
|
||||
|
||||
=== Repository Encryption Configuration
|
||||
|
||||
Configuring repository encryption requires specifying the encryption protocol version and the associated Key Provider
|
||||
properties. Repository encryption can be configured on new or existing installations using standard properties. Records
|
||||
in existing repositories should be readable using standard capabilities, and the encrypted repository will write new
|
||||
records using the specified configuration.
|
||||
|
||||
Disabling repository encryption on existing installations requires removing existing repository contents, and
|
||||
restarting the system after making configuration changes. For this reason, flow administrators should confirm that the
|
||||
system has processed all available FlowFiles to avoid losing information when disabling repository encryption.
|
||||
|
||||
==== Protocol Version Configuration
|
||||
|
||||
Setting the following protocol version property enables encryption for all repositories:
|
||||
|
||||
....
|
||||
nifi.repository.encryption.protocol.version=1
|
||||
....
|
||||
|
||||
==== Key Provider Configuration
|
||||
|
||||
All encrypted repositories require a Key Provider to perform encryption and decryption operations. NiFi supports
|
||||
configuring the Key Provider implementation as well as the Key Identifier that will be used for new encryption
|
||||
operations. Key Provider implementations can hold multiple keys to support using a new key while maintaining access to
|
||||
information encrypted using the previous key.
|
||||
|
||||
Repository encryption supports access to secret keys using standard `java.security.KeyStore` files.
|
||||
See <<secret-key-generation-and-storage-using-keytool>> for details on supported KeyStore types, as well as examples of
|
||||
generating secret keys.
|
||||
|
||||
The following configuration properties provide an example using a PKCS12 KeyStore file named `repository.p12` containing
|
||||
a secret key labeled with an alias of `primary-key`:
|
||||
|
||||
....
|
||||
nifi.repository.encryption.key.id=primary-key
|
||||
nifi.repository.encryption.key.provider=KEYSTORE
|
||||
nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
|
||||
nifi.repository.encryption.key.provider.keystore.password=2fRKmwDyMYmT7P5L
|
||||
....
|
||||
|
||||
[[secret-key-generation-and-storage-using-keytool]]
|
||||
==== Secret Key Generation and Storage using Keytool
|
||||
|
||||
The `KeyStoreKeyProvider` supports reading from a `java.security.KeyStore` using a configured password to load AES Secret Key entries.
|
||||
The `KeyStoreKeyProvider` can be configured with any of the encrypted repository implementations.
|
||||
|
||||
The provider supports the following KeyStore Types:
|
||||
|
||||
* BCFKS
|
||||
* PKCS12
|
||||
|
||||
The keystore filename extension must be either `.p12` indicating PKCS12 or `.bcfks` indicating BCFKS.
|
||||
|
||||
The `keytool` command can be used to generate an AES-256 Secret Key stored in a PKCS12 file for repository encryption:
|
||||
|
||||
keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12
|
||||
|
||||
The `keytool` command requires additional arguments specifying the BouncyCastle Security Provider to store
|
||||
Secret Keys using BCFKS. The arguments must include a reference to the BouncyCastle Security Provider library, which
|
||||
is available in the `lib/bootstrap` directory under the NiFi installation.
|
||||
|
||||
The following command can be used to generate an AES-256 Secret Key stored using BCFKS:
|
||||
|
||||
keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.bcfks -storetype BCFKS -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath lib/bootstrap/bcprov-jdk15on-*.jar
|
||||
|
||||
Enter a keystore password when prompted. The same value must be used for both the keystore password and key password.
|
||||
The keystore password will be used in the provider configuration properties.
|
||||
|
||||
[[experimental_warning]]
|
||||
== Experimental Warning
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user