From f3cd416816ae0d0dc86d7bb7fc97601e4c315619 Mon Sep 17 00:00:00 2001 From: exceptionfactory Date: Wed, 2 Nov 2022 12:30:02 -0500 Subject: [PATCH] NIFI-10506 Moved Repository Encryption to Admin Guide - Removed Repository Encryption section from User Guide - Added Configuration Change Considerations section Signed-off-by: Nathan Gough This closes #6615. --- .../main/asciidoc/administration-guide.adoc | 149 +++++++++++++++++- nifi-docs/src/main/asciidoc/user-guide.adoc | 144 ----------------- 2 files changed, 147 insertions(+), 146 deletions(-) diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index b4fd84dce1..5829b4dbf7 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -1599,11 +1599,156 @@ Being added to both the view and modify policies for the process group, User2 ca image:user2-edit-connection.png["User2 Edit Connection"] +[[repository-encryption]] +== Repository Encryption + +NiFi supports encryption of local repositories using a configurable Key Provider to enable protection of information +on the filesystem. Repository encryption configuration uses a version number to indicate the cipher algorithms, metadata +format, and repository implementation classes. This approach provides a generalized method for configuration without the +need to customize each repository implementation class. + +Repository encryption incurs a performance cost due to the overhead of cipher operations. Filesystem encryption at the +operating system level provides an alternative solution, with different performance characteristics. For deployments +where filesystem encryption is not configured, repository encryption provides an enhanced level of data protection. +Due to increased performance requirements, more computing resources may be necessary to achieve sufficient throughput +when enabling repository encryption. + +The security of repository encryption depends on a combination of the cipher algorithms and the protection of encryption +keys. Key protection and key rotation are important parts of securing an encrypted repository configuration. +Key protection involves limiting access to the Key Provider and key rotation requires manual updates to generate and +specify a new encryption key. + +=== Configuration Change Considerations + +Disabling repository encryption on existing installations requires removing existing repository contents, and +restarting the system after making configuration changes. For this reason, flow administrators should confirm that the +system has processed all available FlowFiles to avoid losing information when disabling repository encryption. + +=== Repository Encryption Protocol Version 1 + +The first version of support for repository encryption includes the following cipher algorithms: + +[options="header"] +|======================= +|Repository Type |Cipher Algorithm +|Content Repository |AES/CTR/NoPadding +|FlowFile Repository |AES/GCM/NoPadding +|FlowFile Swap Manager |AES/GCM/NoPadding +|Provenance Repository |AES/GCM/NoPadding +|======================= + +The following classes provide the direct repository encryption implementation, extending standard classes: + +[options="header"] +|======================= +|Repository Type |Class +|Content Repository |org.apache.nifi.content.EncryptedFileSystemRepository +|FlowFile Repository |org.apache.nifi.wali.EncryptedSequentialAccessWriteAheadLog +|FlowFile Swap Manager |org.apache.nifi.controller.EncryptedFileSystemSwapManager +|Provenance Repository |org.apache.nifi.provenance.EncryptedWriteAheadProvenanceRepository +|======================= + +==== Encryption Metadata Serialization + +Each repository implementation class leverages standard cipher operations to perform encryption and decryption. In order +to support AES, the encryption process writes metadata associated with each encryption operation. Encryption protocol +version 1 uses Java Object serialization to write objects containing the encryption Key Identifier, the cipher +Initialization Vector, and other required properties. Serialized objects include the following required properties: + +[options="header"] +|===================== +|Property Name |Property Type |Description +|keyId |String |Encryption key identifier +|ivBytes |byte[] |Cipher initialization vector +|algorithm |String |Cipher algorithm +|version |String |Encryption protocol version +|cipherByteLength |int |Length of enciphered record +|===================== + +Metadata serialization uses the standard `java.io.ObjectOutputStream.writeObject()` method to write objects to a stream +that can be converted to a byte array. The deserialization process uses a custom extension of the +`java.io.ObjectInputStream` to read objects regardless of the original class name associated with the record. This +approach requires the presence of the standard metadata properties, but provides a compatibility layer that avoids +linking the implementation to a specific Java class. + +The initial implementation of encrypted repositories used different byte array markers when writing metadata. Each +repository implementation uses the following byte array markers before writing a serialized metadata record: + +[options="header"] +|======================= +|Repository Type |Byte Array +|Content Repository |byte[]{0x00, 0x00} +|FlowFile Repository |byte[]{} +|Provenance Repository |byte[]{0x01} +|======================= + +=== Repository Encryption Configuration + +Configuring repository encryption requires specifying the encryption protocol version and the associated Key Provider +properties. Repository encryption can be configured on new or existing installations using standard properties. Records +in existing repositories should be readable using standard capabilities, and the encrypted repository will write new +records using the specified configuration. + +==== Protocol Version Configuration + +Setting the following protocol version property enables encryption for all repositories: + +.... +nifi.repository.encryption.protocol.version=1 +.... + +==== Key Provider Configuration + +All encrypted repositories require a Key Provider to perform encryption and decryption operations. NiFi supports +configuring the Key Provider implementation as well as the Key Identifier that will be used for new encryption +operations. Key Provider implementations can hold multiple keys to support using a new key while maintaining access to +information encrypted using the previous key. + +Repository encryption supports access to secret keys using standard `java.security.KeyStore` files. +See <> for details on supported KeyStore types, as well as examples of +generating secret keys. + +The following configuration properties provide an example using a PKCS12 KeyStore file named `repository.p12` containing +a secret key labeled with an alias of `primary-key`: + +.... +nifi.repository.encryption.key.id=primary-key +nifi.repository.encryption.key.provider=KEYSTORE +nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 +nifi.repository.encryption.key.provider.keystore.password=2fRKmwDyMYmT7P5L +.... + +[[secret-key-generation-and-storage-using-keytool]] +==== Secret Key Generation and Storage using Keytool + +The `KeyStoreKeyProvider` supports reading from a `java.security.KeyStore` using a configured password to load AES Secret Key entries. +The `KeyStoreKeyProvider` can be configured with any of the encrypted repository implementations. + +The provider supports the following KeyStore Types: + +* BCFKS +* PKCS12 + +The keystore filename extension must be either `.p12` indicating PKCS12 or `.bcfks` indicating BCFKS. + +The `keytool` command can be used to generate an AES-256 Secret Key stored in a PKCS12 file for repository encryption: + + keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12 + +The `keytool` command requires additional arguments specifying the BouncyCastle Security Provider to store +Secret Keys using BCFKS. The arguments must include a reference to the BouncyCastle Security Provider library, which +is available in the `lib/bootstrap` directory under the NiFi installation. + +The following command can be used to generate an AES-256 Secret Key stored using BCFKS: + + keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.bcfks -storetype BCFKS -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath lib/bootstrap/bcprov-jdk15on-*.jar + +Enter a keystore password when prompted. The same value must be used for both the keystore password and key password. +The keystore password will be used in the provider configuration properties. + [[encryption]] == Encryption Configuration -This section provides an overview of the capabilities of NiFi to encrypt and decrypt data. - The EncryptContent processor allows for the encryption and decryption of data, both internal to NiFi and integrated with external systems, such as `openssl` and other data sources and consumers. [[key-derivation-functions]] diff --git a/nifi-docs/src/main/asciidoc/user-guide.adoc b/nifi-docs/src/main/asciidoc/user-guide.adoc index 959522ed8b..f62e86c53a 100644 --- a/nifi-docs/src/main/asciidoc/user-guide.adoc +++ b/nifi-docs/src/main/asciidoc/user-guide.adoc @@ -2970,150 +2970,6 @@ Once these property changes have been made, restart NiFi. **Note:** Detailed descriptions for each of these properties can be found in <>. -== Repository Encryption - -NiFi supports encryption of local repositories using a configurable Key Provider to enable protection of information -on the filesystem. Repository encryption configuration uses a version number to indicate the cipher algorithms, metadata -format, and repository implementation classes. This approach provides a generalized method for configuration without the -need to customize each repository implementation class. - -Repository encryption incurs a performance cost due to the overhead of cipher operations. Filesystem encryption at the -operating system level provides an alternative solution, with different performance characteristics. For deployments -where filesystem encryption is not configured, repository encryption provides an enhanced level of data protection. -Due to increased performance requirements, more computing resources may be necessary to achieve sufficient throughput -when enabling repository encryption. - -The security of repository encryption depends on a combination of the cipher algorithms and the protection of encryption -keys. Key protection and key rotation are important parts of securing an encrypted repository configuration. -Key protection involves limiting access to the Key Provider and key rotation requires manual updates to generate and -specify a new encryption key. - -=== Repository Encryption Protocol Version 1 - -The first version of support for repository encryption includes the following cipher algorithms: - -[options="header"] -|======================= -|Repository Type |Cipher Algorithm -|Content Repository |AES/CTR/NoPadding -|FlowFile Repository |AES/GCM/NoPadding -|FlowFile Swap Manager |AES/GCM/NoPadding -|Provenance Repository |AES/GCM/NoPadding -|======================= - -The following classes provide the direct repository encryption implementation, extending standard classes: - -[options="header"] -|======================= -|Repository Type |Class -|Content Repository |org.apache.nifi.content.EncryptedFileSystemRepository -|FlowFile Repository |org.apache.nifi.wali.EncryptedSequentialAccessWriteAheadLog -|FlowFile Swap Manager |org.apache.nifi.controller.EncryptedFileSystemSwapManager -|Provenance Repository |org.apache.nifi.provenance.EncryptedWriteAheadProvenanceRepository -|======================= - -==== Encryption Metadata Serialization - -Each repository implementation class leverages standard cipher operations to perform encryption and decryption. In order -to support AES, the encryption process writes metadata associated with each encryption operation. Encryption protocol -version 1 uses Java Object serialization to write objects containing the encryption Key Identifier, the cipher -Initialization Vector, and other required properties. Serialized objects include the following required properties: - -[options="header"] -|===================== -|Property Name |Property Type |Description -|keyId |String |Encryption key identifier -|ivBytes |byte[] |Cipher initialization vector -|algorithm |String |Cipher algorithm -|version |String |Encryption protocol version -|cipherByteLength |int |Length of enciphered record -|===================== - -Metadata serialization uses the standard `java.io.ObjectOutputStream.writeObject()` method to write objects to a stream -that can be converted to a byte array. The deserialization process uses a custom extension of the -`java.io.ObjectInputStream` to read objects regardless of the original class name associated with the record. This -approach requires the presence of the standard metadata properties, but provides a compatibility layer that avoids -linking the implementation to a specific Java class. - -The initial implementation of encrypted repositories used different byte array markers when writing metadata. Each -repository implementation uses the following byte array markers before writing a serialized metadata record: - -[options="header"] -|======================= -|Repository Type |Byte Array -|Content Repository |byte[]{0x00, 0x00} -|FlowFile Repository |byte[]{} -|Provenance Repository |byte[]{0x01} -|======================= - -=== Repository Encryption Configuration - -Configuring repository encryption requires specifying the encryption protocol version and the associated Key Provider -properties. Repository encryption can be configured on new or existing installations using standard properties. Records -in existing repositories should be readable using standard capabilities, and the encrypted repository will write new -records using the specified configuration. - -Disabling repository encryption on existing installations requires removing existing repository contents, and -restarting the system after making configuration changes. For this reason, flow administrators should confirm that the -system has processed all available FlowFiles to avoid losing information when disabling repository encryption. - -==== Protocol Version Configuration - -Setting the following protocol version property enables encryption for all repositories: - -.... -nifi.repository.encryption.protocol.version=1 -.... - -==== Key Provider Configuration - -All encrypted repositories require a Key Provider to perform encryption and decryption operations. NiFi supports -configuring the Key Provider implementation as well as the Key Identifier that will be used for new encryption -operations. Key Provider implementations can hold multiple keys to support using a new key while maintaining access to -information encrypted using the previous key. - -Repository encryption supports access to secret keys using standard `java.security.KeyStore` files. -See <> for details on supported KeyStore types, as well as examples of -generating secret keys. - -The following configuration properties provide an example using a PKCS12 KeyStore file named `repository.p12` containing -a secret key labeled with an alias of `primary-key`: - -.... -nifi.repository.encryption.key.id=primary-key -nifi.repository.encryption.key.provider=KEYSTORE -nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 -nifi.repository.encryption.key.provider.keystore.password=2fRKmwDyMYmT7P5L -.... - -[[secret-key-generation-and-storage-using-keytool]] -==== Secret Key Generation and Storage using Keytool - -The `KeyStoreKeyProvider` supports reading from a `java.security.KeyStore` using a configured password to load AES Secret Key entries. -The `KeyStoreKeyProvider` can be configured with any of the encrypted repository implementations. - -The provider supports the following KeyStore Types: - -* BCFKS -* PKCS12 - -The keystore filename extension must be either `.p12` indicating PKCS12 or `.bcfks` indicating BCFKS. - -The `keytool` command can be used to generate an AES-256 Secret Key stored in a PKCS12 file for repository encryption: - - keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12 - -The `keytool` command requires additional arguments specifying the BouncyCastle Security Provider to store -Secret Keys using BCFKS. The arguments must include a reference to the BouncyCastle Security Provider library, which -is available in the `lib/bootstrap` directory under the NiFi installation. - -The following command can be used to generate an AES-256 Secret Key stored using BCFKS: - - keytool -genseckey -alias primary-key -keyalg AES -keysize 256 -keystore repository.bcfks -storetype BCFKS -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath lib/bootstrap/bcprov-jdk15on-*.jar - -Enter a keystore password when prompted. The same value must be used for both the keystore password and key password. -The keystore password will be used in the provider configuration properties. - [[experimental_warning]] == Experimental Warning