Commit Graph

474 Commits

Author SHA1 Message Date
Troy Melhase 76648bdc0b NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.

NIFI-7356 - Additional changes from PR feedback.

NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.

NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code cleanup, need to fix IT tests and docs.

NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node NiFi + Quorum.

NIFI-7356 - Checkstyle fixes.

NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS configuration.

NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi properties.

NIFI-7356 - Updated per review, using NiFiProperties keystore strings, classname for ocnnection factory, adjusted TLS configuration checks in NiFiProperties.

NIFI-7356 - Updated configuration validation logic and added tests.

NIFI-7356 - Codestyle check fixes.

NIFI-7356 - Updated some of the log messages.

NIFI-7356 - Updated as per code review.

NIFI-7356 - Fixed max port number.

NIFI-7356 - Updated admin guide and updated small code issues as per code review.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4753.
2021-01-21 13:47:53 -05:00
Andrew Lim 27f57e6463
NIFI-8150 Change Download flow to Download flow definition for process groups (#4766) 2021-01-20 16:00:41 -05:00
Nissim Shiman 3cc8d767b3 NIFI-7738 Reverse Provenance Query
This closes #4563.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 09:35:05 -05:00
exceptionfactory 7d76bcd520 NIFI-8094 Added support for BCFKS Keystore Type
NIFI-8094 Updated Administration Guide to include BCFKS

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4729.
2021-01-12 10:52:49 -05:00
sjyang18 f330078fff NIFI-7924 Add fallback claims for identifying user to OIDC provider
This closes #4630

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-01-07 14:57:57 -06:00
Mark Payne d84583690f NIFI-8070: Added coalesce function to RecordPath 2020-12-03 17:23:11 -05:00
zhangcheng 857eeca3c7
NIFI-8032: fix record-path-guide.adoc
This closes #4679

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-24 18:01:16 -05:00
Mike Thomsen 932496e106 NIFI-7879 Created record path function for UUID v5
This closes #4570.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-11-20 12:10:01 -06:00
Bryan Bende dcc4fb00a5
NIFI-7888 Added support for authenticating via SAML
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page

- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests

- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm

- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client

- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding

- Made atomic replace methods for storing groups and saml credential in database

- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata

- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias

- Fixed unit test

- Added property to specify an optional identity attribute which would be used instead of NameID

- Cleaned up logging

- Fallback to keystore password when key password is blank

- Make signature and digest default to SHA-256 when no value provided in nifi.properties

This closes #4614
2020-11-18 12:44:02 -05:00
Nathan Gough 479ee6e3db
NIFI-7819 - Added ZooKeeperStateProvider TLS properties.
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.

This closes #4613.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 17:05:05 -05:00
Dustin Rodrigues 0805670263
NIFI-7978 - update Homebrew getting started instructions
NIFI-7978 Removed a few more OS X references.

This closes #4647

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-04 07:20:21 -05:00
Mark Payne 4b9014b959
NIFI-1121: Updated backend to perform appropriate validation. Added tests. Updated documentation writer. Updated dev guide to explain how PropertyDescriptor.Builder#dependsOn affects validation. Updated JavaDocs for PropertyDescriptor.Builder#dependsOn
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:42 -05:00
Andrew Lim 718b77c7c4
NIFI-7941 Add NiFi Registry mode options and examples to Encrypt-Conf… (#4616)
* NIFI-7941 Add NiFi Registry mode options and examples to Encrypt-Config section of Toolkit Guide

* Update toolkit-guide.adoc

Replaced references to "master key" with "root key"

* Update toolkit-guide.adoc

Corrected a typo. Anchor was "sensistive_property_key_migration". Changed to "sensitive_property_key_migration". Confirmed nothing was referencing the original.
2020-10-26 09:11:58 -04:00
Joey Frazee 55cb8d73cb NIFI-7401 Add ZooKeeper client TLS to CuratorLeaderElectionManager
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests

NIFI-7401 Added keystore types and changed properties to match nifi.security.*

NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test

NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties

NIFI-7401 Changed connection socket constants to use .class.getName()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4592.
2020-10-20 11:20:43 -04:00
Andrew Lim dee2fcedfe
NIFI-7871 Correct errors for UUID3, UUID5 and hash functions in EL Guide
Added links to UUID function in docs.

This closes #4583.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-10-07 16:35:21 -07:00
abrown b4faf210d8
NIFI-4897 Add documentation to user guide for terminating a processor
NIFI-4897 fixed typos in new section

Fixed some typoes

This closes #4568

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-05 09:17:27 -04:00
Pierre Villard f32405ed16
NIFI-7844 - substring should return empty instead of throwing IndexOutOfBoundsException
This closes #4553

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-01 19:45:26 -04:00
VKadam 7e0bcb98e1
NIFI-7841: Made corrections in the nifi-walkthroughs docs (#4548)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-23 16:37:26 -07:00
Mohammed Nadeem 4040664886
NIFI-7816: Correct documentation example for urlEncode function in Expression Language Guide (#4536)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-21 12:12:23 -07:00
abrown 953abba6d6
NIFI-5061-NiFi documentation incomplete/wrong for EL hierarchy.
Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4530.
2020-09-20 19:48:14 +02:00
Andy LoPresto 9370571131
NIFI-7804 Split nifi-security-utils into sub-module for nifi-security… (#4533)
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.

* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.

* NIFI-7804 Resolved failing unit test.

* NIFI-7804 Removed legacy dependency.

* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
2020-09-17 12:52:22 -04:00
abrown 624678466b
NIFI-7580-Add documentation around autoloading NARs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4529.
2020-09-17 12:50:13 +02:00
Bence Simon 0dff3bc065
NIFI-7429 Adding status history for system level metrics
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4420.
2020-09-10 16:19:11 +02:00
Andrew Lim 29e23e57b9
NIFI-7743 Document Empty all queues option for Process Groups
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4506
2020-09-02 17:14:17 -04:00
VKadam e884b3cdb2
NIFI-7778: Made corrections in descriptions of padLeft, padRight, plus (#4504)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-01 13:07:47 -07:00
Andrew Lim 861b83f314
NIFI-7309 Update Admin Guide by removing unused properties and updating default values
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4488.
2020-08-28 14:38:21 +02:00
Joe Witt 8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt 303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Jaya Aditya 339e09a6e0
NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented commands (#4450)
* NIFI-7681 - Add update-bucket-policy command, add option to specify
timeout and fix documentation to include previously implemented commands

* Fix return type of UpdateBucketPolicy and add missing registry commands to the documentation

Co-authored-by: Jaya Aditya <jchandra@yahoo-corp.jp>
2020-08-07 09:06:56 -04:00
Ruben Laguna 30b328ceab
NIFI-7694 - Move misplaced text related to oidc
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4445.
2020-07-31 11:00:14 +02:00
Tamás Bunth 455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto 716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto 7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Andy LoPresto eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Bryan Bende e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
Mark Payne 44fc4d9f27
NIFI-7552: When Process Group is configured to transfer data in batch, add an attribute to each outbound FlowFile that indicates how many FlowFiles went to each port. Updated user guide to explain the new attributes.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4345.
2020-07-17 10:33:59 +02:00
Andy LoPresto dbee774c5b
NIFI-7304 Removed default value for nifi.web.max.content.size.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:42:00 -07:00
Andy LoPresto 0fa8776f4d
NIFI-7621 & NIFI-7614 Updated terminology throughout codebase and documentation.
Fixed unit test failures introduced from static imports during refactoring.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:39:28 -07:00
mtien 43fb57e7bb
NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.

NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4344.
2020-07-07 15:54:32 -04:00
VKadam 239a2e884c
NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359) 2020-07-01 10:52:22 -07:00
Andy LoPresto 94c98c019f
NIFI-7558 Fixed CatchAllFilter init logic by calling super.init().
Renamed legacy terms.
Updated documentation.

This closes #4351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-22 12:20:28 -07:00
Tamas Palfy a126d0a6b6
NIFI-7442 Added missing use cases (list users and user groups), made update-access-policy use case more in line with the NiFi side. Added some tests. Additional refactor, documentation revision.
This closes #4329.
2020-06-12 12:43:41 -04:00
Bence Simon d99983bdc7
NIFI-7442 Add CLI commands to the registry in order to support automatic registry setup 2020-06-12 12:43:27 -04:00
Mark Payne 463d72117b
NIFI-7507: Added section to User Guide on configuring a Process Group
NIFI-7507: Fixed Flowfile Expiration header in doc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4318
2020-06-08 14:03:00 -04:00
Alessandro D'Armiento 788f8b0389
NIFI-6674 MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

fixed mispositioned # in doc

This closes #3740

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:06:22 -04:00
Alessandro D'Armiento 1ba8f76a44
NIFI-6673 MultiplyEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

This closes #3739

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:01:02 -04:00
Alessandro D'Armiento b025117824
NIFI-6672 PlusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
NIFI-6672 Removed test code.

This closes #3738

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:58:15 -04:00
Phillip Grenier 0f4b79b55e
NIFI-6255 NIFI-6287: Hash function for expression language and record path.
NIFI-6255 NIFI-6287: Rebased to match the new expression language interface
NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports
NIFI-6255 NIFI-6287: Move to the common codec DigetUtils
Update commons-codec

This closes #3624

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:23:00 -04:00