532 Commits

Author SHA1 Message Date
Joe Witt
88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt
4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt
487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Nathan Gough
8057f8f6c5
NIFI-8221 - Set the default HTTP listening interface to 127.0.0.1.
This closes #4817

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-10 14:46:37 -06:00
Mark Payne
03fd59eb2f
NIFI-8196: When node is reconnected to cluster, ensure that it re-registers for election of cluster coordinator / primary node. On startup, if cluster coordinator is already registered and is 'this node' then register silently as coordinator and do not join the cluster until there is no Cluster Coordinator or another node is elected. This allows the zookeeper session timeout to elapse.
Signed-off-by: Bryan Bende <bbende@apache.org>
2021-02-05 15:18:07 -05:00
Peter Gyori
5a8b18b12d NIFI-8123: Added support for --wait-for-init when NiFi started with the 'start' command
This closes #4748.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-03 09:40:09 -05:00
Joe Witt
d826416217
NIFI-8192 updating Copyright years for nifi things to include 2021
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4802.
2021-02-03 12:56:28 +04:00
Mark Payne
4ff9cddf15
NIFI-8173, NIFI-8174: This closes #4784. Updated Parameter Contexts to allow for unsetting a parameter value / setting it to null. Allow Parameters to make use of Expression Language. Updated docs to illustrated how EL is evaluated
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 21:55:18 -07:00
Troy Melhase
76648bdc0b NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.

NIFI-7356 - Additional changes from PR feedback.

NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.

NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code cleanup, need to fix IT tests and docs.

NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node NiFi + Quorum.

NIFI-7356 - Checkstyle fixes.

NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS configuration.

NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi properties.

NIFI-7356 - Updated per review, using NiFiProperties keystore strings, classname for ocnnection factory, adjusted TLS configuration checks in NiFiProperties.

NIFI-7356 - Updated configuration validation logic and added tests.

NIFI-7356 - Codestyle check fixes.

NIFI-7356 - Updated some of the log messages.

NIFI-7356 - Updated as per code review.

NIFI-7356 - Fixed max port number.

NIFI-7356 - Updated admin guide and updated small code issues as per code review.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4753.
2021-01-21 13:47:53 -05:00
Andrew Lim
27f57e6463
NIFI-8150 Change Download flow to Download flow definition for process groups (#4766) 2021-01-20 16:00:41 -05:00
Nissim Shiman
3cc8d767b3 NIFI-7738 Reverse Provenance Query
This closes #4563.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 09:35:05 -05:00
exceptionfactory
7d76bcd520 NIFI-8094 Added support for BCFKS Keystore Type
NIFI-8094 Updated Administration Guide to include BCFKS

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4729.
2021-01-12 10:52:49 -05:00
sjyang18
f330078fff NIFI-7924 Add fallback claims for identifying user to OIDC provider
This closes #4630

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-01-07 14:57:57 -06:00
Mark Payne
d84583690f NIFI-8070: Added coalesce function to RecordPath 2020-12-03 17:23:11 -05:00
zhangcheng
857eeca3c7
NIFI-8032: fix record-path-guide.adoc
This closes #4679

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-24 18:01:16 -05:00
Mike Thomsen
932496e106 NIFI-7879 Created record path function for UUID v5
This closes #4570.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-11-20 12:10:01 -06:00
Bryan Bende
dcc4fb00a5
NIFI-7888 Added support for authenticating via SAML
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page

- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests

- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm

- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client

- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding

- Made atomic replace methods for storing groups and saml credential in database

- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata

- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias

- Fixed unit test

- Added property to specify an optional identity attribute which would be used instead of NameID

- Cleaned up logging

- Fallback to keystore password when key password is blank

- Make signature and digest default to SHA-256 when no value provided in nifi.properties

This closes #4614
2020-11-18 12:44:02 -05:00
Nathan Gough
479ee6e3db
NIFI-7819 - Added ZooKeeperStateProvider TLS properties.
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.

This closes #4613.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 17:05:05 -05:00
Dustin Rodrigues
0805670263
NIFI-7978 - update Homebrew getting started instructions
NIFI-7978 Removed a few more OS X references.

This closes #4647

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-04 07:20:21 -05:00
Mark Payne
4b9014b959
NIFI-1121: Updated backend to perform appropriate validation. Added tests. Updated documentation writer. Updated dev guide to explain how PropertyDescriptor.Builder#dependsOn affects validation. Updated JavaDocs for PropertyDescriptor.Builder#dependsOn
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:42 -05:00
Andrew Lim
718b77c7c4
NIFI-7941 Add NiFi Registry mode options and examples to Encrypt-Conf… (#4616)
* NIFI-7941 Add NiFi Registry mode options and examples to Encrypt-Config section of Toolkit Guide

* Update toolkit-guide.adoc

Replaced references to "master key" with "root key"

* Update toolkit-guide.adoc

Corrected a typo. Anchor was "sensistive_property_key_migration". Changed to "sensitive_property_key_migration". Confirmed nothing was referencing the original.
2020-10-26 09:11:58 -04:00
Joey Frazee
55cb8d73cb NIFI-7401 Add ZooKeeper client TLS to CuratorLeaderElectionManager
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests

NIFI-7401 Added keystore types and changed properties to match nifi.security.*

NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test

NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties

NIFI-7401 Changed connection socket constants to use .class.getName()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4592.
2020-10-20 11:20:43 -04:00
Andrew Lim
dee2fcedfe
NIFI-7871 Correct errors for UUID3, UUID5 and hash functions in EL Guide
Added links to UUID function in docs.

This closes #4583.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-10-07 16:35:21 -07:00
abrown
b4faf210d8
NIFI-4897 Add documentation to user guide for terminating a processor
NIFI-4897 fixed typos in new section

Fixed some typoes

This closes #4568

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-05 09:17:27 -04:00
Pierre Villard
f32405ed16
NIFI-7844 - substring should return empty instead of throwing IndexOutOfBoundsException
This closes #4553

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-01 19:45:26 -04:00
VKadam
7e0bcb98e1
NIFI-7841: Made corrections in the nifi-walkthroughs docs (#4548)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-23 16:37:26 -07:00
Mohammed Nadeem
4040664886
NIFI-7816: Correct documentation example for urlEncode function in Expression Language Guide (#4536)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-21 12:12:23 -07:00
abrown
953abba6d6
NIFI-5061-NiFi documentation incomplete/wrong for EL hierarchy.
Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4530.
2020-09-20 19:48:14 +02:00
Andy LoPresto
9370571131
NIFI-7804 Split nifi-security-utils into sub-module for nifi-security… (#4533)
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.

* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.

* NIFI-7804 Resolved failing unit test.

* NIFI-7804 Removed legacy dependency.

* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
2020-09-17 12:52:22 -04:00
abrown
624678466b
NIFI-7580-Add documentation around autoloading NARs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4529.
2020-09-17 12:50:13 +02:00
Bence Simon
0dff3bc065
NIFI-7429 Adding status history for system level metrics
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4420.
2020-09-10 16:19:11 +02:00
Andrew Lim
29e23e57b9
NIFI-7743 Document Empty all queues option for Process Groups
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4506
2020-09-02 17:14:17 -04:00
VKadam
e884b3cdb2
NIFI-7778: Made corrections in descriptions of padLeft, padRight, plus (#4504)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-09-01 13:07:47 -07:00
Andrew Lim
861b83f314
NIFI-7309 Update Admin Guide by removing unused properties and updating default values
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4488.
2020-08-28 14:38:21 +02:00
Joe Witt
8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt
fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt
303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Jaya Aditya
339e09a6e0
NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented commands (#4450)
* NIFI-7681 - Add update-bucket-policy command, add option to specify
timeout and fix documentation to include previously implemented commands

* Fix return type of UpdateBucketPolicy and add missing registry commands to the documentation

Co-authored-by: Jaya Aditya <jchandra@yahoo-corp.jp>
2020-08-07 09:06:56 -04:00
Ruben Laguna
30b328ceab
NIFI-7694 - Move misplaced text related to oidc
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4445.
2020-07-31 11:00:14 +02:00
Tamás Bunth
455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto
716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto
7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Andy LoPresto
eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Bryan Bende
e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne
a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
Mark Payne
44fc4d9f27
NIFI-7552: When Process Group is configured to transfer data in batch, add an attribute to each outbound FlowFile that indicates how many FlowFiles went to each port. Updated user guide to explain the new attributes.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4345.
2020-07-17 10:33:59 +02:00
Andy LoPresto
dbee774c5b
NIFI-7304 Removed default value for nifi.web.max.content.size.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:42:00 -07:00
Andy LoPresto
0fa8776f4d
NIFI-7621 & NIFI-7614 Updated terminology throughout codebase and documentation.
Fixed unit test failures introduced from static imports during refactoring.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:39:28 -07:00
mtien
43fb57e7bb
NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.

NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4344.
2020-07-07 15:54:32 -04:00
VKadam
239a2e884c
NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359) 2020-07-01 10:52:22 -07:00