Commit Graph

6645 Commits

Author SHA1 Message Date
exceptionfactory 052c60dc53
NIFI-8363 Added Single User Login Identity Provider and Authorizer (#4968)
* NIFI-8363 Added Single User Login Identity Provider and Authorizer

- Reads and writes username and hashed password in login-identity-providers.xml
- Generates random username using java.util.UUID.randomUUID()
- Generates random password using java.security.SecureRandom with Base64 encoding
- Writes generated password hash using bcrypt

* NIFI-8363 Updated SingleUserAuthorizer to require SingleUserLoginIdentityProvider

* NIFI-8363 Added handling of null login identity provider property
2021-04-12 08:48:20 -04:00
Mark Payne 74531fa1ca NIFI-8348: Updated system tests to reference the version of jersey client used elsewhere, as it was missed in the previous commit for NIFI-8348 and system tests are failing 2021-04-10 10:33:16 -04:00
sjyang18 365dde2d11 NIFI-7573 Added Azure Active Directory User Group Provider
This closes #4367

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-09 15:06:14 -07:00
nabmoh123 d0045c13b1
NIFI-8341 Support Multi Hosts in AMQP Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4939.
2021-04-09 17:42:24 +02:00
Joe Gresock 76f33e42c6
NIFI-7198: Adding optional SSL Context Service to RedisConnectionPoolService
This closes #4981

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-09 10:36:31 -05:00
Timea Barna bfd964b9c7 NIFI-8176 - Move the timeout check after we process the response from Splunk to make sure we poll for acknowledgement at least once. (No need for flag.)
This closes #4824.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-09 17:22:56 +02:00
Mark Payne 14e6dc3dc6
NIFI-8405: Added debug logging around how long it takes to establish connections/query dns/read and write headers and body when replication requests; added additional timing around Ranger audits and authorizations and monitoring of long-running tasks because those run often and frequently show up in the logs at the same time as the long requests
This closes #4983

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-08 15:41:44 -05:00
Mark Payne 78b69f10dc NIFI-8404: Added capability to rollback on failure for PublishKafka(Record)_2_0 and _2_6
NIFI-8404: Added unit tests for new Failure Strategy property

This closes #4982.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 17:47:13 +02:00
Matthew Burgess 33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00
Mohammed Nadeem 77a73a603b NIFI-8394 Fix NPE error when NiFi connects to external zookeeper insecurely
This closes #4974

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-07 11:23:25 -07:00
Nathan Gough 9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne b79987918a
NIFI-8380: Allow for an extensions.directory property to specify where to place downloaded files. Also fixed an issue that was encountered, when a Source Processor is scheduled for Primary Node Only but more than 1 task is set. In that case, even though only a single task will should be scheduled, an Exception was getting thrown because @OnScheduled methods of Processors were still called. To avoid this, moved the initialization of the dataflow outside of the creation of the dataflow so that initialization can be triggered only when appropriate.
NIFI-8380: Removed requirement in validation for working directory and extensions directory to exist; removed auto-creation of directories in validation

NIFI-8380: Fixed a few thrading bugs, so that if we have multiple threads trying to download/unpack extensions, we properly synchronize the unpacking and unpack into the correct sub-directory under the working directory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4950.
2021-04-07 13:28:12 +02:00
Pierre Villard f4ad658fae NIFI-8389 - Grok Reader - raw data strategy when no match
added allowable value in property
added docs in additionalDetails

This closes #4966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-07 09:11:44 +02:00
Peter Turcsanyi 4b852ba7c8
NIFI-8390: Handle HBase namespaces in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4977.
2021-04-07 08:57:29 +02:00
exceptionfactory ef60e7e404 NIFI-6714 Corrected OSUtils to use Process.pid interface method
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4970.
2021-04-06 19:48:05 -04:00
Jose Luis Pedrosa fa3da2aa1c
There was some minor changes required to stop using deprecated APIs byu accumulo, inspecting the deprecated makes the fixes really simple
* writerFactory.createWriter receives an extra parameteres with a map, in the implementation it sends an empty one if not passed.
* tests were relying on  getConnector, that is replaced by createAccumuloClient

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4942.
2021-04-06 11:10:01 +02:00
exceptionfactory 64b12176b2
NIFI-8388 Upgrade Hazelcast from 4.0.1 to 4.2
- Removes dependency on Guava using ByteBuffer methods for serializing revision

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4967.
2021-04-05 11:46:32 +02:00
Jose Luis Pedrosa f21c1be60f
NIFI-8374 Refactor readOnly to accessMode in ApiModelProperty usages
This closes #4945

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-02 14:22:07 -04:00
Rob Fellows 87b561f026
[NIFI-8387] - Use the bulletins from the referencing components rathe… (#4969)
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.

* Moving the spinner next to the Parameter/Variable lables

This closes #4969
2021-04-02 09:59:34 -04:00
Mark Payne 7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
Mark Payne 2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory 89dda0b6a4
NIFI-8364 This closes #4953. Set static date for TestQuerySolr
- Static date for TestQuerySolr avoids the potential for missing values in facet range queries

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 16:05:04 -07:00
noblenumbat360 ee1386db02
NIFI-8317 heap and non_heap usage Prometheus metrics now return -1.0 when undefined.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4920
2021-03-30 11:06:52 -04:00
exceptionfactory 02ea103c6b
NIFI-8379 This closes #4949. Improved reliability and performance of DBCPServiceTest
- Removed unreliable H2 network connection test methods
- Removed duplicate and ignored test methods
- Reduced duplication of setting common service properties

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 07:24:35 -07:00
Matthew Burgess a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
Eric Olson bff3e94c01
NIFI-7172 Trim trailing whitespace from NiFi properties
This closes #4854

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-25 15:40:35 -05:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Florimond Manca 66e2741871
NIFI-5535 NIFI-4713 This closes #4858. Change metric tagging in DataDogReportingTask
Remove low-value default tags

Send processor name as tag instead of as part of metric name

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:59:59 -07:00
exceptionfactory 54d2222fb0
NIFI-8364 This closes #4935. Refactored TestQuerySolr and TestGetSolr to reuse embedded SolrClient
- Replaced Gson with Jackson for parsing JSON
- Corrected assertion argument ordering
- Simplified relative path determination for EmbeddedSolrServerFactory
- Replaced SimpleDateFormat with java.time.Instant parsing and formatting

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:33:19 -07:00
exceptionfactory 36e0187648 NIFI-8347 Set Thread Context ClassLoader for provided jetty-server classes
- Updated unit test with WebSocket connect method

NIFI-8347 Replaced init() method with ServletContextHandler.setClassLoader()

This closes #4918.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-25 15:51:22 +01:00
Timea Barna 9df717f69b
NIFI-8283 Value handling in ScanAccumulo processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4931.
2021-03-25 13:00:04 +01:00
Hsin-Ying Lee 0f28702b47
NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file… (#4923)
* NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
* NIFI-8354 Record the log when delete file failed


This closes #4923

Signed-off-by: Otto Fowler <otto@apache.org>
2021-03-24 12:26:56 -04:00
Peter Turcsanyi 4473d23ccd
NIFI-8356: Add unit test for LongRunningTaskMonitor.
This closes #4925

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-24 10:29:59 -05:00
Mark Payne 91313a2e75
NIFI-8360: Fixed an overflow issue where we used an integer to store the number of bytes encountered when reading data and searching for a given pattern
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4929.
2021-03-24 15:30:22 +01:00
Mark Payne 74ea3840ac NIFI-8357: Updated Kafka 2.6 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:59:13 +01:00
Mark Payne 2f08d1f466 NIFI-8357: Updated Kafka 2.0 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:58:51 +01:00
Dmitry Ibragimov 057b4af482
NIFI-8296 - Use API to retrieve all subjects associated with a schema id for Confluent Schema Registry v5.4.0+
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4872.
2021-03-24 14:23:41 +01:00
Mark Bean 1719e36165 NIFI-8348: upgrade jersey version to one fully compatible with Java 11 2021-03-24 09:18:31 -04:00
Joe Witt 7fe8858d22
NIFI-8359 resolving unused import 2021-03-23 14:30:28 -07:00
Joe Witt 788abe9eb3 NIFI-8359 removing test that cannot be reliable. Random data doesn't guarantee it will be detected as octet-stream and there is no obvious 'will never match' set of bytes so this test is not reliable or valuable 2021-03-23 17:18:54 -04:00
Denes Arvay 22fcb594d9 NIFI-8346 Route to failure in PutAzureBlobStorage for auth, IO exceptions
This closes #4917

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-23 11:28:04 -07:00
markap14 321e979673
NIFI-8353: When receiving data via load-balanced connection, throw an Exception (resulting in an ABORT_TRANSACTION status code) when attempting to add received FlowFiles to the FlowFile queue, if the node is not currently connected to cluster. In this case, ensure that we remove the received FlowFiles from the FlowFile Repository and emit a DROP event to the repository (with appropriate details) to coincide with the already-committed Provenance events. Also ensure that when a node is disconnected that we don't keep sending to that node until it reconnects. While testing the fixes via LoadBalancedQueueIT, also noticed that unit test logging was not working properly in nifi-framework-core because of the (erroneous) dependency on logback-class; removed this dependency and updated unit tests that were created that depended on its existence. (#4924) 2021-03-23 14:17:54 -04:00
exceptionfactory e505e8b42d
NIFI-8352 This closes #4922. Changed assertServerConnections to loop and sleep while evaluating
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:15:48 -07:00
Joe Witt 4189a1c1eb
NIFI-8350 This closes 4921. updated worfklow
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:09:59 -07:00
Mark Payne e1b9548ab6 NIFI-8344: Introduced new Rollover Tail Period property
This closes #4916.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-22 18:22:00 +01:00
Karthick Narendran 3508b5b12f NIFI-7850 Adds nifi.cluster.protocol.is.secure to Docker secure.sh
This closes #4896

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-19 15:30:23 -07:00
Nathan Gough 07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
exceptionfactory 0e659981d4 NIFI-8298 Refactored Kerberos and Socket classes from security-utils to new modules
- Created nifi-security-socket-ssl
- Created nifi-security-kerberos
- Removed nifi-security-utils dependency from nifi-processor-utils
- Updated modules to reference new dependencies
- Eliminated unnecessary transitive dependencies on bcprov-jdk15on from over 30 modules

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4881.
2021-03-19 13:12:19 -04:00
Bence Simon 4d68509212 NIFI-8308 Remove unnecessary field adjustments from QuestDB row creation
This closes #4888.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-19 17:55:11 +01:00
s9514171 fedeafb43f NIFI-8300 Make non-source processors invalid if scheduled for Primary Node only
This closes #4913.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-19 10:48:29 -04:00