6396 Commits

Author SHA1 Message Date
Mark Payne
7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
exceptionfactory
052c60dc53
NIFI-8363 Added Single User Login Identity Provider and Authorizer (#4968)
* NIFI-8363 Added Single User Login Identity Provider and Authorizer

- Reads and writes username and hashed password in login-identity-providers.xml
- Generates random username using java.util.UUID.randomUUID()
- Generates random password using java.security.SecureRandom with Base64 encoding
- Writes generated password hash using bcrypt

* NIFI-8363 Updated SingleUserAuthorizer to require SingleUserLoginIdentityProvider

* NIFI-8363 Added handling of null login identity provider property
2021-04-12 08:48:20 -04:00
Mark Payne
74531fa1ca NIFI-8348: Updated system tests to reference the version of jersey client used elsewhere, as it was missed in the previous commit for NIFI-8348 and system tests are failing 2021-04-10 10:33:16 -04:00
sjyang18
365dde2d11 NIFI-7573 Added Azure Active Directory User Group Provider
This closes #4367

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-09 15:06:14 -07:00
nabmoh123
d0045c13b1
NIFI-8341 Support Multi Hosts in AMQP Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4939.
2021-04-09 17:42:24 +02:00
Joe Gresock
76f33e42c6
NIFI-7198: Adding optional SSL Context Service to RedisConnectionPoolService
This closes #4981

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-09 10:36:31 -05:00
Timea Barna
bfd964b9c7 NIFI-8176 - Move the timeout check after we process the response from Splunk to make sure we poll for acknowledgement at least once. (No need for flag.)
This closes #4824.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-09 17:22:56 +02:00
Mark Payne
14e6dc3dc6
NIFI-8405: Added debug logging around how long it takes to establish connections/query dns/read and write headers and body when replication requests; added additional timing around Ranger audits and authorizations and monitoring of long-running tasks because those run often and frequently show up in the logs at the same time as the long requests
This closes #4983

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-08 15:41:44 -05:00
Mark Payne
78b69f10dc NIFI-8404: Added capability to rollback on failure for PublishKafka(Record)_2_0 and _2_6
NIFI-8404: Added unit tests for new Failure Strategy property

This closes #4982.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 17:47:13 +02:00
Matthew Burgess
33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00
Mohammed Nadeem
77a73a603b NIFI-8394 Fix NPE error when NiFi connects to external zookeeper insecurely
This closes #4974

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-07 11:23:25 -07:00
Nathan Gough
9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne
b79987918a
NIFI-8380: Allow for an extensions.directory property to specify where to place downloaded files. Also fixed an issue that was encountered, when a Source Processor is scheduled for Primary Node Only but more than 1 task is set. In that case, even though only a single task will should be scheduled, an Exception was getting thrown because @OnScheduled methods of Processors were still called. To avoid this, moved the initialization of the dataflow outside of the creation of the dataflow so that initialization can be triggered only when appropriate.
NIFI-8380: Removed requirement in validation for working directory and extensions directory to exist; removed auto-creation of directories in validation

NIFI-8380: Fixed a few thrading bugs, so that if we have multiple threads trying to download/unpack extensions, we properly synchronize the unpacking and unpack into the correct sub-directory under the working directory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4950.
2021-04-07 13:28:12 +02:00
Pierre Villard
f4ad658fae NIFI-8389 - Grok Reader - raw data strategy when no match
added allowable value in property
added docs in additionalDetails

This closes #4966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-07 09:11:44 +02:00
Peter Turcsanyi
4b852ba7c8
NIFI-8390: Handle HBase namespaces in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4977.
2021-04-07 08:57:29 +02:00
exceptionfactory
ef60e7e404 NIFI-6714 Corrected OSUtils to use Process.pid interface method
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4970.
2021-04-06 19:48:05 -04:00
Jose Luis Pedrosa
fa3da2aa1c
There was some minor changes required to stop using deprecated APIs byu accumulo, inspecting the deprecated makes the fixes really simple
* writerFactory.createWriter receives an extra parameteres with a map, in the implementation it sends an empty one if not passed.
* tests were relying on  getConnector, that is replaced by createAccumuloClient

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4942.
2021-04-06 11:10:01 +02:00
exceptionfactory
64b12176b2
NIFI-8388 Upgrade Hazelcast from 4.0.1 to 4.2
- Removes dependency on Guava using ByteBuffer methods for serializing revision

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4967.
2021-04-05 11:46:32 +02:00
Jose Luis Pedrosa
f21c1be60f
NIFI-8374 Refactor readOnly to accessMode in ApiModelProperty usages
This closes #4945

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-02 14:22:07 -04:00
Rob Fellows
87b561f026
[NIFI-8387] - Use the bulletins from the referencing components rathe… (#4969)
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.

* Moving the spinner next to the Parameter/Variable lables

This closes #4969
2021-04-02 09:59:34 -04:00
Mark Payne
7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
Mark Payne
2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory
89dda0b6a4
NIFI-8364 This closes #4953. Set static date for TestQuerySolr
- Static date for TestQuerySolr avoids the potential for missing values in facet range queries

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 16:05:04 -07:00
noblenumbat360
ee1386db02
NIFI-8317 heap and non_heap usage Prometheus metrics now return -1.0 when undefined.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4920
2021-03-30 11:06:52 -04:00
exceptionfactory
02ea103c6b
NIFI-8379 This closes #4949. Improved reliability and performance of DBCPServiceTest
- Removed unreliable H2 network connection test methods
- Removed duplicate and ignored test methods
- Reduced duplication of setting common service properties

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 07:24:35 -07:00
Matthew Burgess
a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
Eric Olson
bff3e94c01
NIFI-7172 Trim trailing whitespace from NiFi properties
This closes #4854

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-25 15:40:35 -05:00
exceptionfactory
1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Florimond Manca
66e2741871
NIFI-5535 NIFI-4713 This closes #4858. Change metric tagging in DataDogReportingTask
Remove low-value default tags

Send processor name as tag instead of as part of metric name

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:59:59 -07:00
exceptionfactory
54d2222fb0
NIFI-8364 This closes #4935. Refactored TestQuerySolr and TestGetSolr to reuse embedded SolrClient
- Replaced Gson with Jackson for parsing JSON
- Corrected assertion argument ordering
- Simplified relative path determination for EmbeddedSolrServerFactory
- Replaced SimpleDateFormat with java.time.Instant parsing and formatting

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:33:19 -07:00
exceptionfactory
36e0187648 NIFI-8347 Set Thread Context ClassLoader for provided jetty-server classes
- Updated unit test with WebSocket connect method

NIFI-8347 Replaced init() method with ServletContextHandler.setClassLoader()

This closes #4918.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-25 15:51:22 +01:00
Timea Barna
9df717f69b
NIFI-8283 Value handling in ScanAccumulo processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4931.
2021-03-25 13:00:04 +01:00
Hsin-Ying Lee
0f28702b47
NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file… (#4923)
* NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
* NIFI-8354 Record the log when delete file failed


This closes #4923

Signed-off-by: Otto Fowler <otto@apache.org>
2021-03-24 12:26:56 -04:00
Peter Turcsanyi
4473d23ccd
NIFI-8356: Add unit test for LongRunningTaskMonitor.
This closes #4925

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-24 10:29:59 -05:00
Mark Payne
91313a2e75
NIFI-8360: Fixed an overflow issue where we used an integer to store the number of bytes encountered when reading data and searching for a given pattern
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4929.
2021-03-24 15:30:22 +01:00
Mark Payne
74ea3840ac NIFI-8357: Updated Kafka 2.6 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:59:13 +01:00
Mark Payne
2f08d1f466 NIFI-8357: Updated Kafka 2.0 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:58:51 +01:00
Dmitry Ibragimov
057b4af482
NIFI-8296 - Use API to retrieve all subjects associated with a schema id for Confluent Schema Registry v5.4.0+
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4872.
2021-03-24 14:23:41 +01:00
Mark Bean
1719e36165 NIFI-8348: upgrade jersey version to one fully compatible with Java 11 2021-03-24 09:18:31 -04:00
Joe Witt
7fe8858d22
NIFI-8359 resolving unused import 2021-03-23 14:30:28 -07:00
Joe Witt
788abe9eb3 NIFI-8359 removing test that cannot be reliable. Random data doesn't guarantee it will be detected as octet-stream and there is no obvious 'will never match' set of bytes so this test is not reliable or valuable 2021-03-23 17:18:54 -04:00
Denes Arvay
22fcb594d9 NIFI-8346 Route to failure in PutAzureBlobStorage for auth, IO exceptions
This closes #4917

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-23 11:28:04 -07:00
markap14
321e979673
NIFI-8353: When receiving data via load-balanced connection, throw an Exception (resulting in an ABORT_TRANSACTION status code) when attempting to add received FlowFiles to the FlowFile queue, if the node is not currently connected to cluster. In this case, ensure that we remove the received FlowFiles from the FlowFile Repository and emit a DROP event to the repository (with appropriate details) to coincide with the already-committed Provenance events. Also ensure that when a node is disconnected that we don't keep sending to that node until it reconnects. While testing the fixes via LoadBalancedQueueIT, also noticed that unit test logging was not working properly in nifi-framework-core because of the (erroneous) dependency on logback-class; removed this dependency and updated unit tests that were created that depended on its existence. (#4924) 2021-03-23 14:17:54 -04:00
exceptionfactory
e505e8b42d
NIFI-8352 This closes #4922. Changed assertServerConnections to loop and sleep while evaluating
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:15:48 -07:00
Joe Witt
4189a1c1eb
NIFI-8350 This closes 4921. updated worfklow
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:09:59 -07:00
Mark Payne
e1b9548ab6 NIFI-8344: Introduced new Rollover Tail Period property
This closes #4916.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-22 18:22:00 +01:00
Karthick Narendran
3508b5b12f NIFI-7850 Adds nifi.cluster.protocol.is.secure to Docker secure.sh
This closes #4896

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-19 15:30:23 -07:00
Nathan Gough
07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
exceptionfactory
0e659981d4 NIFI-8298 Refactored Kerberos and Socket classes from security-utils to new modules
- Created nifi-security-socket-ssl
- Created nifi-security-kerberos
- Removed nifi-security-utils dependency from nifi-processor-utils
- Updated modules to reference new dependencies
- Eliminated unnecessary transitive dependencies on bcprov-jdk15on from over 30 modules

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4881.
2021-03-19 13:12:19 -04:00
Bence Simon
4d68509212 NIFI-8308 Remove unnecessary field adjustments from QuestDB row creation
This closes #4888.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-19 17:55:11 +01:00