Commit Graph

3683 Commits

Author SHA1 Message Date
Joe Ferner 62606ff89a NIFI-6873: Added support for replacing a process group
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
 - added replace process group functionality in ProcessGroupResource
 - parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
 - refactored flow update methods to make use of commonality across different update types whenever possible
 - fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
 - improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import

This closes #4023.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-02-11 14:46:40 -05:00
Wouter de Vries a80b2475f7
NIFI-7132: fix handling of UUIDs in PutCassandraQL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4048.
2020-02-11 11:21:27 -08:00
Mark Payne 65b2a9bc2c
NIFI-7117: When SocketLoadBalancedFlowFileQueue creates its array of Queue Partitions in the constructor, it added the local partition as the first element in that list. This list should be ordered the same across all nodes in the cluster. By making the local partition the first in the array, each node had a different ordering of these partitions. As a result, Partition by Attribute strategy would constantly rebalance flowfiles that it received to other node, and Single Node always transferred data to the first partition, which was the local node, instead of whichever node should have been the first in the list. This commit addresses this issue by instead inserting the local partition intot he 'queuePartitions' array based on the local node identifier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4045.
2020-02-11 10:59:19 -08:00
David Savage 5e964fbc47
NIFI-4792: Add support for querying array fields in QueryRecord
Work in progress adding support for array based queries
updated calcite dependency

tidy up unused imports highlighted by checkstyle in travis build

tidy up }s highlighted by checkstyle in travis build

Add test for use case referenced in NIFI-4792

Bumped Calcite version to 1.21.0

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4015
2020-02-11 10:59:20 -05:00
Pierre Villard 58bcd6c5dd
NIFI-7106 - Add parent name and parent path in SiteToSiteStatusReportingTask
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4039
2020-02-10 12:07:21 -05:00
John Highcock f738e19a75
NIFI-2537: Add custom MIME type configuration support to IdentifyMimeType
Add two new properties to IdentifyMimeType (Config File and Config Body).
Specifying one of these properties will override the default NiFi MIME
type configuration and use the configured property's MIME config instead.
Add additional runtime documentation for IdentifyMimeType's usage. The
default behavior of IdentifyMimeType is unchanged.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4016.
2020-02-06 19:34:18 -05:00
Pierre Villard 089013aa59
NIFI-7105 - fix NPE for counters in SiteToSiteStatusReportingTask (#4038)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-06 14:10:48 -05:00
nagasivanath 425ba63e66
NIFI-6968 Create Connection Model Allows Multiple Adds
This closes #4013.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-06 10:32:59 -05:00
Matt Gilman b1b06bf5e1 NIFI-6958:
- Addressing issue causing errors preventing version changes when there is a processor to enable or disable in a sub process group.
2020-02-05 13:49:06 -05:00
Bryan Bende 5d851e6a13
NIFI-7067 Allow a user and group with the same name/identity to exist
This closes #4019
2020-02-05 09:19:18 -05:00
Matthew Burgess dd6bcc7485
NIFI-7054: Add RecordSinkServiceLookup for dynamic sink routing (#4020)
* NIFI-7054: Add RecordSinkServiceLookup for dynamic sink routing

* NIFI-7054: Incorporated review comments
2020-02-03 09:45:08 -05:00
Andy LoPresto d7c29f4637 NIFI-7079 Cleaned up formatting and comments. 2020-01-31 08:58:51 -05:00
Andy LoPresto 95746d346c NIFI-7079 Added SecureHasher interface and Argon2SecureHasher implementation.
Added unit tests.
Fixed JwtService charset references.
Changed FingerprintFactory to mask sensitive values via Argon2 hashing.
Resolved RAT checks.
2020-01-31 08:58:51 -05:00
Matthew Burgess 76e8c51e11
NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file system close
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-30 13:06:46 -05:00
mdayakar 850869c6d2
NIFI-7049 : SFTP processors shouldn't silently try to access known hosts file of the user
Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #4014
2020-01-30 15:15:43 +01:00
Matthew Burgess 04fae9cb5f
NIFI-7069: Add groovy-dateutil module to scripting bundles
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4021.
2020-01-30 14:49:41 +07:00
Mark Payne 37ef10e6fb
NIFI-7076: This closes #4024. Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."
This reverts commit 7731609582.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-29 15:50:55 -05:00
Mark Payne be34767c8a
NIFI-7059: This closes #4007. Fixed bug that results in priorities not properly being set in the SocketLoadBalancedFlowFileQueue. Even though the queue's setPriorities method was called, the underlying may not have contained the localPartition. As a result, when setPriorities() was called, it did not properly delegate that call to . As a result, the queue knew that the Prioritizers were set but the local queue did not apply them. This happened due to a race condition between queue creation and NodeClusterCoordinator learning the Local Node Identifier. Additionally, updated NodeClusterCoordinator so that it will persist its state when it does learn the Local Node Identifier, as that was not being persisted. In testing this, also encounterd a deadlock in a particular Integration Test that was run, around AbstractFlowFileQueue and its synchronization. Because the 'synchronized' keyword synchronizes on 'this' and the concrete implementation also uses 'synchronized' and the result is the same 'this', a deadlock can occur that is difficult to understand. As a result, refactored AbstractFlowFileQueue to instead use read/write locks.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-22 23:30:41 -05:00
Joe Witt 535cafdb83
Merge branch 'NIFI-7021-RC3' bringing the Apache NiFi 1.11.0 to master 2020-01-22 21:15:12 -05:00
Bryan Bende 24b846d08a
NIFI-7051 Protect against empty group membership in ShellUserGroupProvider, and add differentiator to id seeding
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info

NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance

This closes #4003.
2020-01-22 15:51:42 -05:00
nagasivanath 24ef8ba4cb
Update nf-processor-details.js
NIFI-7032:
- Processor Details no longer appears when clicking 'View Processor Details'
- handling the review comments

This closes #3990
2020-01-21 13:54:12 -05:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt 67c36f3c3e
NIFI-7021 apache parent pom and common dep maintenance for build 2020-01-19 13:32:37 -05:00
Gardella Juan Pablo cddaac591b
NIFI-7034 Thise closes #4002. Connection leak with JMSConsumer and JMSPublisher
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 12:08:22 -05:00
mans2singh ddaaded08f
NIFI-6861 - This closes #3882. Added missing display name property
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 12:02:03 -05:00
Marc Parisi de2a286a7a
NIFI-818: This closes #3926. Initial implementation of NiFi-Accumulo ( https://github.com/phrocker/nifi-accumulo ) with connectors to Apache Accumulo 2.x
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:43:19 -05:00
Dirk Ernst van den Biggelaar ab0113664d
NIFI-6997 This closes #3971. connection closing fixed
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:09:52 -05:00
Gal Shinder 4fec54debf
NIFI-6996 This closes #3981. Fixed a stackoverflow that occurs when a subscription fails.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:06:37 -05:00
Tamas Palfy b35ad7cd20
NIFI-7039 - This closes #3995. Fix: PublishJMS outgoing flowfile attribute cleanup could lead to ConcurrentModificationException
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:04:39 -05:00
Joe Gresock e51ab8c9d6
NIFI-7036: This closes #3993. Adding 'Append to Attributes' to QueryElasticsearchHttp
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 10:58:40 -05:00
Gardella Juan Pablo c1301e196c
NIFI-6915 This closes #3961. Jms Durable non shared subscription is broken
Revert NIFI-4834 enhancement for durable non shared consumers only.

Updated also AbstractJMSProcessor class to be public. The testing are not working
without that change, as org.apache.nifi.jms.processors.PublishJMSIT and
org.apache.nifi.jms.processors.ConsumeJMSIT are not working, as @OnSchedule
method is not called (because it is not public).
The method org.apache.nifi.util.StandardProcessorTestRunner.run(int iterations, boolean stopOnFinish, boolean initialize, long runWait) uses ReflectionUtils.invokeMethodsWithAnnotation which does not call non public
methods.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 10:04:39 -05:00
Joe Witt b205b99668
NIFI-7041 This closes #4000. Ensure that if the permissions arent set by the flowfile or processor property that we dont attempt to set perms on the remote host
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 08:47:58 -05:00
Joe Witt 0789ec3a6b
NIFI-6882 from plain patch file in JIRA contributed by 'Josh'
Removed tests which created dirs outside target location and which do not appear unit testable.
2020-01-18 21:02:50 -05:00
Matthew Burgess 71226ce077
NIFI-7044: Fixed 'InputStream not closed' issue in PutElasticsearchRecord and DeleteHBaseCells
This closes #3997

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-18 10:39:47 -05:00
Nathan Gough 4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
bahlulh f8057d0c2f
NIFI-7042: This closes #3996.Azure Reporting Tasking throwing NullReferenceException
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-17 15:29:40 -05:00
Andy LoPresto 34f2a592df
NIFI-7023 This closes #3991. Removed SLF4J and Log4J transitive dependencies from Zookeeper so tests log correctly.
Changed template handling.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-16 13:00:08 -05:00
Peter Turcsanyi ce050f4ecb
NIFI-7009: Atlas reporting task retrieves only the active flow components
Filter out the deleted components before querying them, instead of retrieving
all the components before filtering.

This closes #3979
2020-01-16 11:31:18 -05:00
Matthew Burgess 91e9e65a5c
NIFI-6543: Added unit tests for lifecycle methods
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3974.
2020-01-16 19:11:35 +07:00
wagggz 48d4e6d14c
Add support for onUnscheduled method
Add support for onUnscheduled method1

Add support for onUnscheduled method

Add support for onUnscheduled method
2020-01-16 19:11:26 +07:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Shawn Weeks aa24664801
NIFI-7022 - This closes #3989. Disable Zookeeper Admin Server for Embedded Zookeeper
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 14:43:21 -05:00
Matthew Burgess bb699e7497
NIFI-7017: This closes #3988. Fixed PrometheusReportingTask for nested PG status
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 08:27:49 -05:00
Tamas Palfy 103325354b NIFI-7015 - ConsumeJMS now supports ObjectMessage, MapMessage and StreamMessage types as well. Added optional ERROR_QUEUE property. Result flowfiles get a 'jms.messagetype' attribute that contains the incoming message type (TextMessage, BytesMessage, ObjectMessage, MapMessage or StreamMessage). 2020-01-14 17:07:44 -05:00
Michael Hogue 421bfdd5ff
NIFI-6919: Added relationship attribute to DistributeLoad
NIFI-6919: Cleaned up docs

NIFI-6919: Cleanup

NIFI-6919: Cleanup

NIFI-6919: added negative unit test

NIFI-6919: Removed unnecesary feature flag

Updated attribute description

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3939
2020-01-14 16:14:48 -05:00
Matthew Burgess 824cc0ed77
NIFI-7014: This closes #3985. Add RecordReader/Writer access in ExecuteGroovyScript
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 12:58:21 -05:00
Matthew Burgess 6dbcf9d521
NIFI-6902: This closes #3986. Added unit test for original author's patch
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 11:59:08 -05:00
Tom eb37e7475f
NIFI-6902 Update PrometheusReportingTask to supply correct fields for OutputPort 2020-01-14 11:58:45 -05:00
Martin Šimek 2601f722b3
NIFI-6998 This closes #3972. Batch & Partitioning key
+ Asynchronous sending in limited batches
+ Property to determine attribute name carrying partitioning key
+ Maximum batch size property
+ Carrier object
- Unit test fakery
NIFI-6998 Attributes to User Defined properties
NIFI-6998 Unit tests
NIFI-6998 Review corrections
+ Interruption propagation (& test)
+ Final carrier members
+ Unnecessary generic modifiers removed from generic container
NIFI-6998 checkstyle corrections
+ Tabs to spaces, trailnig spaces
+ Absolute Imports
+ Braces locations
NIFI-6998 imports & license

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 22:37:46 -05:00