1659 Commits

Author SHA1 Message Date
Nathan Gough
69b6c231fd
NIFI-7223 - Added another test for when keyPasswd is not present. 2020-03-16 12:28:18 -04:00
Andy LoPresto
f9d75056fa
NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
2020-03-16 12:28:01 -04:00
Nathan Gough
7374361b5c
NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.
2020-03-16 12:26:50 -04:00
Joe Witt
97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Mark Payne
d4a2afc25c
NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.
2020-03-12 10:25:04 +01:00
Mark Payne
d68720920f
NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.
2020-03-11 21:00:43 +01:00
Mark Payne
afad982e91
NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"
This reverts commit f01668e66ad2e45197915769e966a4be27e1592e.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-10 14:32:43 -04:00
Pierre Villard
12c8402ac3
NIFI-7210 - added PG path in bulletins for S2S Bulletin RT
Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100
2020-03-09 16:29:22 -04:00
Mark Bean
f4b65afb64 NIFI-7231: move controller service validation out of synchronized block for enabling
This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:01:19 -05:00
Joe Witt
4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Joe Witt
bad0f10a52
NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4091.
2020-02-25 17:38:33 -08:00
Andy LoPresto
815e4cf51f
NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085)
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-25 13:14:52 -08:00
Joe Witt
778012412a
Revert "NIFI-6363 Refactors sensitive properties, adds additional providers."
This reverts commit 479fcfdc0bbd97dd0635bbec0273cc25fcf9cc03.

It does not build properly as shown in Github Actions.
2020-02-24 22:07:53 -08:00
Troy Melhase
479fcfdc0b NIFI-6363 Refactors sensitive properties, adds additional providers.
NIFI-6363 Additional fixes.

NIFI-6363 Fix Hadoop compile problem.  Add GCP IT instructions.

NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.

NIFI-6363 - Added some docs and experimental tag to the relevant classes.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4080.
2020-02-24 16:35:19 -05:00
MatthewKnight-NG
acaf321af0
NIFI-5924 Labels should be searchable
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4070
2020-02-24 16:14:21 -05:00
Troy Melhase
0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
M Tien
37614d02cd
NIFI-7136 Added autocomplete="off" to login password input (#4055)
NIFI-7136 Added autocomplete="off" to login password input
Updated nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/login/login-form.jsp

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-14 20:46:08 -08:00
Joe Ferner
62606ff89a NIFI-6873: Added support for replacing a process group
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
 - added replace process group functionality in ProcessGroupResource
 - parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
 - refactored flow update methods to make use of commonality across different update types whenever possible
 - fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
 - improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import

This closes #4023.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-02-11 14:46:40 -05:00
Mark Payne
65b2a9bc2c
NIFI-7117: When SocketLoadBalancedFlowFileQueue creates its array of Queue Partitions in the constructor, it added the local partition as the first element in that list. This list should be ordered the same across all nodes in the cluster. By making the local partition the first in the array, each node had a different ordering of these partitions. As a result, Partition by Attribute strategy would constantly rebalance flowfiles that it received to other node, and Single Node always transferred data to the first partition, which was the local node, instead of whichever node should have been the first in the list. This commit addresses this issue by instead inserting the local partition intot he 'queuePartitions' array based on the local node identifier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4045.
2020-02-11 10:59:19 -08:00
nagasivanath
425ba63e66
NIFI-6968 Create Connection Model Allows Multiple Adds
This closes #4013.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-06 10:32:59 -05:00
Matt Gilman
b1b06bf5e1 NIFI-6958:
- Addressing issue causing errors preventing version changes when there is a processor to enable or disable in a sub process group.
2020-02-05 13:49:06 -05:00
Bryan Bende
5d851e6a13
NIFI-7067 Allow a user and group with the same name/identity to exist
This closes #4019
2020-02-05 09:19:18 -05:00
Andy LoPresto
d7c29f4637 NIFI-7079 Cleaned up formatting and comments. 2020-01-31 08:58:51 -05:00
Andy LoPresto
95746d346c NIFI-7079 Added SecureHasher interface and Argon2SecureHasher implementation.
Added unit tests.
Fixed JwtService charset references.
Changed FingerprintFactory to mask sensitive values via Argon2 hashing.
Resolved RAT checks.
2020-01-31 08:58:51 -05:00
Mark Payne
37ef10e6fb
NIFI-7076: This closes #4024. Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."
This reverts commit 773160958209a8a173b4f741517c4bda63a28e82.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-29 15:50:55 -05:00
Mark Payne
be34767c8a
NIFI-7059: This closes #4007. Fixed bug that results in priorities not properly being set in the SocketLoadBalancedFlowFileQueue. Even though the queue's setPriorities method was called, the underlying may not have contained the localPartition. As a result, when setPriorities() was called, it did not properly delegate that call to . As a result, the queue knew that the Prioritizers were set but the local queue did not apply them. This happened due to a race condition between queue creation and NodeClusterCoordinator learning the Local Node Identifier. Additionally, updated NodeClusterCoordinator so that it will persist its state when it does learn the Local Node Identifier, as that was not being persisted. In testing this, also encounterd a deadlock in a particular Integration Test that was run, around AbstractFlowFileQueue and its synchronization. Because the 'synchronized' keyword synchronizes on 'this' and the concrete implementation also uses 'synchronized' and the result is the same 'this', a deadlock can occur that is difficult to understand. As a result, refactored AbstractFlowFileQueue to instead use read/write locks.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-22 23:30:41 -05:00
Joe Witt
535cafdb83
Merge branch 'NIFI-7021-RC3' bringing the Apache NiFi 1.11.0 to master 2020-01-22 21:15:12 -05:00
Bryan Bende
24b846d08a
NIFI-7051 Protect against empty group membership in ShellUserGroupProvider, and add differentiator to id seeding
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info

NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance

This closes #4003.
2020-01-22 15:51:42 -05:00
nagasivanath
24ef8ba4cb
Update nf-processor-details.js
NIFI-7032:
- Processor Details no longer appears when clicking 'View Processor Details'
- handling the review comments

This closes #3990
2020-01-21 13:54:12 -05:00
Joe Witt
3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt
633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt
67c36f3c3e
NIFI-7021 apache parent pom and common dep maintenance for build 2020-01-19 13:32:37 -05:00
Nathan Gough
4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
Andy LoPresto
34f2a592df
NIFI-7023 This closes #3991. Removed SLF4J and Log4J transitive dependencies from Zookeeper so tests log correctly.
Changed template handling.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-16 13:00:08 -05:00
Joe Witt
23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Shawn Weeks
aa24664801
NIFI-7022 - This closes #3989. Disable Zookeeper Admin Server for Embedded Zookeeper
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 14:43:21 -05:00
Mark Payne
66d5ab80eb
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data structures: activeQueue, swapQueue. activeQueue is intended to be pulled from for processing. swapQueue is intended to hold FlowFiles that are waiting to be swapped out. SinWe want to ensure that we first swap in any data that has already been swapped out before processing the swap queue, in order to ensure that we process the data in the correct order. This fix ddresses an issue where data was being swapped out by writing the lowest priority data to a swap file, then adding the highest priority data to activeQueue and the 'middle' priority data back to swapQueue. As a result, when polling from the queue we got highest priority data, followed by lowest priority data, followed by middle priority data. This is addressed by avoiding putting anything back on swapQueue when we swap out. Instead, write data to the swap file, then push everything else to activeQueue. This way, any new data that comes in will still go to the swapQueue, as it should, but all data that didn't get written to the Swap file will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
2020-01-13 22:02:05 -05:00
nagasivanath
4e2b61efe4
NIFI-6652 UI Fix overflowing text in variables dialog
This closes #3982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-13 12:12:34 -08:00
Tamas Palfy
b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
mdayakar
ccb85cf8b7
NIFI-6923:If FlowFile content is truncated in the Content Repository, NiFi does not throw Exception when reading the content.
NIFI-6923 fixing checkstyle
2020-01-13 12:45:59 -05:00
Shawn Weeks
2c18cf22ad
NIFI-6852 This closes #3973. Don't Validate Processors that accept any ControllerService Implementation
NIFI-6852 Add Integration Tests for Controller Service API Validation

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 11:39:39 -05:00
Andy LoPresto
2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
mtien
0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
Bryan Bende
08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Mark Payne
c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne
5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Mark Payne
2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
Joe Ferner
058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00
Mark Payne
e05d11c0b4 NIFI-6787: Added a comment to the UI tooltip, user guide, and javadocs to indicate that round robin may skip a node if that node is not receiving the data as fast as other nodes in the cluster, in order to maximum throughput. 2019-12-13 15:09:15 -05:00
Tamas Palfy
7731609582 NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize().
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
2019-12-13 14:36:44 -05:00