Commit Graph

1627 Commits

Author SHA1 Message Date
Nathan Gough 4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
Andy LoPresto 34f2a592df
NIFI-7023 This closes #3991. Removed SLF4J and Log4J transitive dependencies from Zookeeper so tests log correctly.
Changed template handling.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-16 13:00:08 -05:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Shawn Weeks aa24664801
NIFI-7022 - This closes #3989. Disable Zookeeper Admin Server for Embedded Zookeeper
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 14:43:21 -05:00
Mark Payne 66d5ab80eb
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data structures: activeQueue, swapQueue. activeQueue is intended to be pulled from for processing. swapQueue is intended to hold FlowFiles that are waiting to be swapped out. SinWe want to ensure that we first swap in any data that has already been swapped out before processing the swap queue, in order to ensure that we process the data in the correct order. This fix ddresses an issue where data was being swapped out by writing the lowest priority data to a swap file, then adding the highest priority data to activeQueue and the 'middle' priority data back to swapQueue. As a result, when polling from the queue we got highest priority data, followed by lowest priority data, followed by middle priority data. This is addressed by avoiding putting anything back on swapQueue when we swap out. Instead, write data to the swap file, then push everything else to activeQueue. This way, any new data that comes in will still go to the swapQueue, as it should, but all data that didn't get written to the Swap file will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
2020-01-13 22:02:05 -05:00
nagasivanath 4e2b61efe4
NIFI-6652 UI Fix overflowing text in variables dialog
This closes #3982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-13 12:12:34 -08:00
Tamas Palfy b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
mdayakar ccb85cf8b7
NIFI-6923:If FlowFile content is truncated in the Content Repository, NiFi does not throw Exception when reading the content.
NIFI-6923 fixing checkstyle
2020-01-13 12:45:59 -05:00
Shawn Weeks 2c18cf22ad
NIFI-6852 This closes #3973. Don't Validate Processors that accept any ControllerService Implementation
NIFI-6852 Add Integration Tests for Controller Service API Validation

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 11:39:39 -05:00
Andy LoPresto 2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
mtien 0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
Bryan Bende 08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Mark Payne c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne 5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Mark Payne 2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
Joe Ferner 058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00
Mark Payne e05d11c0b4 NIFI-6787: Added a comment to the UI tooltip, user guide, and javadocs to indicate that round robin may skip a node if that node is not receiving the data as fast as other nodes in the cluster, in order to maximum throughput. 2019-12-13 15:09:15 -05:00
Tamas Palfy 7731609582 NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize().
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
2019-12-13 14:36:44 -05:00
Iván Rodriguez 65ba4a2d93 NIFI-6853 - flowFileEvent combineCounters hashmap overwritten
- Added unit test to verify behavior of contribution

This closes #3875.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 14:23:50 -05:00
Mark Payne 452ca98c29
NIFI-6924: When seeking to the appropriate offset for a content claim, ensure that if there are not enough bytes in the underlying resource claim that a ContentNotFoundException is thrown. Also cleaned up error-handling case in StandardProcessSession to ensure that we close the existing InputStream before calling handleContenttNotFoundException, since this method may itself throw an Exception
This closes #3924.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-12-10 15:27:53 -05:00
Bryan Bende dd620680b1
NIFI-6904 Moving ClassLoader creation before Authorizer instantiation
This closes #3903
2019-11-25 13:35:21 -05:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Andrew Lim 5bfc86737d
NIFI-6747 Updated images used by Swagger REST API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3901.
2019-11-22 14:05:36 +01:00
Joe Ferner 3f3b778f24 NIFI-6893
Upgraded angular from 1.7.2 to 1.7.9

This closes #3899

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-11-21 16:45:33 -05:00
Mark Payne ea31c1fcbb
NIFI-6868: Ensure that when a property value is obtained by calling ProcessContext.getPropertyValue(...).evaluateAttributeExpressions(...).getValue() that we still retain the appropriate value even if the property references a sensitive parameter
* Added additional unit tests and updated ParameterEvaluator to make sure that it follows the same logic as the ParameterExpression in that sensitive parameters cannot be access from within EL

This closes #3889
Signed-off-by: Rob Fellows <rob.fellows@gmail.com>
2019-11-14 15:37:39 -05:00
Joe Ferner c947e57d5e
NIFI-6814:
- Restored JAXB Adapter

This closes #3880
2019-11-08 15:43:13 -05:00
Mark Payne 5f4ce8f431
NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedNiFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath. Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
NIFI-6707: Added System Test to verify behavior when services depend on one another

NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them

NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group

NIFI-6707: Updated to include java11 subdirectory under lib/

NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT

NIFI-6707: Updates to see if it will help travis build correctly

NIFI-6707: Commenting out java commands in bootstrap.conf files

This closes #3831.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-11-08 12:33:16 -05:00
Mark Payne 49b7a7cd6b
NIFI-6846: If an Exception is thrown while a Processor is writing to a FlowFile, but that Content Claim is not yet eligible for destruction, mark it as a transient claim on the RepositoryRecord so that if it's available when the FlowFile Repository is checkpointed, then it will be cleaned up then
This closes #3872
2019-11-06 09:53:25 -05:00
Andy LoPresto 12bb284453
NIFI-6841 Fixed checkstyle error introduced in a9db5a8. 2019-11-05 13:12:40 -08:00
Mark Payne a9db5a8cb7
NIFI-6841: Fixed bug that resulted in the wrong number of 'Bytes Read' being reported by ByteCountingInputStream in the event that #skip was called between calls to #mark and #reset
This closes #3868
2019-11-05 13:06:06 -05:00
Jan Hentschel d6b4bce668 NIFI-4758 Changed 'Name' to 'Flow Name' in Import Version dialog 2019-11-04 15:26:25 -05:00
mtien 7f1886adad NIFI-4757 Capitalized 'version' in tool tip. 2019-11-04 15:22:52 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 4119279c03
Merge branch 'NIFI-6733-RC3' 2019-11-04 13:24:29 -05:00
Joe Witt 363b751310
Revert "NIFI-6733 bumping master to 1.11.0-SNAPSHOT"
This reverts commit a7c56cfd9a.
2019-11-04 13:24:13 -05:00
Joe Witt a7c56cfd9a
NIFI-6733 bumping master to 1.11.0-SNAPSHOT 2019-11-04 07:54:17 -05:00
Mark Payne 0b3a60ae3e NIFI-6832: When receiving a 0-byte FlowFile via site-to-site, avoid writing to the FlowFile at all 2019-11-01 16:22:17 -04:00
Kotaro Terada 245e055d5d
NIFI-6828 There is an unintentional string in "conf/logback.xml"
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3863.
2019-10-31 11:20:21 +01:00
mtien 3d56e2f26d
NIFI-4348 handled NPE in isExpressionLanguagePresent.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3857.
2019-10-30 09:58:55 +01:00
Mark Payne ce5eae5b2c NIFI-6800: Added unit test to verify behavior of contribution 2019-10-29 10:17:27 -04:00
Ivan Ezequiel Rodriguez 67f1677212 NIFI-6800 - Fix hashmap counter overwritten in highThroughputSession
This closes #3837.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-29 10:16:49 -04:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Joe Witt dda29cd90b
NIFI-6733 updating zk to latest incremental 2019-10-28 12:40:03 -07:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Matt Gilman 9364f7d2e8
NIFI-6811:
- Allowing enterGroup to complete prior to loading the controller service table.

This closes #3846.
2019-10-25 12:07:03 -04:00
Mark Payne 3f270f184c
NIFI-6807: When a Controller Service's state is transitioned to ENABLING, complete the Future successfully, even if the Controller Service is not valid. The Controller Service will remain in the ENABLING state until it is made valid, at which point it will ENABLE (unless explicitly disabled first). This allows us to Enable a Controller Service and its referencing components, even if the referencing component is still invalid due to it not yet recognizing the the referenced service has been enabled.
This closes #3841
2019-10-25 11:38:34 -04:00
Mark Payne 81fc2768c9
NIFI-6798: When checking that referenced Controller Services are valid, use ValidationContext.getProperties() rather than ComponentNode.getSupportedPropertyDescriptors() because the latter does not include dynamic properties
This closes #3836.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-25 09:52:44 -04:00
Mark Payne 8079d2531e
NIFI-6805, NIFI-6806: Updated toString() of StandardControllerServiceNode so that it includes the component's type and UUID, updated the more generic Exception Mappers to include stack traces regardless of the log level that is enabled
This closes #3840

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-24 19:27:30 -04:00
Yolanda M. Davis 147365285c
NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3838
2019-10-24 08:51:50 -04:00