Commit Graph

3670 Commits

Author SHA1 Message Date
Matthew Burgess 76e8c51e11
NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file system close
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-30 13:06:46 -05:00
mdayakar 850869c6d2
NIFI-7049 : SFTP processors shouldn't silently try to access known hosts file of the user
Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #4014
2020-01-30 15:15:43 +01:00
Matthew Burgess 04fae9cb5f
NIFI-7069: Add groovy-dateutil module to scripting bundles
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4021.
2020-01-30 14:49:41 +07:00
Mark Payne 37ef10e6fb
NIFI-7076: This closes #4024. Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."
This reverts commit 7731609582.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-29 15:50:55 -05:00
Mark Payne be34767c8a
NIFI-7059: This closes #4007. Fixed bug that results in priorities not properly being set in the SocketLoadBalancedFlowFileQueue. Even though the queue's setPriorities method was called, the underlying may not have contained the localPartition. As a result, when setPriorities() was called, it did not properly delegate that call to . As a result, the queue knew that the Prioritizers were set but the local queue did not apply them. This happened due to a race condition between queue creation and NodeClusterCoordinator learning the Local Node Identifier. Additionally, updated NodeClusterCoordinator so that it will persist its state when it does learn the Local Node Identifier, as that was not being persisted. In testing this, also encounterd a deadlock in a particular Integration Test that was run, around AbstractFlowFileQueue and its synchronization. Because the 'synchronized' keyword synchronizes on 'this' and the concrete implementation also uses 'synchronized' and the result is the same 'this', a deadlock can occur that is difficult to understand. As a result, refactored AbstractFlowFileQueue to instead use read/write locks.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-22 23:30:41 -05:00
Joe Witt 535cafdb83
Merge branch 'NIFI-7021-RC3' bringing the Apache NiFi 1.11.0 to master 2020-01-22 21:15:12 -05:00
Bryan Bende 24b846d08a
NIFI-7051 Protect against empty group membership in ShellUserGroupProvider, and add differentiator to id seeding
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info

NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance

This closes #4003.
2020-01-22 15:51:42 -05:00
nagasivanath 24ef8ba4cb
Update nf-processor-details.js
NIFI-7032:
- Processor Details no longer appears when clicking 'View Processor Details'
- handling the review comments

This closes #3990
2020-01-21 13:54:12 -05:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt 67c36f3c3e
NIFI-7021 apache parent pom and common dep maintenance for build 2020-01-19 13:32:37 -05:00
Gardella Juan Pablo cddaac591b
NIFI-7034 Thise closes #4002. Connection leak with JMSConsumer and JMSPublisher
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 12:08:22 -05:00
mans2singh ddaaded08f
NIFI-6861 - This closes #3882. Added missing display name property
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 12:02:03 -05:00
Marc Parisi de2a286a7a
NIFI-818: This closes #3926. Initial implementation of NiFi-Accumulo ( https://github.com/phrocker/nifi-accumulo ) with connectors to Apache Accumulo 2.x
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:43:19 -05:00
Dirk Ernst van den Biggelaar ab0113664d
NIFI-6997 This closes #3971. connection closing fixed
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:09:52 -05:00
Gal Shinder 4fec54debf
NIFI-6996 This closes #3981. Fixed a stackoverflow that occurs when a subscription fails.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:06:37 -05:00
Tamas Palfy b35ad7cd20
NIFI-7039 - This closes #3995. Fix: PublishJMS outgoing flowfile attribute cleanup could lead to ConcurrentModificationException
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 11:04:39 -05:00
Joe Gresock e51ab8c9d6
NIFI-7036: This closes #3993. Adding 'Append to Attributes' to QueryElasticsearchHttp
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 10:58:40 -05:00
Gardella Juan Pablo c1301e196c
NIFI-6915 This closes #3961. Jms Durable non shared subscription is broken
Revert NIFI-4834 enhancement for durable non shared consumers only.

Updated also AbstractJMSProcessor class to be public. The testing are not working
without that change, as org.apache.nifi.jms.processors.PublishJMSIT and
org.apache.nifi.jms.processors.ConsumeJMSIT are not working, as @OnSchedule
method is not called (because it is not public).
The method org.apache.nifi.util.StandardProcessorTestRunner.run(int iterations, boolean stopOnFinish, boolean initialize, long runWait) uses ReflectionUtils.invokeMethodsWithAnnotation which does not call non public
methods.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 10:04:39 -05:00
Joe Witt b205b99668
NIFI-7041 This closes #4000. Ensure that if the permissions arent set by the flowfile or processor property that we dont attempt to set perms on the remote host
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 08:47:58 -05:00
Joe Witt 0789ec3a6b
NIFI-6882 from plain patch file in JIRA contributed by 'Josh'
Removed tests which created dirs outside target location and which do not appear unit testable.
2020-01-18 21:02:50 -05:00
Matthew Burgess 71226ce077
NIFI-7044: Fixed 'InputStream not closed' issue in PutElasticsearchRecord and DeleteHBaseCells
This closes #3997

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-18 10:39:47 -05:00
Nathan Gough 4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
bahlulh f8057d0c2f
NIFI-7042: This closes #3996.Azure Reporting Tasking throwing NullReferenceException
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-17 15:29:40 -05:00
Andy LoPresto 34f2a592df
NIFI-7023 This closes #3991. Removed SLF4J and Log4J transitive dependencies from Zookeeper so tests log correctly.
Changed template handling.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-16 13:00:08 -05:00
Peter Turcsanyi ce050f4ecb
NIFI-7009: Atlas reporting task retrieves only the active flow components
Filter out the deleted components before querying them, instead of retrieving
all the components before filtering.

This closes #3979
2020-01-16 11:31:18 -05:00
Matthew Burgess 91e9e65a5c
NIFI-6543: Added unit tests for lifecycle methods
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3974.
2020-01-16 19:11:35 +07:00
wagggz 48d4e6d14c
Add support for onUnscheduled method
Add support for onUnscheduled method1

Add support for onUnscheduled method

Add support for onUnscheduled method
2020-01-16 19:11:26 +07:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Shawn Weeks aa24664801
NIFI-7022 - This closes #3989. Disable Zookeeper Admin Server for Embedded Zookeeper
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 14:43:21 -05:00
Matthew Burgess bb699e7497
NIFI-7017: This closes #3988. Fixed PrometheusReportingTask for nested PG status
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 08:27:49 -05:00
Tamas Palfy 103325354b NIFI-7015 - ConsumeJMS now supports ObjectMessage, MapMessage and StreamMessage types as well. Added optional ERROR_QUEUE property. Result flowfiles get a 'jms.messagetype' attribute that contains the incoming message type (TextMessage, BytesMessage, ObjectMessage, MapMessage or StreamMessage). 2020-01-14 17:07:44 -05:00
Michael Hogue 421bfdd5ff
NIFI-6919: Added relationship attribute to DistributeLoad
NIFI-6919: Cleaned up docs

NIFI-6919: Cleanup

NIFI-6919: Cleanup

NIFI-6919: added negative unit test

NIFI-6919: Removed unnecesary feature flag

Updated attribute description

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3939
2020-01-14 16:14:48 -05:00
Matthew Burgess 824cc0ed77
NIFI-7014: This closes #3985. Add RecordReader/Writer access in ExecuteGroovyScript
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 12:58:21 -05:00
Matthew Burgess 6dbcf9d521
NIFI-6902: This closes #3986. Added unit test for original author's patch
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 11:59:08 -05:00
Tom eb37e7475f
NIFI-6902 Update PrometheusReportingTask to supply correct fields for OutputPort 2020-01-14 11:58:45 -05:00
Martin Šimek 2601f722b3
NIFI-6998 This closes #3972. Batch & Partitioning key
+ Asynchronous sending in limited batches
+ Property to determine attribute name carrying partitioning key
+ Maximum batch size property
+ Carrier object
- Unit test fakery
NIFI-6998 Attributes to User Defined properties
NIFI-6998 Unit tests
NIFI-6998 Review corrections
+ Interruption propagation (& test)
+ Final carrier members
+ Unnecessary generic modifiers removed from generic container
NIFI-6998 checkstyle corrections
+ Tabs to spaces, trailnig spaces
+ Absolute Imports
+ Braces locations
NIFI-6998 imports & license

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 22:37:46 -05:00
Mark Payne 66d5ab80eb
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data structures: activeQueue, swapQueue. activeQueue is intended to be pulled from for processing. swapQueue is intended to hold FlowFiles that are waiting to be swapped out. SinWe want to ensure that we first swap in any data that has already been swapped out before processing the swap queue, in order to ensure that we process the data in the correct order. This fix ddresses an issue where data was being swapped out by writing the lowest priority data to a swap file, then adding the highest priority data to activeQueue and the 'middle' priority data back to swapQueue. As a result, when polling from the queue we got highest priority data, followed by lowest priority data, followed by middle priority data. This is addressed by avoiding putting anything back on swapQueue when we swap out. Instead, write data to the swap file, then push everything else to activeQueue. This way, any new data that comes in will still go to the swapQueue, as it should, but all data that didn't get written to the Swap file will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
2020-01-13 22:02:05 -05:00
nagasivanath 4e2b61efe4
NIFI-6652 UI Fix overflowing text in variables dialog
This closes #3982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-13 12:12:34 -08:00
John Pierce 0dc6439ddb
NIFI-6964 This closes #3975. Use compression level for xz-lzma2 format of the CompressContent processor
NIFI-6965 adding resource warning on memory and highlighting this with the compression level property

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 15:08:13 -05:00
Tamas Palfy b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
mdayakar ccb85cf8b7
NIFI-6923:If FlowFile content is truncated in the Content Repository, NiFi does not throw Exception when reading the content.
NIFI-6923 fixing checkstyle
2020-01-13 12:45:59 -05:00
Tamas Palfy 26fcf8158a NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added "Batch Size property", takes effect only if "Destination" is set to "Content" and Grouping" is set to "None"
NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added validation for 'Batch Size' in 'GetHDFSFileInfo'.
NIFI-6992 - Changed 'GetHDFSFileInfo.BATCH_SIZE' validator from 'NON_NEGATIVE_INTEGER_VALIDATOR' to 'POSITIVE_INTEGER_VALIDATOR'. Added more tests.
NIFI-6992 - Removed 'AllowEmptyValidator'. 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String.
NIFI-6992 - 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String - cont.
NIFI-6992 - Fix: Unused import.

This closes #3966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-01-13 17:50:33 +01:00
Shawn Weeks 2c18cf22ad
NIFI-6852 This closes #3973. Don't Validate Processors that accept any ControllerService Implementation
NIFI-6852 Add Integration Tests for Controller Service API Validation

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 11:39:39 -05:00
Andy LoPresto 2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
Peter Turcsanyi b8ffb54612
NIFI-6945: Use minExtInfo=true to reduce the amount of data queried from Atlas
This closes #3970
2020-01-10 09:03:07 -05:00
Joe Witt fc2e6b3471 NIFI-7006 Changed the Kudu controller service test to an integration test since it starts a server and is not portable across build environments (#3976)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 19:10:03 -08:00
mtien 0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
Bryan Bende 08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Matthew Burgess 8d214f99a4
NIFI-6960: Add Provenance Table to MetricsSqlQueryService
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3945.
2020-01-09 17:26:43 +07:00