Commit Graph

5732 Commits

Author SHA1 Message Date
Peter Turcsanyi cc74534bc0
NIFI-6939: Upgrade Atlas client dependency to 2.0.0
NIFI-6939: Review changes

This closes #3944
2020-01-08 13:25:45 -05:00
Matthew Burgess c604923c0b NIFI-6947: Add PutRecord processor using RecordSinkService (#3943)
* NIFI-6947: Add PutRecord processor using RecordSinkService

* NIFI-6947: Incorporated review comments
2020-01-08 04:25:14 -04:00
Matthew Burgess f641a15c5f
NIFI-6989: Fixed SiteToSiteReportingRecordSink when transaction sent no data
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3965.
2020-01-08 13:56:42 +07:00
Matthew Burgess f4ef45678e
NIFI-6963: Fix ValidateRecord handling of missing required array fields
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3948.
2020-01-08 02:02:04 +07:00
markap14 209c560ff5
Merge pull request #3946 from IlyaKovalev/NIFI-6955
NIFI-6955 close streams in Reader/writer mock
2020-01-07 10:55:27 -05:00
markap14 596e0ffbdc
Merge pull request #3932 from mtien-apache/NIFI-6336
NIFI-6336 Added code to catch port value when it is 0.
2020-01-07 10:48:50 -05:00
Matthew Burgess 9ed4623817
NIFI-6886: Fixed SiteToSiteReportingRecordSink, refactored mocks
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3959.
2020-01-07 15:15:54 +07:00
Eduardo Fontes 1ee6dba00a
NIFI-6886 - Bugfix
Attribute peerPersistence can be null generating Bulletin WARNs "Unable to refresh Remote Group's peers due to null".

Rollback

The fix is inside site-to-site-reporting-task-bundle

Modify getClient()

Get ConfigurationContext and ReportingContext to provide a StateManager.

Modify OnScheduled setup()

The OnSchedule setup() now save the ConfigurationContext to lazily create a SiteToSiteClient with ReportingContext through an overloaded setup().

Modify OnTrigger

Lazily creates SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Retry compile

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Update AbstractSiteToSiteReportingTask.java

Removed the OnSchedule setup(ConfigContext) because it is not needed.

Update SiteToSiteUtils.java

Removed ConfigContext from getClient parameters because ReportContext share the same properties.
2020-01-07 15:15:44 +07:00
Ubuntu 65ca8e6c8d
NIFI-6942 This closes #3934. Added a reporting task to push provenance data to azure log analytics.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-06 15:42:15 -05:00
markap14 e4bdc79ea6
Merge pull request #3853 from markap14/NIFI-6822
NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we p…
2020-01-06 15:01:42 -05:00
Mark Payne c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne b03e5b0520
NIFI-6983: Ensure that if any call to kafka's Producer throws a ProducerFencedException that it's handled properly by poisoning the lease, which in turn will close the client
This closes #6983.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-01-06 11:34:51 -05:00
Shawn Weeks b470db620b
NIFI-6979 Add record.index field to UpdateRecord
This closes #3955

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-03 19:25:01 -05:00
Endre Zoltan Kovacs c15876494a
NIFI-6897: making nifi-toolkit >= java 8 compliant
fixing cli.bat
cygwin fix: on windows ${JAVA} can have spaces in it.
...must wrap in quotes
some jvms doesn't print a dot after the major version

...removing that from patter matching

This closes #3908

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-12-30 08:10:37 -05:00
Peter Turcsanyi 41b47a5156
NIFI-6922: Fix typo in fs.permissions.umask-mode Hadoop property name in PutHDSF
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3911.
2019-12-30 10:01:25 +01:00
r65535 82b4fb0633
NIFI-6957 - Added REGEX header property, and option to allow illegal chars in header names
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3942.
2019-12-30 09:55:39 +01:00
Pierre Villard ad636789f0
NIFI-6928 - add connect/read timeout to RestLookupService
This closes #3920

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-12-29 19:10:30 -05:00
Mark Payne 5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Andrew Lim 7f49c6b76a
NIFI-6907 Added nifi.bootstrap.sensitive.key to Bootstrap Properties section in Admin Guide
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3927.
2019-12-26 16:41:31 +01:00
mans2singh ec5bc7ea7a
NIFI-6971 - Minor typo in capabilities description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3952.
2019-12-26 10:58:54 +01:00
Nathan Gough 42cb6e8489
NIFI-6948 - Removing some debug logging.
This closes #3935.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-12-23 16:07:52 -05:00
Mark Payne 2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
KovalevIV 8f9fc49ca0 NIFI-6955 close streams in Reader/writer mock 2019-12-19 22:26:31 +03:00
mtien 2959c436b1 NIFI-6336 Fixed the Build Failure issue. 2019-12-19 10:47:23 -08:00
Matthew Burgess 0024668dd5
NIFI-6952: Evaluate EL for Hive3ConnectionPool properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3937.
2019-12-19 11:51:18 +01:00
Matthew Burgess 3a5a2da73c
NIFI-6954: Evaluate EL for DBCPConnectionPool properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3940.
2019-12-19 11:44:55 +01:00
mtien ab15536976 NIFI-6336 Fixed Exception message to refer to the correct port. 2019-12-18 15:06:45 -08:00
Joe Ferner 058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00
Matthew Burgess f398299cfe NIFI-6956: Fixed StreamingException handling in PutHive3Streaming (#3941) 2019-12-17 14:49:04 -05:00
Matthew Burgess 6d1bcc22e7 NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession (#3913)
* NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession

* Fixed logic for no records, added unit test

* Fixed PutElasticsearchHttpRecord and PutHive3Streaming, same bug as JoltTransformRecord

* Added null checks
2019-12-17 11:42:24 -05:00
Matthew Formosa 4b08cf116c
NIFI-6824 - Handling NPE of header value when consuming from Kafka
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3859.
2019-12-17 16:40:42 +01:00
Tamas Palfy 67fca6832a NIFI-6910 - ReportLineageToAtlas - Added processor properties for connect timeout and read timeout.
NIFI-6910 - ReportLineageToAtlas - Capitalized Atlas connect and read timeout property display names.

This closes #3936.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-12-16 20:55:13 +01:00
Matthew Burgess 8b3f6fa61c NIFI-6953: Add missing documentation for input.flowfile.uuid attribute of ExecuteSQL processors 2019-12-16 14:48:12 -05:00
Tamas Palfy 599fb98415 NIFI-6937 - NotificationSender uses typedQualifiedName instead of simple qualifiedName as keys in local maps.
NIFI-6937 - Fix NotificationSender: typedQualifiedName handling.

This closes #3929.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-12-16 17:33:47 +01:00
Mark Payne e05d11c0b4 NIFI-6787: Added a comment to the UI tooltip, user guide, and javadocs to indicate that round robin may skip a node if that node is not receiving the data as fast as other nodes in the cluster, in order to maximum throughput. 2019-12-13 15:09:15 -05:00
Tamas Palfy 7731609582 NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize().
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
2019-12-13 14:36:44 -05:00
Iván Rodriguez 65ba4a2d93 NIFI-6853 - flowFileEvent combineCounters hashmap overwritten
- Added unit test to verify behavior of contribution

This closes #3875.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 14:23:50 -05:00
Kourge 43bc6c6ed9 NIFI-6905 Prevent non-primary nodes to connect Twitter API when runnin in primary node only mode.
This closes #3909.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 13:41:21 -05:00
Pierre Villard 3c3eba1909 NIFI-6916 - handle null text message in JMSConsumer
This closes #3919.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 13:03:23 -05:00
Bryan Bende 442858127b NIFI-6938 Fixing import-flow-version CLI command to correctly copy over fields
This closes #3925.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 12:58:31 -05:00
Matthew Knight 29e7adb2b4 NIFI-6686 - Improved exception handling in ResizeImage processor.
This closes #3922.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 12:55:26 -05:00
mtien 92b9a05309 NIFI-6336 Added code to catch port value when it is 0.
Added unit tests validating port value format, valid port value, and catching port value of 0.
2019-12-12 12:09:42 -08:00
Mark Payne 452ca98c29
NIFI-6924: When seeking to the appropriate offset for a content claim, ensure that if there are not enough bytes in the underlying resource claim that a ContentNotFoundException is thrown. Also cleaned up error-handling case in StandardProcessSession to ensure that we close the existing InputStream before calling handleContenttNotFoundException, since this method may itself throw an Exception
This closes #3924.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-12-10 15:27:53 -05:00
Peter Turcsanyi ba6d050ba8
NIFI-6926: Fixed memory leak in NiFiAtlasHook
NIFI-6926: Use new instance of list instead of clearing it
NIFI-6926: Logging the number of messages to be sent to Atlas.
NIFI-6926: Pass a copy of the messages list to send() and clear the original list.

This closes #3915
2019-12-09 16:31:33 -05:00
John Fortin 885f1d1e5e
NIFI-6795 - fix for CouchbaseMapCacheClient to allow new objects to be created
Signed-off-by: John Fortin <johnf@marketamerica.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3833.
2019-12-06 07:04:37 +01:00
Pierre Villard 41fef551c3
NIFI-6917 - fix doc for dynamic properties variables support in JMS Connection Factory Provider
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3914
2019-12-04 16:12:18 -05:00
Grant Henke 1898ad44be
NIFI-6895: Fix PutKudu processor concurrency issues
Calls to `trigger()` may be called concurrently from different threads,
however the PutKudu processor is storing the `kuduSession`
in a class level field. This can result in the logging issue reported in
NIFI-6895 and likely other unusual anomolies including performace
issues depending on the processor configuration.

Additionally the `operationType` was also stored in a class level field
and could be set concurrently resulting in the incorrect operation type
used.

This patch fixes the issue by moving both kuduSession and operationType
to be local. Additionaly some minor code cleanup was included.

An integration test, ITPutKudu, was added and used to manual verify the
logging issue existed and is fixed by this patch. I ran the test using
`mvn -Pintegration-tests verify -Dtest=ITPutKudu`

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3910.
2019-12-03 03:46:12 +01:00
Matthew Burgess 955784031f
NIFI-6912: Add profiles to Hive 3 NAR to optionally include additional Hadoop libs
This closes #3907.
2019-11-27 15:37:48 -05:00
Bryan Bende dd620680b1
NIFI-6904 Moving ClassLoader creation before Authorizer instantiation
This closes #3903
2019-11-25 13:35:21 -05:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00