3371 Commits

Author SHA1 Message Date
Mark Payne
58a25cfa5a
NIFI-6110: Updated StandardProcessSession such that if we fail to update the FlowFile Repository, we do not decrement claimant counts for any FlowFiles that were removed. Doing so can cause an issue where a FlowFile is removed, then the FlowFileRepo update fails, resulting in the flowfile being rolled back, but after its claimant count is decremented. It will then be processed again, which can result in the same thing, and we'll end up decrementing the claimant count repeatedly. Also updated LengthDelimitedJournal so that if the overflow directory already exists, it does not fail when trying to create the directory and instead just moves on. Updated unit tests to test both of these new fixes and updated DummyRecordSerde to be thread-safe because the TestLengthDelimitedJournal now needs it to be thread safe due to the new unit test that was added.
This closes #3358.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-08 10:32:38 -05:00
Peter Turcsanyi
79e05c9f58 NIFI-6078: Create PostSlack processor
Processor for sending messages on Slack and optionally upload and attach
the FlowFile content (e.g. an image) to the message.

NIFI-6078: Remove username/icon properties.

NIFI-6078: Make Text property optional.

NIFI-6078: Documentation changes.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3339
2019-03-07 13:00:31 -05:00
Ferenc Szabó
d53cefa2a7 NIFI-3685 Azure EventHub processor tests fail
Moved test classes into the proper folder.
Fixed outdated mocking and added assertions that actually tests the content.

This closes #3346.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-03-07 14:26:40 +09:00
Kevin Doran
1d06044e3b
NIFI-6108: Fix port names when imported from registry
This closes #3356.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-06 17:07:18 -05:00
Matthew Burgess
c32ea618c5
NIFI-6040: Fixed ExecuteSQL processors when Output Batch Size is set
This closes #3355.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-06 13:44:33 -05:00
thenatog
36bbc77723 NIFI-6097 - Upgraded the fasterxml jackson version to 2.9.8. Ensure that the version is consistent across modules using a maven property defined in the root pom.
This closes #3347

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-04 19:30:35 -05:00
Matthew Burgess
32bd7ed8b4 NIFI-6062: Add support for BLOB, CLOB, NCLOB in record handling
This closes #3329

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-28 08:59:37 -05:00
Matthew Burgess
e5fa18d63c NIFI-6088: Widen type inference for BIGINT and DOUBLE
This closes #3342

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-28 05:51:33 -05:00
Arpad Boda
2a6c3c1686
NIFI-5977 - Add "Minimum/Maximum File Age" Parameter to ListSFTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3324.
2019-02-28 09:15:59 +01:00
Otto Fowler
922da68a23
NIFI-6076 syslog5424 should support missing MSG
- newer lib version and better test

This closes #3331.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-26 09:39:09 -05:00
Joe Percivall
cd7edb1c04 NIFI-5172 Adding the ability to specify a record writer for PutElasticsearchHttpRecord in order to individually handle failed records
Addressing PR feedback

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3299
2019-02-25 17:15:39 -05:00
Bryan Bende
be6cd4f0f2
NIFI-6069 Fixing checkstyle error 2019-02-25 16:52:41 -05:00
Mike Thomsen
b0a93b473b
NIFI-5980 Added HBase_1_1_2_ListLookupService.
NIFI-5980 Added HBase_2_ListLookupService.

This closes #3278.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-25 16:42:11 -05:00
Bryan Bende
4b45e85327 NIFI-6055 Improving error handling during auto-loading of NARs and ensuring only one Jetty NAR is loaded
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api

This closes #3322.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:19:12 -05:00
Bryan Bende
26c00f1080 NIFI-6069 Fixing fingerprinting of connection relationships to be properly sorted
This closes #3325.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:13:13 -05:00
Mark Payne
414bfc94fc NIFI-6068: Fixed unit test 2019-02-22 11:55:25 -05:00
Mark Payne
e26bacf697 NIFI-6068, NIFI-6065: Updated StandardFunnel to avoid looping indefinitely and instead transfer no more than 10,000 FlowFiles before returning from onTrigger. Updated Local Ports to behavior in the same way. Updated Root Group Ports so that instead of blocking for up to 100 milliseconds for an incoming request, it blocks for up to 1 millisecond and if nothing is available yields for the 'bored yield duration'
Signed-off-by: Brandon Devries <devriesb@apache.org>

This closes #3328.
2019-02-22 10:12:20 -05:00
Pierre Villard
2336044b90 NIFI-6059 - Fixed NPE in GCS processors when project ID is not specified
This closes #3327.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 09:45:13 -05:00
Mark Bean
a0c28ee19e NIFI-5660: JMSPublisher should set some header properties in JmsTemplate instead of directly in the message
NIFI-5660: Added lines to integration test to verify these header properties (added by Mike Moser)

Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #3053
2019-02-21 14:45:04 -05:00
Jan Hentschel
60b00a1e38 NIFI-2738 Removed references to nifi.cluster.request.replication.claim.timeout
This closes #2860

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-21 06:34:27 -05:00
joewitt
25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay
76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
Wietze
d59a07245f NIFI-5500 Adding support for array fields when outputting to attributes when using QueryElasticsearchHttp
NIFI-5500 Optimised array-to-string conversion

This closes #2942

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 12:41:57 -05:00
Corey Fritz
af6039a5b0 NIFI-6045: Added support EL support to dyanmic properties for Kafka 2.0 processors
This closes #3316

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 10:34:43 -05:00
Mike Thomsen
53ca9eb7e5 NIFI-5987 Fixed issue where an invalid query pulled from an attribute would cause GetMongo to not route to failure.
Signed-off-by: zenfenan <zenfenan@apache.org>

This closes #3285
2019-02-19 16:24:15 +05:30
joewitt
0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt
45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Mark Payne
82f44155f6
NIFI-6044: This closes #3314. Retain the input data's order in the CSV Reader's inferred schema
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-16 20:57:42 -05:00
Aleksandr Salatich
9e0687ab6b
NIFI-6018 Marking GetHTTP and PostHTTP deprecated. Adding info about deprecation to processors descriptions. Fix small mistake in developer-guide.
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.

This closes #3312.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-02-15 14:09:26 -08:00
Jeff Storck
b7ca59f568
NIFI-6023: Deprecated the "Distributed Cache Service" property in ListHDFS
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS

NIFI-6023: Minor update of the "Distributed Cache Service" description

This closes #3313.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 14:12:04 -05:00
Mark Payne
b508d6bfbc
NIFI-6033, NIFI-6034, NIFI-6035, NIFI-6036, NIFI-6037: Fixed bugs that were found during 1.9.0-RC1 validation. If multiple FlowFiles were written to same Content Claim, and a Processor attempted to read two of them wi within a single session, it would seek to the wrong part of the content or else throw a ContentNotFoundException. Updated logic for considering a processor to be 'running' / having 'active threads' if the processor is invalid upon NiFi restart but scheduled to run. Fixed NPE in FreeFormTextWriter. If MergeRecord reaches minimum number of records, flush writer after writing content out so that its minimum size can accurately be checked.
This closes #3309.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 11:59:47 -05:00
Mike Moser
f61947627e
NIFI-6038 Remove scope parameter from OIDC TokenRequest
This closes #3308
2019-02-14 16:43:15 -05:00
Aleksandr Salatich
e598b30d6d NIFI-6000 Catch also IllegalArgumentException in ConvertAvroToORC hive processor. Added support for Avro null types.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3302
2019-02-14 15:08:54 -05:00
Endre Zoltan Kovacs
ae8a794ff0
NIFI-6031: allow OS level socket keep alive checking
This closes #3307.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-14 13:29:00 -05:00
Patrice Freydiere
8a197e5a96 NIFI-4367 Fix on processor for permit deriving script classes from AbstractProcessor or other Records based base classes
Improve UT

Add licence and fix import

improve filtering properties

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2201
2019-02-14 12:19:13 -05:00
Alex Savitsky
e7ae97797e NIFI-5943 support conversions from List to Avro ARRAY and from Map to Avro RECORD
NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)

This closes #3267

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:31:42 -05:00
Alex Savitsky
45b32e3bc1 NIFI-5947 Elasticsearch lookup service compatible with LookupAttribute
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen

This closes #3268.

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:02:29 -05:00
Mike Thomsen
033b2a1940 NIFI-4975 Add GridFS processors
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2546
2019-02-13 16:51:17 -05:00
Jeff Storck
72244d09ff NIFI-5575: Added Hadoop configuration to MockFileSystem for umask tests in PutHDFSTest
Minor update to Permissions umask property description
Minor code cleanup/formatting

This closes #3252
2019-02-13 16:09:12 -05:00
Key Miyauchi
7a763d3c49 NIFI-5575 Make PutHDFS check fs.permissions.umask-mode if property "Permission umask" is empty. 2019-02-13 16:08:57 -05:00
Kevin Doran
2938454ae4
NIFI-6020: Fix NPE in getAccessPoliciesForUser
This closes #3304
2019-02-13 16:03:46 -05:00
Ed
3492313d0b NIFI-5869 Support Reconnection for JMS
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.

improved controller reset on exception

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3261
2019-02-13 14:18:05 -05:00
Dorian Bugeja
da8c8a14a1 NIFI-6017 - ArrayIndexOutOfBounds Load Balancer CorrelationAttributePartitioner
Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-13 08:53:19 -05:00
Matthew Burgess
b74d71b647
NIFI-4886: Add EL support for Webhook URL in PutSlack
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3303.
2019-02-13 13:54:24 +01:00
Mark Payne
82a0434901 NIFI-5903: Allow RecordPath to be used in QueryRecord processor. Also some code cleanup and improvements to the docs
NIFI-5903: Removed TODO comments that were done

NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3223
2019-02-12 16:44:17 -05:00
Bryan Bende
7278a3970d
NIFI-5859 Add nifi-api dependency to nifi-jetty-bundle pom so the NAR Maven plugin will know the api version
Signed-off-by: Kevin Doran <kdoran@apache.org>

This closes #3291.
2019-02-12 11:54:54 -05:00
Kevin Doran
a940ff6343
NIFI-5950: Use temp port names during flow updates
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.

Add a final step to the update process group logic to set the final name on all ports.

This closes #3301.
2019-02-12 11:12:38 -05:00
Mark Payne
3d408f2b30 NIFI-5984: Enabled Kerberos Authentication for PutKudu
This closes #3279
2019-02-11 18:15:16 -05:00
Mark Payne
36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Mark Payne
3a4c3ccbaa
NIFI-6015: This closes #3300. Updated unit test to sleep for 10 milliseconds before adding last FlowFile to queue so that the first and last FlowFiles will have different timestamps
NIFI-6015: Moved call to sleep() up in unit test to ensure that we sleep before creating the flowfile
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-11 12:40:10 -05:00