Commit Graph

5178 Commits

Author SHA1 Message Date
Bryan Bende be6cd4f0f2
NIFI-6069 Fixing checkstyle error 2019-02-25 16:52:41 -05:00
Mike Thomsen b0a93b473b
NIFI-5980 Added HBase_1_1_2_ListLookupService.
NIFI-5980 Added HBase_2_ListLookupService.

This closes #3278.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-25 16:42:11 -05:00
Bryan Bende 4b45e85327 NIFI-6055 Improving error handling during auto-loading of NARs and ensuring only one Jetty NAR is loaded
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api

This closes #3322.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:19:12 -05:00
Bryan Bende 26c00f1080 NIFI-6069 Fixing fingerprinting of connection relationships to be properly sorted
This closes #3325.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:13:13 -05:00
Mark Payne 414bfc94fc NIFI-6068: Fixed unit test 2019-02-22 11:55:25 -05:00
Mark Payne e26bacf697 NIFI-6068, NIFI-6065: Updated StandardFunnel to avoid looping indefinitely and instead transfer no more than 10,000 FlowFiles before returning from onTrigger. Updated Local Ports to behavior in the same way. Updated Root Group Ports so that instead of blocking for up to 100 milliseconds for an incoming request, it blocks for up to 1 millisecond and if nothing is available yields for the 'bored yield duration'
Signed-off-by: Brandon Devries <devriesb@apache.org>

This closes #3328.
2019-02-22 10:12:20 -05:00
Pierre Villard 2336044b90 NIFI-6059 - Fixed NPE in GCS processors when project ID is not specified
This closes #3327.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 09:45:13 -05:00
Mark Bean a0c28ee19e NIFI-5660: JMSPublisher should set some header properties in JmsTemplate instead of directly in the message
NIFI-5660: Added lines to integration test to verify these header properties (added by Mike Moser)

Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #3053
2019-02-21 14:45:04 -05:00
Joe Skora 270b6d5897 NIFI-4166 Create nifi-toolkit-api.
NIFI-4166 Adjusted dependencies
NIFI-4166 Excluded RAT and checkstyle for generated-sources. (Added by Mike Thomsen)

This closes #1994

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-21 07:09:25 -05:00
Jan Hentschel 60b00a1e38 NIFI-2738 Removed references to nifi.cluster.request.replication.claim.timeout
This closes #2860

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-21 06:34:27 -05:00
Aldrin Piri 0ec69efe9a Bump Docker Maven version for docker-compose file to 1.10
This closes #3323

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-20 20:49:06 -05:00
Craig Davidson ed21d02fa7 Added ability to configure JVM Memory settings via Docker ENV variables
This closes #3321

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-20 20:46:30 -05:00
joewitt d1388bbd7f
NIFI-6029 updated dockerfile as per release guide 2019-02-19 23:17:48 -05:00
joewitt 25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay 76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
Wietze d59a07245f NIFI-5500 Adding support for array fields when outputting to attributes when using QueryElasticsearchHttp
NIFI-5500 Optimised array-to-string conversion

This closes #2942

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 12:41:57 -05:00
Corey Fritz af6039a5b0 NIFI-6045: Added support EL support to dyanmic properties for Kafka 2.0 processors
This closes #3316

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 10:34:43 -05:00
Mike Thomsen 53ca9eb7e5 NIFI-5987 Fixed issue where an invalid query pulled from an attribute would cause GetMongo to not route to failure.
Signed-off-by: zenfenan <zenfenan@apache.org>

This closes #3285
2019-02-19 16:24:15 +05:30
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Mark Payne 82f44155f6
NIFI-6044: This closes #3314. Retain the input data's order in the CSV Reader's inferred schema
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-16 20:57:42 -05:00
Aleksandr Salatich 9e0687ab6b
NIFI-6018 Marking GetHTTP and PostHTTP deprecated. Adding info about deprecation to processors descriptions. Fix small mistake in developer-guide.
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.

This closes #3312.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-02-15 14:09:26 -08:00
Jeff Storck b7ca59f568
NIFI-6023: Deprecated the "Distributed Cache Service" property in ListHDFS
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS

NIFI-6023: Minor update of the "Distributed Cache Service" description

This closes #3313.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 14:12:04 -05:00
Mark Payne 319979f256
NIFI-6039: Avoid NPE if Record Writer does not provide schema
This closes #3310.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 13:06:30 -05:00
Mark Payne b508d6bfbc
NIFI-6033, NIFI-6034, NIFI-6035, NIFI-6036, NIFI-6037: Fixed bugs that were found during 1.9.0-RC1 validation. If multiple FlowFiles were written to same Content Claim, and a Processor attempted to read two of them wi within a single session, it would seek to the wrong part of the content or else throw a ContentNotFoundException. Updated logic for considering a processor to be 'running' / having 'active threads' if the processor is invalid upon NiFi restart but scheduled to run. Fixed NPE in FreeFormTextWriter. If MergeRecord reaches minimum number of records, flush writer after writing content out so that its minimum size can accurately be checked.
This closes #3309.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 11:59:47 -05:00
Mike Moser f61947627e
NIFI-6038 Remove scope parameter from OIDC TokenRequest
This closes #3308
2019-02-14 16:43:15 -05:00
Aleksandr Salatich e598b30d6d NIFI-6000 Catch also IllegalArgumentException in ConvertAvroToORC hive processor. Added support for Avro null types.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3302
2019-02-14 15:08:54 -05:00
Endre Zoltan Kovacs ae8a794ff0
NIFI-6031: allow OS level socket keep alive checking
This closes #3307.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-14 13:29:00 -05:00
Patrice Freydiere 8a197e5a96 NIFI-4367 Fix on processor for permit deriving script classes from AbstractProcessor or other Records based base classes
Improve UT

Add licence and fix import

improve filtering properties

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2201
2019-02-14 12:19:13 -05:00
Alex Savitsky e7ae97797e NIFI-5943 support conversions from List to Avro ARRAY and from Map to Avro RECORD
NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)

This closes #3267

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:31:42 -05:00
Alex Savitsky 45b32e3bc1 NIFI-5947 Elasticsearch lookup service compatible with LookupAttribute
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen

This closes #3268.

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:02:29 -05:00
Mike Thomsen 033b2a1940 NIFI-4975 Add GridFS processors
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2546
2019-02-13 16:51:17 -05:00
Jeff Storck 72244d09ff NIFI-5575: Added Hadoop configuration to MockFileSystem for umask tests in PutHDFSTest
Minor update to Permissions umask property description
Minor code cleanup/formatting

This closes #3252
2019-02-13 16:09:12 -05:00
Key Miyauchi 7a763d3c49 NIFI-5575 Make PutHDFS check fs.permissions.umask-mode if property "Permission umask" is empty. 2019-02-13 16:08:57 -05:00
Kevin Doran 2938454ae4
NIFI-6020: Fix NPE in getAccessPoliciesForUser
This closes #3304
2019-02-13 16:03:46 -05:00
Ed 3492313d0b NIFI-5869 Support Reconnection for JMS
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.

improved controller reset on exception

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3261
2019-02-13 14:18:05 -05:00
Dorian Bugeja da8c8a14a1 NIFI-6017 - ArrayIndexOutOfBounds Load Balancer CorrelationAttributePartitioner
Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-13 08:53:19 -05:00
Matthew Burgess b74d71b647
NIFI-4886: Add EL support for Webhook URL in PutSlack
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3303.
2019-02-13 13:54:24 +01:00
Mark Payne 82a0434901 NIFI-5903: Allow RecordPath to be used in QueryRecord processor. Also some code cleanup and improvements to the docs
NIFI-5903: Removed TODO comments that were done

NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3223
2019-02-12 16:44:17 -05:00
Bryan Bende 7278a3970d
NIFI-5859 Add nifi-api dependency to nifi-jetty-bundle pom so the NAR Maven plugin will know the api version
Signed-off-by: Kevin Doran <kdoran@apache.org>

This closes #3291.
2019-02-12 11:54:54 -05:00
Kevin Doran a940ff6343
NIFI-5950: Use temp port names during flow updates
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.

Add a final step to the update process group logic to set the final name on all ports.

This closes #3301.
2019-02-12 11:12:38 -05:00
Mark Payne 3d408f2b30 NIFI-5984: Enabled Kerberos Authentication for PutKudu
This closes #3279
2019-02-11 18:15:16 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Mark Payne 3a4c3ccbaa
NIFI-6015: This closes #3300. Updated unit test to sleep for 10 milliseconds before adding last FlowFile to queue so that the first and last FlowFiles will have different timestamps
NIFI-6015: Moved call to sleep() up in unit test to ensure that we sleep before creating the flowfile
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-11 12:40:10 -05:00
Jeff Storck 8c8a9b4d53
NIFI-5985: Added capability for DBCPConnectionPool to use KerberosCredentialsService.
Refactored KerberosAction to return a result from execute()
Removed usage of ProcessContext.yield() from KerberosAction, which should instead be handled the component using the KerberosCredentialsService.
Updated SolrProcessor to yield a flowfile on error, rather than the KerberosAction invoking the yield.

NIFI-5985: Updated TestPutSolrContentStream.testUpdateWithKerberosAuth test case to match on PrivilegedExceptionAction instead of PrivilegedAction doAs arguments.

NIFI-5985: Moved kerberosUser logout after closing the datasource in the shutdown method.

NIFI-5985: Removed catching exceptions in DBCPConnectionPool.shutdown
Exception when closing the datasource is prioritized over an exception when logging out the kerberos principal
Added GroovyDBCPServiceTest tests to verify prioritizing datasource.close() exception over kerberosUser.logout() exception

This closes #3288.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-11 09:45:55 -05:00
Mark Payne bc0e68ff2a NIFI-5989: Update to property names to clarify difference between batch sizes 2019-02-08 15:58:43 -05:00
Alex Goos 29d54126e4 NIFI-5989 - PutKudu: Additional FF Queue length setting
Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-08 15:41:02 -05:00
Mark Payne 4e914cea1f
NIFI-6008: When a Processor is Terminated, call its #onPropertyModified method for properties whose value differs from the default value. Previously, it was called if the value differs from the current value (which it never will since we are only restoring the current values for the Processor). This results in ensuring that #onPr #onPropertyModified is correctly called when a component is reloaded
This closes #3296.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-08 10:52:04 -05:00
Jeff Storck e4be06c7c8 NIFI-5967: Removed unused ALLOW_EXPLICIT_KEYTAB private variable
Updated Hive_1_1ConnectionPoolTest.testKerberosAuthException to use a mocked KerberosCredentialsService

This closes #3272
2019-02-08 00:22:56 -05:00
Matthew Burgess cdf3c69208 NIFI-5967: Add Hive 1.1 processors
Removed extra Kerberos properties, added LICENSEs to all Hive NARs, removed unnecessary NOTICE entries
2019-02-08 00:11:38 -05:00