Commit Graph

5158 Commits

Author SHA1 Message Date
Mark Payne 82f44155f6
NIFI-6044: This closes #3314. Retain the input data's order in the CSV Reader's inferred schema
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-16 20:57:42 -05:00
Aleksandr Salatich 9e0687ab6b
NIFI-6018 Marking GetHTTP and PostHTTP deprecated. Adding info about deprecation to processors descriptions. Fix small mistake in developer-guide.
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.

This closes #3312.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-02-15 14:09:26 -08:00
Jeff Storck b7ca59f568
NIFI-6023: Deprecated the "Distributed Cache Service" property in ListHDFS
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS

NIFI-6023: Minor update of the "Distributed Cache Service" description

This closes #3313.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 14:12:04 -05:00
Mark Payne 319979f256
NIFI-6039: Avoid NPE if Record Writer does not provide schema
This closes #3310.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 13:06:30 -05:00
Mark Payne b508d6bfbc
NIFI-6033, NIFI-6034, NIFI-6035, NIFI-6036, NIFI-6037: Fixed bugs that were found during 1.9.0-RC1 validation. If multiple FlowFiles were written to same Content Claim, and a Processor attempted to read two of them wi within a single session, it would seek to the wrong part of the content or else throw a ContentNotFoundException. Updated logic for considering a processor to be 'running' / having 'active threads' if the processor is invalid upon NiFi restart but scheduled to run. Fixed NPE in FreeFormTextWriter. If MergeRecord reaches minimum number of records, flush writer after writing content out so that its minimum size can accurately be checked.
This closes #3309.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 11:59:47 -05:00
Mike Moser f61947627e
NIFI-6038 Remove scope parameter from OIDC TokenRequest
This closes #3308
2019-02-14 16:43:15 -05:00
Aleksandr Salatich e598b30d6d NIFI-6000 Catch also IllegalArgumentException in ConvertAvroToORC hive processor. Added support for Avro null types.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3302
2019-02-14 15:08:54 -05:00
Endre Zoltan Kovacs ae8a794ff0
NIFI-6031: allow OS level socket keep alive checking
This closes #3307.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-14 13:29:00 -05:00
Patrice Freydiere 8a197e5a96 NIFI-4367 Fix on processor for permit deriving script classes from AbstractProcessor or other Records based base classes
Improve UT

Add licence and fix import

improve filtering properties

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2201
2019-02-14 12:19:13 -05:00
Alex Savitsky e7ae97797e NIFI-5943 support conversions from List to Avro ARRAY and from Map to Avro RECORD
NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)

This closes #3267

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:31:42 -05:00
Alex Savitsky 45b32e3bc1 NIFI-5947 Elasticsearch lookup service compatible with LookupAttribute
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen

This closes #3268.

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-14 08:02:29 -05:00
Mike Thomsen 033b2a1940 NIFI-4975 Add GridFS processors
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2546
2019-02-13 16:51:17 -05:00
Jeff Storck 72244d09ff NIFI-5575: Added Hadoop configuration to MockFileSystem for umask tests in PutHDFSTest
Minor update to Permissions umask property description
Minor code cleanup/formatting

This closes #3252
2019-02-13 16:09:12 -05:00
Key Miyauchi 7a763d3c49 NIFI-5575 Make PutHDFS check fs.permissions.umask-mode if property "Permission umask" is empty. 2019-02-13 16:08:57 -05:00
Kevin Doran 2938454ae4
NIFI-6020: Fix NPE in getAccessPoliciesForUser
This closes #3304
2019-02-13 16:03:46 -05:00
Ed 3492313d0b NIFI-5869 Support Reconnection for JMS
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.

improved controller reset on exception

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3261
2019-02-13 14:18:05 -05:00
Dorian Bugeja da8c8a14a1 NIFI-6017 - ArrayIndexOutOfBounds Load Balancer CorrelationAttributePartitioner
Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-13 08:53:19 -05:00
Matthew Burgess b74d71b647
NIFI-4886: Add EL support for Webhook URL in PutSlack
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3303.
2019-02-13 13:54:24 +01:00
Mark Payne 82a0434901 NIFI-5903: Allow RecordPath to be used in QueryRecord processor. Also some code cleanup and improvements to the docs
NIFI-5903: Removed TODO comments that were done

NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3223
2019-02-12 16:44:17 -05:00
Bryan Bende 7278a3970d
NIFI-5859 Add nifi-api dependency to nifi-jetty-bundle pom so the NAR Maven plugin will know the api version
Signed-off-by: Kevin Doran <kdoran@apache.org>

This closes #3291.
2019-02-12 11:54:54 -05:00
Kevin Doran a940ff6343
NIFI-5950: Use temp port names during flow updates
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.

Add a final step to the update process group logic to set the final name on all ports.

This closes #3301.
2019-02-12 11:12:38 -05:00
Mark Payne 3d408f2b30 NIFI-5984: Enabled Kerberos Authentication for PutKudu
This closes #3279
2019-02-11 18:15:16 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Mark Payne 3a4c3ccbaa
NIFI-6015: This closes #3300. Updated unit test to sleep for 10 milliseconds before adding last FlowFile to queue so that the first and last FlowFiles will have different timestamps
NIFI-6015: Moved call to sleep() up in unit test to ensure that we sleep before creating the flowfile
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-11 12:40:10 -05:00
Jeff Storck 8c8a9b4d53
NIFI-5985: Added capability for DBCPConnectionPool to use KerberosCredentialsService.
Refactored KerberosAction to return a result from execute()
Removed usage of ProcessContext.yield() from KerberosAction, which should instead be handled the component using the KerberosCredentialsService.
Updated SolrProcessor to yield a flowfile on error, rather than the KerberosAction invoking the yield.

NIFI-5985: Updated TestPutSolrContentStream.testUpdateWithKerberosAuth test case to match on PrivilegedExceptionAction instead of PrivilegedAction doAs arguments.

NIFI-5985: Moved kerberosUser logout after closing the datasource in the shutdown method.

NIFI-5985: Removed catching exceptions in DBCPConnectionPool.shutdown
Exception when closing the datasource is prioritized over an exception when logging out the kerberos principal
Added GroovyDBCPServiceTest tests to verify prioritizing datasource.close() exception over kerberosUser.logout() exception

This closes #3288.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-11 09:45:55 -05:00
Mark Payne bc0e68ff2a NIFI-5989: Update to property names to clarify difference between batch sizes 2019-02-08 15:58:43 -05:00
Alex Goos 29d54126e4 NIFI-5989 - PutKudu: Additional FF Queue length setting
Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-08 15:41:02 -05:00
Mark Payne 4e914cea1f
NIFI-6008: When a Processor is Terminated, call its #onPropertyModified method for properties whose value differs from the default value. Previously, it was called if the value differs from the current value (which it never will since we are only restoring the current values for the Processor). This results in ensuring that #onPr #onPropertyModified is correctly called when a component is reloaded
This closes #3296.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-08 10:52:04 -05:00
Jeff Storck e4be06c7c8 NIFI-5967: Removed unused ALLOW_EXPLICIT_KEYTAB private variable
Updated Hive_1_1ConnectionPoolTest.testKerberosAuthException to use a mocked KerberosCredentialsService

This closes #3272
2019-02-08 00:22:56 -05:00
Matthew Burgess cdf3c69208 NIFI-5967: Add Hive 1.1 processors
Removed extra Kerberos properties, added LICENSEs to all Hive NARs, removed unnecessary NOTICE entries
2019-02-08 00:11:38 -05:00
Bryan Bende 8c58d51857
NIFI-5990 This closes #3283. Moving Developer docs section above Processors
Signed-off-by: joewitt <joewitt@apache.org>
2019-02-06 21:04:21 -05:00
thenatog f81d6bd63b
NIFI-5968 - Added the X-XSS-Protection and Strict-Transport-Security HTTP headers using Jetty Filters. Added some tests.
Removed bad test.
Refactored filter creation method.
Ensure HSTS header is only applied if NiFi is secured with HTTPS
Small changes to header array list.
Fixed checkstyle errors.

This closes #3273.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-02-07 11:13:28 +11:00
Endre Zoltan Kovacs 24a7d480c8 NIFI-5983: handling parse problems in recordReader implementations
Fixed Checkstyle violation

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3282
2019-02-06 15:54:15 -05:00
Peter Wicks 35147a620f NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is Disabled
This closes #3255

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-06 12:42:43 -05:00
Mark Payne 412c4908e2
NIFI-5997: Recover FlowFile Repository before swap files; then, when recovering swap files, ignore any that are unknown to the flowfile repo. This prevents us from incrementing the size of the flowfile queue for unknown swap files
This closes #3292.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-05 15:23:24 -05:00
Mark Payne 83ac191736
NIFI-5997: If we swap out data, ensure that we do not increment the size of the queue by the size of the data that we failed to swap out. Also, if the FlowFile Repo does not know about a given swap file, do not restore it on restart
This closes #3290.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-04 14:29:17 -05:00
Sujesh Menon 2c2b47ab85
NIFI-5991 Added WebSockets schemes to the broker URI Validator
MQTT over websockets uses ws (and wss for secure websocket connections) schemes in the broker URI.
Added support for the same.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3289.
2019-02-04 15:53:31 +01:00
Peter Wicks ad1f2fb666
NIFI-5722 Expose Penalty Remaining Duration (#3091)
Signed-off-by: Koji Kawamura <ijokarumawak@gmail.com>
2019-02-01 13:04:51 -07:00
Mike Thomsen fef41b3022 NIFI-5995 Updated ScriptedLookupService documentation to warn about Jython and removed Jython from the list of supported script engines for it because it's broken now.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3287
2019-02-01 13:33:33 -05:00
Bryan Bende 79a3696e5e NIFI-5961 Fixing bug where the name of a versioned process group gets incorrectly reset
This closes #3271.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-01-30 11:14:53 -05:00
Pierre Villard 8a3d7a453a NIFI-5867 - add thread() EL function to get thread name
moved thread name evaluation in evaluate method

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3198
2019-01-30 10:14:35 -05:00
a.durov 8e777203a0 NIFI-5974 fix: Fragment Attributes are populated in case no split has occured.
Unit test is implemented: testNoSplitterInString

NIFI-5974: Fixed Checkstyle violations

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3275
2019-01-30 09:40:36 -05:00
Bryan Bende 2eac0e96c7
NIFI-5962 protecting against null Configuration in AbstractHadoopProcessor onStopped
This closes #3270.
2019-01-28 13:13:48 -05:00
Matt Gilman 531c84ac31
NIFI-5972: Converting some warning log messages to debug as they could possibly be due to a valid scenario like NiFi users belonging to a group that is not relevant to NiFi.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3274.
2019-01-25 15:00:12 +01:00
Bryan Bende 410c9a4ecd NIFI-4915 Adding HBase 2.x service bundle
Thise closes #3254

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-01-25 07:19:54 -05:00
Mark Payne fd3d69bc90 NIFI-5949: Addressed problematic error handling logic in PutKudu. Also removed AbstractKudu and moved all logic into PutKudu, as the abstract class was clearly designed to be extended only by PutKudu and no other processors exist for Kudu
Added explicit .toString() call on rowError to avoid IllegalStateException during logging of row errors

This closes #3262.
2019-01-24 19:47:03 -05:00
erichanson5 4ef2251d74 NIFI-3611: Added ability to set Transaction Isolation Level on Database connection for the QueryDatabaseTable processor
NIFI-3611: Make TRANS_ISOLATION_LEVEL property optional

This closes #3248.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-01-21 17:19:42 +09:00
Koji Kawamura 05de73d6a0 NIFI-5951 Fix error logging with rollback on failure
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3264
2019-01-17 17:11:39 -05:00
Mark Payne 89adb039d2
NIFI-5944: If processor is no longer scheduled to run after a failed trigger of @OnScheduled, ensure that we set scheduled status to STOPPED
This closes #3263
2019-01-11 10:22:22 -05:00
Bryan Bende 2bbfb3217b NIFI-5945 Add support for password login to kerberos code in nifi-security-utils
Fixing solr test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3256
2019-01-10 15:05:31 -05:00