Commit Graph

5831 Commits

Author SHA1 Message Date
Mike Thomsen 915617dbe7 NIFI-7299 Add basic OAuth2 token provider service that can fetch access tokens when supplied with appropriate credentials.
Added skeleton of oauth2 provider.
Added copy of our code.
Refactored a few things.
Updated apis to better match flow descriptions.
Updated poms and other artifacts.
Updated copyright notice.
Updated LICENSE.

This closes #4173

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
2020-06-04 08:43:27 -04:00
Mark Payne e0dd6d466e NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord
This closes #4223.
2020-06-02 15:13:14 -04:00
Bence Simon 5c2bfcf7d3 NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 15:13:14 -04:00
Peter Gyori 91dd59dbdf NIFI-7312: Enable search in variable registry of root process group
This closes #4303.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 13:25:00 -04:00
Mark Payne e31c323aa7 NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader. 2020-06-02 12:13:51 -04:00
jahenaor 04711ab466
NIFI-7477 Optionally adding validation details as a new attribute of the flowfile
NIFI-7477 Improving description and unit test now verifies attribute content

NIFI-7477: Fixed checkstyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4301
2020-06-01 14:08:08 -04:00
zhangcheng 1ec6675f38
NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266
2020-06-01 11:33:10 -04:00
Pierre Villard 2e45d959ba
Updated KEYS with new key after previous one expired 2020-05-30 22:03:59 +02:00
Niels Basjes 2a82efc417
NIFI-6666 Add Useragent Header to InvokeHTTP requests
This closes #3734

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-29 21:39:14 -04:00
Pierre Villard af58714606 NIFI-7430 - LookupRecord change coordinate key for in-place replacement 2020-05-29 13:54:11 -04:00
Pierre Villard 8162edfd98 NIFI-6701 - Fix for PublishGCPPubSub 2020-05-29 13:38:11 -04:00
Peter Turcsanyi 06864c830c
NIFI-7422: Support aws_s3_pseudo_dir in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4292.
2020-05-29 14:22:58 +02:00
zhangcheng a1f277fdf7
NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4294.
2020-05-29 10:41:19 +02:00
zhangcheng e04e6793ca
NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4293.
2020-05-29 10:39:04 +02:00
r65535 4ced1775b5
NIFI-7487 - Added batch support and displayName to ModifyBytes processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4302.
2020-05-29 10:28:37 +02:00
Peter Gyori 1dd0e92040 NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
Made warning and error messages more informative.
Refactored flowFile assertion in the tests.

This closes #4287.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-27 09:13:41 +02:00
Mike Thomsen aa804cfceb NIFI-7485 Updated commons-configuration2.
NIFI-7485 Found more instances that needed updating.

This closes #4295
2020-05-26 17:11:24 -04:00
Tamas Palfy ca65bba5d7 NIFI-7453 In PutKudu creating a new Kudu client when refreshing TGT
NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.)
NIFI-7453 Safely closing old Kudu client before creating a new one.
NIFI-7453 Visibility adjustment.

This closes #4276.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-26 18:01:08 +02:00
adyoun2 101387bfaa
NIFI-6785 Support Deflate Compression
NIFI-6785 Remove unused imports

This closes #3822

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 16:22:08 -04:00
Mike Thomsen 443f969d36
NIFI-7211 Added @Ignore with warning message to a test that randomly fails due to timing issues.
This closes #4296
2020-05-25 10:24:38 -04:00
Alessandro D'Armiento 788f8b0389
NIFI-6674 MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

fixed mispositioned # in doc

This closes #3740

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:06:22 -04:00
Alessandro D'Armiento 1ba8f76a44
NIFI-6673 MultiplyEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

This closes #3739

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:01:02 -04:00
Alessandro D'Armiento b025117824
NIFI-6672 PlusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
NIFI-6672 Removed test code.

This closes #3738

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:58:15 -04:00
Phillip Grenier 0f4b79b55e
NIFI-6255 NIFI-6287: Hash function for expression language and record path.
NIFI-6255 NIFI-6287: Rebased to match the new expression language interface
NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports
NIFI-6255 NIFI-6287: Move to the common codec DigetUtils
Update commons-codec

This closes #3624

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:23:00 -04:00
Andy LoPresto 45e626f6a8
Updated pull request template to separate JDK 8 and 11 questions 2020-05-22 12:19:58 -07:00
Andy LoPresto 97a919a3be
NIFI-7482 Changed InvokeHTTP to be extensible.
Added unit test.

This closes #4291.

Signed-off-by: Arpad Boda <aboda@apache.org>
2020-05-22 11:44:53 -07:00
pcgrenier b6ef7e13bf
NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions
NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types

NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4282
2020-05-21 13:13:12 -04:00
eduardofontes c7edcd68e1
NIFI-7463
Create empty relationship for RunMongoAggregation

Fix default autoterminate and condition to redirect to REL_EMPTY

Change from new relationship to write an empty FlowFile to RESULT

Fix MONGO_URI

This closes #4281

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-20 19:54:24 -04:00
Mark Payne d195702ee2 Fixed a couple of typos in the RecordPath guide 2020-05-20 17:09:40 -04:00
Pierre Villard a9e9e5d137 NIFI-6571 Check token length on TLS toolkit server startup
This closes #3659.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-20 15:17:39 -05:00
KovalevIV 7034d7e44c NIFI-7471 fix bug with property validation 2020-05-20 10:49:49 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
Sushil Kumar 5826a09a83 NIFI-6911 Removed default Blob value for PutAzureBlobStorage
This closes #3906

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-19 13:47:08 -05:00
Peter Gyori d6240a1074 NIFI-7336: Add tests for DeleteAzureDataLakeStorage
DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string

NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case

This closes #4272.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 18:35:33 +02:00
sjyang18 852715aadd NIFI-7409: Azure managed identity support to Azure Datalake processors
NIFI-7409: review changes
NIFI-7409: ordering import statements
NIFI-7409: changed validateCredentialProperties logic

This closes #4249.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 11:50:08 +02:00
mtien 3fec4d8c27
NIFI-7331 Fixed grammatical errors in log output.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.
2020-05-19 10:38:45 +02:00
Peter Gyori 9aae58f117 NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory
A newer version (12.1.1) of azure-storage-file-datalake is imported.

This closes #4273.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-18 23:50:05 +02:00
Matthieu Cauffiez 179675f0b4 NIFI-7380 - fix for controller service validation in NiFi Stateless
This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 16:11:40 -04:00
Matthew Burgess a3cc2c58ff NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes
This closes #4275.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 15:11:33 -04:00
Yolanda M. Davis 13418ccb91
NIFI-7437 - created separate thread for preloading predictions, refactors for performance
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274
2020-05-18 14:57:54 -04:00
Mark Payne c51b9051a8 NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows 2020-05-15 16:16:39 -04:00
Matthew Burgess 53a161234e
NIFI-7448: Fix quoting of DDL table name in PutORC
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4269.
2020-05-14 11:41:35 +02:00
Peter Gyori 9608fe33fa NIFI-7367: Add tests for FetchAzureDataLakeStorage
NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.

This closes #4257.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-13 14:50:31 +02:00
Nathan Gough 302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
Peter Gyori 6b73ea48bc
NIFI-7413: Documented REMOTE_INVOCATION provenance event type in user/dev guides
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4267
2020-05-12 16:00:41 -04:00
Sushil Kumar 851359c1fc
NIFI-6913: PutAzureBlobStorage processor will create container if not exists
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4237.
2020-05-09 15:37:14 +02:00
mtien 647a9a60b7
NIFI-7423 Upgraded jquery dependency version.
NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258
2020-05-08 13:13:38 -04:00
Bence Simon 66b175f405
NIFI-7390 Covering Avro type conversion in case of map withing Record
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256
2020-05-08 13:11:19 -04:00
mtien 6e2f86d716 NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml.
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4252.
2020-05-07 14:19:55 -04:00
Matthew Burgess b7c81d6007
NIFI-7428: Switch hive.version property to set Hive 3 version
This closes #4259
2020-05-06 19:38:23 -04:00