Commit Graph

5947 Commits

Author SHA1 Message Date
Mark Payne 8b1a23a99c NIFI-7539: When capturing diagnostics information, capture a thread dump once and then provide this information to ProcessorNode when capturing active threads. Previously, each processor captured a thread dump itself. When this is done thousands of times it can result in a very long delay. 2020-06-16 14:48:14 -04:00
Макаров Василий Николаевич 9828e7dd14 NIFI-7035 The first curator connection issue is logged as ERROR until reconnect 2020-06-16 12:24:41 -04:00
Tamas Palfy e02ffdd99f NIFI-7527 AbstractKuduProcessorrefresh TGT deadlock fix: Redesigned locking.
NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set).
NIFI-7527 Refactor: removed redundant kudu client creation.

This closes #4330.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-06-15 14:17:34 +02:00
Tamas Palfy a126d0a6b6
NIFI-7442 Added missing use cases (list users and user groups), made update-access-policy use case more in line with the NiFi side. Added some tests. Additional refactor, documentation revision.
This closes #4329.
2020-06-12 12:43:41 -04:00
Bence Simon d99983bdc7
NIFI-7442 Add CLI commands to the registry in order to support automatic registry setup 2020-06-12 12:43:27 -04:00
sjyang18 73008451d4 NIFI-7434: Add endpoint suffix to Azure storage processors
This closes #4265.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-06-10 09:45:04 -05:00
wanghq-c 3c757b6ba8
NIFI-7313:fix bug on 'Quote Table Identifiers'
NIFI-7313:add test by wanghongqi
NIFI-7313:edit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4185
2020-06-09 16:02:25 -04:00
Pierre Villard a2f521297f
NIFI-7490 - Add optional raw field to Syslog readers
review

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4299
2020-06-09 15:37:08 -04:00
Matt Gilman eb2ebefc46
NIFI-7514:
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.

This closes #4322
2020-06-09 15:23:58 -04:00
Nathan Gough 01e42dfb32
NIFI-7385 Provided reverse-indexed TokenCache implementation.
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #4271.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-08 19:36:14 -07:00
Tamas Palfy aa7c5e2178 NIFI-7511 In ControllerServiceProxyWrapper extended documentation. Minor refactor in StandardControllerServiceInvocationHandler. Also removed an unused import from NiFiSystemIT.
This closes #4317.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-08 16:04:15 -04:00
Mark Payne 463d72117b
NIFI-7507: Added section to User Guide on configuring a Process Group
NIFI-7507: Fixed Flowfile Expiration header in doc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4318
2020-06-08 14:03:00 -04:00
Wouter de Vries f2368a0dd1 NIFI-7393: Add max idle time and idle connections to InvokeHTTP
This closes #4233.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-06-08 09:29:10 -05:00
Andy LoPresto 845b66ab92
NIFI-7467 Refactored S2S peer selection logic.
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.

This closes #4289.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-05 13:40:16 -07:00
Joey Frazee dfefeb7b18 NIFI-7508: Reset classloader after running TestStandardControllerServiceInvocationHandler
and fix checkstyle violation on NiFiSystemIT
2020-06-05 16:23:01 -04:00
Mark Payne 359fd3ff29
NIFI-7476: Implemented FlowFileGating / FlowFileConcurrency at the ProcessGroup level
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4306.
2020-06-04 23:24:03 +02:00
Mike Thomsen 915617dbe7 NIFI-7299 Add basic OAuth2 token provider service that can fetch access tokens when supplied with appropriate credentials.
Added skeleton of oauth2 provider.
Added copy of our code.
Refactored a few things.
Updated apis to better match flow descriptions.
Updated poms and other artifacts.
Updated copyright notice.
Updated LICENSE.

This closes #4173

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
2020-06-04 08:43:27 -04:00
Mark Payne e0dd6d466e NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord
This closes #4223.
2020-06-02 15:13:14 -04:00
Bence Simon 5c2bfcf7d3 NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 15:13:14 -04:00
Peter Gyori 91dd59dbdf NIFI-7312: Enable search in variable registry of root process group
This closes #4303.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 13:25:00 -04:00
Mark Payne e31c323aa7 NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader. 2020-06-02 12:13:51 -04:00
jahenaor 04711ab466
NIFI-7477 Optionally adding validation details as a new attribute of the flowfile
NIFI-7477 Improving description and unit test now verifies attribute content

NIFI-7477: Fixed checkstyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4301
2020-06-01 14:08:08 -04:00
zhangcheng 1ec6675f38
NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266
2020-06-01 11:33:10 -04:00
Pierre Villard 2e45d959ba
Updated KEYS with new key after previous one expired 2020-05-30 22:03:59 +02:00
Niels Basjes 2a82efc417
NIFI-6666 Add Useragent Header to InvokeHTTP requests
This closes #3734

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-29 21:39:14 -04:00
Pierre Villard af58714606 NIFI-7430 - LookupRecord change coordinate key for in-place replacement 2020-05-29 13:54:11 -04:00
Pierre Villard 8162edfd98 NIFI-6701 - Fix for PublishGCPPubSub 2020-05-29 13:38:11 -04:00
Peter Turcsanyi 06864c830c
NIFI-7422: Support aws_s3_pseudo_dir in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4292.
2020-05-29 14:22:58 +02:00
zhangcheng a1f277fdf7
NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4294.
2020-05-29 10:41:19 +02:00
zhangcheng e04e6793ca
NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4293.
2020-05-29 10:39:04 +02:00
r65535 4ced1775b5
NIFI-7487 - Added batch support and displayName to ModifyBytes processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4302.
2020-05-29 10:28:37 +02:00
Peter Gyori 1dd0e92040 NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
Made warning and error messages more informative.
Refactored flowFile assertion in the tests.

This closes #4287.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-27 09:13:41 +02:00
Mike Thomsen aa804cfceb NIFI-7485 Updated commons-configuration2.
NIFI-7485 Found more instances that needed updating.

This closes #4295
2020-05-26 17:11:24 -04:00
Tamas Palfy ca65bba5d7 NIFI-7453 In PutKudu creating a new Kudu client when refreshing TGT
NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.)
NIFI-7453 Safely closing old Kudu client before creating a new one.
NIFI-7453 Visibility adjustment.

This closes #4276.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-26 18:01:08 +02:00
adyoun2 101387bfaa
NIFI-6785 Support Deflate Compression
NIFI-6785 Remove unused imports

This closes #3822

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 16:22:08 -04:00
Mike Thomsen 443f969d36
NIFI-7211 Added @Ignore with warning message to a test that randomly fails due to timing issues.
This closes #4296
2020-05-25 10:24:38 -04:00
Alessandro D'Armiento 788f8b0389
NIFI-6674 MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

fixed mispositioned # in doc

This closes #3740

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:06:22 -04:00
Alessandro D'Armiento 1ba8f76a44
NIFI-6673 MultiplyEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

This closes #3739

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 07:01:02 -04:00
Alessandro D'Armiento b025117824
NIFI-6672 PlusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
NIFI-6672 Removed test code.

This closes #3738

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:58:15 -04:00
Phillip Grenier 0f4b79b55e
NIFI-6255 NIFI-6287: Hash function for expression language and record path.
NIFI-6255 NIFI-6287: Rebased to match the new expression language interface
NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports
NIFI-6255 NIFI-6287: Move to the common codec DigetUtils
Update commons-codec

This closes #3624

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 06:23:00 -04:00
Andy LoPresto 45e626f6a8
Updated pull request template to separate JDK 8 and 11 questions 2020-05-22 12:19:58 -07:00
Andy LoPresto 97a919a3be
NIFI-7482 Changed InvokeHTTP to be extensible.
Added unit test.

This closes #4291.

Signed-off-by: Arpad Boda <aboda@apache.org>
2020-05-22 11:44:53 -07:00
pcgrenier b6ef7e13bf
NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions
NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types

NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4282
2020-05-21 13:13:12 -04:00
eduardofontes c7edcd68e1
NIFI-7463
Create empty relationship for RunMongoAggregation

Fix default autoterminate and condition to redirect to REL_EMPTY

Change from new relationship to write an empty FlowFile to RESULT

Fix MONGO_URI

This closes #4281

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-20 19:54:24 -04:00
Mark Payne d195702ee2 Fixed a couple of typos in the RecordPath guide 2020-05-20 17:09:40 -04:00
Pierre Villard a9e9e5d137 NIFI-6571 Check token length on TLS toolkit server startup
This closes #3659.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-20 15:17:39 -05:00
KovalevIV 7034d7e44c NIFI-7471 fix bug with property validation 2020-05-20 10:49:49 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
Sushil Kumar 5826a09a83 NIFI-6911 Removed default Blob value for PutAzureBlobStorage
This closes #3906

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-19 13:47:08 -05:00
Peter Gyori d6240a1074 NIFI-7336: Add tests for DeleteAzureDataLakeStorage
DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string

NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case

This closes #4272.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 18:35:33 +02:00