Commit Graph

4029 Commits

Author SHA1 Message Date
Andy LoPresto 94c98c019f
NIFI-7558 Fixed CatchAllFilter init logic by calling super.init().
Renamed legacy terms.
Updated documentation.

This closes #4351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-22 12:20:28 -07:00
dennisjaheruddin 704260755a
NIFI-7501
Update nf-context-menu.js for an intuitive road to parameters
When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now.

Added generateflowfile load tag and description
Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile.

Revert "Update nf-context-menu.js for an intuitive road to parameters"

This reverts commit 3c44b1661f.

This closes #4333

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-06-22 11:12:22 -04:00
Matthew Burgess 32fa9ae51b
NIFI-7516: Catch and log SingularMatrixExceptions in OrdinaryLeastSquares model (#4323) 2020-06-22 09:15:41 -04:00
Mike Thomsen 64b544d5af NIFI-7529 Removed OS and Java information from InvokeHttp's UserAgent field so that it's removed regardless of whether or not this field is kept.
This closes #4332
2020-06-21 12:40:53 -04:00
Mark Payne 57c7883f64
NIFI-7566: Avoid using Thread.sleep() to wait for Site-to-Site connection to be handled. Instead, use TimeUnit.timedWait and use Object.notifyAll when setting the beingServiced flag. This significantly reduces latency and improves throughput for small-batch site-to-site communications
This closes #4353.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-19 15:37:02 -07:00
Mark Payne 0a16002076
NIFI-7509: Added optional Record Writer property to all List* Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4315.
2020-06-19 17:30:40 +02:00
Mark Payne c9d08a76b1 NIFI-7536: Fix to improve performance when determining the run status of processors when needing to wait for all processors to stop for updating parameter context, etc. 2020-06-19 10:14:12 -04:00
Greg Solovyev a1b245e051
NIFI-7551 Add support for VARCHAR to Kudu NAR bundle
- update Kudu dependencies to Kudu 1.12.0
 - add VARCHAR to Kudu Lookup Service and Processor
 - add tests for VARCHAR columns

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4347.
2020-06-19 09:19:12 +02:00
thenatog 27b5bb7a20
NIFI-6094 - Added the X-Content-Type-Options header to all web responses. (#4307)
NIFI-6094 - Added the mime/content type for ttf files.
2020-06-17 17:15:18 -07:00
Nathan Gough 2b25ccc298 NIFI-7537 - Small fix to make the LDAP connection timeout property a String rather than a Long. Fixes cast error in LDAP libs. 2020-06-17 16:33:02 -04:00
Joey c18b27af18
NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment (#4341)
* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment

This also fixes NIFI-4760.

* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-16 17:45:37 -07:00
Mark Payne 8b1a23a99c NIFI-7539: When capturing diagnostics information, capture a thread dump once and then provide this information to ProcessorNode when capturing active threads. Previously, each processor captured a thread dump itself. When this is done thousands of times it can result in a very long delay. 2020-06-16 14:48:14 -04:00
Макаров Василий Николаевич 9828e7dd14 NIFI-7035 The first curator connection issue is logged as ERROR until reconnect 2020-06-16 12:24:41 -04:00
Tamas Palfy e02ffdd99f NIFI-7527 AbstractKuduProcessorrefresh TGT deadlock fix: Redesigned locking.
NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set).
NIFI-7527 Refactor: removed redundant kudu client creation.

This closes #4330.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-06-15 14:17:34 +02:00
sjyang18 73008451d4 NIFI-7434: Add endpoint suffix to Azure storage processors
This closes #4265.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-06-10 09:45:04 -05:00
wanghq-c 3c757b6ba8
NIFI-7313:fix bug on 'Quote Table Identifiers'
NIFI-7313:add test by wanghongqi
NIFI-7313:edit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4185
2020-06-09 16:02:25 -04:00
Pierre Villard a2f521297f
NIFI-7490 - Add optional raw field to Syslog readers
review

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4299
2020-06-09 15:37:08 -04:00
Matt Gilman eb2ebefc46
NIFI-7514:
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.

This closes #4322
2020-06-09 15:23:58 -04:00
Nathan Gough 01e42dfb32
NIFI-7385 Provided reverse-indexed TokenCache implementation.
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #4271.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-08 19:36:14 -07:00
Tamas Palfy aa7c5e2178 NIFI-7511 In ControllerServiceProxyWrapper extended documentation. Minor refactor in StandardControllerServiceInvocationHandler. Also removed an unused import from NiFiSystemIT.
This closes #4317.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-08 16:04:15 -04:00
Wouter de Vries f2368a0dd1 NIFI-7393: Add max idle time and idle connections to InvokeHTTP
This closes #4233.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-06-08 09:29:10 -05:00
Andy LoPresto 845b66ab92
NIFI-7467 Refactored S2S peer selection logic.
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.

This closes #4289.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-05 13:40:16 -07:00
Joey Frazee dfefeb7b18 NIFI-7508: Reset classloader after running TestStandardControllerServiceInvocationHandler
and fix checkstyle violation on NiFiSystemIT
2020-06-05 16:23:01 -04:00
Mark Payne 359fd3ff29
NIFI-7476: Implemented FlowFileGating / FlowFileConcurrency at the ProcessGroup level
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4306.
2020-06-04 23:24:03 +02:00
Mike Thomsen 915617dbe7 NIFI-7299 Add basic OAuth2 token provider service that can fetch access tokens when supplied with appropriate credentials.
Added skeleton of oauth2 provider.
Added copy of our code.
Refactored a few things.
Updated apis to better match flow descriptions.
Updated poms and other artifacts.
Updated copyright notice.
Updated LICENSE.

This closes #4173

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
2020-06-04 08:43:27 -04:00
Mark Payne e0dd6d466e NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord
This closes #4223.
2020-06-02 15:13:14 -04:00
Bence Simon 5c2bfcf7d3 NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 15:13:14 -04:00
Peter Gyori 91dd59dbdf NIFI-7312: Enable search in variable registry of root process group
This closes #4303.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 13:25:00 -04:00
Mark Payne e31c323aa7 NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader. 2020-06-02 12:13:51 -04:00
jahenaor 04711ab466
NIFI-7477 Optionally adding validation details as a new attribute of the flowfile
NIFI-7477 Improving description and unit test now verifies attribute content

NIFI-7477: Fixed checkstyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4301
2020-06-01 14:08:08 -04:00
zhangcheng 1ec6675f38
NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266
2020-06-01 11:33:10 -04:00
Niels Basjes 2a82efc417
NIFI-6666 Add Useragent Header to InvokeHTTP requests
This closes #3734

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-29 21:39:14 -04:00
Pierre Villard af58714606 NIFI-7430 - LookupRecord change coordinate key for in-place replacement 2020-05-29 13:54:11 -04:00
Pierre Villard 8162edfd98 NIFI-6701 - Fix for PublishGCPPubSub 2020-05-29 13:38:11 -04:00
Peter Turcsanyi 06864c830c
NIFI-7422: Support aws_s3_pseudo_dir in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4292.
2020-05-29 14:22:58 +02:00
zhangcheng a1f277fdf7
NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4294.
2020-05-29 10:41:19 +02:00
zhangcheng e04e6793ca
NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4293.
2020-05-29 10:39:04 +02:00
r65535 4ced1775b5
NIFI-7487 - Added batch support and displayName to ModifyBytes processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4302.
2020-05-29 10:28:37 +02:00
Peter Gyori 1dd0e92040 NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
Made warning and error messages more informative.
Refactored flowFile assertion in the tests.

This closes #4287.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-27 09:13:41 +02:00
Mike Thomsen aa804cfceb NIFI-7485 Updated commons-configuration2.
NIFI-7485 Found more instances that needed updating.

This closes #4295
2020-05-26 17:11:24 -04:00
Tamas Palfy ca65bba5d7 NIFI-7453 In PutKudu creating a new Kudu client when refreshing TGT
NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.)
NIFI-7453 Safely closing old Kudu client before creating a new one.
NIFI-7453 Visibility adjustment.

This closes #4276.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-26 18:01:08 +02:00
adyoun2 101387bfaa
NIFI-6785 Support Deflate Compression
NIFI-6785 Remove unused imports

This closes #3822

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 16:22:08 -04:00
Mike Thomsen 443f969d36
NIFI-7211 Added @Ignore with warning message to a test that randomly fails due to timing issues.
This closes #4296
2020-05-25 10:24:38 -04:00
Andy LoPresto 97a919a3be
NIFI-7482 Changed InvokeHTTP to be extensible.
Added unit test.

This closes #4291.

Signed-off-by: Arpad Boda <aboda@apache.org>
2020-05-22 11:44:53 -07:00
pcgrenier b6ef7e13bf
NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions
NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types

NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4282
2020-05-21 13:13:12 -04:00
eduardofontes c7edcd68e1
NIFI-7463
Create empty relationship for RunMongoAggregation

Fix default autoterminate and condition to redirect to REL_EMPTY

Change from new relationship to write an empty FlowFile to RESULT

Fix MONGO_URI

This closes #4281

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-20 19:54:24 -04:00
KovalevIV 7034d7e44c NIFI-7471 fix bug with property validation 2020-05-20 10:49:49 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
Sushil Kumar 5826a09a83 NIFI-6911 Removed default Blob value for PutAzureBlobStorage
This closes #3906

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-19 13:47:08 -05:00
Peter Gyori d6240a1074 NIFI-7336: Add tests for DeleteAzureDataLakeStorage
DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string

NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case

This closes #4272.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 18:35:33 +02:00
sjyang18 852715aadd NIFI-7409: Azure managed identity support to Azure Datalake processors
NIFI-7409: review changes
NIFI-7409: ordering import statements
NIFI-7409: changed validateCredentialProperties logic

This closes #4249.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 11:50:08 +02:00
mtien 3fec4d8c27
NIFI-7331 Fixed grammatical errors in log output.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.
2020-05-19 10:38:45 +02:00
Peter Gyori 9aae58f117 NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory
A newer version (12.1.1) of azure-storage-file-datalake is imported.

This closes #4273.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-18 23:50:05 +02:00
Matthieu Cauffiez 179675f0b4 NIFI-7380 - fix for controller service validation in NiFi Stateless
This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 16:11:40 -04:00
Matthew Burgess a3cc2c58ff NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes
This closes #4275.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 15:11:33 -04:00
Yolanda M. Davis 13418ccb91
NIFI-7437 - created separate thread for preloading predictions, refactors for performance
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274
2020-05-18 14:57:54 -04:00
Mark Payne c51b9051a8 NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows 2020-05-15 16:16:39 -04:00
Matthew Burgess 53a161234e
NIFI-7448: Fix quoting of DDL table name in PutORC
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4269.
2020-05-14 11:41:35 +02:00
Peter Gyori 9608fe33fa NIFI-7367: Add tests for FetchAzureDataLakeStorage
NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.

This closes #4257.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-13 14:50:31 +02:00
Nathan Gough 302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
Sushil Kumar 851359c1fc
NIFI-6913: PutAzureBlobStorage processor will create container if not exists
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4237.
2020-05-09 15:37:14 +02:00
mtien 647a9a60b7
NIFI-7423 Upgraded jquery dependency version.
NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258
2020-05-08 13:13:38 -04:00
Bence Simon 66b175f405
NIFI-7390 Covering Avro type conversion in case of map withing Record
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256
2020-05-08 13:11:19 -04:00
mtien 6e2f86d716 NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml.
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4252.
2020-05-07 14:19:55 -04:00
Matthew Burgess b7c81d6007
NIFI-7428: Switch hive.version property to set Hive 3 version
This closes #4259
2020-05-06 19:38:23 -04:00
Otto Fowler f22cd26cf4 NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large
This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-06 17:20:08 -04:00
Alan Jackoway effa930151
NIFI-7425 Log Message for ReplaceText Over Buffer Size
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.
2020-05-06 09:40:46 +02:00
Mark Payne 0448e23a96
NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz
NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244
2020-05-04 14:49:15 -04:00
Peter Gyori 2a44f0ff2f NIFI-7412: Fixed provenance event types in Azure Fetch/Delete processors
This closes #4245.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-01 22:57:02 +02:00
Matthew Burgess aa986e0bfb
NIFI-7404: Fixed invalid script processors upon thread termination
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4238.
2020-05-01 16:46:52 +02:00
sjyang18 aa1e272052 NIFI-6149: Azure EventHub Managed identities support patch
review changes
additional review changes
NIFI-6149: typo fixes

This closes #4226.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-30 23:17:20 +02:00
Matt Gilman e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Yolanda M. Davis 7784178abd
NIFI-7408 - added percent used metrics for connections
NIFI-7408 - return double value for utilization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4240
2020-04-29 13:11:06 -04:00
Otto Fowler 659a383723 NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-04-29 13:01:06 -04:00
Matthew Burgess 1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Peter Turcsanyi a7008903ed
NIFI-7298: PutAzureDataLakeStorage tests.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4227.
2020-04-28 19:03:56 +02:00
Andy LoPresto 148537d64a
NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 17:29:42 -07:00
Sushil Kumar 996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Sushil Kumar 6c2701abef Do not update status for stopping a deleted node 2020-04-25 11:01:08 -04:00
Matthew Burgess cd10435b9f
NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219) 2020-04-22 10:37:04 -04:00
Tamas Palfy e2716a6c94 NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file.
Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.

NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.

This closes #4213.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-20 22:34:03 +02:00
Mark Payne c19db9d623
NIFI-7375: This closes #4218. Fixed a bug that caused Provenance Events not to show up in specific situations when clicking View Provenance for a Processor.
- Added System-level tests for Provenance repository to reproduce behavior.
- Added a Provenance Client to the CLI, which is necessary for System-level tests.
- Added small additional configuration for Provenance repository to simplify development of system tests
- Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-04-20 13:40:07 -04:00
Bence Simon 923a07a5db NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302) 2020-04-20 11:19:21 -04:00
Jérémie 8e9dbd7335
NIFI-7366 - ConsumeEWS Processor parse EML
https://issues.apache.org/jira/browse/NIFI-7366

This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4215.
2020-04-17 15:59:15 +02:00
muazmaz 58118cf904 NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.
NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-16 11:49:44 +02:00
EndzeitBegins 8e3f42051f
NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration
This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-04-16 15:56:40 +09:00
Alexandre Vautier b113a022e4
NIFI-7359 Fix parent id on process metrics for Prometheus
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4209
2020-04-15 20:27:44 -04:00
Bence Simon 83400789f6
NIFI-7292 Preventing file listing from fail because of insufficient privileges
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.
2020-04-15 18:54:51 +02:00
Rob Fellows c263daf20b
[NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208
2020-04-15 10:06:38 -04:00
sjyang18 9df53e7204
NIFI-6977 - Change the reporting behavior of Azure Reporting task to report report the time when metrics are generated
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4211.
2020-04-15 12:53:18 +02:00
Mark Payne a61a4c2b58
NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.
2020-04-15 12:39:13 +02:00
Andy LoPresto 07a8311b4c
NIFI-7319 Add walkthrough document (#4193)
* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>
2020-04-10 22:25:40 -07:00
Mark Payne c99a187018 NIFI-7346: Ensure that the Provenance Repository doesn't delete the Active Event File 2020-04-09 13:34:37 -04:00
Mark Payne 728bdec0f4 NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. 2020-04-09 13:32:34 -04:00
Matthew Burgess 84968e70d2
NIFI-7087: Use FlowManager.findAllConnections() when available
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4026
2020-04-09 12:59:01 -04:00
Mark Payne 683b1d9952
NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited.
- Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup

This closes #3891
2020-04-09 10:42:05 -04:00
Peter Turcsanyi 08dcd4af14
NIFI-7345: Fixed Hive database and table names case insensitivity in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4198.
2020-04-09 14:13:54 +02:00
muazmaz 63379c3520 NIFI-7259 Adding DeleteDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.
Updated to remove unused variables
NIFI-7259 import and property description changes

This closes #4189.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-08 10:58:48 +02:00
Tamas Palfy 09cece8e99 NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down.
NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown.
NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup.

This closes #4191.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-07 20:12:18 +02:00
Joe Witt 445efcfde6
NIFI-7326 updated URL to find splunk artifacts (#4188)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-06 20:36:56 -07:00
Dustin Rodrigues f0e8daf45d
NIFI-7317 - make .java files non-executable
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4181.
2020-04-06 12:19:46 +02:00
Matthew Burgess a093af2d42
NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)
* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable
2020-04-03 16:12:53 -04:00
Bence Simon c0f5fcb484 NIFI-7188 Extending UI search with filters and refactoring existing solution
This closes #4123.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-02 21:49:40 +02:00
Mike Thomsen 4a2a91135e
NIFI-7293 Add in-memory janusgraph implementation of GraphClientService to help with live testing.
Added new in memory janus graph client for testing.
Added integration test to ExecuteGraphQuery.

NIFI-7293 Added missing getter.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4168
2020-04-01 10:00:41 -04:00
muazmaz daddf400a2 NIFI-7103 Adding PutDataLakeStorage Processor to provide native support for Azure Data Lake Storage Gen 2 Storage.
added data-lake dependency
NIFI-7103 fixed indentation
Update to add IllegalArgumentException
Fixed indentation and logging
nifi-7103 review changes
nifi-7103 root directory and exception change

This closes #4126.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-03-31 11:15:07 +02:00
Mubashir Kazia fee1b8b8e0
NIFI-7294 Address deprecation issues in solrj and httpclient
Some calls to deprecated methods in httpclient were resulting in
UnsupportedOperationException. Use the new API calls in both httpclient
and solrj. Add an integration test to include test coverage for
org.apache.nifi.processors.solr.SolrUtils.createClient

This closes #4171.
2020-03-30 16:49:24 -04:00
Joe Witt 992d990dd2
NIFI-7291 updated enforcement of dependency rules and build command
Reviewed by markap14
This closes #4166.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-27 16:34:59 -04:00
Marc Parisi 1343b00e0b NIFI-7290: omit transitive that is causing the build to fail and not needed for the test scope 2020-03-27 13:12:13 -04:00
Matthew Burgess be2929b8b0
NIFI-7287: Move services-api dependency from Prometheus reporting task to its NAR
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4162
2020-03-27 08:48:41 -04:00
Tamas Palfy c79473baf5 NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled 2020-03-26 11:45:09 -04:00
Troy Melhase 483f23a8aa
NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-25 17:23:22 -07:00
Bryan Bende 975f2bdc4f
NIFI-7279 This closes #4160. Protect against NPE in RedisDistributedMapCacheClientService when value is null
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-25 17:40:20 -04:00
Bryan Bende 5784888082
NIFI-7281 This closes #4159. Use BufferedInputStream in StandardSocketChannelRecordReader in order to support mark/reset
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-25 17:39:44 -04:00
Michael Hogue aeaf953e06
NIFI-5925: Added controller services to set of components that are searched
NIFI-5925: cleanup, add negative test

NIFI-5925: fixed checkstyle

This closes #4105

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-25 08:55:44 -04:00
Bryan Bende 5fd25d6235
NIFI-7278 Adding support for SCRAM-SHA-512 to Kafka 2.0 processors 2020-03-24 16:53:22 -04:00
Mubashir Kazia 9b4292024b
NIFI-7269 - Upgrade solrj version to 7 in nifi-solr-processors
Remove unused imports

Use the latest solrj version(8.4.1)

Setup default schemaFactory for tests
The default schemaFactory ManagedIndexSchemaFactory creates
additional files in test's resources directory. Change it to
ClassicIndexSchemaFactory for classic behavior.

This closes #4152.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-03-24 14:55:51 -04:00
Bryan Bende 6412097eb2
NIFI-7257 Added HadoopDBCPConnectionPool
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback

This closes #4149.
2020-03-24 11:05:18 -04:00
Bryan Bende 7f32aa56db
NIFI-7271 Make command timeout configurable for ShellUserGroupProvider
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback

This closes #4154.
2020-03-23 20:43:06 -04:00
Joe Witt 254b20fa5f
NIFI-7274 add time for test conditions to be met 2020-03-22 09:54:43 -04:00
Bryan Bende 2feeb57159
NIFI-7221 Support v2 and v3 protocol version for Hortonworks Schema Registry
- Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java
- Addressing review feedback

This closes #4120.
2020-03-20 10:56:47 -04:00
Pierre Villard 1fe79021b5
NIFI-7221 Initial work 2020-03-20 10:56:32 -04:00
Joe Witt f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
Eduardo Fontes 943de310ad
NIFI-6293
Add support to Mongo Extended JSON v2
Add org.json lib
Replace evil json
Replace evil json for alternative
Include testExtendedJsonSupport
Style adjustment
Remove unecessary new JSON parser
Fix query in testExtendedJsonSupport
Parse with Jackson and BSON
Back to default MONGO_URI

This closes #4068

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-18 18:36:09 -04:00
M Tien f91d6c420d
NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151)
* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Fixed typo in Javadoc.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 19:49:15 -07:00
Pierre Villard 7105bf36a8
NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 15:51:28 -07:00
Pierre Villard 71bf12976f
NIFI-7258 - fix overflow in PutAzureEventHub when not configured correctly
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4146.
2020-03-16 21:28:12 +01:00
Andy LoPresto 3feb85a030
NIFI-7223 Resolved merge conflicts from additional test case for null key password.
This closes #4145.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-16 12:47:41 -04:00
Nathan Gough 69b6c231fd
NIFI-7223 - Added another test for when keyPasswd is not present. 2020-03-16 12:28:18 -04:00
Andy LoPresto f9d75056fa
NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
2020-03-16 12:28:01 -04:00
Nathan Gough 7374361b5c
NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.
2020-03-16 12:26:50 -04:00
Matthew Burgess ace5754524 NIFI-7208: Restore default timezone in JdbcCommon 2020-03-13 15:00:24 -04:00
Mark Payne 0ad58200af
NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-13 14:05:40 -04:00
Joe Witt 561be89a21
NIFI-7250 activate user.timezone appropriate to each region
NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4140.
2020-03-13 17:30:19 +01:00
Matthew Burgess 798a8eeb50
NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4139.
2020-03-13 17:20:45 +01:00
Adam Taft decb5d062e
NIFI-6742 Use JUnit TemporaryFolder when creating test databases
- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines

Signed-off-by: Marc Parisi <phrocker@apache.org>

This closes #4137.
2020-03-12 20:48:52 -04:00
Joe Witt 97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Peter Turcsanyi dfaef38805
NIFI-7248: Atlas reporting task handles PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4138.
2020-03-12 10:44:48 +01:00
Mark Payne d4a2afc25c
NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.
2020-03-12 10:25:04 +01:00
Mark Payne d68720920f
NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.
2020-03-11 21:00:43 +01:00
Mark Payne 67676ba5b4 Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record 2020-03-11 14:29:44 -04:00
Pierre Villard cd83e70b91 unit test reproducing the issue 2020-03-11 14:29:44 -04:00
Peter Turcsanyi 3de3ad4029
NIFI-7245: JMS processors yield when connection factory initialisation failed
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4133.
2020-03-11 19:24:10 +01:00
Gardella Juan Pablo 89d8b877f9 NIFI-7050 ConsumeJMS is not yielded in case of exception
This closes #4004.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-03-11 10:54:20 +01:00
Peter Turcsanyi 9cb85a4fd0
NIFI-7239: Upgrade the Hive 3 bundle to use Apache Hive 3.1.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4129.
2020-03-11 10:26:10 +01:00
Pierre Villard a679e88b6f
NIFI-4970 - Add a property to deal with empty GZIP HTTP response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4127.
2020-03-11 10:08:51 +01:00
Mark Payne afad982e91
NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"
This reverts commit f01668e66a.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-10 14:32:43 -04:00
Pierre Villard 12c8402ac3
NIFI-7210 - added PG path in bulletins for S2S Bulletin RT
Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100
2020-03-09 16:29:22 -04:00
Mike a33a5e35b3
Merge pull request #4119 from eduardofontes/patch-4
NIFI-7233 - Fix Mongo Processors to avoid cycle in provenance graph
2020-03-08 13:18:12 -04:00
Pierre Villard 0f775f3a57
NIFI-7195 - Catch MongoException to route flow files to failure
This closes #4089

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-08 08:10:54 -04:00
Pierre Villard 578430c9d9 NIFI-7197 - In-place replacement in LookupRecord processor
This closes #4088

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:35:59 -05:00
Mark Bean f4b65afb64 NIFI-7231: move controller service validation out of synchronized block for enabling
This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:01:19 -05:00
Eduardo Fontes 57947cd2a2
Fiz writeBatch to avoid cycle in provenance graph 2020-03-06 16:06:21 -03:00
Mark Payne f283c1191c
Fixed unit test failed with NIFI-7232
This closes #4117.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-06 12:46:17 -05:00
Joe Witt 20dda05f26
NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE 2020-03-06 12:46:02 -05:00
Peter Turcsanyi 7c57e75da4 NIFI-7226: Add Connection Factory configuration properties to PublishJMS and ConsumeJMS processors
Some JMS client libraries may not work with the existing controller services due to incompatible
classloader handling between the 3rd party library and NiFi.
Via configuring the Connection Factory on the processor itself, only the processor's and its
children's classloaders will be used which eliminates the mentioned incompatibility.

This closes #4110.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 09:51:37 -05:00
Joe Witt 040c8a0af9
NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4115.
2020-03-06 12:48:15 +01:00
Matthew Burgess b82fec41d9
NIFI-7055: Removed unit test that is now covered by ListValidator
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4114.
2020-03-05 11:01:48 +01:00
Matthew Burgess 74b1b2fc59 NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) 2020-03-04 11:46:47 -05:00
jstorck 4b6de8d164
NIFI-7025: Wrap Hive 3 calls with UGI.doAs
Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods
Fixed misleading logging message after the principal has been authenticated with the KDC
When connecting to unsecured Hive 3, a UGI with "simple" auth will be used

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4108
2020-03-04 11:21:27 -05:00
Eduardo Fontes bad254ec5b
NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4109.
2020-03-04 10:36:06 +01:00
karthik.kadajji 2cf4fde686
NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4106.
2020-03-03 16:43:56 +01:00
Bryan Bende 852d2f3210 NIFI-7030 Add Kerberos principal and password properties to Solr processors
Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing

This closes #4062.
2020-03-02 16:01:39 -05:00
Bryan Bende e0fc75a963 NIFI-7019 Add kerberos principal and password properties to NiFi DBPCConnectionPool
This closes #4087.
2020-03-02 14:20:20 -05:00
Bryan Bende 5b93e537d5 NIFI-7029 Add kerberos password property to NiFi Kudu components
This closes #4097.
2020-03-02 14:03:37 -05:00
Jeff Storck c74e71306e
Merge pull request #4071 from bbende/NIFI-7026
NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaReg…
2020-03-02 12:50:20 -05:00
jstorck ab1658130d
NIFI-7024: Added Kerberos Password support to HBase_1_1_2_ClientService and HBase_2_ClientService
This closes #4103.
2020-03-02 11:43:13 -05:00
jstorck 1678531638
NIFI-7025: Initial commit adding Kerberos Password feature for Hive components
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation.  If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC

Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components.  An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).

Addressing PR feedback

Addressing PR feedback

This closes #4102.
2020-03-02 11:28:59 -05:00
jstorck 614136ce51
NIFI-7018: Initial commit of processors extending AbstractHadoopProcessor supporting kerberos passwords
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest

Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations

This closes #4095.
2020-02-28 10:10:19 -05:00
Justin Miller 23b04ae968
NIFI-6856 - Make client ID a non-required field for the MQTTConsume and MQTTProduce processors. Generates a
random ID if not set.

Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter>

add expression language support for the MQTT client ID

Setting client id in publish test fails because it is not a flowfile attribute.
Remove client id and autogenerate it when testing.

Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry.

Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3879.
2020-02-26 15:54:53 -08:00
Joe Witt 4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Shawn Weeks 0bb8ce7438
NIFI-7139 Add record.error.message on failure of a record reader or writer
Handle scenario where message might be null.

Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.

Fixed Style Issues and Updated WritesAttributes.

Added Test Case for Error Message

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4052
2020-02-26 08:50:32 -05:00
Joe Witt bad0f10a52
NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4091.
2020-02-25 17:38:33 -08:00
Przemysław Dubaniewicz c092a23bdf
NIFI-7164 Upgrade shyiko/mysql-binlog-connector-java dependency
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4076.
2020-02-25 14:05:47 -08:00
Andy LoPresto 815e4cf51f
NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085)
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-25 13:14:52 -08:00
Pierre Villard cff1e329a2
NIFI-7173 - This closes #4084. add nifi-metrics jar to scripting bundle
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-25 13:12:53 -08:00
Yolanda M. Davis abf223d574
NIFI-7163 - added RulesEngine and RulesEngineProvider interfaces, enhanced easy rules to support provider interface and refactored to extract rules engine implementation
NIFI-7163 - updated documentation and comments

NIFI-7163 - fix checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4081
2020-02-25 09:59:39 -05:00
Joe Witt 778012412a
Revert "NIFI-6363 Refactors sensitive properties, adds additional providers."
This reverts commit 479fcfdc0b.

It does not build properly as shown in Github Actions.
2020-02-24 22:07:53 -08:00
Troy Melhase 479fcfdc0b NIFI-6363 Refactors sensitive properties, adds additional providers.
NIFI-6363 Additional fixes.

NIFI-6363 Fix Hadoop compile problem.  Add GCP IT instructions.

NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.

NIFI-6363 - Added some docs and experimental tag to the relevant classes.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4080.
2020-02-24 16:35:19 -05:00
MatthewKnight-NG acaf321af0
NIFI-5924 Labels should be searchable
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4070
2020-02-24 16:14:21 -05:00
Wouter de Vries 9848eb4409
NIFI-7007: Add update functionality to the PutCassandraRecord processor.
NIFI-7007: Add additional unit tests that hit non-happy path

NIFI-7007: Use AllowableValue instead of string

NIFI-7007: Add the use of attributes for the update method, statement type and batch statement type

NIFI-7007: Add additional tests, mainly for the use of attributes

NIFI-7007: add some ReadsAttribute properties to the PutCassandraRecord processor

NIFI-7007: additional update keys validation logic

NIFI-7007: fix imports

NIFI-7007: Convert fieldValue to long in separate method

NIFI-7007: Add new style of tests checking actual CQL output

NIFI-7007: add license to new test file

NIFI-7007: add customValidate to check for certain incompatible property combinations

NIFI-7007: remove check on updateMethod and replace Set.of with java 8 compatible replacmenet

NIFI-7007: Add test for failure with empty update method via attributes

NIFI-7007: remove unused variable

NIFI-7007: Fix customValidate that incorrectly invalidated a valid config

Fix Checkstyle

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3977
2020-02-24 15:14:16 -05:00
Pierre Villard 4f315edc6e
NIFI-7133 - This closes #4049. Clarification of EnforceOrder description
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:59:49 -08:00
Pierre Villard fd7c01453b
NIFI-7184 - This closes #4074. Added mime type property to GenerateFlowFile
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:54:28 -08:00
Pierre Villard 9a8a551e03
NIFI-7183 - This closes #4073. Improve ReplaceText when removing FF's content
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:50:34 -08:00
Joe Witt 722b99432c
NIFI-7185 relaxed timing constrained for builds on lower resources environments like our Github CI builds
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4075.
2020-02-22 11:18:56 -08:00
Zoltan Kornel Torok e631851e0f
NIFI-7178 - Handle the case when schema is not available
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4067.
2020-02-22 11:07:31 -08:00
Joe Witt 49ba9e8196
NIFI-7181 convert integration test to proper name so it isnt run as unit test 2020-02-21 18:05:40 -05:00
Matthew Burgess 6ea78c5548
NIFI-7114: This closes #4069. Update time-sensitive tests in TestLuceneEventIndex
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-21 17:38:36 -05:00
Matthew Burgess 1b0e3865e9
NIFI-7114: Fix file leaks in StandardCommsSession and S2S Reporting components
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-21 17:37:54 -05:00
Bryan Bende 8fa855c8c3
NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaRegistry 2020-02-21 14:56:33 -05:00
Joe Witt f8a5d1295c
NIFI-7175 removed appveyor and travis builds which have become unreliable. Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4058.
2020-02-20 16:36:51 -08:00
Troy Melhase 0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
Grant Henke 4098404596
NIFI-6551: Improve PutKudu timestamp handling
Uses `DataTypeUtils.toTimestamp` when writing to Kudu
timestamp (`UNIXTIME_MICROS`) columns. This allows
us to use the `row.addTimestamp` API and get much more
intuitive and predictable timestamp ingest behavior.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4054.
2020-02-18 15:30:08 -08:00
Grant Henke 09c7406d18
NIFI-6867: Validate PutKudu operation type property
This patch adds validation to the PutKudu operation type property.
It also improves the description to include the valid values and
adjusts the inputs to be case insensitive.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4063.
2020-02-18 14:36:02 -08:00
Grant Henke 268ba1d23e
NIFI-7142: Automatically handle schema drift in the PutKudu processor
Adds a boolean property to the PutKudu processor to optionally
enable automatic schema drift handling.

If set to true, when fields with names that are not in the target
Kudu table are encountered, the Kudu table will be altered to
include new columns for those fields.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4053.
2020-02-18 12:33:58 -08:00
Peter Turcsanyi 7e5e332842
NIFI-7143: Upgrade GCP dependency
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes NIFI-7143.
2020-02-17 12:11:16 -08:00
M Tien 37614d02cd
NIFI-7136 Added autocomplete="off" to login password input (#4055)
NIFI-7136 Added autocomplete="off" to login password input
Updated nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/login/login-form.jsp

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-14 20:46:08 -08:00
Joe Ferner 62606ff89a NIFI-6873: Added support for replacing a process group
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
 - added replace process group functionality in ProcessGroupResource
 - parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
 - refactored flow update methods to make use of commonality across different update types whenever possible
 - fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
 - improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import

This closes #4023.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-02-11 14:46:40 -05:00
Wouter de Vries a80b2475f7
NIFI-7132: fix handling of UUIDs in PutCassandraQL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4048.
2020-02-11 11:21:27 -08:00
Mark Payne 65b2a9bc2c
NIFI-7117: When SocketLoadBalancedFlowFileQueue creates its array of Queue Partitions in the constructor, it added the local partition as the first element in that list. This list should be ordered the same across all nodes in the cluster. By making the local partition the first in the array, each node had a different ordering of these partitions. As a result, Partition by Attribute strategy would constantly rebalance flowfiles that it received to other node, and Single Node always transferred data to the first partition, which was the local node, instead of whichever node should have been the first in the list. This commit addresses this issue by instead inserting the local partition intot he 'queuePartitions' array based on the local node identifier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4045.
2020-02-11 10:59:19 -08:00
David Savage 5e964fbc47
NIFI-4792: Add support for querying array fields in QueryRecord
Work in progress adding support for array based queries
updated calcite dependency

tidy up unused imports highlighted by checkstyle in travis build

tidy up }s highlighted by checkstyle in travis build

Add test for use case referenced in NIFI-4792

Bumped Calcite version to 1.21.0

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4015
2020-02-11 10:59:20 -05:00