Commit Graph

4450 Commits

Author SHA1 Message Date
Mark Payne e06afbdd22
NIFI-8609: Optimized AvroTypeUtil Record creation and conversion
Added unit test that is ignored so that it can be manually run for testing performance before/after changes to AvroTypeUtil. Updated AvroTypeUtil to be more efficient by not using Record.getValue() and instead iterating over the Map of values directly. getValue() is less efficient here because we know the RecordField's we are iterating over exist in the schema since they are retrieved from there directly; as a result, any null values still have be looked up by aliaases, but that step can be skipped in this situation. Also avoided looking for fields that exist in Avro Schema and not in RecordSchema just to set default values on GenericRecord - there's no need to set them if they are default values.

This closes #5080

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-15 09:05:38 -05:00
Mark Payne 8004aa5e6e
NIFI-8698: If using a Data RecordPath and handling schema drift, ensure that the RecordPath is evaluated against the data before attempting to determine the fields to update. Also refactored to move the handling of schema drift into its own method for the purpose of clarity
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5156
2021-06-15 09:45:18 -04:00
markap14 96a8b2d090
NIFI-8666: Allow users to escape parameter names in Expression Language using quotes. (#5133) 2021-06-15 09:37:34 -04:00
Nathan Gough b49a62e1b5
NIFI-8699 - Upgrade Lucene to 8.8.2 in Provenance Bundle
This closes #5157

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-15 08:30:50 -05:00
Kent Nguyen 87c02c17d2
NIFI-8674: Update Apache Calcite version to 1.27.0
This closes #5138

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-14 15:47:47 -05:00
Mark Payne 576338cd55 NIFI-8697: When login endpoint is encountered and JWT is no longer valid, request cookie to be deleted. Also fixed NPE that was encountered when going to /logout without the expected cookie being present
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5155.
2021-06-14 16:06:18 -04:00
Mark Payne 20c889cf82
NIFI-8620: Ensure that we provider appropriate error messages if attempting to migrate FlowFiles from one session to another without including full hierarchy; added tests to verify behavior
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5099
2021-06-14 10:00:49 -04:00
Peter Turcsanyi 48befe22f6
NIFI-8645: Disable LongRunningTaskMonitor by default
This closes #5111

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-14 08:50:34 -05:00
exceptionfactory ab851b9cde NIFI-8672 Added syncUninterruptibly() to Netty shutdown and close methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5139.
2021-06-12 17:14:34 -04:00
Peter Turcsanyi 1a515ee74a NIFI-8656 Support EL for SAS Token in the ADLS Gen2 processors
This closes #5119

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-06-11 14:11:07 -07:00
markap14 38cf25c7cf
NIFI-8647: Updated VersionedPropertyDescriptor to include information about which properties are referencing resources, how many resource, and the types of resources; updated NiFiRegistryFlowMapper to perform the mapping. (#5114) 2021-06-11 16:48:58 -04:00
Joe Gresock 1ccc4fbb0f
NIFI-8651: Refactor Sensitive Properties Providers for extension
This closes #5131

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-11 15:44:37 -05:00
Mark Payne 172afac6ab
NIFI-8633: This closes #5104. When reading a Content/Resource Claim from FileSystemRepository, avoid the unnecessary Files.exists call and instead just create a FileInputStream, catching FileNotFoundException
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-06-11 13:10:32 -07:00
Matthew Burgess 3e774bc5be
NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues (#5116)
NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues
2021-06-11 14:44:55 -04:00
exceptionfactory db40989b48 NIFI-8516 Enabled HTTPS and Single User Authentication in default configuration
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
2021-06-11 14:18:53 -04:00
Timea Barna d7a8d275c9 NIFI-8642 Select the default Old Gen Memory Pool for Memory Reporting Task
This closes #5115.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-06-11 14:06:15 +02:00
Mark Payne 5c91143dc2
NIFI-8680: Ensure that if commitAsync() is called and an Exception caught that it is rethrown
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5143
2021-06-10 10:10:51 -04:00
exceptionfactory 4df2a79c8a
NIFI-8678 Forced Calcite 1.27.0 and Calcite Avatica 1.6.0 for Hive Bundle
- Overriding Calcite incubating versions avoids dependencies from third party repositories

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5142
2021-06-10 10:02:50 -04:00
Peter Gyori 7cb3bce6a7
NIFI-8622: Reuse of GenericRecord instance eliminated from AvroReaderWithExplicitSchema to avoid issues when parsing fields of type "bytes" (#5090) 2021-06-09 15:22:22 -04:00
markap14 d44dec7345
NIFI-8670: Fixed bug in which a Parameter Context Update would fail if an updated parameter was referenced by a Controller Service whose state was ENABLING. Created system test to verify and addressed bug that was encountered in SingleFlowFileConcurrencyIT. (#5137) 2021-06-09 13:07:33 -04:00
Paul Grey a3c1cd074b
NIFI-8499 - Added encrypted FlowFile repository swap file implementation
This closes #5122

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-08 13:23:22 -05:00
Mark Payne 07ff4f2592
NIFI-8667: When marking a Controller Service as enabled, ensure that we release the write lock before calling validation methods of referencing components. Otherwise, we can encounter a deadlock.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5134
2021-06-08 11:59:24 -04:00
Matthew Burgess 659fda10f3
NIFI-8659: Support multiple output records in JoltTransformRecord (#5126)
NIFI-8659: Support multiple output records in JoltTransformRecord
2021-06-07 15:30:41 -04:00
Dominik Przybysz f07e17ba74
[NIFI-8610] Do not reuse generic records in convertAvroToORC
[NIFI-8610] Simplify decimal test for convertAvroToORC

Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #1081
2021-06-07 11:26:16 +02:00
Mark Payne c9dee30294
NIFI-8658: Allow Filter Functions and expressions to be specified as a RecordPaths
NIFI-8658: Addressed issue where the RecordField that was provided from Function Filters were not accurate

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5125
2021-06-04 17:17:33 -04:00
exceptionfactory 9af136ad13
NIFI-8654 Changed SNMP TestAgent to use getAvailableUdpPort()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5120.
2021-06-04 23:10:51 +02:00
Mark Payne d340ad79ed
NIFI-8661: Updated ReaderLookup and RecordSetWriterLookup to use a property that indicates which reader/writer to use instead of requiring a specific attribute be provided
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5129
2021-06-04 16:21:05 -04:00
s9514171 66f0f90252
NIFI-8471 Parameter Contexts - show referencing process groups
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5048.
2021-06-02 18:22:35 +02:00
Timea Barna f23dcb05f6 NIFI-8640 Regression with NIFI-8522 NiFi can duplicate controller service during template generation
This closes #5109.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-06-01 14:37:51 +02:00
Chris Sampson e0a1cb8a01
NIFI-8133 Add ability to supress null/empty values in ElasticSearchClientService for PutElasticsearchRecord output
NIFI-8133 pass null/empty field handling from PutElasticsearchRecord to ElasticSearchClientImpl

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4755
2021-05-29 13:29:14 -04:00
Timea Barna bf960cae2e
NIFI-8522 NiFi can duplicate controller services when generating templates
Adding final keywords and capitalize comment for process group section

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5086.
2021-05-29 16:42:20 +02:00
Joe Gresock 9fe1f56019
NIFI-8632: Using available port in StandardOidcIdentityProviderGroovyIT.groovy
This closes #5106

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-27 20:42:33 -05:00
exceptionfactory 7fb0293c3e NIFI-8626 Upgraded okhttp-digest to 2.5
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5096.
2021-05-27 12:41:11 -04:00
Mark Payne 08edc33eb7
NIFI-8629: Implemented the LogComponentStatuses task that runs periodically in stateless.
Also noticed a typo in the ControllerStatusReportingTask and found in comparing outputs
that it had a bug that caused it to log counters generated only by processors at the root level so fixed that.

This closes #5101

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-26 20:07:26 -05:00
Mark Payne 46b1f6755c NIFI-8631: Ensure that GCP Pub/Sub messages are not acknowledged until session has been committed, in order ot ensure that we don't have data loss
This closes #5102.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-26 21:54:31 +02:00
exceptionfactory 1e6161c0aa
NIFI-8361 Upgraded Zip4j to 2.8.0
- Upgrade resolves issue unpacking Zip files with temporary spanning markers

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5103.
2021-05-26 20:21:11 +02:00
s9514171 1e1c446243
NIFI-8628 - Variable Registry - Variable count doesn't reset when opening the variable dialog
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5097.
2021-05-26 18:04:03 +02:00
tpalfy dfa683af0e
NIFI-8528 Migrate NiFi Registry into NiFi codebase (#5065)
NIFI-8528 Migrate NiFi Registry fully codebase into NiFi as a module. No changes except certain dependency scopes to preserve the NiFi Registry original by overriding the new parent (nifi).
- Version adjustments. Removed distinct checkstye rules form nifi-registry. (Using nifi's instead.)
- Made some tests Windows-compatible.
- Consolidated LICENSE, NOTICE and README.md.
- Fixed CryptoKeyLoaderGroovyTest.groovy.
- Disable frontend-maven-plugin on Windows.
- Skipping all goals of the frontend-maven-plugin on Windows.
- Registry integration tests not to run in github jobs (same as the original settings). Skip all registry tests (build and run) on Windows.
- Removed Husky from registry.
2021-05-26 09:34:55 -04:00
exceptionfactory 103aae64cb
NIFI-8614 Adjusted NodeClusterCoordinatorFactoryBean to handle null ClusterNodeFirewall (#5100)
- Changed FileBasedClusterNodeFirewallFactoryBean to return null when configuration file is not found
2021-05-25 16:53:02 -04:00
s9514171 beb1d2f445
NIFI-8520 - Parameter Contexts - Show the wrong information of referencing components
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5060.
2021-05-25 19:45:21 +02:00
exceptionfactory a3365c8833 NIFI-8462 Refactored PutSyslog and ListenSyslog using Netty
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog

NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues

NIFI-8462 Removed unused import of ExpressionLanguageScope

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5044.
2021-05-25 10:54:15 -04:00
Tamas Palfy 5108d7cdd0 NIFI-8439 Update parquet-avro to allow reading parquet INT96 timestamps as byte arrays (instead of throwing an exception).
Also allow to write them as such (byte-arrays) - again, instead of throwing an exception.

NIFI-8439 Fixed unit tests.
NIFI-8439 Allow writing parquet INT96 timestamps if they were read by the same parquet-avro library.

This closes #5006.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-25 13:18:46 +02:00
Mohammed Nadeem 622507f994
NIFI-8627: Upgrade apache derby to 10.14.2.0 to mitigate CVE_2015_1832
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5095
2021-05-24 14:35:27 -04:00
exceptionfactory 01783a295c NIFI-8037 Changed SSLContextServices to use runtime supported protocols for TLSv1.3
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4827.
2021-05-23 23:52:01 -04:00
Wouter de Vries 0b29a42991
NIFI-7737: add string array option to putcassandrarecord
NIFI-7737: fix checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5005
2021-05-21 13:55:46 -04:00
exceptionfactory 1c17da562b NIFI-8604 Upgraded Apache Accumulo to 2.0.1
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5077.
2021-05-21 11:49:49 -04:00
Timea Barna b6f84413c4 NIFI-8373 Add Kerberos support to Accumulo processors
This closes #4973.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-05-21 14:11:15 +02:00
exceptionfactory 4888eaf62d
NIFI-8623 Changed repositories from conjars.org to public.nexus.pentaho.org
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5091
2021-05-21 08:07:09 -04:00
Mark Payne 154e4f3df3
NIFI-8617: Created nifi-stateless module at the root level and moved sub-modules into that. Created nifi-stateless-assembly and the associated resources module. Also updated some configuration to be configurable via environment variables
NIFI-8618: Allow for parameters to be set for environment variables. Updated README to indicate this and also corrected some outdated information.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5087
2021-05-20 22:13:53 -04:00
Mohammed Nadeem c113960b81
NIFI-8613: Improve FlattenJson Processor
- Unflattening a flattened json
- Preserving primitive arrays such as strings, numbers, booleans and null in a nested json
- Logging errors when failure
- Pretty printing resulted json

This closes #5083

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-20 15:04:23 -05:00