7144 Commits

Author SHA1 Message Date
exceptionfactory
cf7af1b463
NIFI-9743 Upgraded Jetty from 9.4.44 to 9.4.45
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5819.
2022-03-02 12:48:15 +01:00
Mark Payne
53a35ae4c9
NIFI-9689: When checking FlowFile Availability, consider swap queue and trigger data to be swapped in, since calling poll() will no longer happen if no data is available
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5821.
2022-03-02 06:22:37 -05:00
Mark Payne
a327ba478a
NIFI-9741: Make the close() method of WriteAvroResultWithExternalSchema idempotent
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5818
2022-03-01 17:52:02 -05:00
exceptionfactory
d17fcf6427 NIFI-9735 Corrected Jetty Duplicate Mapping Warning
- Removed duplicate default servlet mapping from addDocsServlets()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5817.
2022-03-01 15:49:23 -05:00
Shane Ardell
4719f78ef4
NIFI-9580: UI work for framework-level retry in Processors (#5666)
* NIFI-9580: initial ui work for framework-level retry in processors

* NIFI-9580: disable retry controls when retry not selected

* NIFI-9580: refactor marshalling to be one function

* NIFI-9580: style tweaks and documentation update

* NIFI-9580: update tooltip content

* NIFI-9580: change tooltip copy for clarity

* NIFI-9580: only submit retry configs if selected

* NIFI-9580: add relationship details to read-only dialog

* NIFI-9580: change styling based on feedback

This closes #5666
2022-03-01 14:57:18 -05:00
Mark Payne
0d0912d34b
NIFI-9400: Ensure that we always use the CollectionUsage metrics in the mbeans instead of the Usage metrics.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5798.
2022-03-01 14:17:32 -05:00
Mark Payne
9d3788ff05
NIFI-9730: Consider a change in value for retry-related fields from 'null' to the default value as an environmental change so that it's not flagged as a Local Modification, which would prevent users from updating the version of the Process Group that they are using
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5809.
2022-03-01 13:49:47 -05:00
Joe Gresock
db28c91cdb
NIFI-9733: Fixing StandardConnection.verifySourceStoppedOrFunnel infinite recursion (#5815) 2022-03-01 09:57:51 -05:00
Nathan Gough
dd3c9be847
NIFI-9641 - Adjusted the extraction of the chroot suffix for solr client connections.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5727.
2022-03-01 06:51:16 -05:00
exceptionfactory
9bb34188c9
NIFI-9734 Standardized exception cause message formatting
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5813.
2022-03-01 06:49:34 -05:00
Paul Grey
875f589548
NIFI-9738 - VersionedComponent data members should derive from Object
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5816.
2022-03-01 06:06:48 -05:00
exceptionfactory
52f21390db
NIFI-9688 Improve Logback shutdown handling (#5805)
* NIFI-9688 Improved Logback shutdown handling
- Added environment variable to disable Logback shutdown on web application termination
- Added shutdown hook to Logback configuration
- Added Logback shutdown hook configuration to system tests
2022-02-28 14:29:44 -05:00
exceptionfactory
ddecfa201b
NIFI-9736 Improved TestRouteText to avoid intermittent failures (#5814)
- Refactored testPatternCache to avoid testing maximum cache size reached
2022-02-28 14:26:19 -05:00
Mark Payne
649e2d2556
NIFI-9731: Updated to use a shorter, simpler output format for FlowFiles when creating bulletins
NIFI-9731: Only simplify flowfile arg in bulletin after extracting the flowfile uuid
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5808
2022-02-28 13:21:10 -05:00
Mike Thomsen
589a1c24d1
NIFI-9124 Refactored nifi-jms-bundle to use JUnit 5
- NIFI-9122 Refactored nifi-influxdb-bundle to use JUnit 5
- NIFI-9121 Refactored nifi-ignite-bundle to use JUnit 5
- NIFI-9120 Refactored nifi-html-bundle to use JUnit 5
- NIFI-9119 Refactored nifi-hl7-bundle to use JUnit 5
- NIFI-9118 Refactored nifi-hive-bundle to use JUnit 5
- NIFI-9117 Refactored nifi-hbase-bundle to use JUnit 5
- NIFI-9116 Refactored nifi-hazelcast-bundle to use JUnit 5
- NIFI-9114 Refactored nifi-hadoop-bundle to use JUnit 5
- Added DisabledOnOs(OS.WINDOWS) to Hive tests with @TempDir from JUnit 5 due to Windows-specific problems in JUnit 5

This closes #5791

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-28 10:39:47 -06:00
Mark Payne
483968bac3
NIFI-6390, NIFI-1825: When we write to a FlowFile and that results in a 0-byte FlowFile, remove the content claim all together. This is more efficient to process, but far more importantly it prevents a 0-byte FlowFile from holding content in the Content Repository. Also fixed issue in which a Provenance Event cannot be replayed if there is no ContentClaim.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5810
2022-02-28 10:24:32 -05:00
Mike Thomsen
4b41570d17
NIFI-9714 Added overloaded toMap to MapRecord that can convert sub-records into maps.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5784
2022-02-28 09:12:51 -05:00
exceptionfactory
1eede30974
NIFI-9732 Upgraded Zip4j from 2.8.0 to 2.9.1
This closes #5811

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-27 06:09:14 -05:00
Mike Thomsen
ae1731c79e
NIFI-9134 nifi-metrics-reporting-bundle to use JUnit 5
- NIFI-9129 Refactored as much of nifi-kudu-bundle to use JUnit 5 as possible without breaking the integration tests which use test libraries from the Kudu project that depend on JUnit 4
- NIFI-9109 Refactored nifi-extension-utils to use JUnit 5. Left two tests with JUnit 4 because they used APIs that don't appear to have analogues in JUnit 5
- NIFI-9112 Refactored nifi-groovyx-bundle to use JUnit 5
- NIFI-9142 Refactored nifi-ranger-bundle to use JUnit 5
- NIFI-9136 Refactored nifi-network-bundle to use JUnit 5

This closes #5794

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-26 08:18:08 -06:00
Matthew Burgess
bcc8d03314
NIFI-9585 Upgraded H2 from 1.4 to 2.1.210
- Added nifi-h2-database module shading H2 1.4.200
- Implemented version checking and migration

This closes #5724

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-25 19:20:48 -06:00
markobean
4e3871fec7
NIFI-9072: improvements to ValidateXML including validate XML in attr… (#5324)
NIFI-9072: Improvements to ValidateXML including validate XML in attributes and non-schema, well-formed XML only validation, and additional documentation
- Updated description and added System Resource Consideration to discourage using large XML documents in attributes
- Migrate from XML DocumentBuilderFactory to SafeXMLConfiguration
2022-02-25 14:57:04 -05:00
Mark Payne
6cea5ea520
NIFI-9729: When restarting components in the VersionedFlowSynchronizer, first filter out any components that are intended to be stopped.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5806.
2022-02-25 14:29:38 -05:00
Paul Grey
e74991e705
NIFI-9727 - IndexOutOfBoundsException in CorrelationAttributePartitioner
This closes #5804

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-25 11:42:02 -05:00
exceptionfactory
49d1c747ca
NIFI-9724 Added set-sensitive-properties-algorithm command
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5801.
2022-02-25 11:03:55 -05:00
exceptionfactory
a6aba3bf8e
NIFI-9726 Removed duplicate nifi-utils dependencies from graph modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5803
2022-02-25 10:40:24 -05:00
Mark Payne
432f98714f
NIFI-9725: On shutdown, instead of spawning a background thread to shutdown Cluster Coordinator, do so in the calling thread. This avoids a race condition whereby the cluster coordinator cannot be determined because the other thread has shutdown the FlowController.
This closes #5800

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-24 22:16:01 -06:00
Mike Thomsen
13be48cfe3
NIFI-9113 Refactored nifi-grpc-bundle to use JUnit 5
- NIFI-9111 Refactored nifi-gcp-bundle to use JUnit 5
- NIFI-9110 Refactored nifi-flume-bundle to use JUnit 5
- NIFI-9108 Refactored nifi-evtx-bundle to use JUnit 5
- NIFI-9106 Refactored nifi-email-bundle to use JUnit 5
- NIFI-9105 Refactored nifi-easyrules-bundle to use JUnit 5
- NIFI-9104 Refactored nifi-druid-bundle to use JUnit 5

This closes #5790

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-24 16:59:44 -06:00
Mark Payne
c453d2ba8f
NIFI-9723: When we add controller-level Controller Services on restart of NiFi, ensure that all Controller Services are updated to include their property values, etc. Also ensure that for these services and reporting tasks we decrypt the property values.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5797.
2022-02-24 15:13:14 -05:00
Hsin-Ying Lee
c331f9cb9a
NIFI-9227 Run Once not working when scheduling strategy is CRON or Event driven (#5445) 2022-02-24 12:27:01 -05:00
Mark Payne
39483b9c12
NIFI-9716, NIFI-9577: Addressed issue in the PathFilter for GetFile / ListFile. For any file that is found in the Input Directory directly, it was previously being listed/fetched even if it didn't match the PathFilter. Additionally, updated the code to create a new File FIlter for every invocation of onTrigger. This was necessary for NIFI-9577 because the directory to monitor supports Expression Language and as a result may change from invocation to invocation, if using a function such as now() but the PathFilter would always relativize the path based on the value that was obtained when the processor was scheduled.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5788.
2022-02-24 09:13:16 -05:00
Mark Payne
9237b4e923
NIFI-9722: Do not throw an Exception from verifyCanUpdateProperties when property descriptor & parameter descriptor's sensitivities don't match - instead allow the set to happen and let processor become invalid. Also, allow values such as abc#{param} for ghost processors.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5795.
2022-02-24 08:58:54 -05:00
Mike Thomsen
51cf0fb0d1
NIFI-9133 Refactored nifi-media-bundle to use JUnit 5
- NIFI-9130 Refactored nifi-language-translation-bundle to use JUnit 5
- NIFI-9126 Refactored nifi-kafka-bundle to use JUnit 5
- NIFI-9125 Refactored nifi-jolt-record-bundle to use JUnit 5

This closes #5792

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-23 21:18:25 -06:00
Mike Thomsen
8b62ebeb76
NIFI-9103 Refactored nifi-datadog-bundle to use JUnit 5
- NIFI-9102 Refactored nifi-cybersecurity-bundle to use JUnit 5
- NIFI-9101 Refactored nifi-couchbase-bundle to use JUnit 5
- NIFI-9100 Refactored nifi-confluent-platform-bundle to use JUnit 5
- NIFI-9099 Refactored nifi-cdc to use JUnit 5
- NIFI-9098 Refactored nifi-ccda-bundle to use JUnit 5
- NIFI-9097 Refactored nifi-cassandra-bundle to use JUnit 5
- NIFI-9096 Refactored nifi-beats-bundle to use JUnit 5

This closes #5789

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-23 21:03:43 -06:00
Matthew Burgess
ab4cadc204
NIFI-9721: Support enum types in AvroTypeUtil.buildAvroSchema()
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5793.
2022-02-23 12:44:57 -05:00
Mark Payne
3511ce3d13
NIFI-9548: When disabling RPG transmission, wait for the ports to complete in a background thread instead of blocking the web thread. Also moved the RPG initialization logic into flow controller instead of flow service and added a delay in order to reduce likelihood of ConnectException happening when pointing to nodes in the same cluster
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5641.
2022-02-23 11:29:26 -05:00
Timea Barna
a3e1f32cae NIFI-9657 Create MoveADLS processor
This closes #5752.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-02-23 16:09:31 +01:00
Mark Payne
ce0122bd25
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5772.
2022-02-22 18:54:32 -05:00
Joe Gresock
dc1117d104
NIFI-9707: Resolved duplicate JLine dependency in nifi-toolkit
- Excluded groovy-groovysh from several toolkit modules

This closes #5782

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-22 14:55:43 -06:00
exceptionfactory
45f8795177
NIFI-9711 Added support for flow.json.gz in SetSensitivePropertiesKey
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5783.
2022-02-22 12:58:30 -05:00
Mark Payne
d0a23bc26b
NIFI-9704: Updated the ContentRepositoryScanTask to show details of how much content in the content repo is retained by each queue in the dataflow. Changed default for nifi.content.claim.max.appendable.size property from 1 MB to 50 KB. Updated docs to reflect the new default value and explain what the property does and how it's used.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5780.
2022-02-22 12:06:54 -05:00
Chris Sampson
04d061a7bd
NIFI-9581 add PutElasticsearchRecord relationship for output of successful Records sent to Elasticsearch
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5665.
2022-02-22 11:42:01 -05:00
xinyuwu4
22b47ccc7b
NIFI-9672: fix flaky tests caused by the use of HashMap.
This closes #5758

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-22 08:06:52 -05:00
lawye1973
0ade040b76
NIFI-9696 : DeleteS3Object don't have provenance
Changed from AmazonS3 to AmazonS3client.
Add REMOTE_INVOCATION provenance event with URL and "Object deleted"

This closes #5778

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-22 08:01:59 -05:00
lawye1973
54b4209798
NIFI-9713 : TagS3Object do not have provenance data
Changed from AmazonS3 to AmazonS3client.
Add REMOTE_INVOCATION provenance event with URL and "Object tagged"

This closes #5785

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-22 07:58:45 -05:00
Chris Sampson
07131a66ea
NIFI-9715 add option to output empty FlowFile from Elasticsearch REST API Json Query processors when there are no hits from query
This closes #5786

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-22 06:32:56 -05:00
Mike Thomsen
4141ed29ec
NIFI-9647 Added ExtractDocumentText Processor
- Based on https://github.com/tspannhw/nifi-extracttext-processor

This closes #5732

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-21 20:01:25 -06:00
Tamas Palfy
635824904d
NIFI-9609 Added nifi-snowflake-bundle with SnowflakeComputingConnectionPool
- Updated DBCPConnectionPool for extension

This closes #5692

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-21 19:38:47 -06:00
lawye1973
8d80fb2930
NIFI-9694 : Correct FetchS3Object.java provenance data URL
Current provenance data URL is incorrect as it's hardcoded.
Follow PutS3Object.java method, using AmazonS3Client (instead of AmazonS3) to use getResourceUrl(bucket,key) to build the correct URL for provenance data

This closes #5779

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-19 18:46:20 -05:00
levilentz
dfbf2e3cea
NIFI-9286: JOLT Expression Language
Fixes NIFI-6213 and adds in functionality to use expression language in class and module specification
NIFI-9286: adding JOLT unit tests
NIFI-9286: addressing PR feedback
Fixes a problem with the scope of the EL for module directory
NIFI-9286: alignment of JOLT processors
NIFI-9286: fix checkstyle

This closes #5444

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-19 14:11:44 -05:00
Nathan Gough
7ef2fd2986
NIFI-7333 Added OIDC trust store strategy property
This closes #5753

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-18 15:19:22 -05:00