Commit Graph

7069 Commits

Author SHA1 Message Date
Peter Turcsanyi dafa03a21a
NIFI-9421: Running NiFi Stateless with local NARs only
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5557.
2021-12-08 15:59:31 -05:00
Lehel Boér bf288f3ba1
NIFI-9454: Moved file renaming after closing the OutputStream in StandardFlowConfigurationDAO (#5581) 2021-12-08 13:38:45 -05:00
exceptionfactory 563df24067
NIFI-9448 Improved S2S HTTP Extend Transaction Exception Handling
- Refactor background transaction extension to ExtendTransactionCommand
- Avoid closing S2S HTTP client for IllegalStateExceptions
- Avoid creating additional S2S HTTP client instance for transaction extension commands
- Add check for extend transaction requests received in client test class
- Add null check for Peer Persistence implementation in PeerSelector

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5577.
2021-12-08 09:33:55 -05:00
Bence Simon 12015a17dd
NIFI-9397 Added Custom Authorization property to JettyWebSocketClient
This closes #5574

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-07 10:54:58 -06:00
ahmed shaaban 70822b9c8d
NIFI-9093 This closes #5578. GetSplunk Processor hangs addressed by adding timeouts.
changed the required flag to false on ConnectTimeout and ReadTimeout properties

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-12-07 08:17:31 -07:00
Lehel 1eb4264e34 NIFI-9447: Fix SNMP related tests to find available UDP ports instead of TCP
This closes #5576.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-12-06 21:00:40 +01:00
Bryan Bende 0f027743d1
NIFI-9443 Update NAR plugin to 1.3.3 and update data model for extension manifest to capture new fields
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5570.
2021-12-06 08:58:40 -05:00
Chris Sampson 53809dd83f
NIFI-9439:
- Add PutElasticsearchJson processor to Elasticsearch REST bundle
- Deprecate PutElasticsearchHttp/PutElasticsearchHttpReccord in favour of Elasticsearch REST processors

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5566.
2021-12-05 15:39:35 -05:00
Mark Payne 2a3f7dafd6
NIFI-9442: When deleting a connection, ensure that when the source of the connection is a funnel that its upstream components are checked (recursively)
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5568.
2021-12-05 06:39:52 -05:00
Mark Payne 7a83c32a01
NIFI-9441: Ensure that we only update our member variable for the latest timestamp after processing all objects within the GCS Bucket
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5567.
2021-12-04 23:49:27 +01:00
exceptionfactory 41ff6f076b NIFI-9409 Updated links for EVP BytesToKey and PBKDF2 2021-12-03 13:27:34 -05:00
Mark Payne f95044ff87
NIFI-9433: When a Connection is unregistered from the NioAsyncLoadBalanceClient, make sure that we only cancel its active transaction if the transaction belongs to the appropriate connection. Also ensure that when we do cancel a transaction / call its failure callback, we purge the collection of any FlowFiles that have been sent in that transaction. This ensures that we cannot later attempt to failure the transaction again, decrementing the count of FlowFiles for the connection more than we should.
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5564.
2021-12-03 10:31:31 -05:00
Mark Payne 90b39b593a
NIFI-9069 Changed framework dataflow serialization to support JSON
- Changed framework so that it serializes the dataflow into a VersionedDataflow using JSON as well as XML, and prefers the JSON representation on load, if it's available. This also results in the need for the cluster protocol to exchange its representation of the dataflow to using JSON. Rather than re-implementing all of the complex logic of Flow Fingerprinting, updated to just inherit the cluster's flow.
- Moved logic to synchronize Process Group with Versioned Process Group into a new ProcessGroupSynchronizer class instead of having all of the logic within StandardProcessGroup
- Reworked versioned components to use an instance id.
- Renamed StandardFlowSynchronizer to XmlFlowSynchronizer; introduced new StandardFlowSynchronizer that delegates to the appropriate (Xml or Versioned)FlowSynchronzer
- Updated to allow import of VersionedProcessGroup even if not all bundles are available - will now use ghost components
- Introduced a VersionedDataflow object to hold controller-level services, reporting tasks, parameter contexts, templates, etc.
- Allow mutable requests to be made while nodes are disconnected. Also fixed issue in AbstractPolicyBasedAuthorizer that caused ClassNotFoundException / NoClassDefFoundError if the authorizations were changed and then a node attempts to rejoin the cluster. The Authorizer was attempting to use XmlUtils, which is in nifi-security-utils and so so by madking nifi-security-utils a provided dependency of nifi-framework-api, but this doesn't work, because nifi-framework-api is loaded by a higher-level classloader, so the classloader that loads AbstractPolicyBasedAuthorizer will never have the appropriate classloader to provide nifi-security-utils. Addressed this by copying the code for creating a safe document builder from XmlUtils to AbstractPolicyBasedAuthorizer.
- Fixed bug that occurred when importing a Process Group that has 2 parameter contexts, one inheriting from another, where neither is pre-defined in the existing flow
- Fixed bug that was encountered when Updating a Versioned Process Group where one version had a disabled processor and the other had the processor running.
- Increased system-tests workflow timeout to 120 minutes
- Added additional exception handling to system tests

This closes #5514

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-02 15:10:38 -06:00
Pierre Villard fede6b9354
NIFI-9432 - fix typo in diagnostic output (#5562) 2021-12-02 15:25:23 -05:00
Tamas Palfy ff864266f5
NIFI-9436 - In AbstractPutHDFSRecord make sure the record writers use the FileSystem object the processor already has.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5565
2021-12-02 14:24:12 -05:00
M Tien 2b415de912
NIFI-9336 - Show icon for property values with leading or trailing whitespace (#5559)
* NIFI-9336 - Show icon in processor and controller services configurations when property values contain leading or trailing whitespace

* - Address PR feedback

* - Fix a bug to clean up tooltips to prevent a DOM leak

This closes #5559
2021-12-02 13:31:19 -05:00
Chris Sampson d45b23d89f
NIFI-8153 custom date/time format properties for PutElasticsearchRecord
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5552.
2021-12-01 07:28:41 -05:00
exceptionfactory 2844652f72
NIFI-9426 Removed unused jackson-mapper-asl from MiNiFi
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5561
2021-11-30 20:39:04 -05:00
exceptionfactory 55e91d607c
NIFI-9420 Upgraded MiNiFi Guava from 26.0 to 31.0.1
- Replaced usage of Guava Lists with standard Java classes in TemplatesIteratorTest

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5556
2021-11-30 15:02:08 -05:00
Matthew Burgess 07c4a05cdf
NIFI-9412: Autogenerate sensitive key in MiNiFi if not present
This closes #5558

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-30 13:57:07 -06:00
Pierre Villard e6b573e79e
NIFI-9419 ParseCEF - Upgraded parcefone and supported empty extensions
- Upgraded com.fluenda:parcefone from 2.0.0 to 2.1.0
- Added Accept empty extensions property to ParseCEF

This closes #5555

Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-30 11:27:50 -06:00
exceptionfactory 6a6b755aaa
NIFI-9394 Removed RequestLogger and TimerFilter
- Removed logger references from logback.xml

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5543.
2021-11-29 12:55:39 -05:00
Duan 8171ecb2c1
NIFI-9381 Updated CompositeUserGroupProviderTestBase with LinkedHashMap
- Replaced HashMap with LinkedHashMap to avoid potential non-deterministic results for user group properties

This closes #5524

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-29 10:45:59 -06:00
Joe Gresock a5581562bf
NIFI-9416: Fixing NPE when updating param context without inheritedParameterContexts (#5553) 2021-11-29 11:29:33 -05:00
Paul Grey d6b6dd1eb4
NIFI-9389 - NPE guard for ExecuteProcess when not triggered
- Added support for TestRunner.run() with zero operations

This closes #5544

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-29 09:27:04 -06:00
Joe Gresock 99954204b6 NIFI-9403: Deprecating FetchElasticsearch processors in favor of GetElasticsearch
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>

This closes #5545.
2021-11-27 13:47:06 +00:00
Lehel Boér bd7536b257 NIFI-3328: SendTrapSNMP and ListenTrapSNMP processors added. 2021-11-24 21:12:15 +01:00
Joe Gresock dfd8adfd8e
NIFI-9411: Adding missing 'setSubtitle' call for Controller Services modal
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5551.
2021-11-24 18:01:48 +01:00
Peter Gyori 2dcff09f7f NIFI-9410: Fix for ConsumeMQTT processor in stateless environment
NIFI-9410: Added displayName to the QoS processor property

This closes #5549.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-11-24 10:05:43 +01:00
Pierre Villard fea6a7f6db
NIFI-9408 - MiNiFi - remove Ignite dependencies
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5546
2021-11-23 10:03:18 -05:00
Robert Kalmar f28e755bc3
NIFI-9392 PutHive3Streaming processor throws java.lang.NoClassDefFoundError on startup
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5536
2021-11-23 10:01:19 -05:00
Pierre Villard bb97ecb2b1
NIFI-9395 - MiNiFi - bump httpclient to 4.5.13
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5537
2021-11-23 09:59:56 -05:00
Mike Thomsen 1ca5a1dbbf
NIFI-9149 Refactored multiple NAR modules to use JUnit 5
- NIFI-9163 Refactored nifi-websocket-bundle to use JUnit 5
- NIFI-9162 Refactored nifi-update-attribute-bundle to use JUnit 5
- NIFI-9161 Refactored nifi-tcp-bundle to use JUnit 5
- NIFI-9160 Refactored nifi-stateful-analysis-bundle to use JUnit 5
- NIFI-9159 Refactored nifi-sql-reporting-bundle to use JUnit 5
- NIFI-9158 Refactored nifi-spring-bundle to use JUnit 5
- NIFI-9157 Refactored nifi-splunk-bundle to use JUnit 5
- NIFI-9156 Refactored nifi-spark-bundle to use JUnit 5
- NIFI-9155 Refactored nifi-solr-bundle to use JUnit 5
- NIFI-9154 Refactored nifi-social-media-bundle to use JUnit 5
- NIFI-9153 Refactored nifi-snmp-bundle to use JUnit 5
- NIFI-9152 Refactored nifi-smb-bundle to use JUnit 5
- NIFI-9151 Refactored nifi-slack-bundle to use JUnit 5
- NIFI-9150 Refactored nifi-site-to-site-reporting-bundle to use JUnit 5
- NIFI-9149 Refactored nifi-single-user-iaa-providers-bundle to use JUnit 5

This closes #5362

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-23 08:25:30 -06:00
Chris Sampson 6259269f73
NIFI-8779 correctly pass MIRROR_BASE_URL from DockerBuild to Dockerfile; address shellcheck issues with Docker-related scripts
Allow override of DISTRO_PATH for NiFi dockerhub build to enable image creation from the Apache Dev server during Release Candidate voting

Rationalise nifi-docker builds and integration-tests (remove duplicate script directories)

This closes #5213.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2021-11-22 19:19:19 -05:00
exceptionfactory c1e54276b8 NIFI-9355 Upgraded Apache Curator from 4.2.0 to 5.2.0
- Removed unused Curator dependency declaration from MiNiFi

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5503.
2021-11-22 18:24:39 -05:00
exceptionfactory 0e09750a4d NIFI-9393 Set Port Scheduled State for Flow Definitions
- Set Scheduled State for Versioned Port and Versioned Remote Port when mapping Flow Definition
- Updated StandardProcessGroup to set disable Port based on Scheduled State of DISABLED
- Updated StandardProcessGroup to set Remote Port transmitting based on Scheduled State of ENABLED

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5534.
2021-11-22 18:16:42 -05:00
markap14 839fbf7d19
NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which H… (#5533)
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests

* NIFI-9382: Renamed interface based on review feedback

* NIFI-9382: Removed ReentrantKerberosUser.
2021-11-22 14:55:19 -05:00
exceptionfactory c033debdf3
NIFI-9399 This closes #5542. Apply Secure Processing to TransformXml XSLT
- Added XML Stream Reader processing for XSLT with external entities disabled
- Removed unused XsltValidator
- Upgraded Saxon-HE from 9.6.0-5 to 10.6

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-11-22 09:18:57 -07:00
Chris Sampson 1745d2a88e
- NIFI-9388: add GetElasticsearch to fetch Elasticsearch document using the ElasticsearchClientService
- NIFI-9387: add Proxy capability to ElasticsearchClientService
- NIFI-1576: allow GetElasticsearch to run without requiring FlowFile input

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5535.
2021-11-20 06:26:55 -05:00
Pierre Villard 1724cb0d3b
NIFI-9396 - MiNiFi - bump junit to 4.13.2
This closes #5538

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-19 09:09:50 -06:00
Timea Barna 848a22eb29
Revert "NIFI-9385 Add Flow Metrics producer for Cloudera Manager"
This reverts commit 785021dade.

This closes #5540

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-19 08:52:20 -06:00
Timea Barna 785021dade
NIFI-9385 Add Flow Metrics producer for Cloudera Manager
This closes #5528

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-18 11:51:53 -06:00
exceptionfactory 605346ae0f
NIFI-9291 Added NiFi HTTP request logging
- Added nifi.web.request.log.format property
- Added Filters to set and retrieve authenticated username for logging

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5527.
2021-11-18 10:30:50 -05:00
Bryan Bende 18fc492e4c
NIFI-9378 Create new artifact in nifi-assembly that packages all exte… (#5521)
* NIFI-9378 Create new artifact in nifi-assembly that packages all extension manifests

* Minor changes to clean up manifest zip creation

This closes #5521
2021-11-17 14:38:26 -05:00
M Tien 5016eac010
NIFI-9205 - Update prioritizer configuration (#5516)
* NIFI-9205 - Update prioritizer configuration

* - Address review feedback

This closes #5516
2021-11-17 13:55:02 -05:00
exceptionfactory d66219739b
NIFI-5821 Added Engine Name to Script Engine property descriptions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5529
2021-11-17 12:32:55 -05:00
exceptionfactory 0cf515c9c0 NIFI-9384 Corrected usage and generics in ListenTCP
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5526.
2021-11-17 11:54:51 -05:00
Bryan Bende 77f235bf1c
NIFI-9379 Add dependent properties and resource definitions to manifest model
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5522.
2021-11-17 06:36:10 -05:00
Bryan Bende 15465bb4b8
NIFI-9380 Exclude snappy-java used in parquet bundle and rely on snappy-java from Hadoop libs NAR
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5525
2021-11-16 15:51:20 -05:00
Nathan Gough f7466fd0f1
NIFI-8806 Refactored ListenTCP using Netty
- Deprecated Max Number of Receiving Message Handler Threads property
- Deprecated Pool Receive Buffers property
- Updated TestListenTCP using Netty EventSender

This closes #5493

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-11-16 11:27:38 -06:00