Commit Graph

7358 Commits

Author SHA1 Message Date
Mark Payne 38b51b0dde
NIFI-10037: When system test fails to clean up flow, destroy the entire environment so that the next test starts in a healthy state. Name troubleshooting directories with the name of the test class to avoid ambiguity. Also added a log statement so that we know which test is running when looking at the log output from the tests themselves. Finally, found an issue in AbstractComponentNode in which we iterate over the elements in a Map and call setProperty, which can update the underlying Map - updated to first create a copy of the HashMap. Updated that in this Jira because I suspect it is causing one of the tests failures that I've been investigating.
This closes #6059

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-19 13:14:30 -05:00
Joe Gresock 1cfca0d978
NIFI-10036: Corrected Elasticsearch Client Service to prefix paths for all requests
- Prefixing endpoint paths with a forward slash ensures correct HTTP request formatting required for some deployments with a forwarding proxy

This closes #6058

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-19 13:02:40 -05:00
exceptionfactory 7754e8903e
NIFI-10015 Upgraded Hadoop from 3.3.0 to 3.3.2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6051.
2022-05-19 17:33:53 +02:00
markap14 88e53eae8c
NIFI-10001: Fixed bugs that caused some components to not have their scheduled state updated. (#6049)
* NIFI-10001: Fixed issue in which some components may fail to update the scheduled state when comparing flows

* NIFI-10001: Fixed bugs that caused some components to not have their scheduled state updated. When comparing two flows, now allow specifying how to determine a VersionedComponent's ID for comparison. When comparing local flow against flow from registry, use Versioned Component ID. But when comparing two instantiated flows, such as local flow vs. cluster flow, use the VersionedComponent's Instance ID instead. This ensures that we can properly compare two components even if there are several instances of a given flow
2022-05-18 13:53:36 -04:00
exceptionfactory fe424a2d42
NIFI-9805 Refactored Distributed Cache Servers using Netty
- Added Map and Set Cache Servers based on nifi-event-transport components
- Removed custom servers and unused socket stream components
- Reduced duplication on protocol classes
- Added checks for readable bytes
- Added mark and reset handling for buffer reads

This closes #6040
Signed-off-by: Paul Grey <greyp@apache.org>
2022-05-18 12:40:16 -04:00
exceptionfactory 85d5e2071b
NIFI-10031 Downgraded Jakarta Mail from 2.0.1 to 1.6.7 (#6052)
- Downgraded Jakarta Activation from 2.0.1 to 1.2.2 in nifi-bootstrap
- Updated assembly configurations to include Jakarta Activation in java11 directory
2022-05-18 11:38:36 -04:00
tpalfy 2a8d07641e
NIFI-9895 Allow parameters to reference controller services (#5958)
* NIFI-9895 Allow parameter to reference controller service. Check read and write authorization for both previous and newly set controller service. Authorization done for both property or parameter change. Import/export handled by switching between instance id and versioned id.
2022-05-16 17:55:03 -04:00
Lehel eba1318529
NIFI-10028: Added QuerySalesforceObject additionalDetails and improved age field property descriptions
This closes #6047

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-16 16:24:20 -05:00
Matthew Burgess bd1e033d55 NIFI-9967: Added FlowSerializationStrategy to determine which flow formats (XML,JSON) to save
This closes #6001.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2022-05-16 16:22:19 -04:00
Peter Gyori 3f16a41ca1
NIFI-10010: ListenTCP adds Certificate Subject and Issuer FlowFile attributes
This closes #6032

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-13 15:48:12 -05:00
markap14 8031b62351
NIFI-10001: When enabling a collection of Controller Services, change… (#6042)
* NIFI-10001: When enabling a collection of Controller Services, changed logic. Instead of enabling dependent services and waiting for them to complete enablement before starting a given service, just start the services given. The previous logic was necessary long ago because we couldn't enable a service unless all dependent services were fully enabled. But that changed a while ago. Now, we can enable a service when it's invalid. It'll just keep trying to enable until it becomes valid. At that point, it will complete its transition from ENABLING to ENABLED.

* NIFI-10001: Restored previous implementation for StandardControllerServiceProvider, as the changes were not ultimately what we needed. Changed StandardProcessGroup to use a ConcurrentHashMap for controller services instead of a HashMap with readLock. This was causing a deadlock when we enable a Controller Service that references another service during flow synchronization. Flow Synchronization was happening within a write lock and enabling the service required a read lock on the group. Eventually the thread holding the write lock would timeout and release the write lock. But this caused significant delays on startup. By changing to a ConcurrentHashMap, we alleviate the need for the Read Lock. Also noticed in testing that the StandardNiFiServiceFacade did not save flow changes when enabling dependent services so added call to controllerFacade.save().
2022-05-13 15:09:23 -04:00
exceptionfactory 30f7c1ba1e
NIFI-9990 This closes #6030. Improved FTP 550 file unavailable handling
- Improved File Not Found reply detection
- Added Permission Denied reply handling

Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-13 07:44:22 -07:00
Chris Sampson 313d70520d
NIFI-9470 This closes #6027. Allow creation of Parameter Context without any Inherited Parameter Contexts
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-13 07:39:11 -07:00
exceptionfactory 5215cc0ca0
NIFI-10026 This closes #6043. Added jakarta.activation runtime dependency to bootstrap
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-13 06:56:14 -07:00
exceptionfactory eecdf38158
NIFI-10014 Upgraded Registry Test JDBC Drivers
- Upgraded MySQL from 8.0.15 to 8.0.29
- Upgraded MariabDB from 2.4.1 to 3.0.4
- Upgraded PostgreSQL from 42.2.19 to 42.3.5
- Removed unsupported database version references from project README.md
- Added MariaDB 10.3 to list of Registry examples

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6033.
2022-05-12 22:41:00 +02:00
exceptionfactory e826aca8f2
NIFI-10024 This closes #6041. Upgraded Netty from 4.1.74 to 4.1.77
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-12 12:48:21 -07:00
exceptionfactory 7cdbde1eba
NIFI-10020 This closes #6039. Corrected ClassNotFoundException in EmailNotificationService
- Added activation-api runtime dependency to nifi-bootstrap

Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-12 12:46:41 -07:00
Mark Payne 4a60d1673e
NIFI-10023: Ensure that any files that are archived upon startup properly increment the archival count in the content repo
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6037
2022-05-12 15:38:49 -04:00
Paul Grey a528b89169
NIFI-10003 - NiFi build fails in UTC+ time zones
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6038
2022-05-12 13:55:51 -04:00
Bryan Bende e2b5773027
NIFI-9969 Add extension restrictions to C2 component definition model
Update test to verify ListHDFS does not have restrictions

Fix checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6031
2022-05-10 18:56:22 -04:00
Pierre Villard dfc8b9c4c8
NIFI-10009 - bump Kudu client from 1.15 to 1.16
This closes #6029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-10 13:21:07 -05:00
Lehel Boér 8247910a81 NIFI-9865: Add Atlas lineage for GCS processors (PutHDFS)
This closes #5926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-05-10 11:04:30 +00:00
Tamas Palfy 8aaa3d2f10 NIFI-9991 Fix: The asn1 StandardRecordModelIteratorProvider's interator returns only the first object read from the input stream.
This closes #6019.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-05-09 20:37:41 +00:00
dependabot[bot] a038309ba4
NIFI-10006 Upgraded async from 2.6.3 to 2.6.4 for Registry
Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

This closes #6007

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-09 14:02:05 -05:00
Mark Payne 9311188785
NIFI-10004: Allow loading .class files that are relative to the NAR directory, in addition to the .jar / .nar files
NIFI-10004: When determining classname from a file, consider File.separator instead of just /
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6026
2022-05-09 14:54:01 -04:00
exceptionfactory c3829e2198
NIFI-9999 Refactored MiNiFi tests using JUnit 5
- Replaced docker-compose-rule-junit4 with docker-compose-junit-jupiter

NIFI-9999 Removed JUnit 4 dependencies from MiNiFi Maven configuration

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6024
2022-05-09 14:05:38 -04:00
Matthew Burgess 728744a49b
NIFI-9998: This closes #6023. Upgrade Hive3 to 3.1.3
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-05-09 10:53:32 -07:00
Mark Payne b25201dd22
NIFI-9993: Fixed bug in initialization in which the Content Repo did not properly increment the counter for how many files exist in the archive directories. This was causing the counter to become negative in some cases, which caused processors to incorrectly pause, waiting for content archive cleanup to occur when, in fact, there were no files archived
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes #6021.
2022-05-09 12:22:00 -04:00
exceptionfactory 9c2e255fae
NIFI-9986 Upgraded commons-configuration2 to 2.7 for Accumulo
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6014.
2022-05-09 16:53:34 +02:00
exceptionfactory 0c8305950c
NIFI-9987 Standardize Commons IO on version 2.11.0
- Removed version declarations from multiple modules
- Adjusted PutDynamoDB to remove catch for IOException not thrown in Commons IO 2.11.0

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6015.
2022-05-09 12:06:37 +02:00
Matthieu Bertin 65dbbc9e76 NIFI-9966 Corrects the registry loading of large flowfiles from git
This closes #6012

Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
2022-05-07 13:37:28 +01:00
exceptionfactory 8f3445d48f
NIFI-10000 Updated Pull Request Template Summary
- Removed italicized formatting for example summary paragraph and updated example issue number

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #6025.
2022-05-06 18:36:27 -04:00
exceptionfactory 5e8b4d3144
NIFI-9997 Upgraded Ranger from 2.1.0 to 2.2.0 (#6022)
- Upgraded Hadoop from 3.1.1 to 3.3.2 for Ranger Plugins
2022-05-06 16:07:53 -04:00
exceptionfactory ea75a0a996 NIFI-9995 Replaced Custom Filters with Spring Security HeaderWriter
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6020.
2022-05-06 12:22:30 -04:00
exceptionfactory 64f9b66141
NIFI-9984 Allow 200-series responses in OAuth2 Access Token Provider
This closes #6016

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-05-04 17:31:07 -04:00
exceptionfactory 272325cb4e NIFI-9988 Corrected Property Decryption for Authorizers and Providers
- Updated Protection Scheme Resolver to support both Name matching and Path matching

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6017.
2022-05-04 15:19:17 -04:00
Emilio Setiadarma e20aa0ea2a
NIFI-9755 Added ConsumeTwitter Processor
This closes #5947

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-03 21:36:48 -05:00
dependabot[bot] 3699d42116
NIFI-9932 Upgrade minimist from 1.2.5 to 1.2.6
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

This closes #5971

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-03 13:49:45 -05:00
exceptionfactory 0f6679f2df
NIFI-9980 Corrected conflicting GCP dependencies
- Replaced google-cloud-bom 0.172.0 with libraries-bom 25.2.0 in nifi-gcp-bundle
- Removed specific versions from Google dependencies in nifi-gcp-processors and nifi-gcp-services-api

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6010.
2022-05-03 13:51:34 +02:00
Matthew Burgess 04121c9d6a
NIFI-8533: Replace deprecated jython-shaded with jython-slim
This closes #5531

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-02 15:47:23 -05:00
Matt Gilman e175b146cd NIFI-9978:
- Updating the condition under which we show the Download Flow menu items.

This closes #6008.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-05-02 19:40:48 +02:00
Juldrixx b04632129f
NIFI-9798 Added Proxy Support for ConsumeGCPubSub and PublishGCPubSub
This closes #5868

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-02 11:37:47 -05:00
exceptionfactory 21b263bd71
NIFI-9979 Removed nifi-druid-bundle and associated components
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6009.
2022-04-30 16:43:36 +02:00
Tamas Palfy 5aa71570ff
NIFI-9977 In StandardOauth2AccessTokenProvider add new property to be able to set "scope".
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6006.
2022-04-30 16:41:27 +02:00
Mike Thomsen 7271e8cea7
NIFI-7234 Standardized on Avro 1.11.0
NIFI-7234 Replaced Jackson 1.X references with Jackson 2.X references in various classes.
NIFI-7234 Added jackson-annotations to nifi-hdfs-processors.
NIFI-7234 Various updates to bring our test cases into better alignment with the Avro specification as of 1.11.
Fixed a checkstyle issue.

NIFI-7234 Made changes requested in a review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5900
2022-04-29 17:48:59 -04:00
exceptionfactory fc2f539c7d
NIFI-9976 Upgraded json-smart to 2.4.8
- Replaced nifi-framework-bundle managed dependency to root managed dependency

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6005.
2022-04-29 10:12:41 +02:00
exceptionfactory 2d5e24c0a2
NIFI-9975 Upgraded OWASP Dependency Check from 6.5.3 to 7.1.0
- Removed unnecessary suppression configurations due to detection improvements

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6004.
2022-04-29 10:08:49 +02:00
Lehel 569929269a NIFI-9956: Fix QuerySalesforceObject schema parsing with nested field strategy
This closes #5998.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2022-04-28 17:17:24 +02:00
Lehel a97c20cdb2
NIFI-9936 Added DescribedValue in PropertyDescriptor Builder for AllowableValues
This closes #5977

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-04-28 10:04:44 -05:00
exceptionfactory fbfdcdca9f
NIFI-9970 Restructured Pull Request Template
- Added Summary, Tracking, and Verification section headings
- Reworded checklist to declarative statements instead of questions
- Added JDK 17 to list of build platforms

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6002.
2022-04-28 10:04:11 +02:00