Removed Provenance Repository from the stateless RepositoryContextFactory and added it to the DataflowTriggerContext. This was necessary because the previous design overlooked the possibility of many threads concurrently running the same dataflow. They all shared the same StatelessProvenanceRepository, but the code was designed as if only a single thread would be using the repository. As a result, the events that were registered with the stateless prov repo were being copied many times into NiFi's underlying provenance repository. This refactoring also led to the discovery of some old Java 8 syntax that could be cleaned up, and it led to the discovery of some methods that were no longer being used and could be cleaned up. Finally, in testing, I found that when a Stateless Group was scheduled, it scheduled the triggering of the stateless group before marking the state as RUNNING; as a result, the second thread could run, determine that the state is STARTING instead of RUNNING, and return without triggering the stateless group. This was addressed by ensuring that we set the state to RUNNING before triggering the stateless group to be triggered.
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Updated Standard Controller Service Provider to enable requested Controller Services that do not depend on a disabled Controller Service, instead of failing to enable the entire set of requested Controller Services
- Updated enabling process improves behavior when restarting NiFi and attempting to enable large numbers of Controller Services, some of which depend on disabled Services
Signed-off-by: David Handermann <exceptionfactory@apache.org>
This closes#9392
- Added EphemeralKeyStoreBuilder to nifi-security-ssl
- Removed nifi-security-utils
- Moved StandardTlsConfiguration to nifi-ssl-context-service
- Refactored tests to use EphemeralKeyStoreBuilder and nifi-security-cert-builder for TLS
Signed-off-by: Joseph Witt <joewitt@apache.org>
- Fixed Reporting Task property migration: in case of sensitive properties the encrypted form was migrated to the new property value
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Only synchronize flow with the coordinator after getting cluster response
This closes#9317
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-13721: Migrating additional details to markdown.
* NIFI-13721: Rebuilding documentation to generate mark up on the client.
* NIFI-13721: Removing old documentation approach.
* NIFI-13721: Rebase main and final fixing button colors.
* NIFI-13721: Fixing rat issues.
* NIFI-13721: Removing generated comonent documentation.
- Removing links and images from additional details markdown.
* NIFI-13721: Recreating package lock following rebase.
* NIFI-13721: Updating documentation for M3.
* NIFI-13721: Fixing broken test.
* NIFI-13721: Addressing review feedback.
* NIFI-13721: Addressing review feedback.
* NIFI-13721: Ensuring the selected item is visible after routing.
* NIFI-13721: Fixing reportingTaskTypes in finding extensions for a given type.
* NIFI-13721: Updating the see also component to handle when the referenced extension is not installed.
* NIFI-13721: Fixing expansion panel header styles.
- Ensure NARs are loaded in parent-first order during restart
- Allow replacing a NAR when other NARs are dependent on it
- Improvements to catch Throwable in several places to handle NoClassDefFoundError
Signed-off-by: David Handermann <exceptionfactory@apache.org>
When inheriting Registry Clients from a cluster's flow, any missing registry clients are removed at the end instead of the beginning of the synchronization logic. This allows those registry clients to still be referenced while performing synchronization. Also found that if a Processor is missing from cluster's flow but is running in local flow, on startup we get an error indicating that the processor does not belong to the associated process group so fixed that in tandem. Finally, noticed while verifying the fix that we check if the proposed flow is empty with a null check instead of using the isFlowEmpty() method - this could result in inheriting an empty flow from cluster even when a flow is loaded locally.
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Fixed stateless group startup around Controller Services by changing the startup logic in two ways. First, we enable Controller Services before we create the Stateless Flow. Secondly, we do not bother enabling the ephemeral Controller Services that are created during the Stateless Flow creation/synchronization. This ensures that if Processors make use of a Controller Service during its @OnScheduled method (which is part of the Stateless Flow initialization process) that the Controller Service is Enabled. It also avoids calling @OnEnabled methods of Controller Services that will never actually be used, as Processors make use of the existing Controller Services when running within standard NiFi .
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-13632: Updating content access.
- Removing existing content viewer application.
- Introduced new front end based content viewer applicaiton.
- Introduced new bundled content viewers for hex and images.
- Deleted previous image content viewer.
- Migrated existing standard content viewer (which handles json, xml, yaml, csv, and text) to utilize new content access interface.
- Moved standard content viewer into its own NAR.
- Moved and renamed custom ui utils and content viewer utils into nifi-common.
- Added mime type to FlowFileSummary response payload to help drive the initially opened content viewer.
* NIFI-13632: Fixing rat issue.
* NIFI-13632: Fixing CI issue.
* NIFI-13632: Removing standard content viewer war from code coverage.
* NIFI-13632: Addressing review feedback.
* NIFI-13632: Fixing import.
* NIFI-13632: Further simplification of content viewer state.
* NIFI-13632: Removing unneeded code.
* NIFI-13632: Adding a min height around codemirror.
* NIFI-13632: Addressing review feedback.
- Rendering mime type in the content viewer.
- Preventing additions to the browser history as the user changes how the content is viewed.
- Appending forward slash to content viewer UIs to prevent unnecessary redirect.
- Removing Standard Content Viewer title based on review feedback.
- Fixing double scroll bar in the Standard Content Viewer on small browser heights.
- Removed unnecessary application.yml.
- Bumping spring boot version.
This closes#9221
- Added perform validation for Controller-level Services
- Added wait on Parameter Provider validation status to allow for Controller Service enabling
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#9210.
Fixed issue in which we recursively validate that we can change Execution Engine of a group, but without changing the top level, inherited groups can't actually be changed; instead, replaced logic to ensure that we can change the top level, that we don't validate the constraint that a STATELESS group cannot have a STATEFUL child, and then validate that all descendant groups can be updated, but without checking the descendants' parent groups.
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added REST resource methods for managing assets associated with Parameter Contexts
- Added CLI commands for managing assets
- Added and updated system test for assets and Parameter Contexts
- Updated flow synchronization to consider referenced asset changes
- Adjusted authorization for cluster nodes to read Parameter Contexts
This closes#9138
Co-authored-by: Bryan Bende <bbende@apache.org>
Co-authored-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added ParameterValueMapper for handling serialization of Parameter Values for Flow Configuration
- Added Parameter Group retrieval method for Flow Synchronizer
- Changed conditional from evaluating HTTPS port to evaluating store path and password properties
This closes#9083
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Return copied set of Extensions from ExtensionManager to avoid concurrent modification
This closes#9089
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Ensure on startup that Content Repo archive directories are created, if configured to do so, rather than creating them on demand. If IOException thrown when archiving file, delete it instead. Cleaned up some code duplication between remove(ResourceClaim) and archive(ResourceClaim) methods
This closes#9079
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Add NarPersistenceProvider to framework API with standard implementation
- Add NarManager to orchestrate interactions with the persistence provider and framework
- Add REST API for upload, download, list, and delete of custom NARs
- Add CLI commands for call these REST APIs
- Implement an upload request replicator for replicating a file upload across the cluster
- Update ExtensionManager, NarLoader, and JettyServer to support unloading a NAR
- Add ability to replace a NAR in place by unloading components, loading new NAR, and reloading components
- Add system tests for uploading a NAR and verifying other APIs
- Add NAR digest to NarNode and summary DTO, fix bug in digest utils
- Sync NARs from cluster coordinator when joining the cluster
This closes#9017
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Remove nifi-socket-utils and moved remaining classes to referencing framework components
This closes#9059
Signed-off-by: David Handermann <exceptionfactory@apache.org>
This closes#8978
- Removed nifi-property-protection-api and implementation modules
- Removed nifi-toolkit-encrypt-config and minifi-toolkit-encrypt-config modules
- Removed extra bootstrap.conf configuration files for property protection implementations
Signed-off-by: Joseph Witt <joewitt@apache.org>
Removed the entire integration-tests-ci profile and reference in GitHub workflow.
Removed all tests which require specific local env configs/cloud credentials/ or simply do not work and thus have been blocked from the build for a long time.
Set Disabled annotation on a couple key tests which are by design manual and must be retained.
The move to testcontainers has been a dramatic improvement in test reliability/behaviors and should be a model for all such tests going forward.
This closes#8818
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Methods for getBranches, getDefaultBranch, and createBranch in flow registry client
- Add branch as a field to all relevant objects
- Add optional branch argument to REST API for retrieving buckets
- Add new REST API for retrieving branches for a given registry client
- Enforce NiFi Registry can only receive default branch
This closes#8726
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Bryan Bende
Mark Payne
jrsteinebrey
exceptionfactory
Peter Turcsanyi
Matt Gilman
dan-s1
Ferenc Erdei
Joseph Witt
Ferenc Kis
Pierre Villard
EndzeitBegins
tpalfy