- Upgraded Spring from 5.3.30 to 6.0.13
- Upgraded Spring Boot from 2.7.16 to 3.1.5
- Upgraded Spring Security from 5.8.7 to 6.1.5
- Upgraded Jetty from 10.0.18 to 11.0.18
- Upgraded Servlet API from 3.1.0 to 5.0.0
- Upgraded JAX-RS from 2.1.1 to 3.1.0
- Upgraded Jersey from 2.41 to 3.1.3
- Upgraded JAXB from 2.3.2 to 4.0.4
- Upgraded Java Validation from 2.0.1 to 3.0.2
- Upgraded Flyway from 8.5.13 to 9.22.3
- Upgraded ASM from 9.3 to 9.6 for Java 21
- Upgraded NiFi Toolkit CLI to JAX RS 3 and Jersey 3
- Set Jersey 3 and JAX RS 3 dependency versions in nifi-flow-registry-client-bundle to align with updates to nifi-registry-client classes
Signed-off-by: Joseph Witt <joewitt@apache.org>
When calling migrateProperties, provide the properties that were configured in the VersionedComponent, rather than creating a new property map based on the component's new properties
This closes#7964
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Corrected system test failures for PythonProcessorIT and Stateless CreateFlowFileIT
This closes#7935
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>
- Added ability to get raw property values from PropertyConfiguration instead of just effective values
- Updated TestRunner to allow for testing these migration methods
- Auto-enable newly created controller services if they are valid
- Eliminated Proxy properties in all AWS processors and instead just make use of the Proxy Configuration controller service
- Eliminated authentication properties from AWS processors and migrated all processors to using Controller Service or authentication
This closes#7874
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-12206 Refactored Flow History using JetBrains Xodus
- Replaced H2 Database Engine with JetBrains Xodus for persistent storage of FlowConfigurationHistory
- Added EntityStoreAuditService implementation using Xodus PersistentEntityStore
- Removed nifi.h2.url.append from properties
- Ensure that after restoring nars in the lib/ directory we restart NiFi so that they take effect. This is important if this test is not the last one to run
This closes#7837
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Created NIFI-12203 to evaluate issues with flow comparison surfaced in JoinClusterWithDifferentFlow
This closes#7661
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Imported nifi-bom to nifi-commons and nifi-nar-bundles parent modules
- Removed explicit version and scope properties for dependencies declared in nifi-bom
Fixed issue in logic of LocalComponentLifecycle, which was waiting for all PGs to reach desired stateless run schedule, even when the group itself was not stateless
Fixed JavaScript to specify correct value for recursivity when updating process group parameter context
- Added nifi-security-cert for reusable components without dependencies
- Added nifi-security-cert-builder for certificate generation
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7651
- Timing issues in system integration tests could sometimes fail with a NullPointerException when calling the poison method multiple times
This closes#7618
Signed-off-by: David Handermann <exceptionfactory@apache.org>
When shutting down FlowController, wait up to gracefulShutdownSeconds seconds for the components to stop before shutting down thread pools. This allows for asynchronous operations such as disableControllerServicesAsync to complete during shutdown. Updated StandardStatelessFlow so that on shutdown it catches more general Exception to ensure that shutdown succeeds
Ensure that Max Concurrent Tasks cannot be set less than 1 for stateless group; fixed typo in ProcessGroupDTO's docs; on shutdown, we may need to disable controller services asynchronously. At that point, the thread pool used to do so may already be shutdown. If so, catch this and create a new single-thread pool, disable the service, and immediately shutdown the pool. Also, if we fail to disable services on shutdown of a stateless flow, instead of throwing an Exception, just log it and move on - it doesn't make much sense for shutdown() to throw an Exception in that case.
Updated system tests so that when emptying a queue, we check the result and if the queue still has data (because a Processor hasn't acknowledged the data, for example) then continue issuing request until the queue fully becomes empty.
When shutting down input/output ports for stateless group, ensure that we wait for the ports' active threads to complete before returning
When stopping StatelessGroupNode, ensure that all the Processors, Controller Services (even those not executed by the Stateless Engine) are stopped/disabled before considering the Stateless Group to be fully STOPPED.
This closes#7253
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Ensure that testComponentsRecreatedOnRestart only counts components in its own group
This closes#7466
Signed-off-by: David Handermann <exceptionfactory@apache.org>
Also changed some properties around so that we can have both a clustered and a standalone instance running at the same time.
This closes#7427
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- FlowSynchronizationIT no longer requires isDestroyEnvironmentAfterEachTest to return true
This closes#7420
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Removed hamcrest-all from default dependencies
- Added groovy-test to selected modules with Groovy test classes
- Added junit-vintage-engine to selected modules with JUnit 4 test classes
- Corrected references to JUnit 4 assertions in JUnit 5 test classes
- Removed several unnecessary test classes from nifi-socket-utils
- Removed duplicative Registry toolkit test classes
- Removed Kudu integration tests
NIFI-11532 Corrected scope for junit-vintage-engine for Elasticsearch
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#7233
Rather than creating many FlowFiles with the same Content Claim, refactored content repos' OutputStreams and ClaimWriteCache so that a new ContentClaim is created for each FlowFile. This ensures that we have a content claim offset of 0. The poor performance was due to having to use StreamUtils.skip() in conjunction with the CipherInputStream, which would only skip a max of 511 bytes at a time. By using a separate Content Claim per FlowFile, we no longer need to seek after creating the CipherInputStream
This closes#7363
Signed-off-by: David Handermann <exceptionfactory@apache.org>
After terminating Processor in DynamicallyModifiedClasspathIT system test, keep trying to empty queue until successful instead of assuming success after the first iteration.
When writing to a replacement file in AsyncCommitCallbackIT, write to a temp file and then rename. This prevents the processor from picking up the file while it's still being written.
This closes#7302
Signed-off-by: David Handermann <exceptionfactory@apache.org>
When waiting for a Processor to stop completely in system tests, also consider any terminated threads. Consider the processor stopped only when all active threads and all terminated threads have gone to 0. Additionally, when stopping all components in a Process Group, wait for the PG to reflect that there are no running components. Previously, we checked for processor active thread counts. However, this was problematic because the flow that was fetched was not fetched recursively and as a result, processors running in embedded groups were not always stopped when the waitForProcessorsStopped(String groupId) method returned. Finally, removed the step for gathering surefire-reports in failed system tests because it was not working as expected and was unneeded because the tests' logs were already gathered into the diagnostic dump that is uploaded.
Added fix for BatchFlowBetweenGroupsIT. The test would intermittently fail, based on timing. The test assumed that when Input Port in Group B was scheduled to run, it would trigger before FlowFiles in Group A finished processing. The Sleep processor caused a 10 ms delay for each of 5 FlowFiles which gives the Input Port about 50 ms to trigger. If it took longer to trigger the first time, the test failed. Adjusted the Sleep processor to use 2 seconds. So now instead of 50 milliseconds to start, it has 10 seconds. Overall this adds about 30-45 seconds to the run time of the test but ensures more-than-adequate time for the test to run, even on extremely slow hardware
This closes#7289
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-11557: Added an additional system test and updated github actions to include surefire-report in order to help diagnose problem that occurred in one of the last system-test runs in Github. Could not replicate problem locally
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7265
- Updated GitHub workflow so that system tests include Python 3.9
- Updated GitHub actions to build necessary modules for system tests
This closes#7003
Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Fixed system tests so that they work properly in Clustered version of RegistryClientIT
- Fixed system test - ensure that we wait for processors to become valid before attempting to start them; also added an additional system test around Controller Services in versioned flows
This closes#7095
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Ensure that components are notified that primary node has changed in a background thread instead of the Leader Election thread and activate/deactivate the thread in the case of Processors so that they can be viewed in the UI and terminated
- Fixed system tests that would fail intermittently because they did not wait for node disconnection to complete and did not properly switch the client to look at the connected node before checking cluster status
This closes#7052
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-11310: Fixed META-INF/services file that was mistakenly listing an extra extension point, due to rebase
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#7061
- Updated algorithm from PBEWITHMD5AND256BITAES-CBC-OPENSSL to NIFI_PBKDF2_AES_GCM_256
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#7041.
- Replaced OSUtils.getProcessId() with java.lang.Process.pid() available on Java 9 and following
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6989.
NIFI-11192: If a failure is encountered when changing the version of a flow from 1 version to another, attempt to rollback the changes instead of just failing with the flow in a bad state
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#6981
exceptionfactory
Mark Payne
Bence Simon
Joe Gresock
Nandor Soma Abonyi
Pierre Villard
Timea Barna
Robert Kalmar
Lehel Boér
Bryan Bende
Joe Witt