Commit Graph

34 Commits

Author SHA1 Message Date
Joe Witt 2f95eed8cb
NIFI-9575 This closes #5667. updating copyright year to 2022
Signed-off-by: Joe Witt <joewitt@apache.org>
2022-01-19 09:14:09 -07:00
Mark Payne 72e54f4fab NIFI-9390: Updates to MergeContent / MergeRecord so that they play nicely within Stateless
NIFI-9390: Addressed underlying condition in stateless framework that caused Merge-related processors and similar to not properly be triggered as necessary. Added several system tests to verify different configurations.

NIFI-9390: Simplified the logic for how to iterate over the components in a Stateless flow that are ready to be triggered

This closes #5634.

Co-authored-by: Peter Turcsanyi <turcsanyi@apache.org>
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-11 09:45:00 +01:00
Lehel Boér be138ab656
NIFI-9434: Added bulletin to alert FileSystemRepository backpressure (#5596) 2022-01-04 15:59:32 -05:00
exceptionfactory 7c6bdcb035
NIFI-9491 Excluded commons-logging and added jcl-over-slf4j references
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5608
2021-12-17 07:21:03 -05:00
Joe Gresock 429087c11d
NIFI-9386: Adding status task schedule to Stateless engine config
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5532.
2021-12-13 12:08:54 +01:00
Peter Turcsanyi dafa03a21a
NIFI-9421: Running NiFi Stateless with local NARs only
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5557.
2021-12-08 15:59:31 -05:00
markap14 839fbf7d19
NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which H… (#5533)
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests

* NIFI-9382: Renamed interface based on review feedback

* NIFI-9382: Removed ReentrantKerberosUser.
2021-11-22 14:55:19 -05:00
Joe Witt 9e845d87ff
Merge branch 'NIFI-9344-RC3' 2021-11-07 16:23:10 -07:00
Mark Payne 373498445f
NIFI-9364: Ensure that we delegate calls to write(byte[]) and write(byte[], int, int) to the underlying OutputStream when writing to the file-based content repository for stateless
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5508.
2021-11-04 16:48:24 -04:00
Joe Witt c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt 7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Matthew Burgess 104078868e
NIFI-7012: Refactored OnConfigurationRestored to support sensitive property validation (#5415) 2021-10-27 14:40:17 -04:00
markap14 5766d33fce
NIFI-9309: Include a uiOnly flag when requesting flow for a given pro… (#5462)
NIFI-9309: Include a uiOnly flag when requesting flow for a given process group. In that case, do not include the property descriptors, property values, etc. for Processors. When fetching Variable Registry, improved logic to cache the VariableImpact for each property value instead of parsing/recreating it every time. When fetching bulletins for a component, avoid filtering through all components' bulletins and instead only look at bulletins that might pertain to the appropriate component
2021-10-23 19:42:31 -04:00
Mark Payne be2b80d909
NIFI-9292: Added getInputPortNames() and getOutputPortNames() to StatelessDataflowDefinition and getReadOnlyExtensionsDirectories() to StatelessEngineConfiguration
- Addressed issue identified in system test where a flow can fail due to FailurePortEncounteredException but then purge the flow in the background after the call to trigger completes

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5450.
2021-10-18 12:22:12 -04:00
Mark Payne 607480c1a0
NIFI-9239: Updated Consume/Publish Kafka processors to support Exactly Once Semantics. Added ExecuteStateless processor/bundle
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5412.
2021-10-04 15:32:21 -04:00
exceptionfactory a94b47ecf8
NIFI-9254 Updated default Stateless Sensitive Property configuration
- Set NIFI_PBKDF2_AES_GCM_256 as property encryption method
- Replaced static default sensitive properties key with random UUID
- Added unit test for PropertiesFileEngineConfigurationParser
- Added random encryption key generation method
- Changed Stateless to use PropertyEncryptionMethod enum

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5424
2021-10-01 15:55:06 -04:00
Mark Payne baf29e59f1
NIFI-9009: Created VerifiableProcessor, VerifiableControllerService, VerifiableReportingTask components; implemented backend work to call the methods. Added REST APIs and created/updated data models for component configuration verification
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5288
2021-09-22 19:00:23 -04:00
Mike Thomsen 6a55c6ca44
NIFI-9081 Updated nifi-framework-api tests to JUnit 5
- Updated nifi-external tests to JUnit 5
- Updated nifi-stateless to use JUnit 5
- Updated nifi-api to use JUnit 5

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-11 14:57:38 -05:00
Hsin-Ying Lee 59f0b6bac3
NIFI-8739 Penalized flowfiles should be able to be polled from the queue in some cases (#5189) 2021-08-30 11:26:02 -04:00
Joe Gresock 4a3e81531b
NIFI-8490: Adding inherited parameter contexts (#5072)
- Allowing inherited param contexts on creation, updating PC authorization
2021-08-30 10:44:31 -04:00
Mark Payne fb4edfa0d6
NIFI-8950: Added ability to optionally use a file-system backed Content Repository for Stateless NiFi
This closes #5254

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-11 12:01:07 -05:00
tpalfy 77cf9a11df
NIFI-8671 Create nifi-versioned-components module. (#5145)
NIFI-8671 Moved versioned components class into nifi-api
- Removed @XmlRootElement from VersionedProcessGroup.
- Fixed nifi-api dependency version in nifi-registry-data-model. Changed logic of handling instances of un-annotated classes during xml serialization in JAXBSerializer.
2021-08-09 16:55:39 -04:00
Joe Gresock 82bd72b83d
NIFI-9002: Renaming Stateless' ParamaterProvider to ParameterValueProvider (#5282) 2021-08-06 13:01:26 -04:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
Mark Payne a4cfdbb695
NIFI-8774: Fixed NullPointerExceptions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5208
2021-07-09 21:56:00 -04:00
Mark Bean ea31634ea7
NIFI-8195: add default connection settings to process group configuration
- include new process group property support in NiFi Registry
  - updated documentation to describe and show new feature
  - added elements to XSD schema definition

NIFI-8195: update to DAO to fix PG move and copy/paste

update condition to not null vice null

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5192
2021-07-07 17:45:54 -04:00
Mark Payne 6df07df3b2
NIFI-8644: Introduced a notion of ParameterProviderDefinition
- Refactored stateless to use this when creating a dataflow so that Parameter Provider implementations can be externalized into NARs. Also updated ExtensionDiscoveringManager such that callers are able to provide a new type of class to be discovered (e.g., ParameterProvider) so that the extensions will be automatically discovered
- Put specific command-line overrides as highest precedence for parameter overrides
- Make ParameterOverrideProvider valid by allowing for dynamically added parameters
- Fixed bug in validation logic, added new system tests to verify proper handling of Required and Optional properties
- Addressed review feedback and fixed some bugs. Also added system test to verify Parameter Providers are working as expected

This closes #5113

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-06 16:45:02 -05:00
markap14 88cc232f15
NIFI-8731: If a processor is running but made invalid due to a parameter update, it ends up in a state of STARTING. If the parameter is then updated again, it transitions the state to STOPPING but the processor is not fully stopped yet. At that point, the parameter is updated and the processor is attempted to be started again. Fixed this by keeping the number of active threads to >= 1 if processor is STOPPING in order to convey that it is not fully stopped. Also addressed a few minor bugs discovered in the process: when stopping a processor, if status == invalid, it should be skipped instead of waiting for the status to become stopped since it never will be. In the DTO's run status use Stopped instead of Invalid if there is at least 1 active thread / if stopping but not stopped. When considering if a processor has transitioned to the desired state for parameter updates, do not consider validation status if still transitioning to stopped or if the desired state has already been reached. Added new system tests to verify behavior. (#5180)
This closes #5180
2021-06-25 12:40:49 -04:00
Matthew Burgess 5e4f32663e Revert "NIFI-8195: add default connection settings to process group configuration"
This reverts commit 5ebbe0028b.
2021-06-24 18:52:17 -04:00
Mark Bean 5ebbe0028b
NIFI-8195: add default connection settings to process group configuration
NIFIDEVS-8195: fixed properties not properly inheriting from template/snippet values

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5094
2021-06-24 14:03:51 -04:00
Mark Payne 08edc33eb7
NIFI-8629: Implemented the LogComponentStatuses task that runs periodically in stateless.
Also noticed a typo in the ControllerStatusReportingTask and found in comparing outputs
that it had a bug that caused it to log counters generated only by processors at the root level so fixed that.

This closes #5101

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-26 20:07:26 -05:00
tpalfy dfa683af0e
NIFI-8528 Migrate NiFi Registry into NiFi codebase (#5065)
NIFI-8528 Migrate NiFi Registry fully codebase into NiFi as a module. No changes except certain dependency scopes to preserve the NiFi Registry original by overriding the new parent (nifi).
- Version adjustments. Removed distinct checkstye rules form nifi-registry. (Using nifi's instead.)
- Made some tests Windows-compatible.
- Consolidated LICENSE, NOTICE and README.md.
- Fixed CryptoKeyLoaderGroovyTest.groovy.
- Disable frontend-maven-plugin on Windows.
- Skipping all goals of the frontend-maven-plugin on Windows.
- Registry integration tests not to run in github jobs (same as the original settings). Skip all registry tests (build and run) on Windows.
- Removed Husky from registry.
2021-05-26 09:34:55 -04:00
Mark Payne 154e4f3df3
NIFI-8617: Created nifi-stateless module at the root level and moved sub-modules into that. Created nifi-stateless-assembly and the associated resources module. Also updated some configuration to be configurable via environment variables
NIFI-8618: Allow for parameters to be set for environment variables. Updated README to indicate this and also corrected some outdated information.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5087
2021-05-20 22:13:53 -04:00