* NIFI-9958 Implemented Sensitive Dynamic Properties
- Added SupportsSensitiveDynamicProperties annotation for components
- Added optional sensitive query parameter to Property Descriptor REST Resource methods
- Added system tests for components supporting sensitive dynamic properties
- Updated REST Resources to support Sensitive Dynamic Property Names
- Updated Documentation Writer to indicate component support for Sensitive Dynamic Properties
- Updated InvokeHTTP to support Sensitive Dynamic Properties
- Updated Auditor components to handle masking Sensitive Dynamic Properties
* Refactored Property Descriptor REST method handling
- Corrected AbstractDocumentationWriter evaluation of support for sensitive dynamic properties
- Refactored Controller Service Dynamic Properties system tests to new class
* Updated AbstractComponentNode.getProperties() to get canonical descriptor
- Updated stateless and StandardProcessGroup, etc. to make use of VersionedExternalFlow
- Updated StatelessDataflowDefinition to use ExternalVersionedFlow instead of generic type
- Updated Stateless Bootstrap to avoid loading stateless engine libs from root class path but instead use a NarClassLoader to load the statelss nar
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5832.
* NIFI-9543: add bring-to-front functionality to labels
* NIFI-9543: update zIndex changes to persist after shutdown
* NIFI-9543: remove commented code
* NIFI-9543: update flow comparator and snippet to account for zIndex
* NIFI-9543: changed add function to sort labels when a new label is added
* NIFI-9543: remove unnecessary call to position
* NIFI-9543: only register difference if zIndex is not default value
* NIFI-9543: add null checks to prevent primitive Long
This closes#5765
- Applied new permission restrictions to AWSCredentialsProviderControllerService and GCPCredentialsControllerService
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5796.
- Write additional fields to extnesion-manifest.xml for processors
- Update C2 model classes to support new fields for processors, properties, and scheduling
- Create converter between NiFi model and C2 model
- Create generator and execute during the build
Add profile to nifi-assembly that skips the binary assembly, update github workflow to enable this profile instead of excluding nifi-assembly
Add additional documentation on new fields in processor definition and reporting task definition
Set charset to UTF-8 on the OutputStreamWriter
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5612
- Changed framework so that it serializes the dataflow into a VersionedDataflow using JSON as well as XML, and prefers the JSON representation on load, if it's available. This also results in the need for the cluster protocol to exchange its representation of the dataflow to using JSON. Rather than re-implementing all of the complex logic of Flow Fingerprinting, updated to just inherit the cluster's flow.
- Moved logic to synchronize Process Group with Versioned Process Group into a new ProcessGroupSynchronizer class instead of having all of the logic within StandardProcessGroup
- Reworked versioned components to use an instance id.
- Renamed StandardFlowSynchronizer to XmlFlowSynchronizer; introduced new StandardFlowSynchronizer that delegates to the appropriate (Xml or Versioned)FlowSynchronzer
- Updated to allow import of VersionedProcessGroup even if not all bundles are available - will now use ghost components
- Introduced a VersionedDataflow object to hold controller-level services, reporting tasks, parameter contexts, templates, etc.
- Allow mutable requests to be made while nodes are disconnected. Also fixed issue in AbstractPolicyBasedAuthorizer that caused ClassNotFoundException / NoClassDefFoundError if the authorizations were changed and then a node attempts to rejoin the cluster. The Authorizer was attempting to use XmlUtils, which is in nifi-security-utils and so so by madking nifi-security-utils a provided dependency of nifi-framework-api, but this doesn't work, because nifi-framework-api is loaded by a higher-level classloader, so the classloader that loads AbstractPolicyBasedAuthorizer will never have the appropriate classloader to provide nifi-security-utils. Addressed this by copying the code for creating a safe document builder from XmlUtils to AbstractPolicyBasedAuthorizer.
- Fixed bug that occurred when importing a Process Group that has 2 parameter contexts, one inheriting from another, where neither is pre-defined in the existing flow
- Fixed bug that was encountered when Updating a Versioned Process Group where one version had a disabled processor and the other had the processor running.
- Increased system-tests workflow timeout to 120 minutes
- Added additional exception handling to system tests
This closes#5514
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests
* NIFI-9382: Renamed interface based on review feedback
* NIFI-9382: Removed ReentrantKerberosUser.
NIFI-9309: Include a uiOnly flag when requesting flow for a given process group. In that case, do not include the property descriptors, property values, etc. for Processors. When fetching Variable Registry, improved logic to cache the VariableImpact for each property value instead of parsing/recreating it every time. When fetching bulletins for a component, avoid filtering through all components' bulletins and instead only look at bulletins that might pertain to the appropriate component
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5412.
- Updated nifi-external tests to JUnit 5
- Updated nifi-stateless to use JUnit 5
- Updated nifi-api to use JUnit 5
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-8974 Integrate KerberosUserService with HDFS processors
NIFI-8980 Integrate KerberosUserService with Kafka 2.6 processors
- Introduced SelfContainerKerberosUserService to restrict which impls can be used with Kafka
- Add variations of KerberosUser doAs that allow setting the context ClassLoader
- Add additional unit tests for configurations
This closes#5277
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-8671 Moved versioned components class into nifi-api
- Removed @XmlRootElement from VersionedProcessGroup.
- Fixed nifi-api dependency version in nifi-registry-data-model. Changed logic of handling instances of un-annotated classes during xml serialization in JAXBSerializer.
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
- Added totalActiveQueuedDuration and maxActiveQueuedDuration to the ConnectionStatus object
- Updated FlowFileQueue implementations and supporting code to properly calculate and provide the totalActiveQueuedDuration and maxActiveQueuedDuration for their active queues
- Fixing failing unit test. In examining this it appears that the unit test only accidentally passed in the past and that the object mocked to always throw an exception was not actually being used in the test.
- Adding UI component via ConnectionStatusDescriptor along with updates based on PR comments.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Updated components to make use of new feature
NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it
NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems
This closes#4890.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Removed schema properties from FreeFormTextRecordSetWriter causing validation issues
This closes#4851.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission
This closes#4642.
Signed-off-by: Mark Payne <markap14@hotmail.com>
greyp9
simonbence
Joe Gresock
Lehel Boér
Nissim Shiman
exceptionfactory
Bence Simon
Paul Grey
Bryan Bende
Mark Payne
Shane Ardell
timeabarna
Matthew Burgess
Peter Turcsanyi
Mike Thomsen
tpalfy
Jon Kessler
Mark Bean
Tamas Palfy
exceptionfactory