NIFI-9382: Fixed issue with SharedInstanceClassLoader where the classloader may get closed but then get used again. When the SharedInstanceClassLoader is closed, we will now ensure that we don't use anymore and instead create a new one.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Changed framework so that it serializes the dataflow into a VersionedDataflow using JSON as well as XML, and prefers the JSON representation on load, if it's available. This also results in the need for the cluster protocol to exchange its representation of the dataflow to using JSON. Rather than re-implementing all of the complex logic of Flow Fingerprinting, updated to just inherit the cluster's flow.
- Moved logic to synchronize Process Group with Versioned Process Group into a new ProcessGroupSynchronizer class instead of having all of the logic within StandardProcessGroup
- Reworked versioned components to use an instance id.
- Renamed StandardFlowSynchronizer to XmlFlowSynchronizer; introduced new StandardFlowSynchronizer that delegates to the appropriate (Xml or Versioned)FlowSynchronzer
- Updated to allow import of VersionedProcessGroup even if not all bundles are available - will now use ghost components
- Introduced a VersionedDataflow object to hold controller-level services, reporting tasks, parameter contexts, templates, etc.
- Allow mutable requests to be made while nodes are disconnected. Also fixed issue in AbstractPolicyBasedAuthorizer that caused ClassNotFoundException / NoClassDefFoundError if the authorizations were changed and then a node attempts to rejoin the cluster. The Authorizer was attempting to use XmlUtils, which is in nifi-security-utils and so so by madking nifi-security-utils a provided dependency of nifi-framework-api, but this doesn't work, because nifi-framework-api is loaded by a higher-level classloader, so the classloader that loads AbstractPolicyBasedAuthorizer will never have the appropriate classloader to provide nifi-security-utils. Addressed this by copying the code for creating a safe document builder from XmlUtils to AbstractPolicyBasedAuthorizer.
- Fixed bug that occurred when importing a Process Group that has 2 parameter contexts, one inheriting from another, where neither is pre-defined in the existing flow
- Fixed bug that was encountered when Updating a Versioned Process Group where one version had a disabled processor and the other had the processor running.
- Increased system-tests workflow timeout to 120 minutes
- Added additional exception handling to system tests
This closes#5514
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Set Scheduled State for Versioned Port and Versioned Remote Port when mapping Flow Definition
- Updated StandardProcessGroup to set disable Port based on Scheduled State of DISABLED
- Updated StandardProcessGroup to set Remote Port transmitting based on Scheduled State of ENABLED
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5534.
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests
* NIFI-9382: Renamed interface based on review feedback
* NIFI-9382: Removed ReentrantKerberosUser.
- ConcurrentHashMap collection for Active Threads eliminates the need for method synchronization
- Replaced System.out.println() with Logger.info() in nifi-system-test-suite classes
- Methods without the synchronized keyword, such as terminate(), iterate over activeThreads
This closes#5509
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Ensure that we consider all property values to determine if a component references a variable, not just 'supported' (i.e., well-known/non-user-defined) properties
This closes#5499
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-9309: Include a uiOnly flag when requesting flow for a given process group. In that case, do not include the property descriptors, property values, etc. for Processors. When fetching Variable Registry, improved logic to cache the VariableImpact for each property value instead of parsing/recreating it every time. When fetching bulletins for a component, avoid filtering through all components' bulletins and instead only look at bulletins that might pertain to the appropriate component
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5412.
* NIFI-9229 Flow upgrade not possible if a Output Port changes to a funnel
* NIFI-9229 Addressing review comments modified log message and added comments
- Removed nifi.minifi.sensitive.props.provider Property from MiNiFi
- Removed property from example NiFi properties files
- Removed provider from MiNiFi SensitivePropsSchema
- Removed BC provider value from MiNiFi test cases
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5422.
* NIFI-8491:
- Adding support for configuring parameter context inheritance.
* NIFI-8491:
- Allowing changes to the parameter context inheritance to drive Apply disabled state.
* NIFI-8491: Updating StandardParameterContext#isAuthorized check
* NIFI-8491:
- Showing selected inherited parameter contexts in ready only form when appropriate.
- Allowing available parameter contexts to be inherited by double clicking.
- Removing support for rendering unauthorized inherited parameter contexts as they can no longer be opened.
* NIFI-8491: Adding inherited param context verification earlier
* NIFI-8491:
- Addressing CI failures by rolling back to some order JS language spec to allow yui-compress to minify and compress.
* NIFI-8491:
- Ensuring selected context sort order is honored.
- Ensuring the Apply button is correctly enabled.
- Showing Pending Apply message when selected Parameter Context changes.
- Ensuring the Parameter's tab is selected now that there is a third tab.
* Updates to inherited param context verification
* Improving validation between parameters/inherited parameters
* NIFI-8491:
- Ensuring the available parameter contexts are loaded whether the edit dialog is opened from the listing or outside of the listing.
* NIFI-8491:
- Fixing conditions we check if the parameter context listing is currently open.
* NIFI-8491:
- Waiting for the parameter contexts to load prior to rendering the parameter context inheritance tab and showing the dialog.
* NIFI-8491:
- Fixing pending apply message clipping.
- Hiding pending apply message after clicking Apply.
Co-authored-by: Joe Gresock <jgresock@gmail.com>
This closes#5371
NIFI-8671 Moved versioned components class into nifi-api
- Removed @XmlRootElement from VersionedProcessGroup.
- Fixed nifi-api dependency version in nifi-registry-data-model. Changed logic of handling instances of un-annotated classes during xml serialization in JAXBSerializer.
- include new process group property support in NiFi Registry
- updated documentation to describe and show new feature
- added elements to XSD schema definition
NIFI-8195: update to DAO to fix PG move and copy/paste
update condition to not null vice null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5192
- Refactored stateless to use this when creating a dataflow so that Parameter Provider implementations can be externalized into NARs. Also updated ExtensionDiscoveringManager such that callers are able to provide a new type of class to be discovered (e.g., ParameterProvider) so that the extensions will be automatically discovered
- Put specific command-line overrides as highest precedence for parameter overrides
- Make ParameterOverrideProvider valid by allowing for dynamically added parameters
- Fixed bug in validation logic, added new system tests to verify proper handling of Required and Optional properties
- Addressed review feedback and fixed some bugs. Also added system test to verify Parameter Providers are working as expected
This closes#5113
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFIDEVS-8195: fixed properties not properly inheriting from template/snippet values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5094
NIFI-8528 Migrate NiFi Registry fully codebase into NiFi as a module. No changes except certain dependency scopes to preserve the NiFi Registry original by overriding the new parent (nifi).
- Version adjustments. Removed distinct checkstye rules form nifi-registry. (Using nifi's instead.)
- Made some tests Windows-compatible.
- Consolidated LICENSE, NOTICE and README.md.
- Fixed CryptoKeyLoaderGroovyTest.groovy.
- Disable frontend-maven-plugin on Windows.
- Skipping all goals of the frontend-maven-plugin on Windows.
- Registry integration tests not to run in github jobs (same as the original settings). Skip all registry tests (build and run) on Windows.
- Removed Husky from registry.
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
- Added totalActiveQueuedDuration and maxActiveQueuedDuration to the ConnectionStatus object
- Updated FlowFileQueue implementations and supporting code to properly calculate and provide the totalActiveQueuedDuration and maxActiveQueuedDuration for their active queues
- Fixing failing unit test. In examining this it appears that the unit test only accidentally passed in the past and that the object mocked to always throw an exception was not actually being used in the test.
- Adding UI component via ConnectionStatusDescriptor along with updates based on PR comments.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Retained legacy default Sensitive Properties Key in ConfigEncryptionTool to support migration
- Streamlined default file path and moved key generation conditional
- Refactored with getDefaultProperties()
- Cleared System Property in ConfigEncryptionToolTest
- Added checking and error handling for clustered status
- Added set-sensitive-properties-key command
- Refactored PropertyEncryptor classes to nifi-property-encryptor
- Added nifi-flow-encryptor
- Refactored ConfigEncryptionTool to use FlowEncryptor for supporting AEAD algorithms
- Added Admin Guide section Updating the Sensitive Properties Key
This closes#4857.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Updated components to make use of new feature
NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it
NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems
This closes#4890.
Signed-off-by: Bryan Bende <bbende@apache.org>
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary
This closes#4852
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Configuration based on Sensitive Properties Algorithm defaults to Argon2
- Added SensitiveValueEncoder interface
- Standard implementation uses existing approach with HmacSHA256
This closes#4867
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor
NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor
NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor
NIFI-7668 Added support for bcrypt secure hashing algorithm
NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4809.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches
NIFI-7646: Removed TODO and unused Jackson dependency
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4818
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup
NIFI-7897: Addressing review feedback
NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly
This closes#4669.
Signed-off-by: Bryan Bende <bbende@apache.org>
Mark Payne
exceptionfactory
Peter Turcsanyi
Bryan Bende
Joe Witt
Lehel Boér
Matthew Burgess
timeabarna
Matt Gilman
Hsin-Ying Lee
Joe Gresock
tpalfy
Mark Bean
Jon Kessler
Nathan Gough
Tamas Palfy
Bryan Bende