* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils
Added QueryNiFiReportingTask to SQL query NiFi status and metrics
Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS
Add CONNECTION_STATUS_PREDICTIONS
check for null predictions
Add ConnectionStatusRecursiveIterator
Fix issue w/ duplicate iterator outputs
Refactored query interfaces, fixed assembly POM
Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors
Add DatabaseRecordSink service (#13)
* Add DatabaseRecordSink service
* Incorporated review comments
* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions
* Added bundle profile, remove predictions table if not enabled
* Added doc for which tables are available when
This closes#3826.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6778 - added comments and component description annotations
NIFI-6778 - added license and notice details
NIFI-6778 - added additional details section and updated description of service
NIFI-6778 - fixed checkstyle error
NIFI-6778 - addressed review comments for documentation and description corrections
NIFI-6778 - added include-rules profile for assembly build
NIFI-6778 - add MVEL and ASM to license and notice
NIFI-6778 - switch to use Jackson for NiFi Rules Json deserialization
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3824
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes
This closes#3795.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6773 Adding two unit tests for lz4-framed compression/decompression
NIFI-6773 Adding compressed lz4 file for decompression test
Signed-off-by: Joe Witt <joewitt@apache.org>
* NIFI-6760: When writing/reading the length of a DataFrame, do so using a 4-byte integer instead of a 2-byte short. When using uncompressed data, we know that the length of the DataFrame will be no more than 64 KB so a 2-byte short is sufficient. However, if data is compresed, there's a chance that the compressed form of the data will be larger than the uncompressed form (for example, with random binary data or with encrypted data). In this situation, we can end up overflowing the 2-byte short, which causes the length that is written to be wrong. Using a 4-byte integer avoids this situation.
* NIFI-6760: Fixed unit tests
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.
This closes#3805.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes#3804.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.
This closes#3798.
Removed unnecessary code from S3 CSE* encryptions.
S3 Encryption Service documentation fixes and improvements.
Renamed region property of StandardS3EncryptionService to kms-region.
Renamed Client-side Customer Master Key in StandardS3EncryptionService.
Use Client-side Customer Key on the GUI / documentation (similar to
Server-side Customer Key).
Use C suffix in constants and class names (similar to SSE_C).
Fixed / extended StandardS3EncryptionService validation.
FetchS3Object encryption strategy changes.
Disable SSE S3 and SSE KMS for FetchS3Object. In case of fetching the
S3 object, these strategies are handled implicitly / automatically.
Set the encryption strategy on the fetched FF that was used to store
the S3 object, instead of the one that is used to read the object (eg.
non-encrypted or SSE S3 encrypted objects can be fetched with a CSE client).
Typo fix.
This closes#3787.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Updated description for "Full Path" filter mode to state that it will ignore scheme and authority
Added tests to TestListHDFS for listing an empty and nonexistent dirs
Updated TestListHDFS' mock file system to track state properly when FileStatus instances are added, and updated listStatus to work properly with the underlying Map that contains FileStatus instances
Updated ListHDFS' additional details to document "Full Path" filter mode ignoring scheme and authority, with an example
Updated TestRunners, StandardProcessorTestRunner, MockProcessorInitializationContext to support passing in a logger.
NIFI-6275 Updated the "Full Path" filter mode to check the full path of a file with and without its scheme and authority against the filter regex
Added additional documentation for how ListHDFS handles scheme and authority when "Full Path" filter mode is used
Added test case for "Full Path" filter mode with a regular expression that includes scheme and authority
This closes#3483.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.
This closes#3677.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6662: Cleaning up Kudu logic
NIFI-6662: Minor enhancements and build fixes
NIFI-6662: This closes#3732.
Signed-off-by: Joe Witt <joewitt@apache.org>
Added AzureStorageCredentialsService controller service interface
with 2 implementations:
- AzureStorageCredentialsControllerService: holds the credential properties
(account name, account key, sas token)
- AzureStorageCredentialsControllerServiceLookup: can be used to dynamically
look up another AzureStorageCredentialsService (similar to
DBCPConnectionPoolLookup)
The controller service can be used by the Azure Storage processors:
{List|Fetch|Put|Delete}AzureBlobStorage, {Get|Put}AzureQueueStorage
NIFI-6550: Review changes.
NIFI-6550: Review changes #2.
This closes#3742.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.
This closes#3715.
Signed-off-by: Mark Payne <markap14@hotmail.com>
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
the same field where some (but not all) were in a wider-than-the-other
relationship.
Before: Some datatypes could be lost. String was wider than any other.
After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.
This closes#3724.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
NIFI-6295: Incorporated review comments
NIFI-6295: Refactored unit tests for NiFiRecordSerDe
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3684
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.
This closes#3718
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
NIFI-6649 - documentation update
NIFI-6649 - add debug logging for score and prediction information
NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3719
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.
This closes#3713
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.
Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.
This closes#3702.
Signed-off-by: Mark Payne <markap14@hotmail.com>
* NIFI-6510 Implement initial analytic engine
* NIFI-6510 Implemented basic linear regression model for queue counts
* NIFI-6510 Initial analytics REST endpoint and supporting objects
* NIFI-6510 Connect the dots for StatusAnalytics -> API
* NIFI-6510 Added poc engine with prediction model caching
(cherry picked from commit e013b91)
DFA-9 - updated logging and corrected logic for checking if not in backpressure
(cherry picked from commit a1f8e70)
* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics
(cherry picked from commit 050e0fc)
(cherry picked from commit 9fd365f)
* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id
(cherry picked from commit 14854ff)
DFA-9 - reduced snapshot interval to 1 minute
(cherry picked from commit 36abb0a)
* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions
* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly
* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"
This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.
* NIFI-6510 Added prediction fields for use by UI, still need to be populated
* NIFI-6510 Analytics Framework Introduction (#10)
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.
DFA-9 - added logging
* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval
* DFA-9 - checkstyle fixes
* NIFI-6510 Add prediction percent values and predicted interval seconds
(cherry picked from commit e60015d)
* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.
(cherry picked from commit 6fae058)
* NIFI-6510 Added tests for engine
(cherry picked from commit 6d7a13b)
* NIFI-6150 Added tests for connection status analytics class, corrected variable names
(cherry picked from commit 58c7c81)
* NIFI-6150 Make checkstyle happy
(cherry picked from commit b6e35ac)
* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing
* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics
* NIFI-6510 Adjusted interval and incorporated R-squared check
Updates to support multiple variables for features, clearing cached regression model based on r-squared values
Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).
Ratcheck fixes
Added test for SimpleRegression. Minor fix for OLS model
fixed test errors
fixed checkstyle errors
(cherry picked from commit fab411b)
* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)
* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties
NIFI-6566 - changes to allow scoring configurations for model in nifi.properties
NIFI-6566 - added default implementation value to NiFiProperties
NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3663
* NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type
* NIFI-6586 - documentation and comments
This closes NIFI-6586
Signed-off-by: Andrew I. Christianson <andy@andyic.org>
* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.
* NIFI-6510 - add an enable/disable property for analytics
* NIFI-6510 - documentation updates for enable/disable property
* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)
* NIFI-6510 - admin guide updates to further describe model functionality
* NIFI-6510 - code quality fixes (if statement and constructor)
* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold
* NIFI-6510 Extract out predictions into their own DTO
* NIFI-6510 Optimize imports
* NIFI-6510 Fix formatting
* NIFI-6510 Optimize imports
* NIFI-6510 Optimize imports
* NIFI-6510 - Notice updates for Commons math and Caffeine
* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)
* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3705
* NIFI-6510 Rip out useless members
* NIFI-6510 - dto updates to check for -1 value
* NIFI-6510 - checkstyle fix
* NIFI-6510 - rolled back last change and applied minNonNegative method
* NIFI-6510 Rip out useless members
* fixed issues converting sensitive props to parameters
* only include parameter value if it has changed (even for non-sensitive params)
* do not allow null parameter values when editing parameters.
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into parameters
* Refactor addNewParameter method to be able to share common parts with convertPropertyToParameter
- Renamed PARTITION_VALUES to STATIC_PARTITION_VALUES for correctness and better understanding
- STATIC_PARTITION_VALUES descriptions clearly states that having that property filler implies Hive Static Partitioning
NIFI-6536: Additional documentation for Static Partition Values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3631
Additional test case in TestEncryptContent
Apply suggestions from code review
nit picking
Apply suggestions from code review
nitpicking from mgaido91
Small fixes
Additional test case to check the provided cipher is actually used by inferring it from the ciphertext
Updated `EncryptContent` to use a pre-defined list of allowable PGP ciphers
Updated `EncryptContent` to check the validity of `PGP_SYMMETRIC_ENCRYPTION_CIPHER`
Updated test cases
Minor fixes
NIFI-6536 Fixed typo and strengthened validity checking for PGP symmetric cipher.
This closes#3664.
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
NIFI-6412 Upgraded Spring Security version in nifi-framework-bundle pom.xml.
NIFI-6412 Updated version for spring.security.version in nifi-kerberos-iaa-providers and nifi-ldap-iaa-providers pom.xml files.
This closes#3564.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
As NIFI-6508 has already solved the original issue, turned this PR into a refactor within in the affected Test class.
This closes#3621.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.
This closes#3657.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context
NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry
NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException
This closes#3642.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4775: Rolling back fix for NIFI-6416
NIFI-4775: Documentation improvements
+1 from markobean
This closes#3648
Signed-off-by: Brandon <devriesb@apache.org>
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build. Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.
NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service
NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11
This closes#3404.
<h1>Changes</h1>
- Instead of keeping a Set of referencing `ComponentNode`, keep instead a Map which stores, for each referencing component, also the name of the property for which the service is referenced
- This is done to keep the `addReference()` idempotent
- `StandardControllerService.referencingComponents` changed from HashSet to HashMap
- `Key` is the combination of `ComponentNode.hashCode()` and property name
- `Value` is the referencing `ComponentNode`
- `ControllerServiceNode.addReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.removeReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.getReferences()` signature is left untouched, when it's called, the values of the `referencingComponents` map are turned into a Set, meaning a referencing component will keep being returned by this method until at least one of its properties still reference the `ControllerService`
- `StandardSchedulingContext.leaseControllerService()` uses a dedicated constant
- `FrameworkIntegrationTest` has a test which create a processor with two optional controller service properties and then verifies that having them referencing and dereferencing the same controller service doesn't cause the reported bug anymore
Removed SchedulingContext from ClojureScriptEngineConfigurator
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3600.
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations
This closes#3630.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.
NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify
NIFI-6380: Added Description to Parameter Context
NIFI-6380: Fixed checkstyle violations
NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged
NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.
NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context
NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO
NIFI-6380: Added additional integration tests around escaped parameter references
NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression
NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update
NIFI-6380: Addressed review feedback
NIFI-6380: Addressed additional review feedback, mostly around code cleanup
NIFI-6380: Bug fix
NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing
NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource
NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior
NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context
NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version
NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update
NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service
NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task
NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.
NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.
NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context
This closes#3536
Fix imports auto formatted by intellij
NIFI-6487 Fix WriteAttribute documentation
This closes#3603.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Unified unit tests
Added custom validation cases for MIN_RECORDS and MAX_RECORDS enforcing they are greater than zero.
While MIN_RECORDS > 0 can fail individually, MAX_RECORDS > 0 validator cannot fail without having also at least another validation step (either the MIN_RECORDS > 0 or the MAX_RECORDS > MIN_RECORDS) to fail, since MIN_RECORDS is a required property with default value 1
This closes#3607.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5856: Fixed checkstyle violation
NIFI-5856: Addressed review feedback
NIFI-5856: When assigning a matching Controller Service at a higher level for Process Group import/update, ensure that we consider the User's read permissions when selecting the Controller Service
This closes#3553.
Signed-off-by: Bryan Bende <bbende@apache.org>
This modification updates the getStatusHistory method of
VolatileComponentStatusRepository to utilize the start, end, and
preferredDataPoints parameters when retrieving status histories.
When calling the various get<XXXX>StatusHistory methods of
VolatileComponentStatusRepository, they are all passed start, end,
and preferredDataPoints values to allow the filtering of histories
according to the provided parameters. But the follow-on method calls ignore
those values completely. This ticket updates those methods to honor the
parameters and filter the returned Dates accordingly.
This closes#3579.
Signed-off-by: Mark Payne <markap14@hotmail.com>
CSVReader and CSVRecordSetWriter controller services and also ConvertExcelToCSVProcessor
support EL for Value Separator, Quote Character and Escape Character properties.
NIFI-6318: Fixed null checks and compound OR expression.
NIFI-6318: RecordSetWriterFactory.createWriter() changes.
NIFI-6318: Initialize CSVFormat in onEnabled() if there are no dynamic formatting properties.
NIFI-6318: Comment Marker supports EL.
NIFI-6318: Various review changes.
This closes#3504.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Matt Gilman
Mark Payne
Koji Kawamura
Matthew Burgess
Yolanda M. Davis
Andy LoPresto
samhjelmfelt
Andrew Rodriguez
Mark Bean
Bryan Bende
JF Beauvais
Grzegorz Kołakowski
Joe Witt
Endre Zoltan Kovacs
Kevin Doran
Peter Turcsanyi
Rob Fellows
Peter Wicks
Kotaro Terada
Mike
James Cheng
Pierre Villard
Jeff Storck
Alessandro D'Armiento
Koji Kawamura
David Mollitor
archongum
eduardofontes
Scott Aslan
thenatog
Tamas Palfy
FaizFizy Rosle
Troy Melhase
korir
Hsin-Ying Lee
Andy I. Christianson
Arpad Boda
Brandon
Alessandro D'Armiento
Mike Thomsen
SandishKumarHN
Evan Reynolds
Malthe Borch
andrew
Joe Gresock
Ferenc Szabó
Mark Owens