This PR introduces 2 new properties for the ConsumeAMQP processor
And one new property for PublishAMQP
This allows to configure the processors to use escaping for commas and to consistently not use curly braces in the amqp$header attribute.
The default values ensure backwards compatibility.
This closes#5458.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Added generic type to AbstractPutEventProcessor for compiler checking of event types
- Refactored createTransitUri to shared method in AbstractPutEventProcessor
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5658.
- Added optional includedRegistries query parameter to Flow Metrics Resource method supporting one or more registries
- Added optional includedNames query parameter to Flow Metrics Resource method supporting one or more metric family names
- Added sampleName and sampleLabelValue optional pattern parameters
- Added FilteringMetricFamilySamplesEnumeration to support streamed filtering
- Added PrometheusMetricsWriter and TextFormat implementation
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5571.
- Removed deprecation from ListenTCP Pool Receive Buffers property
- Added BufferAllocator configuration property for NettyEventServerFactory
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5653.
- Added nifi-property-protection-api with provider interfaces
- Added nifi-property-protection-factory with implementation references
- Added ProtectionSchemeResolver for abstracting conversion from command arguments
- Refactored PropertyProtectionScheme to package private visibility
- Refactored multiple unit test and removed provider integration tests
- Renamed AESSensitivePropertyProvider to AesGcmSensitivePropertyProvider
- Added getSupportedProtectionSchemes() to StandardProtectionSchemeResolver
- Updated command argument descriptions for protection schemes to include supported values
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5650.
NIFI-7749 Added authenticated HTTP proxy support for SFTP
- Added StandardSocketFactoryProvider to return SocketFactory based on credentials
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5624.
NIFI-9390: Addressed underlying condition in stateless framework that caused Merge-related processors and similar to not properly be triggered as necessary. Added several system tests to verify different configurations.
NIFI-9390: Simplified the logic for how to iterate over the components in a Stateless flow that are ready to be triggered
This closes#5634.
Co-authored-by: Peter Turcsanyi <turcsanyi@apache.org>
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Upgraded Apache HttpCore to 4.4.15
- Added dependency management declarations in root Maven configuration for HttpClient and HttpCore
- Removed version numbers from multiple modules
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5647.
- Removed nifi-elasticsearch-5-bundle
- Removed include-elasticsearch-5-bundle profile from nifi-assembly
This closes#5636
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Removed incorrect usage of TriggerWhenEmpty
- Allow for 0 seconds of gracefully waiting for incoming threads on shutdown since we won't have any
- Updated unit tests to no longer have arbitrary sleep statements
This closes#5639
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Centralized Spring Framework and Spring Security versions using BOM dependencies
- Upgraded Spring Security from 5.5.2 to 5.6.1
- Upgraded Spring Boot from 2.5.5 to 2.5.8 in Registry
- Upgraded Spring Integration from 5.5.2 to 5.5.7
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5631.
* NIFI-9391: Modified MergeRecord to process FlowFiles within a loop in a single onTrigger
MergeRecord processed the FlowFiles in multiple onTrigger-s and it needed an extra onTrigger call
(with no incoming FFs) to realize that no more FFs are available and it is time to send the merged FF downstream.
It was not compatible with Stateless Runtime which does not trigger the flow any more if no FFs available.
Also changed "unschedule" logic in StandardProcessorTestRunner: @OnUnscheduled methods were called immediately after
the 1st FlowFile was processed. Unschedule the processor only at the end of the execution (onTrigger finished)
and only if stopOnFinish has been requested by the test case.
change the default value of auto commit function to true
Changed the auto commit property name and add more details in the description
If the auto commit is set to false, commit() is called for consistency
adds unit tests
Fix the check style issue of having more than 200 characters in single line
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5554
- SSHJ 0.32.0 introduced support for rename flags in SFTP commands without checking the protocol version
- PatchedSFTPEngine overrides the rename method to check the SFTP protocol version
Signed-off-by: Joe Witt <joewitt@apache.org>
- Refactored SSH Client configuration and connection to SSHClientProvider
- Implemented exception handling for configuration and connection failures
- Named SSH keep-alive thread for improved runtime tracking
- Closed SSH Client and interrupted keep-alive thread on configuration failures
- Added missing Compression Property to ListSFTP
- Corrected Hostname and Port property descriptors in ListSFTP
Signed-off-by: Joe Witt <joewitt@apache.org>
- Added log4j-core to list of banned dependencies
- Added log4j-to-slf4j for Elasticsearch 5 processors to support runtime logging
Signed-off-by: Joe Witt <joewitt@apache.org>
- Replaced JUnit 4 and TestNG references with JUnit 5
- Added test method for bulletin sorting
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5607
- Changed display name of Max Number of TCP Connections to Max Number of Worker Threads for ListenTCP
- Set Netty Socket Receive Buffer using Max Socket Buffer Size in ListenTCP
This closes#5599
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-9423 - Show icon and tooltip for Parameter values that have leading and/or trailing whitespaces
NIFI-9429 - Parameters should allow blank values that are non-null (only whitespace)
* - Update areas to clean up tooltips in parameter values
- Show whitespaces and ellipsis in parameter and property values and tooltips
- Update serializeValue to accommodate for blank values
* - Address review findings
* - Remove commented out code
* - Add multiline check for ellipsis
* NIFI-9459 - Empty string checked will disable Edit Parameter value field on dialog open
* - Add multi-line style to parameter and property table
* - Safely insert title attribute content
* - Fix Edit Parameter bug that clears textarea for sensitive and empty string values on dialog open
This closes#5569
- Add log4j-bom to root Maven configuration
- Remove previous overrides in favor of log4j-bom in root Maven configuration
This closes#5598
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Implemented override for Timestamp Record Field Type format handling to add support for optional microseconds
- Added FieldConverter and ObjectTimestampFieldConverter implementation for generalized Timestamp parsing using DateTimeFormatter
- Updated PutKudu unit tests for standard Timestamp and Timestamp with microseconds
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5589.
NIFI-9382: Fixed issue with SharedInstanceClassLoader where the classloader may get closed but then get used again. When the SharedInstanceClassLoader is closed, we will now ensure that we don't use anymore and instead create a new one.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Add PutElasticsearchJson processor to Elasticsearch REST bundle
- Deprecate PutElasticsearchHttp/PutElasticsearchHttpReccord in favour of Elasticsearch REST processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5566.
- Changed framework so that it serializes the dataflow into a VersionedDataflow using JSON as well as XML, and prefers the JSON representation on load, if it's available. This also results in the need for the cluster protocol to exchange its representation of the dataflow to using JSON. Rather than re-implementing all of the complex logic of Flow Fingerprinting, updated to just inherit the cluster's flow.
- Moved logic to synchronize Process Group with Versioned Process Group into a new ProcessGroupSynchronizer class instead of having all of the logic within StandardProcessGroup
- Reworked versioned components to use an instance id.
- Renamed StandardFlowSynchronizer to XmlFlowSynchronizer; introduced new StandardFlowSynchronizer that delegates to the appropriate (Xml or Versioned)FlowSynchronzer
- Updated to allow import of VersionedProcessGroup even if not all bundles are available - will now use ghost components
- Introduced a VersionedDataflow object to hold controller-level services, reporting tasks, parameter contexts, templates, etc.
- Allow mutable requests to be made while nodes are disconnected. Also fixed issue in AbstractPolicyBasedAuthorizer that caused ClassNotFoundException / NoClassDefFoundError if the authorizations were changed and then a node attempts to rejoin the cluster. The Authorizer was attempting to use XmlUtils, which is in nifi-security-utils and so so by madking nifi-security-utils a provided dependency of nifi-framework-api, but this doesn't work, because nifi-framework-api is loaded by a higher-level classloader, so the classloader that loads AbstractPolicyBasedAuthorizer will never have the appropriate classloader to provide nifi-security-utils. Addressed this by copying the code for creating a safe document builder from XmlUtils to AbstractPolicyBasedAuthorizer.
- Fixed bug that occurred when importing a Process Group that has 2 parameter contexts, one inheriting from another, where neither is pre-defined in the existing flow
- Fixed bug that was encountered when Updating a Versioned Process Group where one version had a disabled processor and the other had the processor running.
- Increased system-tests workflow timeout to 120 minutes
- Added additional exception handling to system tests
This closes#5514
Signed-off-by: David Handermann <exceptionfactory@apache.org>
* NIFI-9336 - Show icon in processor and controller services configurations when property values contain leading or trailing whitespace
* - Address PR feedback
* - Fix a bug to clean up tooltips to prevent a DOM leak
This closes#5559
- Upgraded com.fluenda:parcefone from 2.0.0 to 2.1.0
- Added Accept empty extensions property to ParseCEF
This closes#5555
Co-authored-by: David Handermann <exceptionfactory@apache.org>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Replaced HashMap with LinkedHashMap to avoid potential non-deterministic results for user group properties
This closes#5524
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- NIFI-9163 Refactored nifi-websocket-bundle to use JUnit 5
- NIFI-9162 Refactored nifi-update-attribute-bundle to use JUnit 5
- NIFI-9161 Refactored nifi-tcp-bundle to use JUnit 5
- NIFI-9160 Refactored nifi-stateful-analysis-bundle to use JUnit 5
- NIFI-9159 Refactored nifi-sql-reporting-bundle to use JUnit 5
- NIFI-9158 Refactored nifi-spring-bundle to use JUnit 5
- NIFI-9157 Refactored nifi-splunk-bundle to use JUnit 5
- NIFI-9156 Refactored nifi-spark-bundle to use JUnit 5
- NIFI-9155 Refactored nifi-solr-bundle to use JUnit 5
- NIFI-9154 Refactored nifi-social-media-bundle to use JUnit 5
- NIFI-9153 Refactored nifi-snmp-bundle to use JUnit 5
- NIFI-9152 Refactored nifi-smb-bundle to use JUnit 5
- NIFI-9151 Refactored nifi-slack-bundle to use JUnit 5
- NIFI-9150 Refactored nifi-site-to-site-reporting-bundle to use JUnit 5
- NIFI-9149 Refactored nifi-single-user-iaa-providers-bundle to use JUnit 5
This closes#5362
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Set Scheduled State for Versioned Port and Versioned Remote Port when mapping Flow Definition
- Updated StandardProcessGroup to set disable Port based on Scheduled State of DISABLED
- Updated StandardProcessGroup to set Remote Port transmitting based on Scheduled State of ENABLED
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5534.
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests
* NIFI-9382: Renamed interface based on review feedback
* NIFI-9382: Removed ReentrantKerberosUser.
- Added XML Stream Reader processing for XSLT with external entities disabled
- Removed unused XsltValidator
- Upgraded Saxon-HE from 9.6.0-5 to 10.6
Signed-off-by: Joe Witt <joewitt@apache.org>
- NIFI-9387: add Proxy capability to ElasticsearchClientService
- NIFI-1576: allow GetElasticsearch to run without requiring FlowFile input
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5535.
- Added nifi.web.request.log.format property
- Added Filters to set and retrieve authenticated username for logging
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5527.
- Addressed compiler warnings in ListenTCP and EventBatcher
- Adjusted ListenTCP property order to match previous version
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5526.
- Deprecated Max Number of Receiving Message Handler Threads property
- Deprecated Pool Receive Buffers property
- Updated TestListenTCP using Netty EventSender
This closes#5493
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Removed unnecessary spaces from initialization log
- Changed bootstrap temporary password file processing messages to debug
- Updated several log statements using parameterized strings
- Refactored NiFi unit test class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5515.
- ConcurrentHashMap collection for Active Threads eliminates the need for method synchronization
- Replaced System.out.println() with Logger.info() in nifi-system-test-suite classes
- Methods without the synchronized keyword, such as terminate(), iterate over activeThreads
This closes#5509
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- changed default credential type to SAS Token
- added endpointSuffix property in integration tests
- added AzureServiceEndpoints to provide endpoint suffix constants and methods to get endpoint urls
- fixed typos, validators
- downgraded azure-identity to 1.3.7 due to a bug in the latest version
- added Access Token credential type (similar to the existing Blob/ADLS processors)
- used JUnit 5 in tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5486.
Added <scope>test</scope> tag to the nifi-web-api pom.xml and corrected imports.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5447
Modifying unit test to avoid systemdefault timezone usuage
NIFI-9185 Applying review recommendations removing duplicate dependency from pom.xml
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5358
- Ensure that we consider all property values to determine if a component references a variable, not just 'supported' (i.e., well-known/non-user-defined) properties
This closes#5499
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Removed parent AccessResource from OIDCAccessResource and SAMLAccessResource to avoid unexpected inherited methods
- Moved Token Expiration validation from AccessResource to StandardBearerTokenProvider
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5489.
- Replaced Netty 3.6.9 and 3.7.1 with 3.10.6
- Replaced Netty 4.1 with 4.1.69
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5490.
- Add AllowableValue objects for write concern values
- Remove warnings for using deprecated write concerns in PutMongo
This closes#5392
Signed-off-by: David Handermann <exceptionfactory@apache.org>
NIFI-9309: Include a uiOnly flag when requesting flow for a given process group. In that case, do not include the property descriptors, property values, etc. for Processors. When fetching Variable Registry, improved logic to cache the VariableImpact for each property value instead of parsing/recreating it every time. When fetching bulletins for a component, avoid filtering through all components' bulletins and instead only look at bulletins that might pertain to the appropriate component
- Allowing the user to submit a verification request for Processors, Controller Services, and Reporting Tasks.
- Tracking progress of verification requests.
- Showing the verification results.
NIFI-9288:
- Fixing class name which prevented styles from being applied.
NIFI-9288:
- Ensuring that previously entered referenced attribute values take precedence.
NIFI-9288:
- Positioning the property listing and verification results based on percentages instead of fixed values.
- Removing the additional dialog height.
NIFI-9288:
- Allowing attribute value entry to be skipped when appropriate.
NIFI-9288:
- Working around an issue caused by css minification.
NIFI-9288:
- Adding some padding to the verifying progress dialog.
This closes#5461
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
Also fixed ITPutAzureDataLakeStorage (was broken due to a former dependency upgrade).
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5472.
- Similar to NIFI-9215, converting integer identifiers to strings to ensure the items are successfully retrieved when attempting to apply a tooltip.
This closes#5454
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Added Decryption Strategy property to DecryptContentPGP
- Added OpenPGP Packet detection to EncryptContentPGP to avoid unnecessary packaging
- Refactored shared processing to EncodingStreamCallback
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5457.
- Upgraded Spring Security from 5.5.1 to 5.5.2
- Upgraded Spring Boot from 2.5.2 to 2.5.5 in Registry
- Upgraded Spring Data Redis from 2.5.3 to 2.5.5 in Redis modules
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5463.
- Addressed issue identified in system test where a flow can fail due to FailurePortEncounteredException but then purge the flow in the background after the call to trigger completes
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5450.
- Excluded org.apache.hive:hive-exec and added the "core" classifier version
- Added the removed dependencies explicitly
NIFI-9248 Add additional log4j excludes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5414
- During mouse over events the items in the Controller Service Types table could not be looked up because the identifier of the item was an integer value and the identifier was a string value. Addressing the issue by always using a string.
This closes#5439
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Populating the empty state when a parameter is not referenced by any component.
- Tweaking margins so the references are slightly more compact.
This closes#5442
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Migrating away from forceFitColumns which is deprecated. Updating to use the same strategy in the new configuration which will prevent the warning logs to the console.
This closes#5428
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
- Updated documentation with new properties
- Refactored cipher operations to common RepositoryEncryptor classes
- Abstracted record metadata serialization for better compatibility
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5407.
- NIFI-8003: UpdateByQueryElasticsearchProcessor
- Addressed various warnings and inefficiencies found in existing processor code
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#4693.
- Creation of a PaginatedJsonQueryElasticsearch and ConsumeElasticsearch processors
- Integration Tests for ES Client Service with version and flavour-specific tests for applicable functionality
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5193.
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5412.
- Removed c3p0 exclusions since Maven coordinates changed and c3p0 is also licensed under the compatible Eclipse Public License 1.0
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5430.
* NIFI-9229 Flow upgrade not possible if a Output Port changes to a funnel
* NIFI-9229 Addressing review comments modified log message and added comments
- Removed nifi.minifi.sensitive.props.provider Property from MiNiFi
- Removed property from example NiFi properties files
- Removed provider from MiNiFi SensitivePropsSchema
- Removed BC provider value from MiNiFi test cases
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5422.
- Refactored NettyEventServerFactory to accept nullable InetAddress
- Updated unit tests referencing NettyEventServerFactory
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#5426.
- Replaced use of Authorization header with custom Request-Token header for CSRF mitigation
- Added Request-Token cookie for CSRF mitigation
- Replaced session storage of JWT with expiration in seconds
- Removed and disabled CORS configuration
- Disabled HTTP OPTIONS method
- Refactored HTTP Proxy URI construction using RequestUriBuilder
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5417.
* NIFI-8491:
- Adding support for configuring parameter context inheritance.
* NIFI-8491:
- Allowing changes to the parameter context inheritance to drive Apply disabled state.
* NIFI-8491: Updating StandardParameterContext#isAuthorized check
* NIFI-8491:
- Showing selected inherited parameter contexts in ready only form when appropriate.
- Allowing available parameter contexts to be inherited by double clicking.
- Removing support for rendering unauthorized inherited parameter contexts as they can no longer be opened.
* NIFI-8491: Adding inherited param context verification earlier
* NIFI-8491:
- Addressing CI failures by rolling back to some order JS language spec to allow yui-compress to minify and compress.
* NIFI-8491:
- Ensuring selected context sort order is honored.
- Ensuring the Apply button is correctly enabled.
- Showing Pending Apply message when selected Parameter Context changes.
- Ensuring the Parameter's tab is selected now that there is a third tab.
* Updates to inherited param context verification
* Improving validation between parameters/inherited parameters
* NIFI-8491:
- Ensuring the available parameter contexts are loaded whether the edit dialog is opened from the listing or outside of the listing.
* NIFI-8491:
- Fixing conditions we check if the parameter context listing is currently open.
* NIFI-8491:
- Waiting for the parameter contexts to load prior to rendering the parameter context inheritance tab and showing the dialog.
* NIFI-8491:
- Fixing pending apply message clipping.
- Hiding pending apply message after clicking Apply.
Co-authored-by: Joe Gresock <jgresock@gmail.com>
This closes#5371
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Unit tests added to test the use of useLogicalType flag
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER
NIFI-9192: Fixed checkstyle violations
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Added unit tests for Array types with and without using logical types
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
NIFI-9192: Review comments considered and unit tests refactored
NIFI-9192: Added further unit tests
NIFI-9192: Minor modifications based on review comments
Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5388
- Implemented ApplicationCookieService for adding and retrieving HTTP Cookies
- Added getCookieResourceUri() leveraging allowed proxy headers to support optional Cookie Paths
- Refactored Access Resources to use ApplicationCookieService for processing
- Changed __Host- prefix to __Secure- prefix for Bearer Token cookie to support Cookie Path processing
- Removed unnecessary jetty-http dependency from nifi-web-api
- Corrected NiFi path references in JavaScript to support prefixed paths
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5329.