NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
Updated documentation.
Added unit tests.
NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.
NIFI-7669 Fixed failing test due to nifi.properties initialization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4435.
Added unit tests.
NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.
NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4436
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4431.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.
This closes#4416.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.
This closes#4421.
Co-authored-by: mtien <mtien.apache@gmail.com>
Added exception mapper.
NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4418.
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4243.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Tar format allows us to archive files with their original permission,
owner, group name and last modification time.
When unpacking with Tar unpacker, these information are stored in new
attributes with names: "file.inner.*". This way, it preserves backward
compatibility when using parallel with GetFile processor (which stores
information in "file.*").
NIFI-6128 Tar unpackContent: assert date of last modification of content is a valid date format.
NIFI-6128 UnpackContent: use original attributes
In case of tar format:
- Use "file.*" attributes instead of "file.inner.*" (which eventually
lead to overwrite if tar had been fetched with GetFile)
- Store file permission in "rwx" format, instead of integer
representation.
- Also replace SimpleDateFormat with DateTimeFormatter.
Replace generic error with IllegalArgumentException
Also refactor: move permission string to top of file as static constant.
Update test
Remove 'file systems' text
Unpackcontent: Fill file.creationTime attribute
which always holds the very same value as file.lastModifiedTime.
This closes#4370.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7581 Separated Controller Service for providing Azure credentials for ADLS (ADLSCredentialsControllerService) form the one that does the same for Blob storages (AzureStorageCredentialsDetails). (This was done due to the considerable difference in the APIs of the libraries used to connect to both.)
NIFI-7581 Fix: Register controller service in META-INF. Minor fixes.
NIFI-7581 Minor changes (documentation, type etc.)
NIFI-7581 Updated integration tests.
NIFI-7581 Minor changes (renaming).
This closes#4369.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Added properties to enable/disable chunked encoding and path-style access
for endpoints that do not support chunked encoding / only support path-style access.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4386.
NIFI-6934 Added more documentation and unit tests.
NIFI-6934 Added missing license for new test class.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4350
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4344.
The introduced changes prevent creating unnecesary sessions and producers
in some scenarios.
This closes#4378.
Signed-off-by: Joey Frazee <jfrazee@apache.org>
In QueryCassandra when writing flowfile to the sesion it's done on the raw OutputStream.
Wrapped it in a BufferedOutputStream for better performance.
This closes#4368.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
ListenHTTP processor now binds port and creates a HTTP connection only
if one of the following conditions apply:
- Primary node execution is 'false'
- Primary node execution is 'true' and node is elected as primary node.
Changes:
- Connection is established in 'onTrigger' annotated method instead of
'onSchedule'. (This is similar to how handleHTTPRequest processor
handles connections.)
- 'onPrimaryNodeStateChange' annotated method is introduced to tear down
server on reelection of primary node
This closes#4356.
Signed-off-by: Mark Payne <markap14@hotmail.com>
* NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task
Also fixing ControllerServiceDisabledException-s when validating the Kerberos config
* NIFI-7523: Fixed test failure on Windows
* NIFI-7523: Added license headers.
* NIFI-7523: Fixed another test failure on Windows
* NIFI-7523: Review changes
NIFI-7542 Override additional jackson-databind versions.
NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml.
This closes#4343
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
Update nf-context-menu.js for an intuitive road to parameters
When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now.
Added generateflowfile load tag and description
Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile.
Revert "Update nf-context-menu.js for an intuitive road to parameters"
This reverts commit 3c44b1661f.
This closes#4333
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- update Kudu dependencies to Kudu 1.12.0
- add VARCHAR to Kudu Lookup Service and Processor
- add tests for VARCHAR columns
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4347.
* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment
This also fixes NIFI-4760.
* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set).
NIFI-7527 Refactor: removed redundant kudu client creation.
This closes#4330.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.
This closes#4322
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#4271.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.
This closes#4289.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4306.
Added skeleton of oauth2 provider.
Added copy of our code.
Refactored a few things.
Updated apis to better match flow descriptions.
Updated poms and other artifacts.
Updated copyright notice.
Updated LICENSE.
This closes#4173
Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
NIFI-7477 Improving description and unit test now verifies attribute content
NIFI-7477: Fixed checkstyle errors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4301
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function
NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.
NIFI-7403: fix reroute logic in AdjustFailed function
NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property
NIFI-7403:Update for PR recheck
NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior
NIFI-7403: Fix Checkstyle issue
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4266
NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
Made warning and error messages more informative.
Refactored flowFile assertion in the tests.
This closes#4287.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.)
NIFI-7453 Safely closing old Kudu client before creating a new one.
NIFI-7453 Visibility adjustment.
This closes#4276.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types
NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4282
Create empty relationship for RunMongoAggregation
Fix default autoterminate and condition to redirect to REL_EMPTY
Change from new relationship to write an empty FlowFile to RESULT
Fix MONGO_URI
This closes#4281
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes#4263.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case
This closes#4272.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4274
NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.
This closes#4257.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.
This closes#4192.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4252.
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4255.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
without the flowfile
- change the content name and content file name from dynamic properties
to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests
This closes#4234.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.
This closes#4222.
Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.
NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.
This closes#4213.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Added System-level tests for Provenance repository to reproduce behavior.
- Added a Provenance Client to the CLI, which is necessary for System-level tests.
- Added small additional configuration for Provenance repository to simplify development of system tests
- Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests
Signed-off-by: Joe Witt <joewitt@apache.org>
https://issues.apache.org/jira/browse/NIFI-7366
This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4215.
Tamás Bunth
Andy LoPresto
Karthikeyan Singaravelan
Mark Payne
Peter Turcsanyi
Bryan Bende
Nathan Gough
Pierre Villard
Mike Thomsen
zhangcheng
Joey Frazee
Vasily Makarov
Joe Witt
Peter Gyori
Cuong Nguyen
Tamas Palfy
neptunesalt
mtien
Matthew Burgess
KovalevIV
Gardella Juan Pablo
Peter Turcsanyi
dennisjaheruddin
Greg Solovyev
thenatog
sjyang18
wanghq-c
Matt Gilman
Wouter de Vries
Bence Simon
jahenaor
Niels Basjes
r65535
adyoun2
pcgrenier
eduardofontes
Sushil Kumar
Matthieu Cauffiez
Yolanda M. Davis
Otto Fowler
Alan Jackoway
Jérémie
muazmaz
EndzeitBegins
Alexandre Vautier