Added method to validate the OIDC Access Token for the revoke endpoint.
Created a new callback URI of oidc/logoutCallback to handle certain OIDC logout cases.
Changed method to exchange the Authorization Code for a Login Authentication Token.
Added a new method to exchange the AuthN Code for an Access Token.
Changed method to convert OIDC Token to a Login AuthN Token instead of a NiFi JWT.
Created new OidcServiceGroovyTest class.
NIFI-7584-rebase Added test.
NIFI-7584 Fixed a checkstyle issue.
NIFI-7584 Removed a dependency not in use.
NIFI-7584 Made revisions based on PR review.
Refactored revoke endpoint POST request to a private method.
Removed unnecessary dependencies.
Fixed Regex Pattern to search for literal dot character.
Fixed logging the Exception message.
Fixed caught Exception.
Changed timeout value to a static variable.
Changed repeating error messages to a static string.
Reduced sleep duration in unit test.
Refactored cookie generation to private method.
NIFI-7584 Fixed the snapshot version.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4593.
Use an AspectJ aspect and agent to intercept the load native library calls
at runtime, copy the native library file to temp folder and proceed with
the newly created file in order to provide classloader isolation.
Remove AspectJ jars from lib directory, move the necessary jar to lib/aspectj subdirectory.
This closes#4540.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-7549 Refining documentation; Changing explicit HA mode; Smaller review comments
NIFI-7549 Code review responses about license, documentation and dependencies
NIFI-7549 Fixing issue when explicit HA; Some further review based adjustments
NIFI-7549 Response to code review comments
NIFI-7549 Adding extra serialization test
NIFI-7549 Minor changes based on review comments
NIFI-7549 Adding hook point to the shutdown
This closes#4510.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests
NIFI-7401 Added keystore types and changed properties to match nifi.security.*
NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test
NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties
NIFI-7401 Changed connection socket constants to use .class.getName()
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4592.
* NIFI-7592: Allow NiFi to be started without a GUI/REST interface
* NIFI-7592: Enable all controller services when starting headless
* NIFI-7592: Marked duplicate dependencies as provided
* NIFI-7592: Incorporated additional review comments
NIFI-7843 Recursive avro schemas fail to write with RecordWriter
Add new test case to TestSimpleRecordSchema to test the scenario
when schema name and schema namespace match.
This closes#4550.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- update Kudu dependencies to Kudu 1.13.0
- add support for passing java.sql.Date for Kudu DATE columns
- add tests for passing java.sql.Date to DATE columns
more about DATE type support in Kudu:
https://issues.apache.org/jira/browse/KUDU-2632
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4549.
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
Add the following functionalities:
- Custom value separator (default is comma)
- Custom quote char to use (default is " i.e. quote sign)
- Quote mode
- Escape character to use (default is no escape character)
- Comment marker
- Trim fields
- Character set to use
The above features use a common implementation with "CSVReader".
Also append a sentence to the capability description that first line of
csv file is considered header. Setting custom header instead of using
the first line is not supported (yet).
Also, a minor refactor: CSVRecordLoopkupService and
SimpleCsvFileLookupService now share common logic in implementation.
CSV Format is extended to the same list as CSVReader. In addition,
lookup services still have the "default" csv format for compatibility
reasons.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4494.
The TikaInputStream and FileInputStream instances utilized in IdentifyMimeType are now explicitly closed. The FileInputStream is additionally wrapped by a BufferedInputStream.
Signed-off-by: Joe Witt <joewitt@apache.org>
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
Updated the patch based on @tpalfy's review
Updated the patch based on @mattyb149's review
Rename DATE_FORMAT_PATTERN to JSON_TIMESTAMP_FORMAT_PATTERN
Changed convertToJsonStream method's visibility to package private.
Removed json prefix from timestamp-format-pattern property to make it more generic
This closes#4463.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Because even though permissions are only the 7 least significant bits of the file
mode but the file mode can be wider and can contain further info (like the
sticky bit).
Extend unit test for converting file mode with sticky bit into 'rwx' style
permission string.
Remove old test cases
This closes#4490.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7729: Updated docs to explain how to add new fields to Records via ScriptedTransformRecord and added example. Fixed checkstyle violation.
NIFI-7729: Always call Record.incorporateInactiveFields with ScriptedTransformRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4470
NIFI-7663 Minor changes (variable name refactor, javadoc, GUI message). Merging Drop All Flowfiles responses across all nodes in a cluster.
NIFI-7663 Reloading the canvas after completing a Drop All Flowfiles request.
NIFI-7663 Fixed typos.
This closes#4425.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Also added validator for Directory Name property in AbstractAzureDataLakeStorageProcessor
Fix Tracking Entities strategy: use milliseconds for lastModified
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4438.
Disable connection automatic recovery which can lead to uncontrolled/stale threads. Handle the recovery in the processors instead.
Use poisoning in case of errors, then discarding and recreating the poisoned consumer/publisher.
NIFI-6312: Use conventional exception handling instead of poisoning
Use component logger in workers.
Remove basicNack()/basicReject() calls as they are not needed because all unacknowledged messages will be redelivered.
NIFI-6312: Further improve exception handling and error logging.
NIFI-6312: Fix consumer closing in previous commit
NIFI-6312: Use custom executor with a single thread (no more is used by the processor)
Reviewed by tamas palfy and simon bence
Signed-off-by: Joe Witt <joewitt@apache.org>
Add new property 'Content Disposition' to allow user
to set the content-disposition http header on the S3 object.
Allowed values are 'inline' (default) and 'attachment'.
If 'attachment' is selected, the filename will be set to the S3 Object key.
Remove default value and keep backward compatibility
Update fetchS3Object filename attribute settin
Update constant names
Update order of if-else condition
NIFI-7664 Update condition in FetchS3Processor
NIFI-7664 Undo the unexpected indent
NIFI-7664 Update international chars unit test
NIFI-7664 Set fetchS3 file path name
NIFI-7664 Update code style
This closes#4423.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.
This property is not required, and has no default value.
The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.
Update property description
Add support EL for cache-control property
This closes#4422.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
Updated documentation.
Added unit tests.
NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.
NIFI-7669 Fixed failing test due to nifi.properties initialization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4435.
Added unit tests.
NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.
NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4436
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4431.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.
This closes#4416.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.
This closes#4421.
Co-authored-by: mtien <mtien.apache@gmail.com>
Added exception mapper.
NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4418.
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4243.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Tar format allows us to archive files with their original permission,
owner, group name and last modification time.
When unpacking with Tar unpacker, these information are stored in new
attributes with names: "file.inner.*". This way, it preserves backward
compatibility when using parallel with GetFile processor (which stores
information in "file.*").
NIFI-6128 Tar unpackContent: assert date of last modification of content is a valid date format.
NIFI-6128 UnpackContent: use original attributes
In case of tar format:
- Use "file.*" attributes instead of "file.inner.*" (which eventually
lead to overwrite if tar had been fetched with GetFile)
- Store file permission in "rwx" format, instead of integer
representation.
- Also replace SimpleDateFormat with DateTimeFormatter.
Replace generic error with IllegalArgumentException
Also refactor: move permission string to top of file as static constant.
Update test
Remove 'file systems' text
Unpackcontent: Fill file.creationTime attribute
which always holds the very same value as file.lastModifiedTime.
This closes#4370.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7581 Separated Controller Service for providing Azure credentials for ADLS (ADLSCredentialsControllerService) form the one that does the same for Blob storages (AzureStorageCredentialsDetails). (This was done due to the considerable difference in the APIs of the libraries used to connect to both.)
NIFI-7581 Fix: Register controller service in META-INF. Minor fixes.
NIFI-7581 Minor changes (documentation, type etc.)
NIFI-7581 Updated integration tests.
NIFI-7581 Minor changes (renaming).
This closes#4369.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Added properties to enable/disable chunked encoding and path-style access
for endpoints that do not support chunked encoding / only support path-style access.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4386.
NIFI-6934 Added more documentation and unit tests.
NIFI-6934 Added missing license for new test class.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4350
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4344.
The introduced changes prevent creating unnecesary sessions and producers
in some scenarios.
This closes#4378.
Signed-off-by: Joey Frazee <jfrazee@apache.org>
In QueryCassandra when writing flowfile to the sesion it's done on the raw OutputStream.
Wrapped it in a BufferedOutputStream for better performance.
This closes#4368.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
ListenHTTP processor now binds port and creates a HTTP connection only
if one of the following conditions apply:
- Primary node execution is 'false'
- Primary node execution is 'true' and node is elected as primary node.
Changes:
- Connection is established in 'onTrigger' annotated method instead of
'onSchedule'. (This is similar to how handleHTTPRequest processor
handles connections.)
- 'onPrimaryNodeStateChange' annotated method is introduced to tear down
server on reelection of primary node
This closes#4356.
Signed-off-by: Mark Payne <markap14@hotmail.com>
* NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task
Also fixing ControllerServiceDisabledException-s when validating the Kerberos config
* NIFI-7523: Fixed test failure on Windows
* NIFI-7523: Added license headers.
* NIFI-7523: Fixed another test failure on Windows
* NIFI-7523: Review changes
NIFI-7542 Override additional jackson-databind versions.
NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml.
This closes#4343
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
Update nf-context-menu.js for an intuitive road to parameters
When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now.
Added generateflowfile load tag and description
Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile.
Revert "Update nf-context-menu.js for an intuitive road to parameters"
This reverts commit 3c44b1661f.
This closes#4333
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- update Kudu dependencies to Kudu 1.12.0
- add VARCHAR to Kudu Lookup Service and Processor
- add tests for VARCHAR columns
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4347.
* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment
This also fixes NIFI-4760.
* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set).
NIFI-7527 Refactor: removed redundant kudu client creation.
This closes#4330.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Peter Turcsanyi
Mark Bean
Mark Payne
mtien
Bryan Bende
Pierre Villard
Fabio Torchetti
Bence Simon
Denes Arvay
Joey Frazee
eylonronen
Indrajeet Gour
Dan Kim
dependabot[bot]
Matthew Burgess
kpsngh1998
Andy LoPresto
Peter Gyori
exceptionfactory
M Tien
Ofirmgr
Denes Arvay
Mohammed Nadeem
Mike Thomsen
doodzio
Greg Solovyev
Paul Kelly
Peter Wicks
Andy LoPresto
yoshiata
abrown
Shane Ardell
Tamás Bunth
Andreas Schöneck
noedetore
Cameron E. Tidd
Kotaro Terada
jmconte
Kristjan Antunovic
Joe Witt
Tamas Palfy
Kyle Miracle
bibistroc
si
Scott Aslan
Mathieu
shreeju
Kent Nguyen
Karthikeyan Singaravelan
Jan Hentschel
Nathan Gough
zhangcheng
Vasily Makarov
Cuong Nguyen
neptunesalt
KovalevIV
Gardella Juan Pablo
Peter Turcsanyi
dennisjaheruddin
thenatog
sjyang18
wanghq-c