NIFI-7906: addressing PR concerns
NIFI-7906: code styling fixes
NIFI-7906: adding in license information to new files
+ enables processor in META-INF
NIFI-7906: exclude test files from RAT
NIFI-7906: PR refactor to streamline graph response
NIFI-7906: removing ERRORS output
Unused after refactor
Did a few cleanups for the contributor.
This closes#4638
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-8039 Review findings; refining thread pool to be able to scale down properly when not under load
NIFI-8039 Answers to PR comments
This closes#4689.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7821 Updated configuration documentation.
NIFI-7821 Fixed getAndPutIfAbsent and added int test.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4635
Addressed PR#4153 comments; removed ES Version property and made Type optional in all ES HTTP/Record processors, applying sensible default values where required; use _source queyr parameter instead of _source_include/s as it's compatible between ES versions
Fix unit test compilation to use JDK8-compatible library/method
Better optional type and id handling for PutElasticsearchRecord; update nifi-elasticsearch-client-service build dependencies to use latest versions of Elasticsearch in each supported major version (5/6/7); addressed several warnings in ElasticSearchClientServiceImpl
This closes#4667.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup
NIFI-7897: Addressing review feedback
NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly
This closes#4669.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page
- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests
- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm
- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client
- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding
- Made atomic replace methods for storing groups and saml credential in database
- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata
- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias
- Fixed unit test
- Added property to specify an optional identity attribute which would be used instead of NameID
- Cleaned up logging
- Fallback to keystore password when key password is blank
- Make signature and digest default to SHA-256 when no value provided in nifi.properties
This closes#4614
NIFI-7989: Allow for optional blank line after optional column and partition headers
NIFI-7989: Incorporated review comments
NIFI-7989: Close Statement when finishing processing
NIFI-7989: Remove database name property, update output table attribute
This closes#4653.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7922: Fixes based on review comments
ListenHTTP: changed if(!sslRequired) to if(sslRequired) so that the positive case comes first.
HealthCheckServlet: response body for GET contains "OK".
ContentAcknowledgmentServlet: super.doDelete() is called when a DELETE should be rejected because of port mismatch.
NIFI-7922: Refactored, based on review comments
NIFI-7922: Fixed a checkstyle violation (organized the imports)
This closes#4603.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.
This closes#4613.
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() and taking care of TGT renewal.
* NIFI-7954 Simplified SecurityUtil.callWithUgi a little.
* NIFI-7954 Simplified SecurityUtil.callWithUgi more.
* NIFI-7954 Removed unnecessary code.
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission
This closes#4642.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added a 'dependent' attribute to determine whether or not to save dependent property values
Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
Bumped icu4j dependency to 60.2.
Replaced jackson-mapper-asl dependency with jackson-databind.
Fixed an error comparing key identities in TestKeyService.
Replaced jackson-mapper-asl ObjectMapper with jackson-databind ObjectMapper in LivySessionController.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4640.
Added method to validate the OIDC Access Token for the revoke endpoint.
Created a new callback URI of oidc/logoutCallback to handle certain OIDC logout cases.
Changed method to exchange the Authorization Code for a Login Authentication Token.
Added a new method to exchange the AuthN Code for an Access Token.
Changed method to convert OIDC Token to a Login AuthN Token instead of a NiFi JWT.
Created new OidcServiceGroovyTest class.
NIFI-7584-rebase Added test.
NIFI-7584 Fixed a checkstyle issue.
NIFI-7584 Removed a dependency not in use.
NIFI-7584 Made revisions based on PR review.
Refactored revoke endpoint POST request to a private method.
Removed unnecessary dependencies.
Fixed Regex Pattern to search for literal dot character.
Fixed logging the Exception message.
Fixed caught Exception.
Changed timeout value to a static variable.
Changed repeating error messages to a static string.
Reduced sleep duration in unit test.
Refactored cookie generation to private method.
NIFI-7584 Fixed the snapshot version.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4593.
Use an AspectJ aspect and agent to intercept the load native library calls
at runtime, copy the native library file to temp folder and proceed with
the newly created file in order to provide classloader isolation.
Remove AspectJ jars from lib directory, move the necessary jar to lib/aspectj subdirectory.
This closes#4540.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-7549 Refining documentation; Changing explicit HA mode; Smaller review comments
NIFI-7549 Code review responses about license, documentation and dependencies
NIFI-7549 Fixing issue when explicit HA; Some further review based adjustments
NIFI-7549 Response to code review comments
NIFI-7549 Adding extra serialization test
NIFI-7549 Minor changes based on review comments
NIFI-7549 Adding hook point to the shutdown
This closes#4510.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests
NIFI-7401 Added keystore types and changed properties to match nifi.security.*
NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test
NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties
NIFI-7401 Changed connection socket constants to use .class.getName()
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4592.
* NIFI-7592: Allow NiFi to be started without a GUI/REST interface
* NIFI-7592: Enable all controller services when starting headless
* NIFI-7592: Marked duplicate dependencies as provided
* NIFI-7592: Incorporated additional review comments
NIFI-7843 Recursive avro schemas fail to write with RecordWriter
Add new test case to TestSimpleRecordSchema to test the scenario
when schema name and schema namespace match.
This closes#4550.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- update Kudu dependencies to Kudu 1.13.0
- add support for passing java.sql.Date for Kudu DATE columns
- add tests for passing java.sql.Date to DATE columns
more about DATE type support in Kudu:
https://issues.apache.org/jira/browse/KUDU-2632
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4549.
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
Add the following functionalities:
- Custom value separator (default is comma)
- Custom quote char to use (default is " i.e. quote sign)
- Quote mode
- Escape character to use (default is no escape character)
- Comment marker
- Trim fields
- Character set to use
The above features use a common implementation with "CSVReader".
Also append a sentence to the capability description that first line of
csv file is considered header. Setting custom header instead of using
the first line is not supported (yet).
Also, a minor refactor: CSVRecordLoopkupService and
SimpleCsvFileLookupService now share common logic in implementation.
CSV Format is extended to the same list as CSVReader. In addition,
lookup services still have the "default" csv format for compatibility
reasons.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4494.
The TikaInputStream and FileInputStream instances utilized in IdentifyMimeType are now explicitly closed. The FileInputStream is additionally wrapped by a BufferedInputStream.
Signed-off-by: Joe Witt <joewitt@apache.org>
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
Updated the patch based on @tpalfy's review
Updated the patch based on @mattyb149's review
Rename DATE_FORMAT_PATTERN to JSON_TIMESTAMP_FORMAT_PATTERN
Changed convertToJsonStream method's visibility to package private.
Removed json prefix from timestamp-format-pattern property to make it more generic
This closes#4463.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Because even though permissions are only the 7 least significant bits of the file
mode but the file mode can be wider and can contain further info (like the
sticky bit).
Extend unit test for converting file mode with sticky bit into 'rwx' style
permission string.
Remove old test cases
This closes#4490.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7729: Updated docs to explain how to add new fields to Records via ScriptedTransformRecord and added example. Fixed checkstyle violation.
NIFI-7729: Always call Record.incorporateInactiveFields with ScriptedTransformRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4470
Mike Thomsen
Levi Lentz
Matthew Burgess
exceptionfactory
markap14
Bence Simon
Koji Kawamura
Chris Sampson
Joe Gresock
Arek Burdach
Denes Arvay
humpfhumpf
Bryan Bende
Tamas Palfy
Kevin Doran
r65535
Peter Gyori
s9514171
Nathan Gough
Peter Turcsanyi
tpalfy
Pierre Villard
Siyao Meng
mtien
sjyang18
Mark Bean
Fabio Torchetti
Joey Frazee
eylonronen
Indrajeet Gour
Dan Kim
dependabot[bot]
kpsngh1998
Andy LoPresto
M Tien
Ofirmgr
Denes Arvay
Mohammed Nadeem
doodzio
Greg Solovyev
Paul Kelly
Peter Wicks
Andy LoPresto
yoshiata
abrown
Shane Ardell
Tamás Bunth
Andreas Schöneck
noedetore
Cameron E. Tidd
Kotaro Terada
jmconte
Kristjan Antunovic
Joe Witt
Kyle Miracle
bibistroc
si