111 Commits

Author SHA1 Message Date
exceptionfactory
70432507df
NIFI-6617 Refactored Encrypted Repository configuration
- Updated documentation with new properties
- Refactored cipher operations to common RepositoryEncryptor classes
- Abstracted record metadata serialization for better compatibility

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5407.
2021-10-08 11:57:10 -04:00
exceptionfactory
00b2358c27
NIFI-9252 Removed unused nifi.sensitive.props.provider Property
- Removed nifi.minifi.sensitive.props.provider Property from MiNiFi
- Removed property from example NiFi properties files
- Removed provider from MiNiFi SensitivePropsSchema
- Removed BC provider value from MiNiFi test cases

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5422.
2021-10-01 22:20:13 -04:00
markap14
73f88b3239
NIFI-9061: Eliminated the nifi.cluster.node.protocol.threads property in favor of nifi.cluster.node.protocol.max.threads property so that we can properly scale out the number of threads used for HTTP request replication. Implementing a caching mechanism for creating the DateTimeFormatter used by TimeAdapter in order to improve performance when parsing timestamps in web requests. Implementing caching logic for caching the number of characters that can rendered without needing an ellipsis for some components in the UI (#5316)
This closes #5316
2021-09-13 14:36:35 -04:00
exceptionfactory
0cf09840ea
NIFI-9082 Add nifi.zookeeper.jute.maxbuffer property (#5335)
* NIFI-9082 Added nifi.zookeeper.jute.maxbuffer property

- Updated ZooKeeperStateProvider to check jute.maxbuffer instead of hard-coded number
- Updated internal ZooKeeperClientConfig class to include new juteMaxBuffer property

* NIFI-9082 Updated jute.maxbuffer documentation and added hexadecimal representation

- Renamed DefaultWatcher to NoOpWatcher
- Removed unnecessary validateDataSize() from createNode()
2021-08-26 21:12:34 +02:00
exceptionfactory
a652280fbb NIFI-8766 Implemented RS512 Algorithm for JWT Signing
- Replaced per-user symmetric-key HS256 with shared and rotated RSA asymmetric-key RS512 implementation
- Added nifi.security.user.jws.key.rotation.period property for RSA Key Pair rotation
- Added JSON Web Tokens section to Administration Guide
- Implemented persistent storage of RSA Public Keys for verification using Local State Manager
- Implemented JWT revocation on logout with persistence using Local State Manager
- Refactored JWT implementation using Spring Security OAuth2 and Nimbus JWT
- Refactored Spring Security Provider configuration using Java instead of XML
- Removed H2 storage of per-user keys
- Upgraded nimbus-jose-jwt from 7.9 to 9.11.2

NIFI-8766 Corrected AuthenticationException handling in AccessResource.getAccessStatus

- Added nifi.user.security.jws.key.rotation.period to default nifi.properties
- Updated logging statements and clarified configuration and method documentation

NIFI-8766 Changed Algorithm to PS512 and updated documentation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5262.
2021-08-19 12:26:12 -04:00
exceptionfactory
c668d3df1b NIFI-8782 Added Rate-Limiting for Access Token Requests
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5215.
2021-07-14 14:24:45 -04:00
Joe Witt
97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt
fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
exceptionfactory
db40989b48 NIFI-8516 Enabled HTTPS and Single User Authentication in default configuration
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
2021-06-11 14:18:53 -04:00
exceptionfactory
09e54c1dad
NIFI-8246 Set NIFI_PBKDF2_AES_GCM_256 as default properties encryption algorithm
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5055.
2021-05-12 12:58:50 +02:00
Joe Gresock
54a0e27c93
NIFI-7134: Adding auto-reloading of Keystore and Truststore
- NIFI-7261 Included TrustStoreScanner for auto-reloading of truststore

This closes #4991

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 08:29:42 -05:00
Joe Gresock
90c7d03ed3
NIFI-8403: Generating Self-signed cert on startup when applicable (#4986)
* NIFI-8403: Implementing auto-generated certificates for secure startup

* Adding check for passwords in SecureNiFiConfigUtil
2021-04-27 09:34:44 -04:00
Nathan Gough
9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Peter Turcsanyi
f00f0ad269 NIFI-8314: Generate warning for long-running processor tasks 2021-03-16 14:36:51 -04:00
Bence Simon
7730777d66 NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-25 15:59:01 -05:00
Joe Witt
88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt
4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt
487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Nathan Gough
8057f8f6c5
NIFI-8221 - Set the default HTTP listening interface to 127.0.0.1.
This closes #4817

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-10 14:46:37 -06:00
sjyang18
f330078fff NIFI-7924 Add fallback claims for identifying user to OIDC provider
This closes #4630

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-01-07 14:57:57 -06:00
Bryan Bende
dcc4fb00a5
NIFI-7888 Added support for authenticating via SAML
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page

- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests

- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm

- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client

- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding

- Made atomic replace methods for storing groups and saml credential in database

- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata

- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias

- Fixed unit test

- Added property to specify an optional identity attribute which would be used instead of NameID

- Cleaned up logging

- Fallback to keystore password when key password is blank

- Make signature and digest default to SHA-256 when no value provided in nifi.properties

This closes #4614
2020-11-18 12:44:02 -05:00
Joey Frazee
55cb8d73cb NIFI-7401 Add ZooKeeper client TLS to CuratorLeaderElectionManager
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests

NIFI-7401 Added keystore types and changed properties to match nifi.security.*

NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test

NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties

NIFI-7401 Changed connection socket constants to use .class.getName()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4592.
2020-10-20 11:20:43 -04:00
Joe Witt
8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt
fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt
303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Mark Payne
eca7f153d0
NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is unknown. This way, if a Flow is inadvertently removed, updated, etc., and NiFi is restarted, the data will not be dropped by default. The old mechanism of dropping data is exposed via a property
This closes #4454.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-08-06 10:55:56 -04:00
Bryan Bende
e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne
e371f4ac7c
NIFI-7521: Removed unused properties from nifi properties. Updated the defaults to use for File System's archive cleanup, flowfile checkpoint interval 2020-07-23 08:14:21 -04:00
Andy LoPresto
dbee774c5b
NIFI-7304 Removed default value for nifi.web.max.content.size.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:42:00 -07:00
Nathan Gough
302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
Matt Gilman
e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Sushil Kumar
996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Mark Payne
728bdec0f4 NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. 2020-04-09 13:32:34 -04:00
Troy Melhase
483f23a8aa
NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-25 17:23:22 -07:00
Joe Witt
3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt
633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Andy LoPresto
2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
Joe Witt
f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt
418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt
b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Yolanda M. Davis
147365285c
NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3838
2019-10-24 08:51:50 -04:00
Yolanda M. Davis
8e1452a3f3
NIFI-6649 - added separate query interval configuration for observation queries
NIFI-6649 - documentation update

NIFI-6649 - add debug logging for score and prediction information

NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3719
2019-09-12 12:13:19 -04:00
Andy I. Christianson
8a8b9c1d08
NIFI-6510 - Analytics framework (#3681)
* NIFI-6510 Implement initial analytic engine

* NIFI-6510 Implemented basic linear regression model for queue counts

* NIFI-6510 Initial analytics REST endpoint and supporting objects

* NIFI-6510 Connect the dots for StatusAnalytics -> API

* NIFI-6510 Added poc engine with prediction model caching

(cherry picked from commit e013b91)

DFA-9 - updated logging and corrected logic for checking if not in backpressure

(cherry picked from commit a1f8e70)

* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics

(cherry picked from commit 050e0fc)

(cherry picked from commit 9fd365f)

* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id

(cherry picked from commit 14854ff)

DFA-9 - reduced snapshot interval to 1 minute

(cherry picked from commit 36abb0a)

* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions

* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly

* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"

This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.

* NIFI-6510 Added prediction fields for use by UI, still need to be populated

* NIFI-6510 Analytics Framework Introduction (#10)

* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.

DFA-9 - added logging

* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval

* DFA-9 - checkstyle fixes

* NIFI-6510 Add prediction percent values and predicted interval seconds

(cherry picked from commit e60015d)

* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.

(cherry picked from commit 6fae058)

* NIFI-6510 Added tests for engine

(cherry picked from commit 6d7a13b)

* NIFI-6150 Added tests for connection status analytics class, corrected variable names

(cherry picked from commit 58c7c81)

* NIFI-6150 Make checkstyle happy

(cherry picked from commit b6e35ac)

* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing

* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics

* NIFI-6510 Adjusted interval and incorporated R-squared check

Updates to support multiple variables for features, clearing cached regression model based on r-squared values

Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).

Ratcheck fixes

Added test for SimpleRegression. Minor fix for OLS model

fixed test errors

fixed checkstyle errors

(cherry picked from commit fab411b)

* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)

* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties

NIFI-6566 - changes to allow scoring configurations for model in nifi.properties

NIFI-6566 - added default implementation value to NiFiProperties

NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3663

* NIFI-6585 - Refactored tests to use mocked models and extract functions.  Added check in ConnectionStatusAnalytics to confirm expected model by type

* NIFI-6586 - documentation and comments

This closes NIFI-6586

Signed-off-by: Andrew I. Christianson <andy@andyic.org>

* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns

* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.

* NIFI-6510 - add an enable/disable property for analytics

* NIFI-6510 - documentation updates for enable/disable property

* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)

* NIFI-6510 - admin guide updates to further describe model functionality

* NIFI-6510 - code quality fixes (if statement and constructor)

* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold

* NIFI-6510 Extract out predictions into their own DTO

* NIFI-6510 Optimize imports

* NIFI-6510 Fix formatting

* NIFI-6510 Optimize imports

* NIFI-6510 Optimize imports

* NIFI-6510 - Notice updates for Commons math and Caffeine

* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)

* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3705

* NIFI-6510 Rip out useless members

* NIFI-6510 - dto updates to check for -1 value

* NIFI-6510 - checkstyle fix

* NIFI-6510 - rolled back last change and applied minNonNegative method

* NIFI-6510 Rip out useless members
2019-09-09 15:37:11 +00:00
Andy LoPresto
e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
simonl
0650521eb2
NIFI-6171 always send email scope for OIDC
NIFI-6171 re-added lookupEmail() as fallback
NIFI-6171 additional OIDC scopes via nifi.properties
NIFI-6171 alternative user identification (instead of email)  via nifi.properties
NIFI-6171 changed lookupEmail() so that any configured claim can be fetched fro the UserInfo endpoint

This closes #3398
This closes #2346
2019-04-26 16:40:37 -04:00
Jan Hentschel
60b00a1e38 NIFI-2738 Removed references to nifi.cluster.request.replication.claim.timeout
This closes #2860

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-21 06:34:27 -05:00
joewitt
0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt
45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Bryan Bende
fdd8cdbb31 NIFI-5673 Set up property/assembly for new auto-load directory
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface

This closes #3119.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-11-01 14:41:10 -04:00
Jeff Storck
c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00