* NIFI-9082 Added nifi.zookeeper.jute.maxbuffer property
- Updated ZooKeeperStateProvider to check jute.maxbuffer instead of hard-coded number
- Updated internal ZooKeeperClientConfig class to include new juteMaxBuffer property
* NIFI-9082 Updated jute.maxbuffer documentation and added hexadecimal representation
- Renamed DefaultWatcher to NoOpWatcher
- Removed unnecessary validateDataSize() from createNode()
- Replaced per-user symmetric-key HS256 with shared and rotated RSA asymmetric-key RS512 implementation
- Added nifi.security.user.jws.key.rotation.period property for RSA Key Pair rotation
- Added JSON Web Tokens section to Administration Guide
- Implemented persistent storage of RSA Public Keys for verification using Local State Manager
- Implemented JWT revocation on logout with persistence using Local State Manager
- Refactored JWT implementation using Spring Security OAuth2 and Nimbus JWT
- Refactored Spring Security Provider configuration using Java instead of XML
- Removed H2 storage of per-user keys
- Upgraded nimbus-jose-jwt from 7.9 to 9.11.2
NIFI-8766 Corrected AuthenticationException handling in AccessResource.getAccessStatus
- Added nifi.user.security.jws.key.rotation.period to default nifi.properties
- Updated logging statements and clarified configuration and method documentation
NIFI-8766 Changed Algorithm to PS512 and updated documentation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5262.
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5215.
- KeyStoreKeyProvider supports PKCS12 and BCFKS
- Refactored KeyProvider and implementations to nifi-security-kms
- Updated Admin Guide and User Guide with KeyStoreKeyProvider details
NIFI-8511 Improved documentation and streamlined several methods
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#5110.
- include new process group property support in NiFi Registry
- updated documentation to describe and show new feature
- added elements to XSD schema definition
NIFI-8195: update to DAO to fix PG move and copy/paste
update condition to not null vice null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5192
NIFIDEVS-8195: fixed properties not properly inheriting from template/snippet values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5094
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
* NIFI-8137 Record Path EscapeJson/UnescapeJson functions
* Correct jackson-databind dependency version
* Add negative tests for RecordPath JSON handling; rename RecordPath JSON classes to better match existing functions
Signed-off-by: Otto Fowler <ottobackwards@gmail.com>
This closes#4756.
- Retained legacy default Sensitive Properties Key in ConfigEncryptionTool to support migration
- Streamlined default file path and moved key generation conditional
- Refactored with getDefaultProperties()
- Cleared System Property in ConfigEncryptionToolTest
- Added checking and error handling for clustered status
- Added set-sensitive-properties-key command
- Refactored PropertyEncryptor classes to nifi-property-encryptor
- Added nifi-flow-encryptor
- Refactored ConfigEncryptionTool to use FlowEncryptor for supporting AEAD algorithms
- Added Admin Guide section Updating the Sensitive Properties Key
This closes#4857.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds
This closes#4972
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor
NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor
NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor
NIFI-7668 Added support for bcrypt secure hashing algorithm
NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4809.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings, classname for ocnnection factory, adjusted TLS configuration checks in NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code review.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4753.
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page
- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests
- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm
- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client
- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding
- Made atomic replace methods for storing groups and saml credential in database
- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata
- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias
- Fixed unit test
- Added property to specify an optional identity attribute which would be used instead of NameID
- Cleaned up logging
- Fallback to keystore password when key password is blank
- Make signature and digest default to SHA-256 when no value provided in nifi.properties
This closes#4614
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.
This closes#4613.
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-7941 Add NiFi Registry mode options and examples to Encrypt-Config section of Toolkit Guide
* Update toolkit-guide.adoc
Replaced references to "master key" with "root key"
* Update toolkit-guide.adoc
Corrected a typo. Anchor was "sensistive_property_key_migration". Changed to "sensitive_property_key_migration". Confirmed nothing was referencing the original.
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests
NIFI-7401 Added keystore types and changed properties to match nifi.security.*
NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test
NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties
NIFI-7401 Changed connection socket constants to use .class.getName()
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4592.
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
* NIFI-7681 - Add update-bucket-policy command, add option to specify
timeout and fix documentation to include previously implemented commands
* Fix return type of UpdateBucketPolicy and add missing registry commands to the documentation
Co-authored-by: Jaya Aditya <jchandra@yahoo-corp.jp>
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
Updated documentation.
Added unit tests.
NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.
NIFI-7669 Fixed failing test due to nifi.properties initialization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4435.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.
This closes#4421.
Co-authored-by: mtien <mtien.apache@gmail.com>
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4344.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY
MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY
The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
fixed mispositioned # in doc
This closes#3740
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY
The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
This closes#3739
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY
The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
NIFI-6672 Removed test code.
This closes#3738
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-6255 NIFI-6287: Rebased to match the new expression language interface
NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports
NIFI-6255 NIFI-6287: Move to the common codec DigetUtils
Update commons-codec
This closes#3624
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
* NIFI-7319 Added first draft of walkthroughs doc.
* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.
* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.
* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.
* NIFI-7319 Added instructions and screenshots for building NiFi from source.
* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.
* NIFI-7319 Completed secure cluster walkthrough.
* NIFI-7319 Added walkthroughs to documentation navigation list.
* NIFI-7319 Incorporated PR feedback on broken links.
* NIFI-7319 Removed line number helpers from update sections.
* NIFI-7319 Incorporated final PR review items.
Co-authored-by: Sandra Pius <spiusapache@gmail.com>
- Refactored Flow Synchronization to make code cleaner
- Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
- Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging
NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior
NIFI-6849: Addressed review feedback
NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup
This closes#3891
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.
This closes#4125.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6363 Additional fixes.
NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions.
NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.
NIFI-6363 - Added some docs and experimental tag to the relevant classes.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4080.
* NIFI-7053 Update Toolkit Guide with macOS 10.15 requirements for trusted certificates
* Simplified note about trusted certs in macOS 10.15
Signed-off-by: Andy LoPresto <alopresto@apache.org>
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.
This closes#3894.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
added a function in Expression Language to
Evaluate Variable value containing expression Language
NIFI-6792
Build related changes
added license to newly added class StringElEvaluator.java
restyled the classes having maven-checkstyle related issues
NIFI-6792
made changes suggested by reviewer MikeThomsen
Expression Language function name changed from stringEL to evaluateELString
This closes#3829
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-1474 Added unit test for constructor p boundaries. Added p boundary conditions.
NIFI-1474 Updated documentation regarding p boundary.
NIFI-1474 Added r validation and unit tests. Added Javadoc for new methods.
This closes#3844.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.
Signed-off-by: Joe Witt <joewitt@apache.org>
Update the links to the quartz documentation in the User Guide section
for cron trigger scheduling.
This closes#3802.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.
This closes#3715.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6649 - documentation update
NIFI-6649 - add debug logging for score and prediction information
NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3719
* NIFI-6510 Implement initial analytic engine
* NIFI-6510 Implemented basic linear regression model for queue counts
* NIFI-6510 Initial analytics REST endpoint and supporting objects
* NIFI-6510 Connect the dots for StatusAnalytics -> API
* NIFI-6510 Added poc engine with prediction model caching
(cherry picked from commit e013b91)
DFA-9 - updated logging and corrected logic for checking if not in backpressure
(cherry picked from commit a1f8e70)
* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics
(cherry picked from commit 050e0fc)
(cherry picked from commit 9fd365f)
* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id
(cherry picked from commit 14854ff)
DFA-9 - reduced snapshot interval to 1 minute
(cherry picked from commit 36abb0a)
* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions
* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly
* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"
This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.
* NIFI-6510 Added prediction fields for use by UI, still need to be populated
* NIFI-6510 Analytics Framework Introduction (#10)
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.
DFA-9 - added logging
* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval
* DFA-9 - checkstyle fixes
* NIFI-6510 Add prediction percent values and predicted interval seconds
(cherry picked from commit e60015d)
* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.
(cherry picked from commit 6fae058)
* NIFI-6510 Added tests for engine
(cherry picked from commit 6d7a13b)
* NIFI-6150 Added tests for connection status analytics class, corrected variable names
(cherry picked from commit 58c7c81)
* NIFI-6150 Make checkstyle happy
(cherry picked from commit b6e35ac)
* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing
* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics
* NIFI-6510 Adjusted interval and incorporated R-squared check
Updates to support multiple variables for features, clearing cached regression model based on r-squared values
Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).
Ratcheck fixes
Added test for SimpleRegression. Minor fix for OLS model
fixed test errors
fixed checkstyle errors
(cherry picked from commit fab411b)
* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)
* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties
NIFI-6566 - changes to allow scoring configurations for model in nifi.properties
NIFI-6566 - added default implementation value to NiFiProperties
NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3663
* NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type
* NIFI-6586 - documentation and comments
This closes NIFI-6586
Signed-off-by: Andrew I. Christianson <andy@andyic.org>
* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.
* NIFI-6510 - add an enable/disable property for analytics
* NIFI-6510 - documentation updates for enable/disable property
* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)
* NIFI-6510 - admin guide updates to further describe model functionality
* NIFI-6510 - code quality fixes (if statement and constructor)
* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold
* NIFI-6510 Extract out predictions into their own DTO
* NIFI-6510 Optimize imports
* NIFI-6510 Fix formatting
* NIFI-6510 Optimize imports
* NIFI-6510 Optimize imports
* NIFI-6510 - Notice updates for Commons math and Caffeine
* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)
* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3705
* NIFI-6510 Rip out useless members
* NIFI-6510 - dto updates to check for -1 value
* NIFI-6510 - checkstyle fix
* NIFI-6510 - rolled back last change and applied minNonNegative method
* NIFI-6510 Rip out useless members
NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init)
NIFI-6546 - Refactored test and added log error for JsonPath exception based on review
NIFI-6546 - Refactored tests based on review comments
NIFI-6546 - Removed redundant phone check and added constant for empty path
NIFI-6546 - Added brackets based on review comments
This closes#3646
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
Expression Language now supports padding functions:
- padLeft:
- attr:padLeft(Int n, Char c) will prepend to the `attr` attributes the `c` character until the size `n` is reached
- attr:padLeft(Int n) will prepend to the `attr` attributes the `'_'` character until the size `n` is reached
- padRight:
- attr:padRight(Int n, Char c) will append to the `attr` attributes the `c` character until the size `n` is reached
- attr:padRight(Int n) will append to the `attr` attributes the `'_'` character until the size `n` is reached
- In both cases, the padding function returns the `attr` `String` as is if its length is already equal of higher than the desired size `n`
- Returns null if `attr` does not exist or in case desiredLenght is higher than Integer.MAX_INT
Further test cases:
- Returns null if the input string is null
- Returns a string full of padding if the input string is empty
Supports PaddingString instead of PaddingCharacter
Apply suggestions from code review
Applying style suggestions
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
style fixes
style fixes
Padding returns input string instead of null in case desired length is missing, is negative, or is overflowing
Better tests
doc update
less verbose parser notation
Doc and style fixes
Fixed `StringEvaluator.evaluate()` after rebase
Applying nitpicking suggestion
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Fixed `PaddingEvaluator` constructor issue
Removed unused import
This closes#3615
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- PadLeft(label, desiredLength, paddingChar) prepends the paddingChar (or the default value '_' if a paddingChar is not provided) to the label string until desiredLength is reached.
- PadRight(label, desiredLength, paddingChar) appends the paddingChar (or the default value '_' if a paddingChar is not provided) to the label string until desiredLength is reached.
Added Apache license disclaimers
checkstyle fixes
Replaced functional interface with abstract method
Fixes and Further test cases:
- Returns null if the input string is null
- Returns a string full of padding if the input string is empty
wip support padding string
In order to be consistent with the feature introduced in #3615, RecordPath padLeft and padRight supports String padding.
Since nifi-record-path doesn't have the Apache Commons StringUtils dependency, the padding methods have been added to the available NiFi commons StringUtils class.
NIFI-6502 Updated top level NOTICE file to include citation for code borrowed from commons-lang3.
borrowed pad methods from lang3 StringUtils
Replaced `PadLeft` and `PadRight` record path functions with borrowed Apache Lang `StringUtils` padding methods and updated `nifi-assembly/NOTICE` accordingly
This closes#3613
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.
This closes#3657.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4775: Rolling back fix for NIFI-6416
NIFI-4775: Documentation improvements
+1 from markobean
This closes#3648
Signed-off-by: Brandon <devriesb@apache.org>
NIFI-5973 More comments and better defaults for the shell provider.
NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided.
NIFI-5973 Fixed a formatting string in the OS X shell commands.
Updated testing conditions to run IT in OS X environment.
Changed unit test to provide identity rather than identifier.
This closes#3537.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
NIFI-6304 Updated code based on code review.
NIFI-6304 Updated documentation.
NIFI-6304 Refactored to make it simpler
NIFI-6304 Reverted Concat to its last state.
This closes#3478.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.
This closes#3351.
Signed-off-by: Mark Payne <markap14@hotmail.com>
There are two superscript references in the JsonPath documentation that don't seem to mean any thing. They are not clickable and there are no other similar reference markers in the document.
This closes#2907
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
This closes#3464
Corrects small typo's in ReplaceAll entry of Expression Language Guide.
Removes an extraneous instance of 'Arguments:' and adds a missing 'T' to
the properly spell the word 'The'.
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6171 re-added lookupEmail() as fallback
NIFI-6171 additional OIDC scopes via nifi.properties
NIFI-6171 alternative user identification (instead of email) via nifi.properties
NIFI-6171 changed lookupEmail() so that any configured claim can be fetched fro the UserInfo endpoint
This closes#3398
This closes#2346
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.
This closes#3312.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
Clear up the documentation around the PriorityAttributePrioritizer to make it clearer that if the attributes are both number the largest wins but if they are not numbers they sort the other way around.
I had to go and look at the code to work out what was going on after reading the existing documentation.
Simplify PriorityAttributePrioritzer docs
Simplify the main section of documentation for PriorityAttributePrioritzer and then add a notes section with more details about the ordering.
Fix example.
Put back comment setting the priority attribute
Further improvements to the Priority ordering
This closes#3205.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.
This closes#3102.
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback
This closes#2947
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.
Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads
Refactored what API exposes
The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.
Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.
Refactor RPG same as ControllerService.
WIP incorporating review comments.
Incorporated review comments
- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit
Restrict enabling ControllerService without read permission
Revert the last commit.
Fix review comments.
- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition
Fixed delete action display condition.
Fixed NPE at Summary.
Apply operation policy to activateControllerServices.
Removed OperationPermissible from ComponentEntity.
This closes#2990
Moved utility code to TlsHelper.
Added unit tests.
Added command-line parsing for additional CA certificate path.
Added documentation on using the TLS Toolkit to generate and sign certificates using an externally-signed CA.
Updated toolkit external CA documentation to be inline with additional context from NIFI-5473.
Cleaned up toolkit documentation.
Improved error message by changing to absolute path.
Added Javadoc to and removed unthrown exception declarations from TlsHelper#verifyCertificateSignature().
Cleaned up unit tests with utility method.
Fixed checkstyle error.
Support conversion of a PKCS#8 formatted private key automatically to avoid forcing the user to do that. Also add some log messages for debugging when the parser fails to parse the appropriate object
Incorporated Peter's contribution for PKCS #8 to PKCS #1 conversion.
Added documentation and refactored methods.
Refactored unit test.
Added RAT exclusion for test resource.
This closes#2935.
Co-authored-by: pepov <peterwilcsinszky@gmail.com>
Signed-off-by: Matt Gilman <mcgilman@apache.org>
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.
This closes#2703
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly
This closes#2509.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added logic to check for simultaneous configuration of HTTP and HTTPS connectors in JettyServer.
- Added test logging resources. Added unit tests.
- Refactored shared functionality to generic method which accepts lambdas.
Fixed unit test with logging side effects.
- Added note about exclusive HTTP/HTTPS behavior to Admin Guide. Fixed typos.
This closes#2683.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
nifi.database.directory should be moved out of default location <root-level-nifi-dir>/database_repository to ease the update process.
Signed-off-by: joewitt <joewitt@apache.org>
Adds authorizers.xml to the files understood by the encrypt-config
tool in the NiFi Toolkit. If enabled, then the sensitive properties
for LdapUserGroupProvider in authorizers.xml will be encrypted.
Also fixes a bug wherein encrypt-config replaces multiple XML nodes
in login-indentity-providers.xml when LdapProvider is not the first
provider listed in the file.
Enable properties in authorizers.xml to be encrypted by the master key.
This closes#2350.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
Corrects time unit conversion for the Sync Interval config property
for LdapUserGroupProvider in authorizers.xml.
Also enforces a minimum value of 10 secs for the Sync Interval to help
catch unintentional misconfigurations, for example users upgrading
from previous versions, where tiny Sync Interval values could be set
as a workaround for NIFI-4667.
This closes#2341
- Adding new properties to allow the referenced attribute of a user/group to be configurable when detecting group membership.
- Expanding on documentation regarding the new properties.
This closes#2274.
Signed-off-by: Bryan Bende <bbende@apache.org>
theBestAndrew
exceptionfactory
Lehel Boér
Emilio Setiadarma
Joe Gresock
Joe Witt
Mark Bean
Matthew Burgess
Bence Simon
Mathieu Rossignol
Andrew Lim
Peter Turcsanyi
Paul Grey
Chris Sampson
Sushil Kumar
sjyang18
Nathan Gough
Joey Frazee
Mark Payne
Tamas Palfy
Peter Gyori
Troy Melhase
Nissim Shiman
exceptionfactory
zhangcheng
Mike Thomsen
Bryan Bende
Dustin Rodrigues
abrown
Pierre Villard
VKadam
Mohammed Nadeem
Andy LoPresto
Jaya Aditya
Ruben Laguna
Tamás Bunth
Andy LoPresto
mtien
Alessandro D'Armiento
Phillip Grenier
Matt Gilman
Joey Frazee
Alex Voitau
Tamas Palfy
Manoj Mamidyala
Bahlul Haider
mans2singh
Yolanda M. Davis
Andy I. Christianson
Alessandro D'Armiento
Brandon Devries
Mike Thomsen
Koji Kawamura
archon
Wil Selwood
Mark Owens
Endre Zoltan Kovacs
simonl
Lars Francke
Peter Turcsanyi
Jan Hentschel
Denes Arvay
Aleksandr Salatich
Peter Wicks
Jeff Storck
Nick Lewis
郑书国10209198
Otto Fowler
zenfenan
Mike Moser
jugi92
Willie Engelbrecht
Leah Anderson
Marco Gaido
Kevin Doran
edwarzjl