NIFI-8171 Increased response and idle timeouts for HTTP unit tests
NIFI-8171 Increased TestServer idle timeout to 45 seconds for HTTP unit tests
NIFI-8171 Adjusted timeout and sleep on TestPutTCPCommon.testPruneSenders
NIFI-8171 Increased TestServer idle timeout to 60 seconds and removed 500ms Thread.sleep() in TestInvokeHttpSSL
NIFI-8171 Optimized OkHttpClientUtils to avoid reading trust store twice during initialization
NIFI-8171 Added static variable for server startup sleep
NIFI-8171 Increased TestInvokeHTTP Connect Timeout and TestListenHTTP Response Timeout to 30 seconds
NIFI-8171 Refactored unit tests for InvokeHTTP and ListenHTTP to optimize SSLContext creation
NIFI-8171 Updated TestListenHTTP for static creation of SSLContext
NIFI-8171 Added started check for ListenHTTP Server in TestListenHTTP
NIFI-8171 Refactored TestPutTCP classes to optimize SSLContext creation
NIFI-8171 Increased TestListenHTTP timeout for server start to 120 seconds and added exception when not connected
NIFI-8171 Increased Connect and Read Timeouts for InvokeHTTP SSL unit tests
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings, classname for ocnnection factory, adjusted TLS configuration checks in NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code review.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4753.
- Added SSLContextService.createContext() and refactored referencing components
- Removed references to ClientAuth from SslContextFactory methods
This closes#4737.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Changed to check the length of all unfiltered properties instead of only filtered properties.
Added additional check if descriptor is a dynamic property.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4707
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup
NIFI-7897: Addressing review feedback
NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly
This closes#4669.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page
- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests
- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm
- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client
- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding
- Made atomic replace methods for storing groups and saml credential in database
- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata
- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias
- Fixed unit test
- Added property to specify an optional identity attribute which would be used instead of NameID
- Cleaned up logging
- Fallback to keystore password when key password is blank
- Make signature and digest default to SHA-256 when no value provided in nifi.properties
This closes#4614
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.
This closes#4613.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission
This closes#4642.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added a 'dependent' attribute to determine whether or not to save dependent property values
Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
Bumped icu4j dependency to 60.2.
Replaced jackson-mapper-asl dependency with jackson-databind.
Fixed an error comparing key identities in TestKeyService.
Replaced jackson-mapper-asl ObjectMapper with jackson-databind ObjectMapper in LivySessionController.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4640.
Added method to validate the OIDC Access Token for the revoke endpoint.
Created a new callback URI of oidc/logoutCallback to handle certain OIDC logout cases.
Changed method to exchange the Authorization Code for a Login Authentication Token.
Added a new method to exchange the AuthN Code for an Access Token.
Changed method to convert OIDC Token to a Login AuthN Token instead of a NiFi JWT.
Created new OidcServiceGroovyTest class.
NIFI-7584-rebase Added test.
NIFI-7584 Fixed a checkstyle issue.
NIFI-7584 Removed a dependency not in use.
NIFI-7584 Made revisions based on PR review.
Refactored revoke endpoint POST request to a private method.
Removed unnecessary dependencies.
Fixed Regex Pattern to search for literal dot character.
Fixed logging the Exception message.
Fixed caught Exception.
Changed timeout value to a static variable.
Changed repeating error messages to a static string.
Reduced sleep duration in unit test.
Refactored cookie generation to private method.
NIFI-7584 Fixed the snapshot version.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4593.
Use an AspectJ aspect and agent to intercept the load native library calls
at runtime, copy the native library file to temp folder and proceed with
the newly created file in order to provide classloader isolation.
Remove AspectJ jars from lib directory, move the necessary jar to lib/aspectj subdirectory.
This closes#4540.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-7549 Refining documentation; Changing explicit HA mode; Smaller review comments
NIFI-7549 Code review responses about license, documentation and dependencies
NIFI-7549 Fixing issue when explicit HA; Some further review based adjustments
NIFI-7549 Response to code review comments
NIFI-7549 Adding extra serialization test
NIFI-7549 Minor changes based on review comments
NIFI-7549 Adding hook point to the shutdown
This closes#4510.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests
NIFI-7401 Added keystore types and changed properties to match nifi.security.*
NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test
NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties
NIFI-7401 Changed connection socket constants to use .class.getName()
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4592.
* NIFI-7592: Allow NiFi to be started without a GUI/REST interface
* NIFI-7592: Enable all controller services when starting headless
* NIFI-7592: Marked duplicate dependencies as provided
* NIFI-7592: Incorporated additional review comments
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
NIFI-7663 Minor changes (variable name refactor, javadoc, GUI message). Merging Drop All Flowfiles responses across all nodes in a cluster.
NIFI-7663 Reloading the canvas after completing a Drop All Flowfiles request.
NIFI-7663 Fixed typos.
This closes#4425.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Updated documentation.
Added unit tests.
NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.
NIFI-7669 Fixed failing test due to nifi.properties initialization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4435.
Added unit tests.
NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.
NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4436
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4431.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.
This closes#4416.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.
This closes#4421.
Co-authored-by: mtien <mtien.apache@gmail.com>
Added exception mapper.
NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4418.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.
NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4344.
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.
This closes#4322
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#4271.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.
This closes#4289.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4306.
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes#4263.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4274
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.
This closes#4192.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.
This closes#4222.
Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
* NIFI-7319 Added first draft of walkthroughs doc.
* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.
* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.
* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.
* NIFI-7319 Added instructions and screenshots for building NiFi from source.
* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.
* NIFI-7319 Completed secure cluster walkthrough.
* NIFI-7319 Added walkthroughs to documentation navigation list.
* NIFI-7319 Incorporated PR feedback on broken links.
* NIFI-7319 Removed line number helpers from update sections.
* NIFI-7319 Incorporated final PR review items.
Co-authored-by: Sandra Pius <spiusapache@gmail.com>
- Refactored Flow Synchronization to make code cleaner
- Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
- Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging
NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior
NIFI-6849: Addressed review feedback
NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup
This closes#3891
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.
This closes#4125.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback
This closes#4149.
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback
This closes#4154.
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE
This closes#4086
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4132.
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6363 Additional fixes.
NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions.
NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.
NIFI-6363 - Added some docs and experimental tag to the relevant classes.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4080.
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
- added replace process group functionality in ProcessGroupResource
- parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
- refactored flow update methods to make use of commonality across different update types whenever possible
- fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
- improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import
This closes#4023.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info
NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance
This closes#4003.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
This closes#3823.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.
This closes#3894.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful
NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication
This closes#3931
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
* Added additional unit tests and updated ParameterEvaluator to make sure that it follows the same logic as the ParameterExpression in that sensitive parameters cannot be access from within EL
This closes#3889
Signed-off-by: Rob Fellows <rob.fellows@gmail.com>
NIFI-6707: Added System Test to verify behavior when services depend on one another
NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them
NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group
NIFI-6707: Updated to include java11 subdirectory under lib/
NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT
NIFI-6707: Updates to see if it will help travis build correctly
NIFI-6707: Commenting out java commands in bootstrap.conf files
This closes#3831.
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils
Added QueryNiFiReportingTask to SQL query NiFi status and metrics
Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS
Add CONNECTION_STATUS_PREDICTIONS
check for null predictions
Add ConnectionStatusRecursiveIterator
Fix issue w/ duplicate iterator outputs
Refactored query interfaces, fixed assembly POM
Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors
Add DatabaseRecordSink service (#13)
* Add DatabaseRecordSink service
* Incorporated review comments
* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions
* Added bundle profile, remove predictions table if not enabled
* Added doc for which tables are available when
This closes#3826.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes
This closes#3795.
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-6760: When writing/reading the length of a DataFrame, do so using a 4-byte integer instead of a 2-byte short. When using uncompressed data, we know that the length of the DataFrame will be no more than 64 KB so a 2-byte short is sufficient. However, if data is compresed, there's a chance that the compressed form of the data will be larger than the uncompressed form (for example, with random binary data or with encrypted data). In this situation, we can end up overflowing the 2-byte short, which causes the length that is written to be wrong. Using a 4-byte integer avoids this situation.
* NIFI-6760: Fixed unit tests
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.
This closes#3805.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes#3804.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.
This closes#3798.
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.
This closes#3677.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback
Signed-off-by: Joe Witt <joewitt@apache.org>
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.
This closes#3715.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ
Signed-off-by: Joe Witt <joewitt@apache.org>
Tamas Palfy
Joe Witt
Pierre Villard
Nathan Gough
Bryan Bende
Peter Gyori
Mark Payne
Mark Bean
Jon Kessler
exceptionfactory
Troy Melhase
Andrew Lim
s9514171
Nissim Shiman
tlsmith
Matthew Burgess
sjyang18
Shane Ardell
mtien
humpfhumpf
Kevin Doran
Bence Simon
Peter Turcsanyi
Joey Frazee
dependabot[bot]
Andy LoPresto
M Tien
Andy LoPresto
Mohammed Nadeem
Kotaro Terada
jmconte
Kristjan Antunovic
si
Scott Aslan
shreeju
Karthikeyan Singaravelan
Jan Hentschel
Vasily Makarov
thenatog
Matt Gilman
KovalevIV
Matthieu Cauffiez
Yolanda M. Davis
Sushil Kumar
Rob Fellows
Michael Hogue
MatthewKnight-NG
Joe Ferner
nagasivanath
Shawn Weeks
Tamas Palfy
mdayakar
Iván Rodriguez
Kotaro Terada
Ivan Ezequiel Rodriguez
samhjelmfelt
Endre Zoltan Kovacs
Alessandro D'Armiento
Koji Kawamura