Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.
Signed-off-by: Joe Witt <joe.witt@gmail.com>
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes#4263.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4132.
NIFI-6363 Additional fixes.
NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions.
NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.
NIFI-6363 - Added some docs and experimental tag to the relevant classes.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
This closes#4080.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.
Signed-off-by: Joe Witt <joewitt@apache.org>
Updated pom files regarding groovy dependencies. groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter
This closes#3547
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.
This closes#3102.
Added HostHeaderHandler regression unit tests before adding new functionality.
Added logic for parsing nifi.web.proxy.host property.
Added default property in nifi.properties and pom.xml.
Added logic for IPv6 and custom default hostnames.
Improved error messaging.
Added HostHeaderHandler unit tests.
Disabled HostHeaderSanitizationCustomizer in HTTP mode.
Fixed HTML escaping in error message.
Improved error message.
Added failing unit test for parsing custom hostnames.
Fixed custom hostname parsing.
Fixed unit tests.
Added TODO for IPv6 custom hostname parsing and unit test.
Added IPv6 custom hostname parsing and unit tests.
Fixed checkstyle issues.
Removed empty element in host list when no value defined.
Improved error message formatting.
Added unit tests.
Removed HostHeaderSanitizationCustomizer.
Removed InvalidPropertiesFormatException from NiFiProperties.
Removed InvalidPropertiesFormatException from HostHeaderHandler.
This closes#2415
Adds support for NiFI Registry config files to the encrypt-config tool
in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
This closes#2376.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.
This closes#1686.
Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate
- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.
Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up
- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.
Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services
- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.
Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework
- Finishing clean up following rebase.
Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests
- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.
Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle
- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.
Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled
- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.
Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory
- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.
Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times
Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.
Improving error messages when looking for bundles
Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.
Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.
Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0
Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster
NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.
Fixing checkstyles issue and adding back assume checks to distributed cache server test
Ensuring new component types are sorted correctly when shown initially.
This closes#1585.
Joe Witt
Andy LoPresto
Troy Melhase
mtien
Jan Hentschel
Bryan Bende
Jeff Storck
Matt Gilman
Kevin Doran