Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.
Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3968.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.
NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify
NIFI-6380: Added Description to Parameter Context
NIFI-6380: Fixed checkstyle violations
NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged
NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.
NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context
NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO
NIFI-6380: Added additional integration tests around escaped parameter references
NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression
NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update
NIFI-6380: Addressed review feedback
NIFI-6380: Addressed additional review feedback, mostly around code cleanup
NIFI-6380: Bug fix
NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing
NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource
NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior
NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context
NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version
NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update
NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service
NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task
NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.
NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.
NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context
This closes#3536
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.
This closes#2703
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes#2468
* Added low water mark for purgeOldEvents() to more quickly purge after a large spike in events.
* Adjusted rollover high water mark to avoid overrunning "nifi.provenance.repository.max.storage.size".
* Adjusted looping logic in mergeJournals() to use ".firstKey()" instead of ".entrySet().iterator().next()" to avoid unnecessary object creation.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1240
Joe Witt
Andy LoPresto
Mark Payne
Jeff Storck
Bryan Rosander
Denes Arvay
Matt Gilman
Mark Bean
Joe Skora
Pierre Villard
zhengsg