- Moved StringUtils from nifi-properties to nifi-property-utils
- Moved Peer Identity methods from CertificateUtils to specific Site-to-Site classes
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#6977.
- Replaced Groovy asserts with JUnit 5 assertions and Groovy shouldFail method Junit 5 with assertThrow method
This closes#6880
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Updated impacted classes to remove redundant import lines
- Removed WebUtilsGroovyTest.groovy class due to use of internal sun.security classes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#6804.
- Refactored XML parsing to use providers from nifi-xml-processing
- Configured spotbugs-maven-plugin with findsecbugs-plugin in nifi-xml-processing
- Disabled Validate DTD in default configuration for EvaluateXPath and EvaluateXQuery
- Replaced configuration of DocumentBuilder and streaming XML Readers with shared components
- Removed XML utilities from nifi-security-utils
- Moved Commons Configuration classes to nifi-lookup-services
This closes#5962
Signed-off-by: Paul Grey <greyp@apache.org>
- Added Ubuntu Zulu JDK 17 GitHub build
- Adjusted MiNiFi C2 FileSystemConfigurationCache test to avoid using environment variables
- Adjusted MiNiFi StatusLogger and StatusLoggerTest to avoid overriding private logger
- Adjusted failure reason attribute check in TestGetIgniteCache
- Adjusted TestRangerAuthorizer and TestRangerNiFiAuthorizer to avoid checking nested exceptions
- Adjusted encrypt-config TestUtil to avoid unnecessary comparison of different types
- Disabled Javascript tests on Java 15 and higher
- Disabled several Hive 3 tests on Java 17 for StringInternUtils illegal access
- Refactored nifi-enrich-processors tests to use Mockito without Powermock
- Refactored nifi-toolkit-tls tests to avoid illegal reflective access
- Removed deprecated X509Certificate test in CertificateUtilsTest
- Removed kryo serialization from nifi-site-to-site-client test
- Updated TestHashContent to use SHA-1 instead of SHA for hash algorithm
- Upgraded maven-war-plugin from 2.5 to 3.3.2
- Upgraded nifi-graph-bundle dependencies from Groovy 2.5.14 to 3.0.8
- Upgraded QuestDB from 4.2.1 to 6.2.1 in nifi-framework-core
This closes#5870
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
- Refactor background transaction extension to ExtendTransactionCommand
- Avoid closing S2S HTTP client for IllegalStateExceptions
- Avoid creating additional S2S HTTP client instance for transaction extension commands
- Add check for extend transaction requests received in client test class
- Add null check for Peer Persistence implementation in PeerSelector
Signed-off-by: Joe Gresock <jgresock@gmail.com>
This closes#5577.
- Added TemporaryKeyStoreBuilder with File.deleteOnExit() for KeyStore and TrustStore files
- Removed JKS files from nifi-security-utils tests
- Refactored usage of KeyStoreUtils.createKeyStoreAndGetX509Certificate() to TemporaryKeyStoreBuilder
- Removed unnecesary hadoop-minikdc test dependency in security-utils
- Replaced Mini KDC Hex utility with Bouncy Castle Hex utility in unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5406
- Refactored multiple tests using KeyStoreUtils
- Removed static KeyStore and TrustStore files
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5401
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.
This closes#4289.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
This closes#4263.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.
This closes#3677.
Signed-off-by: Bryan Bende <bbende@apache.org>
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.
This closes#3018
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.
This closes#2841.
Signed-off-by: Joe Witt <joewitt@apache.org>
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added XmlUtils class.
- Added unit test.
- Added XXE test resource.
- Refactored JAXB unmarshalling globally to prevent XXE attacks.
- Refactored duplicated/legacy code.
- Cleaned up commented code.
- Switched from FileInputStream back to StreamSource in AuthorizerFactoryBean.
- This closes#2134
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.
Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.
Signed-off-by: joewitt <joewitt@apache.org>
- Added batchCount, batchSize, batchDuration to limit flow files to be
included in a single Site-to-Site transaction.
- Added batch throttling logic when StandardRemoteGroupPort transfers
flow files to a remote input port using the batch limit configurations,
so that users can limit batch not only for pulling data, but also pushing data.
- Added destination list shuffle to provide better load distribution.
Previously, the load distribution algorithm produced the same host consecutively.
- Added new batch settings to FlowConfiguration.xsd.
- Added new batch settings to Flow Fingerprint.
- Added new batch settings to Audit.
- Sort ports by name at 'Remote Process Group Ports' dialog.
- Show 'No value set' when a batch configuration is not set
- Updated batch settings tooltip to clearly explain how it works the configuration works differently for input and output ports.
- Updated DTO by separating batch settings to BatchSettingsDTO to indicate count, size and duration are a set of configurations.
- This closes#1306
- Fixed SiteInfoProvider and HttpClient to use specified local address with its SiteToSiteRestApiClient
- Removed setupRequestConfig method call from connection and read timeout setter methods at SiteToSiteRestApiClient, because it created config object before local address was set
- Null clear StandardRemoteProcessGroup localAddress when user clears Local Network Interface
Signed-off-by: joewitt <joewitt@apache.org>
- Removed host and port field from Peer since the same information is
available in PeerDescription
- Refactored variable names in SocketRemoteSiteListener to improve readability
- Changed how SocketRemoteSiteListener constructs PeerDescription
instance. It used to use hard-coded 'localhost' as hostname, and
getPort() which returns server's port. Since the peer is a remote peer,
i.e the client, it should be client hostname and port.
- Added hostname resolution at DataTransferResource to make s2s.host
value consistent with RAW transport. Without this, RAW uses hostname
while HTTP uses IP address. It will be hard to be used from downstream flows.
- Replaced heavy use of mockito which was difficult to maintain, with
nifi-mock
- Added SiteToSiteAttributes and more assertions in unit tests
This closes#1342.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added urls in addition to the existing url, to support multiple target
URLs
- Backward compatibility is provided by returning the first url if
multipe urls are specified, but component accessing the url doesn't
support multiple urls
- UI is not fully updated yet. Following UI components are planned to be updated
by different commits
- Search component: only the first URL is searchable and shown
- Component status: RPG status shows only the first URL
- Component action history: only the first URL is searchable and shown
- Updated Search component to use URLs.
This closes#1208.
- Changed AtomicBoolean to CountDownLatch to avoid sleeping thread in
some test cases
- Specified less number of threads for Jetty and LittleProxy than
default to lower resource usage
- Added try catch for the specific gateway timeout case (504) so that
test can pass even it happens while it fails with other errors
- Consolidated the target cluster URL resolving logic into
SiteToSiteRestApiClient's as a common method
- Changed to more descriptive error message
- Added more unit test cases
This commit fixes following two issues, that happens when a Root Group Port
policy for S2S data transfer is removed at a remote NiFi, after a client NiFi has
connected to that port:
1. At client side, Remote Process Group should show that authorization
is failing on its bulletin, but the Exception is caught and
ignored. Nothing is shown on the UI with HTTP transport protocol.
RAW S2S shows error on RPG bulletin. This commit fixes HTTP S2S to
behave the same.
2. At server side, corresponding input-port or output-port should show
that it is accessed by an unauthorized client on its bulletin, but it's
not shown with HTTP transport protocol.
RAW S2S shows warning messages for this. This commit fixes HTTP S2S to
behave the same.
In order to fix the 2nd issue above, request authorization at
DataTransferResource is changed from using DataTransferAuthorizable
directly, to call RootGroupPort.checkUserAuthorization().
Because the blettin is tied to the Port instance and it's
difficult to produce blettin message from this resource.
Since RootGroupPort.checkUserAuthorization uses
DataTransferAuthorizable inside, the check logic stays the same as
before.
Adding a RootGroupPortAuthorizable to provide access to necessary components for performing the authorization.
This closes#996
exceptionfactory
dan-s1
Mike Thomsen
Paul Grey
Mark Payne
exceptionfactory
s9514171
Andy LoPresto
Kotaro Terada
Andy LoPresto
Koji Kawamura
Arpad Boda
Jeff Storck
Matt Gilman
Joey Frazee
joewitt
Pierre Villard
Bryan Rosander