- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE
This closes#4086
Signed-off-by: Mark Payne <markap14@hotmail.com>
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation. If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC
Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components. An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).
Addressing PR feedback
Addressing PR feedback
This closes#4102.
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest
Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations
This closes#4095.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.
This closes#3894.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6896 Adding hadoop-aws, hadoop-azure, and hadoop-azure-datalake to nifi-hadoop-libraries-nar and nifi-ranger-nar so other filesystem impls will be available
NIFI-6896 Introduce profiles to optionally include additional hadoop modules to reduce default build size
NIFI-6896 Updating licenses and notices to account for new optional dependencies
NIFI-6896 Add profile for hadoop-cloud-storage
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3900
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
the same field where some (but not all) were in a wider-than-the-other
relationship.
Before: Some datatypes could be lost. String was wider than any other.
After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.
This closes#3724.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ
Signed-off-by: Joe Witt <joewitt@apache.org>
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build. Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.
NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service
NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11
This closes#3404.
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations
This closes#3630.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
CSVReader and CSVRecordSetWriter controller services and also ConvertExcelToCSVProcessor
support EL for Value Separator, Quote Character and Escape Character properties.
NIFI-6318: Fixed null checks and compound OR expression.
NIFI-6318: RecordSetWriterFactory.createWriter() changes.
NIFI-6318: Initialize CSVFormat in onEnabled() if there are no dynamic formatting properties.
NIFI-6318: Comment Marker supports EL.
NIFI-6318: Various review changes.
This closes#3504.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Updated pom files regarding groovy dependencies. groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter
This closes#3547
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
In nifi-metrics, added build-helper-maven-plugin to enable groovy tests to run, since there are no java tests or src/test/java path in the module
Replaced yammer/dropwizard's VirtualMachineMetrics usage in multiple modules with JvmMetrics interface and JmxJvmMetrics implementation in nifi-metrics
Updated nifi-datadog-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-scripting-bundle to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTask to use JmxJvmMetrics instead of VirtualMachineMetrics
Updated nifi-reporting-utils pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-ambari-reporting-task pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated SiteToSiteMetricsReportingTask to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTaskGroovyTest to use regular strings instead of GStrings due to class cast exception
Updated nifi-prometheus-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated PrometheusMetricsUtil to use LF endings instead of CRLF
Updated nifi-metrics-reporting-bundle and its submodules to use nifi-metrics:1.10.0-SNAPSHOT
nifi-metrics-reporting-task metrics-graphite version upgraded from 3.1.2 to 4.1.0
Removed unused property yammer.metrics.version from nifi-jolt-record-bundle pom
Updated root NOTICE to include derived work in nifi-metrics
Removed duplicate nifi-assembly NOTICE clauses for yammer/dropwizard metrics
Removed unnecessary inclusion of Yammer Metrics in Kafka 0-9, 0-10, 0-11, 1-0, and 2-0 NARs' NOTICE files since the Kafka dependency that transitively includes it is test-scoped.
Removed unnecessary mock interaction in GraphiteMetricReporterServiceTest, Mockito 2.28.2 operates in strict mode and flags unused unused interactions.
NIFI-6354: Added LICENSE and NOTICE for nifi-kafka-2-0-nar
Updated NOTICE in kafka nar modules to reflect the actual module names at the top of the NOTICE
Added yammer metrics NOTICE clauses to nifi-kafka-0-11-nar
Added yammer metrics NOTICE clauses to nifi-kafka-1-0-nar
NIFI-6355: Extended NOTICE clauses for yammer/dropwizard metrics in multiple modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3520
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
Bence Simon
Yolanda M. Davis
Matthew Burgess
Alexandre Vautier
Bryan Bende
Pierre Villard
Joe Witt
jstorck
Tamas Palfy
KovalevIV
Koji Kawamura
Atul Mahajan
Jan Hentschel
archon
Mark Payne
Peter Turcsanyi
mike
Andy LoPresto
Aaron Leon