Commit Graph

140 Commits

Author SHA1 Message Date
Tamas Palfy 2baafcc2e6
NIFI-6534 Improve logging in PutHive3Streaming processor
Added logging for PutHive3Streaming when routing to failure

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3640
2019-08-09 17:19:26 -04:00
Jeff Storck 4783b12a9c
NIFI-5254 Upgraded Groovy to version 2.5.4
Updated pom files regarding groovy dependencies.  groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter

This closes #3547

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-11 14:17:35 -04:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Jeff Storck 65c41ab917 NIFI-6196: Upgrade Jetty version to 9.4.15.v20190215
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore.  Also updated the tests to separately test one-way and two-way SSL

NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.

This closes #3426.
2019-05-09 12:05:12 -04:00
Matthew Burgess 97690275ac
NIFI-6210: Applied NIFI-5134 Kerberos TGT renewal to Hive3ConnectionPool
This closes #3432.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-12 15:09:52 -04:00
Joe Witt 1bdab73201 NIFI-6121 Updated the easy dependencies to update
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-22 09:01:21 -04:00
joewitt 25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay 76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Aleksandr Salatich e598b30d6d NIFI-6000 Catch also IllegalArgumentException in ConvertAvroToORC hive processor. Added support for Avro null types.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3302
2019-02-14 15:08:54 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Jeff Storck e4be06c7c8 NIFI-5967: Removed unused ALLOW_EXPLICIT_KEYTAB private variable
Updated Hive_1_1ConnectionPoolTest.testKerberosAuthException to use a mocked KerberosCredentialsService

This closes #3272
2019-02-08 00:22:56 -05:00
Matthew Burgess cdf3c69208 NIFI-5967: Add Hive 1.1 processors
Removed extra Kerberos properties, added LICENSEs to all Hive NARs, removed unnecessary NOTICE entries
2019-02-08 00:11:38 -05:00
Koji Kawamura 05de73d6a0 NIFI-5951 Fix error logging with rollback on failure
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3264
2019-01-17 17:11:39 -05:00
Alex Savitsky 3e52ae952d NIFI-5909 added optional settings for date, time, and timestamp formats used to write Records to Elasticsearch
NIFI-5909 added content checks to the unit tests

NIFI-5937 use explicit long value for test dates/times (to not depend on the timezone of test executor)

NIFI-5937 tabs to spaces

Fixing checkstyle violations introduced by https://github.com/apache/nifi/pull/3249 PR)

NIFI-5937 adjusted property descriptions for consistency; limited EL scope to variable registry; added an appropriate validator along with its Maven dependency; moved format initialization to @OnScheduled

NIFI-5909 tabs to spaces

Signed-off-by: Ed <edward.berezitsky@gmail.com>

This closes #3227
2019-01-09 21:55:09 -05:00
kei miyauchi e6e4175d71 NIFI-5841 Fix memory leak of PutHive3Streaming.
This closes #3249.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-01-09 12:06:27 +09:00
Koji Kawamura 595a2decc6
NIFI-5917 Fix TestSelectHiveQL.testNoTimeLimit
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3237.
2019-01-03 11:31:38 +01:00
Matthew Burgess d34789881b NIFI-5904: Fix PutHive3Streaming handling of RecordReaderFactoryException 2018-12-17 14:12:18 -05:00
gkkorir c51512f5e3 NIFI-5891 fix handling of null logical types in Hive3Streaming processor
NIFI-5891: Fixed Checkstyle issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3216
2018-12-13 10:23:18 -05:00
Matthew Burgess 68a49cfad0 NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors
NIFI-5845: Incorporated review comments

This closes #3184.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-29 09:50:21 +09:00
Matthew Burgess 455e3c1bc8 NIFI-5834: Restore default PutHiveQL error handling behavior
NIFI-5834: Incorporated review comments

This closes #3179.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-27 18:10:26 +09:00
yjhyjhyjh0 75906226a6
NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
Signed-off-by: Peter Wicks <patricker@gmail.com>

This closes #3156.
2018-11-15 13:18:31 -07:00
Pierre Villard 9e7610ac70 NIFI-5815 - PutORC processor 'Restricted' still requires access to restricted components regardless of restriction
This closes #3169.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-14 13:50:00 +09:00
Jeff Storck 4f14e517db Merge branch 'NIFI-5720-RC3' as part of the NiFi 1.8.0 release process 2018-10-26 13:56:23 -04:00
Pierre Villard 2201f7746f NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
add @Ignore on unit test...

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3086
2018-10-24 13:33:30 -04:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Matthew Burgess c752ec8394
NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs
This closes #3076
2018-10-17 12:24:32 -04:00
Matthew Burgess ce25ae5419
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name

NIFI-5667: Fixed handling of binary types

NIFI-5667: Added backticks in Hive DDL generation

This closes #3057.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-15 10:10:47 -04:00
thenatog 5eb5e96b16
NIFI-5665 - Changed netty versions to more closely match the original netty dependency version.
NIFI-5665 - Fixed version for nifi-spark-bundle.
NIFI-5665 - Fixing copy and paste error.

This closes #3067
2018-10-12 15:56:52 -04:00
Kotaro Terada fc5c8baeb3 NIFI-5676: Fix a timezone-dependent test in PutORCTest
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3059.
2018-10-10 09:38:06 +02:00
joewitt 8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
Matthew Burgess cfc858c901 NIFI-5517 - Add support for remaining Hive types to PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2950.
2018-08-24 11:09:05 +02:00
zenfenan 82c5126f3c NIFI-5494 - Added ExpressionLanguage support to 'Hive Configuration Resources'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2946.
2018-08-22 13:04:21 +02:00
Jan Hentschel c2657ee043 NIFI-5423 Removed duplicated dependencies in sub-modules
This closes #2887

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-15 11:01:51 -04:00
Matthew Burgess 9ee2316ff6 NIFI-5491: Fixed PutHive3Streaming handling of Byte, Short, and Struct
This closes #2938.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-08-06 14:33:49 -04:00
Matthew Burgess 3d546b8d87
NIFI-5484: Fixed PutHive3Streaming to use the Hive Metastore URI property (to include multiple URIs)
NIFI-5484: Incorporated review comments, added unit test for new validator

This closes #2934
2018-08-03 10:36:56 -04:00
Matthew Burgess b4894c5572
NIFI-5475: Upgraded Hive 3 bundle to Apache Hive 3.1.0 2018-08-01 13:42:31 -04:00
Matthew Burgess 921403ff55 NIFI-5336: Exclude hbase-server from Hive 3 bundle
This closes #2811

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-26 05:33:07 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Matthew Burgess 97f71fd6c7 NIFI-5044: Applied changes in SelectHiveQL to SelectHive3QL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2799.
2018-06-15 22:42:59 +02:00
Ed B 187417d077 NIFI-5044 SelectHiveQL accept only one statement
SelectHiveQL support only single SELECT statement.
This change adds support for pre- and post- select statements.
It will be useful for configuration queries, i.e. "set tez.queue.name=default", and others.

renamed selectQuery to hqlStatement for better readability

style check correction in comment

removed meaningless examples for post-queries

execute query without result set

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2695
2018-06-15 15:48:16 -04:00
amitt90 8370a56dce NIFI-5309 update the logger message to get the output format from the outputFormat variable
This closes #2793

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-14 07:09:23 -04:00
Matthew Burgess da99f873a7
NIFI-4963: Added Hive3 bundle
- Incorporated review comments
- Added more defensive code for PutHive3Streaming error handling

This closes #2755.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-06-13 14:32:58 -04:00
Matthew Burgess 099bfcdf3a NIFI-5121: Added DBCPService API method for passing in flow file attributes when available
This closes #2658

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 08:19:16 -04:00
Jeff Storck 3ad3243511 NIFI-5134 Explicitly requesting UGI to relogin before attempting to get a DB connection in HiveConnectionPool
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2667
2018-05-03 09:12:07 -04:00
Koji Kawamura c575a98936 NIFI-5095: Suppress SET property parse failure at Hive processors
Log debug message when ParseException is thrown.

Log warning message if other unknown Exception is thrown.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2645
2018-04-20 11:24:02 -04:00