Commit Graph

138 Commits

Author SHA1 Message Date
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Jeff Storck 65c41ab917 NIFI-6196: Upgrade Jetty version to 9.4.15.v20190215
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore.  Also updated the tests to separately test one-way and two-way SSL

NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.

This closes #3426.
2019-05-09 12:05:12 -04:00
Matthew Burgess 97690275ac
NIFI-6210: Applied NIFI-5134 Kerberos TGT renewal to Hive3ConnectionPool
This closes #3432.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-12 15:09:52 -04:00
Joe Witt 1bdab73201 NIFI-6121 Updated the easy dependencies to update
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-22 09:01:21 -04:00
joewitt 25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay 76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Aleksandr Salatich e598b30d6d NIFI-6000 Catch also IllegalArgumentException in ConvertAvroToORC hive processor. Added support for Avro null types.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3302
2019-02-14 15:08:54 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Jeff Storck e4be06c7c8 NIFI-5967: Removed unused ALLOW_EXPLICIT_KEYTAB private variable
Updated Hive_1_1ConnectionPoolTest.testKerberosAuthException to use a mocked KerberosCredentialsService

This closes #3272
2019-02-08 00:22:56 -05:00
Matthew Burgess cdf3c69208 NIFI-5967: Add Hive 1.1 processors
Removed extra Kerberos properties, added LICENSEs to all Hive NARs, removed unnecessary NOTICE entries
2019-02-08 00:11:38 -05:00
Koji Kawamura 05de73d6a0 NIFI-5951 Fix error logging with rollback on failure
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3264
2019-01-17 17:11:39 -05:00
Alex Savitsky 3e52ae952d NIFI-5909 added optional settings for date, time, and timestamp formats used to write Records to Elasticsearch
NIFI-5909 added content checks to the unit tests

NIFI-5937 use explicit long value for test dates/times (to not depend on the timezone of test executor)

NIFI-5937 tabs to spaces

Fixing checkstyle violations introduced by https://github.com/apache/nifi/pull/3249 PR)

NIFI-5937 adjusted property descriptions for consistency; limited EL scope to variable registry; added an appropriate validator along with its Maven dependency; moved format initialization to @OnScheduled

NIFI-5909 tabs to spaces

Signed-off-by: Ed <edward.berezitsky@gmail.com>

This closes #3227
2019-01-09 21:55:09 -05:00
kei miyauchi e6e4175d71 NIFI-5841 Fix memory leak of PutHive3Streaming.
This closes #3249.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-01-09 12:06:27 +09:00
Koji Kawamura 595a2decc6
NIFI-5917 Fix TestSelectHiveQL.testNoTimeLimit
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3237.
2019-01-03 11:31:38 +01:00
Matthew Burgess d34789881b NIFI-5904: Fix PutHive3Streaming handling of RecordReaderFactoryException 2018-12-17 14:12:18 -05:00
gkkorir c51512f5e3 NIFI-5891 fix handling of null logical types in Hive3Streaming processor
NIFI-5891: Fixed Checkstyle issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3216
2018-12-13 10:23:18 -05:00
Matthew Burgess 68a49cfad0 NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors
NIFI-5845: Incorporated review comments

This closes #3184.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-29 09:50:21 +09:00
Matthew Burgess 455e3c1bc8 NIFI-5834: Restore default PutHiveQL error handling behavior
NIFI-5834: Incorporated review comments

This closes #3179.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-27 18:10:26 +09:00
yjhyjhyjh0 75906226a6
NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
Signed-off-by: Peter Wicks <patricker@gmail.com>

This closes #3156.
2018-11-15 13:18:31 -07:00
Pierre Villard 9e7610ac70 NIFI-5815 - PutORC processor 'Restricted' still requires access to restricted components regardless of restriction
This closes #3169.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-14 13:50:00 +09:00
Jeff Storck 4f14e517db Merge branch 'NIFI-5720-RC3' as part of the NiFi 1.8.0 release process 2018-10-26 13:56:23 -04:00
Pierre Villard 2201f7746f NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authentication issue/misleading
add @Ignore on unit test...

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3086
2018-10-24 13:33:30 -04:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Matthew Burgess c752ec8394
NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs
This closes #3076
2018-10-17 12:24:32 -04:00
Matthew Burgess ce25ae5419
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name

NIFI-5667: Fixed handling of binary types

NIFI-5667: Added backticks in Hive DDL generation

This closes #3057.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-15 10:10:47 -04:00
thenatog 5eb5e96b16
NIFI-5665 - Changed netty versions to more closely match the original netty dependency version.
NIFI-5665 - Fixed version for nifi-spark-bundle.
NIFI-5665 - Fixing copy and paste error.

This closes #3067
2018-10-12 15:56:52 -04:00
Kotaro Terada fc5c8baeb3 NIFI-5676: Fix a timezone-dependent test in PutORCTest
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3059.
2018-10-10 09:38:06 +02:00
joewitt 8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
Matthew Burgess cfc858c901 NIFI-5517 - Add support for remaining Hive types to PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2950.
2018-08-24 11:09:05 +02:00
zenfenan 82c5126f3c NIFI-5494 - Added ExpressionLanguage support to 'Hive Configuration Resources'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2946.
2018-08-22 13:04:21 +02:00
Jan Hentschel c2657ee043 NIFI-5423 Removed duplicated dependencies in sub-modules
This closes #2887

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-15 11:01:51 -04:00
Matthew Burgess 9ee2316ff6 NIFI-5491: Fixed PutHive3Streaming handling of Byte, Short, and Struct
This closes #2938.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-08-06 14:33:49 -04:00
Matthew Burgess 3d546b8d87
NIFI-5484: Fixed PutHive3Streaming to use the Hive Metastore URI property (to include multiple URIs)
NIFI-5484: Incorporated review comments, added unit test for new validator

This closes #2934
2018-08-03 10:36:56 -04:00
Matthew Burgess b4894c5572
NIFI-5475: Upgraded Hive 3 bundle to Apache Hive 3.1.0 2018-08-01 13:42:31 -04:00
Matthew Burgess 921403ff55 NIFI-5336: Exclude hbase-server from Hive 3 bundle
This closes #2811

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-26 05:33:07 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Matthew Burgess 97f71fd6c7 NIFI-5044: Applied changes in SelectHiveQL to SelectHive3QL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2799.
2018-06-15 22:42:59 +02:00
Ed B 187417d077 NIFI-5044 SelectHiveQL accept only one statement
SelectHiveQL support only single SELECT statement.
This change adds support for pre- and post- select statements.
It will be useful for configuration queries, i.e. "set tez.queue.name=default", and others.

renamed selectQuery to hqlStatement for better readability

style check correction in comment

removed meaningless examples for post-queries

execute query without result set

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2695
2018-06-15 15:48:16 -04:00
amitt90 8370a56dce NIFI-5309 update the logger message to get the output format from the outputFormat variable
This closes #2793

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-14 07:09:23 -04:00
Matthew Burgess da99f873a7
NIFI-4963: Added Hive3 bundle
- Incorporated review comments
- Added more defensive code for PutHive3Streaming error handling

This closes #2755.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-06-13 14:32:58 -04:00
Matthew Burgess 099bfcdf3a NIFI-5121: Added DBCPService API method for passing in flow file attributes when available
This closes #2658

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 08:19:16 -04:00
Jeff Storck 3ad3243511 NIFI-5134 Explicitly requesting UGI to relogin before attempting to get a DB connection in HiveConnectionPool
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2667
2018-05-03 09:12:07 -04:00
Koji Kawamura c575a98936 NIFI-5095: Suppress SET property parse failure at Hive processors
Log debug message when ParseException is thrown.

Log warning message if other unknown Exception is thrown.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2645
2018-04-20 11:24:02 -04:00
Matthew Burgess 090e748726 NIFI-4647: Fix support for strings in unions for ConvertAvroToORC
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2644.
2018-04-19 09:49:32 +02:00
Matthew Burgess ae1d3e3945 NIFI-4862: Keep incoming flowfile attributes in outgoing flowfiles from SelectHiveQL
This closes #2605.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-04-09 12:31:05 +09:00