Commit Graph

57 Commits

Author SHA1 Message Date
Peter Turcsanyi 08dcd4af14
NIFI-7345: Fixed Hive database and table names case insensitivity in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4198.
2020-04-09 14:13:54 +02:00
Peter Turcsanyi dfaef38805
NIFI-7248: Atlas reporting task handles PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4138.
2020-03-12 10:44:48 +01:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Peter Turcsanyi ce050f4ecb
NIFI-7009: Atlas reporting task retrieves only the active flow components
Filter out the deleted components before querying them, instead of retrieving
all the components before filtering.

This closes #3979
2020-01-16 11:31:18 -05:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Peter Turcsanyi b8ffb54612
NIFI-6945: Use minExtInfo=true to reduce the amount of data queried from Atlas
This closes #3970
2020-01-10 09:03:07 -05:00
Peter Turcsanyi cc74534bc0
NIFI-6939: Upgrade Atlas client dependency to 2.0.0
NIFI-6939: Review changes

This closes #3944
2020-01-08 13:25:45 -05:00
Tamas Palfy 67fca6832a NIFI-6910 - ReportLineageToAtlas - Added processor properties for connect timeout and read timeout.
NIFI-6910 - ReportLineageToAtlas - Capitalized Atlas connect and read timeout property display names.

This closes #3936.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-12-16 20:55:13 +01:00
Tamas Palfy 599fb98415 NIFI-6937 - NotificationSender uses typedQualifiedName instead of simple qualifiedName as keys in local maps.
NIFI-6937 - Fix NotificationSender: typedQualifiedName handling.

This closes #3929.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-12-16 17:33:47 +01:00
Peter Turcsanyi ba6d050ba8
NIFI-6926: Fixed memory leak in NiFiAtlasHook
NIFI-6926: Use new instance of list instead of clearing it
NIFI-6926: Logging the number of messages to be sent to Atlas.
NIFI-6926: Pass a copy of the messages list to send() and clear the original list.

This closes #3915
2019-12-09 16:31:33 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Joe Witt 0f02de6002
NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This closes #3791.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-10-07 22:20:48 -04:00
Jeff Storck 82a6c430a4
NIFI-5373 Created new module nifi-metrics
In nifi-metrics, added build-helper-maven-plugin to enable groovy tests to run, since there are no java tests or src/test/java path in the module
Replaced yammer/dropwizard's VirtualMachineMetrics usage in multiple modules with JvmMetrics interface and JmxJvmMetrics implementation in nifi-metrics
Updated nifi-datadog-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-scripting-bundle to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTask to use JmxJvmMetrics instead of VirtualMachineMetrics
Updated nifi-reporting-utils pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-ambari-reporting-task pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated SiteToSiteMetricsReportingTask to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTaskGroovyTest to use regular strings instead of GStrings due to class cast exception
Updated nifi-prometheus-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated PrometheusMetricsUtil to use LF endings instead of CRLF
Updated nifi-metrics-reporting-bundle and its submodules to use nifi-metrics:1.10.0-SNAPSHOT
nifi-metrics-reporting-task metrics-graphite version upgraded from 3.1.2 to 4.1.0
Removed unused property yammer.metrics.version from nifi-jolt-record-bundle pom
Updated root NOTICE to include derived work in nifi-metrics
Removed duplicate nifi-assembly NOTICE clauses for yammer/dropwizard metrics
Removed unnecessary inclusion of Yammer Metrics in Kafka 0-9, 0-10, 0-11, 1-0, and 2-0 NARs' NOTICE files since the Kafka dependency that transitively includes it is test-scoped.
Removed unnecessary mock interaction in GraphiteMetricReporterServiceTest, Mockito 2.28.2 operates in strict mode and flags unused unused interactions.

NIFI-6354: Added LICENSE and NOTICE for nifi-kafka-2-0-nar
Updated NOTICE in kafka nar modules to reflect the actual module names at the top of the NOTICE
Added yammer metrics NOTICE clauses to nifi-kafka-0-11-nar
Added yammer metrics NOTICE clauses to nifi-kafka-1-0-nar

NIFI-6355: Extended NOTICE clauses for yammer/dropwizard metrics in multiple modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3520
2019-06-19 15:50:42 -04:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
pushpavanthar 98ca28d50d
NIFI-6083 - Improving documentation for Kafka Service Name property
This closes #3365.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-12 12:29:22 -04:00
joewitt 25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay 76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
thenatog 5eb5e96b16
NIFI-5665 - Changed netty versions to more closely match the original netty dependency version.
NIFI-5665 - Fixed version for nifi-spark-bundle.
NIFI-5665 - Fixing copy and paste error.

This closes #3067
2018-10-12 15:56:52 -04:00
joewitt 77edddd988
NIFI-5666 Updated all usages of Spring, beanutils, collections to move beyond deps with cves
This closes #3052
2018-10-09 14:14:48 -04:00
joewitt 8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
Jan Hentschel c2657ee043 NIFI-5423 Removed duplicated dependencies in sub-modules
This closes #2887

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-15 11:01:51 -04:00
Koji Kawamura e94f0757db
NIFI-4654: Support reporting RAS S2S lineage to Atlas
- Added 's2s.port.id' FlowFile attribute to track target remote Port id
- Use 's2s.port.id' to analyze RAW S2S provenance events
- This closes #2863
2018-07-10 11:44:00 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Koji Kawamura 79a7014a95 NIFI-4971: ReportLineageToAtlas complete path can miss one-time lineages
- Separated Hook message de-duplication logic from NiFiAtlasHook to
NotificationSender
- NiFiAtlasHook used to send individual CREATE_ENTITY messages for each entities,
this commit changed it to bundle all new entities into a single
CREATE_ENTITY to preserve entity creation order, so that new DataSet
entities can be referred from new nifi_flow_path entities
- Added more unit tests

This closes #2542

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-15 05:50:26 -04:00
Koji Kawamura ff04f8efee NIFI-4980: Typo in ReportAtlasLineage kafka kerberos service name property
This closes #2550

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-07 06:08:59 -04:00
joewitt 59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Pierre Villard 4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Mark Payne 641ba29cc8 NIFI-4993: Accept null user at querying ProvenanceRepository
Pass null user from ReportLineageToAtlas.

Address NullPointer dereferences

This closes #2589.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-28 12:15:05 -04:00
joewitt 7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Mark Payne 0b0aebe148
NIFI-4917: Externalize Keytab and Principal configuration from Processors to a Controller Service. This gives us the ability to allow users to interact with those Keytabs/Principals to which they've been given access without allowing them access to all Keytabs and Principals
- Addressed review feedback; rebased against master

This closes #2552.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-21 14:23:41 -04:00
Koji Kawamura 2590c88318 NIFI-4979 - Fix ReportLineageToAtlas documentation errors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2549.
2018-03-19 17:56:13 +01:00
Koji Kawamura ada069ce42 NIFI-4978: Fixed ReportLineageToAtlas NPE when unscheduled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2548.
2018-03-16 10:19:59 +01:00
Koji Kawamura d3f54994a6 NIFI-4972 - SelectHiveQL to emit FETCH provenance event
SelectHiveQL should emit FETCH instead of CONTENT_MODIFIED when it has
incoming connections.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2543.
2018-03-14 10:17:19 +01:00
Ed c2616e6fe7
NIFI-4833 Add ScanHBase Processor
- New processor for scanning HBase records based on verious params like range of rowkeys, range of timestamps. Supports result limit and reverse scan.
- Adds Atlas Support for ScanHBase processor
- Fixed not recent version of FF
- Formatting and Style changes
- Single line to multiline if-then statements
- Removed HTML formatting that is not used for doc generation
- Fixed issue with limitRows
- Fixed issue with filter expression
- Refactored "processRows"
- Fixed possible NPE for bulkSize var
- Changed provenance to "receive" to indicate new data from external source.
- Updated min/max timestamp custom validation
- JSON array support
- Removed in-memory caching for records. Now records are being written directly to FF
- Removed unfinished flowfile from session, transfered original to "failure". Test cases update as well

This closes #2478.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-13 10:07:07 -04:00
joewitt c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
2018-03-09 16:34:53 -05:00
Koji Kawamura f16cbd462b
NIFI-4818: Fix transit URL parsing at Hive2JDBC and KafkaTopic for ReportLineageToAtlas
- Hive2JDBC: Handle connection parameters and multiple host entries
correctly
- KafkaTopic: Handle multiple host entries correctly
- Avoid potential "IllegalStateException: Duplicate key" exception
when NiFiAtlasHook analyzes existing NiFiFlowPath input/output entries
- This closes #2435
2018-02-07 08:49:35 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00