2014-12-08 15:22:14 -05:00
|
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
|
|
<!--
|
|
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
|
|
this work for additional information regarding copyright ownership.
|
|
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
(the "License"); you may not use this file except in compliance with
|
|
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
See the License for the specific language governing permissions and
|
|
|
|
limitations under the License.
|
|
|
|
-->
|
2015-02-16 23:55:36 -05:00
|
|
|
<modelVersion>4.0.0</modelVersion>
|
2014-12-08 15:22:14 -05:00
|
|
|
<parent>
|
|
|
|
<groupId>org.apache.nifi</groupId>
|
2015-01-21 01:47:48 -05:00
|
|
|
<artifactId>nifi-nar-bundles</artifactId>
|
2016-11-25 23:49:27 -05:00
|
|
|
<version>1.2.0-SNAPSHOT</version>
|
2014-12-08 15:22:14 -05:00
|
|
|
</parent>
|
2015-01-21 01:47:48 -05:00
|
|
|
<artifactId>nifi-kafka-bundle</artifactId>
|
2014-12-08 15:22:14 -05:00
|
|
|
<packaging>pom</packaging>
|
2016-09-07 02:09:36 -04:00
|
|
|
|
|
|
|
<properties>
|
|
|
|
<kafka8.version>0.8.2.2</kafka8.version>
|
|
|
|
<kafka9.version>0.9.0.1</kafka9.version>
|
NIFI-3189: ConsumeKafka 0.9 and 0.10 with downstream backpressure
Currently, NiFi Kafka consumer processors have following issue.
While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger.
It stopps executing poll to tell Kafka server that this client is alive.
Thus, after a while in that situation, Kafka server rebalances the client.
When downstream connections back to normal, although ConsumeKafka is scheduled again,
the client is no longer a part of a consumer group.
If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset.
Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated.
In order to address above issue:
- For ConsumeKafka_0_10, use latest client library
Above issue has been addressed by KIP-62.
The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages.
- For ConsumeKafka (0.9), added manual retention logic using pause/resume
Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually.
Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive.
Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed.
This closes #1527.
Signed-off-by: Bryan Bende <bbende@apache.org>
2017-02-27 22:57:04 -05:00
|
|
|
<kafka10.version>0.10.2.0</kafka10.version>
|
2016-09-07 02:09:36 -04:00
|
|
|
</properties>
|
|
|
|
|
2014-12-08 15:22:14 -05:00
|
|
|
<modules>
|
2016-08-23 18:12:59 -04:00
|
|
|
<module>nifi-kafka-0-8-processors</module>
|
|
|
|
<module>nifi-kafka-0-9-processors</module>
|
|
|
|
<module>nifi-kafka-0-10-processors</module>
|
|
|
|
<module>nifi-kafka-0-8-nar</module>
|
|
|
|
<module>nifi-kafka-0-9-nar</module>
|
|
|
|
<module>nifi-kafka-0-10-nar</module>
|
2014-12-08 15:22:14 -05:00
|
|
|
</modules>
|
2015-01-21 01:47:48 -05:00
|
|
|
<dependencyManagement>
|
2015-04-25 09:20:35 -04:00
|
|
|
<dependencies>
|
|
|
|
<dependency>
|
|
|
|
<groupId>org.apache.nifi</groupId>
|
2016-08-23 18:12:59 -04:00
|
|
|
<artifactId>nifi-kafka-0-8-processors</artifactId>
|
2016-11-25 23:49:27 -05:00
|
|
|
<version>1.2.0-SNAPSHOT</version>
|
2015-04-25 09:20:35 -04:00
|
|
|
</dependency>
|
2016-04-07 07:15:25 -04:00
|
|
|
<dependency>
|
|
|
|
<groupId>org.apache.nifi</groupId>
|
2016-08-23 18:12:59 -04:00
|
|
|
<artifactId>nifi-kafka-0-10-processors</artifactId>
|
2016-11-25 23:49:27 -05:00
|
|
|
<version>1.2.0-SNAPSHOT</version>
|
2016-08-23 18:12:59 -04:00
|
|
|
</dependency>
|
|
|
|
<dependency>
|
|
|
|
<groupId>org.apache.nifi</groupId>
|
|
|
|
<artifactId>nifi-kafka-0-9-processors</artifactId>
|
2016-11-25 23:49:27 -05:00
|
|
|
<version>1.2.0-SNAPSHOT</version>
|
2016-04-07 07:15:25 -04:00
|
|
|
</dependency>
|
2015-04-25 09:20:35 -04:00
|
|
|
</dependencies>
|
2017-03-01 09:43:32 -05:00
|
|
|
</dependencyManagement>
|
2014-12-08 15:22:14 -05:00
|
|
|
</project>
|