nifi/nifi-nar-bundles/nifi-kafka-bundle/pom.xml

59 lines
2.5 KiB
XML
Raw Normal View History

2014-12-08 15:22:14 -05:00
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<modelVersion>4.0.0</modelVersion>
2014-12-08 15:22:14 -05:00
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-nar-bundles</artifactId>
<version>1.2.0-SNAPSHOT</version>
2014-12-08 15:22:14 -05:00
</parent>
<artifactId>nifi-kafka-bundle</artifactId>
2014-12-08 15:22:14 -05:00
<packaging>pom</packaging>
<properties>
<kafka8.version>0.8.2.2</kafka8.version>
<kafka9.version>0.9.0.1</kafka9.version>
NIFI-3189: ConsumeKafka 0.9 and 0.10 with downstream backpressure Currently, NiFi Kafka consumer processors have following issue. While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger. It stopps executing poll to tell Kafka server that this client is alive. Thus, after a while in that situation, Kafka server rebalances the client. When downstream connections back to normal, although ConsumeKafka is scheduled again, the client is no longer a part of a consumer group. If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset. Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated. In order to address above issue: - For ConsumeKafka_0_10, use latest client library Above issue has been addressed by KIP-62. The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages. - For ConsumeKafka (0.9), added manual retention logic using pause/resume Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually. Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive. Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed. This closes #1527. Signed-off-by: Bryan Bende <bbende@apache.org>
2017-02-27 22:57:04 -05:00
<kafka10.version>0.10.2.0</kafka10.version>
</properties>
2014-12-08 15:22:14 -05:00
<modules>
<module>nifi-kafka-0-8-processors</module>
<module>nifi-kafka-0-9-processors</module>
<module>nifi-kafka-0-10-processors</module>
<module>nifi-kafka-0-8-nar</module>
<module>nifi-kafka-0-9-nar</module>
<module>nifi-kafka-0-10-nar</module>
2014-12-08 15:22:14 -05:00
</modules>
<dependencyManagement>
2015-04-25 09:20:35 -04:00
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-kafka-0-8-processors</artifactId>
<version>1.2.0-SNAPSHOT</version>
2015-04-25 09:20:35 -04:00
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-kafka-0-10-processors</artifactId>
<version>1.2.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-kafka-0-9-processors</artifactId>
<version>1.2.0-SNAPSHOT</version>
</dependency>
2015-04-25 09:20:35 -04:00
</dependencies>
</dependencyManagement>
2014-12-08 15:22:14 -05:00
</project>