druid/extensions-core
David Palmer 2855fb6ff8
Change Kafka Lookup Extractor to not register consumer group (#12842)
* change kafka lookups module to not commit offsets

The current behaviour of the Kafka lookup extractor is to not commit
offsets by assigning a unique ID to the consumer group and setting
auto.offset.reset to earliest. This does the job but also pollutes the
Kafka broker with a bunch of "ghost" consumer groups that will never again be
used.

To fix this, we now set enable.auto.commit to false, which prevents the
ghost consumer groups being created in the first place.

* update docs to include new enable.auto.commit setting behaviour

* update kafka-lookup-extractor documentation

Provide some additional detail on functionality and configuration.
Hopefully this will make it clearer how the extractor works for
developers who aren't so familiar with Kafka.

* add comments better explaining the logic of the code

* add spelling exceptions for kafka lookup docs
2022-08-09 16:14:22 +05:30
..
avro-extensions Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
azure-extensions Able to filter Cloud objects with glob notation. (#12659) 2022-06-24 11:40:08 +05:30
datasketches Adding withName implementation to AggregatorFactory (#12862) 2022-08-08 18:31:56 +05:30
druid-aws-rds-extensions Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
druid-basic-security Improve build performance of modules (#12486) 2022-05-01 22:43:11 +08:00
druid-bloom-filter Adding withName implementation to AggregatorFactory (#12862) 2022-08-08 18:31:56 +05:30
druid-kerberos Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
druid-pac4j Mark specific nimbus.lang.tag.version. (#12751) 2022-07-07 09:58:35 +05:30
druid-ranger-security Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
ec2-extensions Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
google-extensions Able to filter Cloud objects with glob notation. (#12659) 2022-06-24 11:40:08 +05:30
hdfs-storage Add authentication call before cleaning up intermediate files in hadoop ingestions (#12030) 2022-05-02 08:40:44 -05:00
histogram Adding withName implementation to AggregatorFactory (#12862) 2022-08-08 18:31:56 +05:30
kafka-extraction-namespace Change Kafka Lookup Extractor to not register consumer group (#12842) 2022-08-09 16:14:22 +05:30
kafka-indexing-service Kinesis ingestion with empty shards (#12792) 2022-08-05 22:38:58 +05:30
kinesis-indexing-service Kinesis ingestion with empty shards (#12792) 2022-08-05 22:38:58 +05:30
kubernetes-extensions Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
lookups-cached-global Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
lookups-cached-single Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
mysql-metadata-storage deps: upgrade mysql-connector-java to v5.1.49 (#12704) 2022-06-29 23:15:46 +08:00
orc-extensions Improved Java 17 support and Java runtime docs. (#12839) 2022-08-03 23:16:05 -07:00
parquet-extensions Perform lazy initialization of parquet extensions module (#12827) 2022-08-02 13:41:12 +05:30
postgresql-metadata-storage Optimize overlord GET /tasks memory usage (#12404) 2022-06-16 22:30:37 +05:30
protobuf-extensions Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
s3-extensions Tidy up construction of the Guice Injectors (#12816) 2022-08-04 00:05:07 -07:00
simple-client-sslcontext Tidy up construction of the Guice Injectors (#12816) 2022-08-04 00:05:07 -07:00
stats Adding withName implementation to AggregatorFactory (#12862) 2022-08-08 18:31:56 +05:30
testing-tools Tidy up construction of the Guice Injectors (#12816) 2022-08-04 00:05:07 -07:00