druid/docs/development/extensions-core
David Palmer 2855fb6ff8
Change Kafka Lookup Extractor to not register consumer group (#12842)
* change kafka lookups module to not commit offsets

The current behaviour of the Kafka lookup extractor is to not commit
offsets by assigning a unique ID to the consumer group and setting
auto.offset.reset to earliest. This does the job but also pollutes the
Kafka broker with a bunch of "ghost" consumer groups that will never again be
used.

To fix this, we now set enable.auto.commit to false, which prevents the
ghost consumer groups being created in the first place.

* update docs to include new enable.auto.commit setting behaviour

* update kafka-lookup-extractor documentation

Provide some additional detail on functionality and configuration.
Hopefully this will make it clearer how the extractor works for
developers who aren't so familiar with Kafka.

* add comments better explaining the logic of the code

* add spelling exceptions for kafka lookup docs
2022-08-09 16:14:22 +05:30
..
approximate-histograms.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
avro.md clarify avro support & general style improvements (#11975) 2021-11-28 16:10:18 +08:00
azure.md Adding Shared Access resource support for azure (#12266) 2022-02-22 18:27:43 +05:30
bloom-filter.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
datasketches-extension.md cleaning up and fixing links (#10528) 2020-12-17 13:37:43 -08:00
datasketches-hll.md Update datasketches-hll.md (#12010) 2021-11-30 18:52:00 -08:00
datasketches-quantiles.md Configurable maxStreamLength for doubles sketches (#11574) 2021-08-31 14:56:37 -07:00
datasketches-theta.md cleaning up and fixing links (#10528) 2020-12-17 13:37:43 -08:00
datasketches-tuple.md fix: fix broken link to Class TTest (#12836) 2022-07-31 10:18:14 +08:00
druid-aws-rds.md AWS RDS token based password provider (#9518) 2021-01-06 21:15:29 -08:00
druid-basic-security.md Docs update for druid-basic-security (#11782) 2021-10-08 14:45:09 -07:00
druid-kerberos.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
druid-lookups.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
druid-pac4j.md Update screenshots for Druid console doc (#12593) 2022-06-15 16:42:20 -07:00
druid-ranger-security.md Refactor SQL docs (#12239) 2022-02-11 14:43:30 -08:00
examples.md De-incubation cleanup in code, docs, packaging (#9108) 2020-01-03 12:33:19 -05:00
google.md Docs: Add multi-dimension partitioning doc; refactor native batch and separate into smaller topics. (#11983) 2021-12-03 16:37:14 +05:30
hdfs.md Docs: Add multi-dimension partitioning doc; refactor native batch and separate into smaller topics. (#11983) 2021-12-03 16:37:14 +05:30
kafka-extraction-namespace.md Change Kafka Lookup Extractor to not register consumer group (#12842) 2022-08-09 16:14:22 +05:30
kafka-ingestion.md Docs: Splits Kafka topic. Adds detailed example for kafka inputFormat (#11912) 2021-11-12 13:02:23 -08:00
kafka-supervisor-operations.md Docs updates (#12069) 2021-12-14 14:38:18 -08:00
kafka-supervisor-reference.md Adding zstandard compression library (#12408) 2022-05-28 17:01:44 -07:00
kinesis-ingestion.md Kinesis ingestion with empty shards (#12792) 2022-08-05 22:38:58 +05:30
kubernetes.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
lookups-cached-global.md Warn if cache size of lookup is beyond max size (#11863) 2021-11-03 21:32:22 -04:00
mysql.md deps: upgrade mysql-connector-java to v5.1.49 (#12704) 2022-06-29 23:15:46 +08:00
orc.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
parquet.md Doc update for the new input source and the new input format (#9171) 2020-01-17 15:52:05 -08:00
postgresql.md Docs: Add multi-dimension partitioning doc; refactor native batch and separate into smaller topics. (#11983) 2021-12-03 16:37:14 +05:30
protobuf.md MySQL extension with MariaDB connector docs (#11608) 2021-08-19 01:52:26 -07:00
s3.md fix(docs): clarify what s3 permissions are needed based on the access management type (#12405) 2022-04-07 16:22:56 -07:00
simple-client-sslcontext.md cleaning up and fixing links (#10528) 2020-12-17 13:37:43 -08:00
stats.md Fix formatting in stats.md (#12470) 2022-04-23 11:35:08 +08:00
test-stats.md De-incubation cleanup in code, docs, packaging (#9108) 2020-01-03 12:33:19 -05:00