druid/docs/development/extensions-contrib
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
..
aliyun-oss-extensions.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
ambari-metrics-emitter.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
cassandra.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
cloudfiles.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
compressed-big-decimal.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
ddsketch-quantiles.md New: Add DDSketch in extensions-contrib (#15049) 2024-01-23 20:17:07 +05:30
delta-lake.md Extension to read and ingest Delta Lake tables (#15755) 2024-01-30 21:53:50 -08:00
distinctcount.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
gce-extensions.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
graphite.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
iceberg.md Extension to read and ingest iceberg data files (#14329) 2023-07-18 08:59:57 +05:30
influx.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
influxdb-emitter.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
k8s-jobs.md Ability to send task types to k8s or worker task runner (#15196) 2023-10-25 09:55:56 -07:00
kafka-emitter.md Adding SegmentMetadataEvent and publishing them via KafkaEmitter (#14281) 2023-06-02 21:28:26 +05:30
materialized-view.md Make NodeRole available during binding; add support for dynamic registration of DruidService (#12012) 2021-12-03 11:59:00 -08:00
momentsketch-quantiles.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
moving-average-query.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
opentsdb-emitter.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
prometheus.md Prometheus config property doc fixup (#15613) 2024-01-02 16:28:42 -08:00
redis-cache.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
spectator-histogram.md Add SpectatorHistogram extension (#15340) 2024-01-14 09:52:30 -08:00
sqlserver.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
statsd.md Fix for segment/count Metric Not Emitting with Statsd-emitter (#15347) 2023-11-10 08:08:58 -08:00
tdigestsketch-quantiles.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
thrift.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
time-min-max.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00