druid/extensions-contrib
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
..
aliyun-oss-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
ambari-metrics-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
cassandra-storage Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
cloudfiles-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
compressed-bigdecimal Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
ddsketch Fix minor build issues and stabilize intellij-inspections runs (#15747) 2024-01-24 15:17:33 +05:30
distinctcount Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
dropwizard-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
druid-deltalake-extensions Extension to read and ingest Delta Lake tables (#15755) 2024-01-30 21:53:50 -08:00
druid-iceberg-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
gce-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
graphite-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
influx-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
influxdb-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
kafka-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
kubernetes-overlord-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
materialized-view-maintenance Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
materialized-view-selection Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
momentsketch Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
moving-average-query Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
opentelemetry-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
opentsdb-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
prometheus-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
redis-cache Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
spectator-histogram Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
sqlserver-metadata-storage Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
statsd-emitter Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
tdigestsketch Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
thrift-extensions Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
time-min-max Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
virtual-columns Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
README.md fix broken links (#9537) 2020-03-22 17:41:18 -07:00

README.md

Community Extensions

Please contribute all community extensions in this directory and include a doc of how your extension can be used under docs/development/extensions-contrib/.

Please note that community extensions are maintained by their original contributors and are not packaged with the core Druid distribution. If you'd like to take on maintenance for a community extension, please post on dev@druid.apache.org to let us know!