druid/docs/ingestion
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
..
data-formats.md Add docs for ingesting Kafka topic name (#14894) 2023-08-24 19:19:59 +05:30
faq.md API reference refactor (#14372) 2023-06-26 15:48:54 -07:00
hadoop.md Removes support for Hadoop 2 (#14763) 2023-08-09 17:47:52 +05:30
index.md docs: Anchor link checker (#15624) 2024-01-08 15:19:05 -08:00
ingestion-spec.md docs: Anchor link checker (#15624) 2024-01-08 15:19:05 -08:00
input-sources.md Extension to read and ingest Delta Lake tables (#15755) 2024-01-30 21:53:50 -08:00
native-batch-firehose.md Docusaurus2 upgrade for master (#14411) 2023-08-16 19:01:21 -07:00
native-batch-simple-task.md Docusaurus2 upgrade for master (#14411) 2023-08-16 19:01:21 -07:00
native-batch.md [Docs] Improvements to JSON-based batch Ingestion page (#15286) 2023-10-31 14:50:45 -07:00
partitioning.md Add missing MSQ error code fields to docs (#13308) 2022-11-10 21:03:04 +05:30
rollup.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
schema-design.md fix array presenting columns to not match single element arrays to scalars for equality (#15503) 2023-12-08 01:22:07 -08:00
schema-model.md Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
standalone-realtime.md Reduce visibility of Tranquility documentation (#11134) 2021-05-03 16:48:24 -07:00
tasks.md Docusaurus2 upgrade for master (#14411) 2023-08-16 19:01:21 -07:00
tranquility.md Reduce visibility of Tranquility documentation (#11134) 2021-05-03 16:48:24 -07:00