druid/docs
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
..
api-reference Remove documentation for unused segments retrieval API (#15721) 2024-01-18 18:25:48 -08:00
assets Add SpectatorHistogram extension (#15340) 2024-01-14 09:52:30 -08:00
comparisons remove ref to plywood repo (#12809) 2022-07-26 10:12:13 +08:00
configuration Disable eager initialization for non-query connection requests (#15751) 2024-01-25 14:38:50 +05:30
data-management Kill tasks honor the buffer period of unused segments (#15710) 2024-01-18 22:23:50 -08:00
design docs: Fix broken anchor links (#15621) 2024-01-03 15:28:27 -08:00
development Extension to read and ingest Delta Lake tables (#15755) 2024-01-30 21:53:50 -08:00
ingestion Extension to read and ingest Delta Lake tables (#15755) 2024-01-30 21:53:50 -08:00
misc Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
multi-stage-query Fix up typo </br /> -> <br /> and adjust interpolated exception msg in InvalidNullByteFault. (#15804) 2024-01-30 12:44:51 -08:00
operations docs: clean up some rolling updates stuff (#15762) 2024-01-26 14:10:53 -08:00
querying Document pivot and unpivot operators (#15669) 2024-01-25 09:53:39 -08:00
release-info docs: Anchor link checker (#15624) 2024-01-08 15:19:05 -08:00
tutorials Doc fixes for query from deep storage and MSQ (#15313) 2023-11-03 10:52:20 +05:30