Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
14,652 Commits 111 Branches 504 Tags 700 MiB
Commit Graph

5 Commits

Author SHA1 Message Date
Shekhar Prasad Rajak ae049a4bab
AWS Glue Catalog for Iceberg ingest extension (#17392) 
* iceberg glue catalog dependencies added

* GlueIcebergCatalog added in druid module

* default version of iceberg glue catalog implementation - basics

* basic tests added

* removed dependecy iceberg-aws-bundle

* glue catalog support - docs update for iceberg

* Update IcebergDruidModule.java

* Update IcebergDruidModule.java

* updates in dependencies and warehousePath must be under catalogProp

* removed some dependencies - which not required

* only glue sdk added

* update license

* avro exclusion removed

* doc update

* doc update

* set the type to glue

* minor change

* minor change

* fixing codestyle

* checkstyle fixes

* checkstyle fixes

* checkstyle fixes

* dependency check fixes

* update pom for ignore warning for glue catalog

* compile scope needed - iceberg-aws and awssdk

* updates pom with comment

* minor change

* mvn dependency check in iceberg extension

* revert pom.xml changes

* aws sdk sts and s3 for gluecatalog initialize

* dependency check - ignore aws sdk s3 and sts

---------

Co-authored-by: SHEKHAR PRASAD RAJAK <shekhar_rajak@apple.com>
 2024-11-10 18:43:55 -08:00
Atul Mohan c1f8ae25b5
Support Iceberg ingestion from REST based catalogs (#17124) 
Adds support to the iceberg input source to read from Iceberg REST Catalogs.
 2024-09-23 22:13:24 -07:00
Charles Smith e562dd3ac6
Docs: note on iceberg (#16955) 
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
 2024-08-27 14:27:23 -07:00
Abhishek Radhakrishnan 1a5b57df84
Update `groupId` for delta-lake and iceberg extensions (#15843) 
* Update the group id to org.apache.druid.extensions.contrib for contrib exts.

* Note iceberg and delta lake extensions in extensions.md

* properties and shell backticks

* Update groupId in distribution/pom.xml

* remove delta-lake from dist.

* Add note on downloading extension.
 2024-02-07 23:54:06 -08:00
Atul Mohan 03d6d395a0
Extension to read and ingest iceberg data files (#14329) 
This adds a new contrib extension: druid-iceberg-extensions which can be used to ingest data stored in Apache Iceberg format. It adds a new input source of type iceberg that connects to a catalog and retrieves the data files associated with an iceberg table and provides these data file paths to either an S3 or HDFS input source depending on the warehouse location.

Two important dependencies associated with Apache Iceberg tables are:

Catalog : This extension supports reading from either a Hive Metastore catalog or a Local file-based catalog. Support for AWS Glue is not available yet.
Warehouse : This extension supports reading data files from either HDFS or S3. Adapters for other cloud object locations should be easy to add by extending the AbstractInputSourceAdapter.
 2023-07-18 08:59:57 +05:30