druid/extensions-core
Kashif Faraz ef7c461722
Add support for selective loading of broadcast datasources in the task layer (#17027) (#17206)
Tasks control the loading of broadcast datasources via BroadcastDatasourceLoadingSpec getBroadcastDatasourceLoadingSpec(). By default, tasks download all broadcast datasources, unless there's an override as with kill and MSQ controller task.

The CLIPeon command line option --loadBroadcastSegments is deprecated in favor of --loadBroadcastDatasourceMode.

Broadcast datasources can be specified in SQL queries through JOIN and FROM clauses, or obtained from other sources such as lookups.To this effect, we have introduced a BroadcastDatasourceLoadingSpec. Finding the set of broadcast datasources during SQL planning will be done in a follow-up, which will apply only to MSQ tasks, so they load only required broadcast datasources. This PR primarily focuses on the skeletal changes around BroadcastDatasourceLoadingSpec and integrating it from the Task interface via CliPeon to SegmentBootstrapper.

Currently, only kill tasks and MSQ controller tasks skip loading broadcast datasources.

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2024-10-01 11:09:52 +05:30
..
avro-extensions Better error handling when retrieving Avro schemas from registry (#16684) 2024-07-02 16:48:34 -07:00
azure-extensions Allow druid.azure.account to be nullable (#16960) 2024-09-02 12:05:51 +05:30
datasketches Window Functions : Remove enable windowing flag (#17087) (#17128) 2024-09-24 10:28:11 +02:00
druid-aws-rds-extensions Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
druid-basic-security Fix excessive logging from druid-basic-security (#16767) 2024-07-22 08:33:00 +05:30
druid-bloom-filter transition away from StorageAdapter (#16985) (#17024) 2024-09-09 21:43:41 -07:00
druid-catalog Quidem record (#16624) 2024-08-05 14:58:32 +02:00
druid-kerberos Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
druid-pac4j Revert "[CVE Fixes] Update version of Nimbus.jose.jwt (#16320)" (#16986) 2024-09-09 10:11:58 +05:30
druid-ranger-security Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
ec2-extensions Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
google-extensions Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
hdfs-storage Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
histogram generic block compressed complex columns (#16863) 2024-08-27 00:34:41 -07:00
kafka-extraction-namespace Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
kafka-indexing-service Track IngestionState more accurately in realtime tasks. (#16934) 2024-08-22 11:43:46 +05:30
kinesis-indexing-service Track IngestionState more accurately in realtime tasks. (#16934) 2024-08-22 11:43:46 +05:30
kubernetes-extensions #16717 defer provider instantiation in Kubernetes Module (#16726) 2024-07-16 13:05:28 -07:00
lookups-cached-global upgrade mysql:mysql-connector-java to 8.2.0 (#16024) 2024-05-06 21:58:37 +08:00
lookups-cached-single upgrade mysql:mysql-connector-java to 8.2.0 (#16024) 2024-05-06 21:58:37 +08:00
multi-stage-query Add support for selective loading of broadcast datasources in the task layer (#17027) (#17206) 2024-10-01 11:09:52 +05:30
mysql-metadata-storage remove Firehose and FirehoseFactory (#16758) 2024-07-19 14:37:21 -07:00
orc-extensions Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
parquet-extensions Refactor: Clean up `DecimalParquetInputTest` using Assume (#16436) 2024-05-14 21:13:07 +05:30
postgresql-metadata-storage remove Firehose and FirehoseFactory (#16758) 2024-07-19 14:37:21 -07:00
protobuf-extensions Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
s3-extensions Reduce logging in RetryableS3OutputStream (#16853) 2024-08-08 10:42:40 +05:30
simple-client-sslcontext Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
stats Window Functions : Remove enable windowing flag (#17087) (#17128) 2024-09-24 10:28:11 +02:00
testing-tools Quidem record (#16624) 2024-08-05 14:58:32 +02:00