2018-12-13 14:47:20 -05:00
|
|
|
---
|
2019-08-21 00:48:59 -04:00
|
|
|
id: joins
|
2018-12-13 14:47:20 -05:00
|
|
|
title: "Joins"
|
|
|
|
|
---
|
|
|
|
|
|
2018-11-13 12:38:37 -05:00
|
|
|
<!--
|
|
|
|
|
~ Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
|
~ or more contributor license agreements. See the NOTICE file
|
|
|
|
|
~ distributed with this work for additional information
|
|
|
|
|
~ regarding copyright ownership. The ASF licenses this file
|
|
|
|
|
~ to you under the Apache License, Version 2.0 (the
|
|
|
|
|
~ "License"); you may not use this file except in compliance
|
|
|
|
|
~ with the License. You may obtain a copy of the License at
|
|
|
|
|
~
|
|
|
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
~
|
|
|
|
|
~ Unless required by applicable law or agreed to in writing,
|
|
|
|
|
~ software distributed under the License is distributed on an
|
|
|
|
|
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
|
|
|
~ KIND, either express or implied. See the License for the
|
|
|
|
|
~ specific language governing permissions and limitations
|
|
|
|
|
~ under the License.
|
|
|
|
|
-->
|
|
|
|
|
|
2022-05-17 19:56:31 -04:00
|
|
|
Apache Druid has two features related to joining of data:
|
2015-08-21 16:54:00 -04:00
|
|
|
|
2020-04-15 19:12:20 -04:00
|
|
|
1. [Join](datasource.md#join) operators. These are available using a [join datasource](datasource.md#join) in native
|
2022-05-17 19:56:31 -04:00
|
|
|
queries, or using the [JOIN operator](sql.md) in Druid SQL. Refer to the
|
Sort-merge join and hash shuffles for MSQ. (#13506)
* Sort-merge join and hash shuffles for MSQ.
The main changes are in the processing, multi-stage-query, and sql modules.
processing module:
1) Rename SortColumn to KeyColumn, replace boolean descending with KeyOrder.
This makes it nicer to model hash keys, which use KeyOrder.NONE.
2) Add nullability checkers to the FieldReader interface, and an
"isPartiallyNullKey" method to FrameComparisonWidget. The join
processor uses this to detect null keys.
3) Add WritableFrameChannel.isClosed and OutputChannel.isReadableChannelReady
so callers can tell which OutputChannels are ready for reading and which
aren't.
4) Specialize FrameProcessors.makeCursor to return FrameCursor, a random-access
implementation. The join processor uses this to rewind when it needs to
replay a set of rows with a particular key.
5) Add MemoryAllocatorFactory, which is embedded inside FrameWriterFactory
instead of a particular MemoryAllocator. This allows FrameWriterFactory
to be shared in more scenarios.
multi-stage-query module:
1) ShuffleSpec: Add hash-based shuffles. New enum ShuffleKind helps callers
figure out what kind of shuffle is happening. The change from SortColumn
to KeyColumn allows ClusterBy to be used for both hash-based and sort-based
shuffling.
2) WorkerImpl: Add ability to handle hash-based shuffles. Refactor the logic
to be more readable by moving the work-order-running code to the inner
class RunWorkOrder, and the shuffle-pipeline-building code to the inner
class ShufflePipelineBuilder.
3) Add SortMergeJoinFrameProcessor and factory.
4) WorkerMemoryParameters: Adjust logic to reserve space for output frames
for hash partitioning. (We need one frame per partition.)
sql module:
1) Add sqlJoinAlgorithm context parameter; can be "broadcast" or
"sortMerge". With native, it must always be "broadcast", or it's a
validation error. MSQ supports both. Default is "broadcast" in
both engines.
2) Validate that MSQs do not use broadcast join with RIGHT or FULL join,
as results are not correct for broadcast join with those types. Allow
this in native for two reasons: legacy (the docs caution against it,
but it's always been allowed), and the fact that it actually *does*
generate correct results in native when the join is processed on the
Broker. It is much less likely that MSQ will plan in such a way that
generates correct results.
3) Remove subquery penalty in DruidJoinQueryRel when using sort-merge
join, because subqueries are always required, so there's no reason
to penalize them.
4) Move previously-disabled join reordering and manipulation rules to
FANCY_JOIN_RULES, and enable them when using sort-merge join. Helps
get to better plans where projections and filters are pushed down.
* Work around compiler problem.
* Updates from static analysis.
* Fix @param tag.
* Fix declared exception.
* Fix spelling.
* Minor adjustments.
* wip
* Merge fixups
* fixes
* Fix CalciteSelectQueryMSQTest
* Empty keys are sortable.
* Address comments from code review. Rename mux -> mix.
* Restore inspection config.
* Restore original doc.
* Reorder imports.
* Adjustments
* Fix.
* Fix imports.
* Adjustments from review.
* Update header.
* Adjust docs.
2023-03-08 17:19:39 -05:00
|
|
|
[join datasource](datasource.md#join) documentation for information about how joins work in Druid native queries,
|
|
|
|
|
or the [multi-stage query join documentation](../multi-stage-query/reference.md#joins) for information about how joins
|
|
|
|
|
work in multi-stage query tasks.
|
2020-04-15 19:12:20 -04:00
|
|
|
2. [Query-time lookups](lookups.md), simple key-to-value mappings. These are preloaded on all servers that are involved
|
|
|
|
|
in queries and can be accessed with or without an explicit join operator. Refer to the [lookups](lookups.md)
|
|
|
|
|
documentation for more details.
|
2015-08-21 16:54:00 -04:00
|
|
|
|
2020-04-15 19:12:20 -04:00
|
|
|
Whenever possible, for best performance it is good to avoid joins at query time. Often this can be accomplished by
|
|
|
|
|
joining data before it is loaded into Druid. However, there are situations where joins or lookups are the best solution
|
|
|
|
|
available despite the performance overhead, including:
|
2015-08-21 16:54:00 -04:00
|
|
|
|
2020-04-15 19:12:20 -04:00
|
|
|
- The fact-to-dimension (star and snowflake schema) case: you need to change dimension values after initial ingestion,
|
|
|
|
|
and aren't able to reingest to do this. In this case, you can use lookups for your dimension tables.
|
|
|
|
|
- Your workload requires joins or filters on subqueries.
|
