Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-06 10:08:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
druid/docs/multi-stage-query/known-issues.md
Soumyava 524842a3bb
Window function on msq (#15470) 
This PR aims to introduce Window functions on MSQ by doing the following:

    Introduce a Window querykit for handling window queries along with its factory and a processor for window queries
    If a window operator is present with a partition by clause, pushes the partition as a shuffle spec of the previous stage
    In presence of empty OVER() clause lets all operators loose on a single rac
    In presence of no empty OVER() clause, breaks down each window into individual stages
    Associated machinery to handle window functions in MSQ
    Introduced a separate hidden engine feature WINDOW_LEAF_OPERATOR which is set only for MSQ engine. In presence of this feature, the planner plans without the leaf operators by creating a window query over an inner scan query. In case of native this is set to false and the planner generates the leafOperators
    Guardrails around materialization
    Comprehensive UTs
2024-03-28 14:58:34 +05:30

3.3 KiB
Raw Blame History

id title sidebar_label
known-issues SQL-based ingestion known issues Known issues

:::info This page describes SQL-based batch ingestion using the druid-multi-stage-query extension, new in Druid 24.0. Refer to the ingestion methods table to determine which ingestion method is right for you. :::

Multi-stage query task runtime

  • Fault tolerance is partially implemented. Workers get relaunched when they are killed unexpectedly. The controller does not get relaunched if it is killed unexpectedly.

  • Worker task stage outputs are stored in the working directory given by druid.indexer.task.baseDir. Stages that generate a large amount of output data may exhaust all available disk space. In this case, the query fails with an UnknownError with a message including "No space left on device".

SELECT Statement

  • GROUPING SETS are not implemented. Queries using these features return a QueryNotSupported error.

INSERT and REPLACE Statements

  • The INSERT and REPLACE statements with column lists, like INSERT INTO tbl (a, b, c) SELECT ..., is not implemented.

  • INSERT ... SELECT and REPLACE ... SELECT insert columns from the SELECT statement based on column name. This differs from SQL standard behavior, where columns are inserted based on position.

  • INSERT and REPLACE do not support all options available in ingestion specs, including the createBitmapIndex and multiValueHandling dimension properties, and the indexSpec tuningConfig property.

EXTERN Function

  • The schemaless dimensions feature is not available. All columns and their types must be specified explicitly using the signature parameter of the EXTERN function.

  • EXTERN with input sources that match large numbers of files may exhaust available memory on the controller task.

  • EXTERN refers to external files. Use FROM to access druid input sources.

WINDOW Function

  • The maximum number of elements in a window cannot exceed a value of 100,000.
  • To avoid leafOperators in MSQ engine, window functions have an extra scan stage after the window stage for cases where native engine has a non-empty leafOperator.