14653 Commits

Author SHA1 Message Date
Zoltan Haindrich
0d09c18cab undo trials; re-introduce expr if needed (will be trimmed in most cases) 2024-11-13 16:48:31 +00:00
Zoltan Haindrich
8dfd57f922 update test 2024-11-13 16:30:08 +00:00
Zoltan Haindrich
1379c9aa2c Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-13 16:24:35 +00:00
Zoltan Haindrich
247ee17f17 fixups after merge 2024-11-13 15:35:00 +00:00
Zoltan Haindrich
a1b9f522fe Merge remote-tracking branch 'apache/master' into rename-d1-dbl1 2024-11-13 15:30:58 +00:00
Zoltan Haindrich
fc31c8a84a update iq files 2024-11-13 14:51:56 +00:00
Zoltan Haindrich
ced72b3ab7 fix nondefault 2024-11-13 14:40:12 +00:00
Zoltan Haindrich
08748436a5 fix some more 2024-11-13 14:20:41 +00:00
Zoltan Haindrich
62664e36e7 fix windowtests 2024-11-13 14:16:48 +00:00
Zoltan Haindrich
88e57f972a fix some more 2024-11-13 14:12:25 +00:00
Zoltan Haindrich
bd72b95777 fix some 2024-11-13 14:01:45 +00:00
Zoltan Haindrich
0d07cf137e dbl1/etc 2024-11-13 13:55:38 +00:00
Akshat Jain
390c2d68c8
Remove intellij-inspections check from CI (#17469) 2024-11-13 18:58:17 +05:30
Zoltan Haindrich
7777e48b77 dbl1/etc 2024-11-13 13:11:10 +00:00
Kiran Gadhave
1dbd005df6
updated docs with behavior for empty collections in pod template selector config (#17464) 2024-11-12 13:21:27 -08:00
zachjsh
1f3b1f85f9
Add documentation for Druids catalog extension (#17459)
* SQL syntax error should target USER persona

* * revert change to queryHandler and related tests, based on review comments

* * add test

* Add documentation for druid-catalog extension

* * fix error

* * fix error

* Apply suggestions from code review

Co-authored-by: Andreas Maechler <amaechler@gmail.com>

* * fix spelling error

* * fix spelling

---------

Co-authored-by: Andreas Maechler <amaechler@gmail.com>
2024-11-12 14:50:55 -05:00
Zoltan Haindrich
d4b4c94a3c Reapply "rename: d1/dbl1"
This reverts commit 0d03cc558781cfd4093c0ff7c87f03c7cc65e027.
2024-11-12 15:04:41 +00:00
Zoltan Haindrich
0d03cc5587 Revert "rename: d1/dbl1"
This reverts commit c49fc2796b5062d5886bc1ddd36ac3e524507c21.
2024-11-12 15:04:14 +00:00
Zoltan Haindrich
c49fc2796b rename: d1/dbl1 2024-11-12 15:04:11 +00:00
Zoltan Haindrich
04ab28f4cd one way 2024-11-12 14:28:25 +00:00
Zoltan Haindrich
e8f06bf90d make helper method 2024-11-12 14:19:29 +00:00
Zoltan Haindrich
322ec81fb0 make some assert function and fail with that 2024-11-12 14:12:30 +00:00
Zoltan Haindrich
d38215fd5a ok-but alon trim should have affected native query - right? 2024-11-12 11:46:30 +00:00
Zoltan Haindrich
f296102f05
ScanQuery should not ignore columnTypes in equals/hashCode (#17463)
* ScanQuery: equals/hashCode/toString
* DruidQuery: changes of Align ScanQuery column order with its desired signature #17457
* ScanQueryTest: add equalsverifer test
2024-11-12 14:26:59 +05:30
Akshat Jain
c571e6905d
Refactor WindowOperatorQueryKit to use WindowStage class for representing different window stages (#17158) 2024-11-12 14:18:16 +05:30
Virushade
8278a1f7df
Fix Javadocs in ColumnCapablities.java (#17462) 2024-11-12 11:30:33 +05:30
Akshat Jain
3f56b57c7e
MSQ WF: Pass a flag from broker to determine operator chain transformation (#17443) 2024-11-12 09:28:28 +05:30
Zoltan Haindrich
b45b3ba495 run trimmer 2024-11-11 13:56:06 +00:00
Zoltan Haindrich
e7061ad04a more wood 2024-11-11 13:10:52 +00:00
Zoltan Haindrich
4e0a37a59b more wood 2024-11-11 12:39:30 +00:00
Zoltan Haindrich
2ceefdc079 try-unnestfiled-instead-rowtype
(cherry picked from commit c4eb2cf1b82c7a332d57701e430ac582a986593f)
2024-11-11 11:15:40 +00:00
Zoltan Haindrich
314d3f7a17 some updates 2024-11-11 10:28:11 +00:00
Shekhar Prasad Rajak
ae049a4bab
AWS Glue Catalog for Iceberg ingest extension (#17392)
* iceberg glue catalog dependencies added

* GlueIcebergCatalog added in druid module

* default version of iceberg glue catalog implementation - basics

* basic tests added

* removed dependecy iceberg-aws-bundle

* glue catalog support - docs update for iceberg

* Update IcebergDruidModule.java

* Update IcebergDruidModule.java

* updates in dependencies and warehousePath must be under catalogProp

* removed some dependencies - which not required

* only glue sdk added

* update license

* avro exclusion removed

* doc update

* doc update

* set the type to glue

* minor change

* minor change

* fixing codestyle

* checkstyle fixes

* checkstyle fixes

* checkstyle fixes

* dependency check fixes

* update pom for ignore warning for glue catalog

* compile scope needed - iceberg-aws and awssdk

* updates pom with comment

* minor change

* mvn dependency check in iceberg extension

* revert pom.xml changes

* aws sdk sts and s3 for gluecatalog initialize

* dependency check - ignore aws sdk s3 and sts

---------

Co-authored-by: SHEKHAR PRASAD RAJAK <shekhar_rajak@apple.com>
2024-11-10 18:43:55 -08:00
Zoltan Haindrich
00837c56ed clenaup 2024-11-09 16:54:13 +00:00
jtuglu-netflix
f906d0d446
Fix query failed metric double count bug (#17454) 2024-11-08 23:15:03 -08:00
Vivek Dhiman
0dcc2bc469
Fixed NPE in array_overlap and array_contains. (#17465) 2024-11-08 20:39:14 -08:00
Zoltan Haindrich
54b057e81d x 2024-11-08 17:53:59 +00:00
Zoltan Haindrich
7021b3f42c working? 2024-11-08 17:33:31 +00:00
George Shiqi Wu
5764183d4e
k8s-based-ingestion: Wait for task lifecycles to enter RUNNING state before returning from KubernetesTaskRunner.start (#17446)
* Add a wait on start() for task lifecycle to go into running

* handle exceptions

* Fix logging messages

* Don't pass in the settable future as a arg

* add some unit tests
2024-11-08 11:13:35 -05:00
Gian Merlino
d8162163c8
Run JDK 21 workflows with 21.0.4. (#17458)
* Run JDK 21 workflows with 21.0.4.

To work around #17429, run our JDK 21 workflows with
version 21.0.4. It does not appear to have this problem.

* Undo changes in standard-its.yml

* Add comments.

---------

Co-authored-by: Zoltan Haindrich <kirk@rxd.hu>
2024-11-07 10:53:52 -08:00
Nandini Anagondi
32394e55f9
Upgrading org.codehaus to com.fasterxml (#17371) 2024-11-07 10:55:47 +01:00
Akshat Jain
73cbce9109
WindowOperatorQueryFrameProcessorFactory: Pass QueryContext instead of WindowOperatorQuery to WindowOperatorQueryFrameProcessor (#17405)
* WindowOperatorQueryKit: Pass QueryContext instead of WindowOperatorQuery to subsequent layers

* Add serializer for QueryContext class

* Revert changes of WindowOperatorQueryFrameProcessorFactory json param

* Fix checkstyle

* Address review comment: Remove older method in favor of calling new method inline
2024-11-07 11:29:49 +05:30
Gian Merlino
9c25226e06
QueryableIndexSegment: Re-use time boundary inspector. (#17397)
This patch re-uses timeBoundaryInspector for each cursor holder, which
enables caching of minDataTimestamp and maxDataTimestamp.

Fixes a performance regression introduced in #16533, where these fields
stopped being cached across cursors. Prior to that patch, they were
cached in the QueryableIndexStorageAdapter.
2024-11-06 09:27:59 -08:00
Abhishek Radhakrishnan
d8e4be654f
ManageLifecycle DropwizardEmitter instantiation. (#17451) 2024-11-05 18:57:22 -08:00
George Shiqi Wu
8850023811
Fix error where communication failures to k8s can lead to stuck tasks (#17431)
* Fix save logs error

* Update extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* make things final

* fix merge conflicts

---------

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2024-11-05 09:58:30 -08:00
Zoltan Haindrich
2eac8318f8
Support Union in Decoupled planning (#17354)
* introduces `UnionQuery`
* some changes to enable a `UnionQuery` to have multiple input datasources
* `UnionQuery` execution is driven by the `QueryLogic` - which could later enable to reduce some complexity in `ClientQuerySegmentWalker`
* to run the subqueries of `UnionQuery` there was a need to access the `conglomerate` from the `Runner`; to enable that some refactors were done
* renamed `UnionQueryRunner` to `UnionDataSourceQueryRunner`
* `QueryRunnerFactoryConglomerate` have taken the place of `QueryToolChestWarehouse` which shaves of some unnecessary things here and there
* small cleanup/refactors
2024-11-05 16:58:57 +01:00
Virushade
ba76264244
Update build documentation (#17444)
Add build instructions for developers
Follow up from issue #17375, add instructions solely for distribution profile. Note that this build command is mostly used by me, everyone is welcome to add further optimizations for a faster distribution build.

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* Update docs/development/build.md

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* Update docs/development/build.md

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

---------

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2024-11-04 18:31:46 -08:00
Tom
e4cdbca23c
make planner errors be user persona (#17437)
Change the persona for errors within the planner from Admin to User. The ADMIN persona is meant to be "a persona who is interacting with admin APIs and understands Druid query concepts". This isn't an admin API, it's a query API. Low quality error messages being returned to the correct audience is better than hiding all error messages.

The errors that can be returned back can be user solvable, and other times requires a druid expert. But the errors do not leak information that should only be seen by more expert/privileged personas.

The original ADMIN persona showed some reticence to tag low-quality error messages with a USER persona. but it really does seem user-directed to me so USER to me would make sense.
2024-11-04 10:48:35 -08:00
Kiran Gadhave
5fcf4205e4
Handle empty values for task and datasource conditions in pod template selector (#17400)
* handling empty sets for dataSourceCondition and taskTypeCondition

* using new HashSet<>() to fix forbidden api error in testCheck

* fixing style issues
2024-10-30 20:18:20 -07:00
Vadim Ogievetsky
4b7902e74a
Web console: Improve workbench view with resizable side panels (#17387) 2024-10-30 11:50:52 -07:00