Commit Graph

14799 Commits

Author SHA1 Message Date
Zoltan Haindrich a1b9f522fe Merge remote-tracking branch 'apache/master' into rename-d1-dbl1 2024-11-13 15:30:58 +00:00
Zoltan Haindrich fc31c8a84a update iq files 2024-11-13 14:51:56 +00:00
Zoltan Haindrich ced72b3ab7 fix nondefault 2024-11-13 14:40:12 +00:00
Zoltan Haindrich 08748436a5 fix some more 2024-11-13 14:20:41 +00:00
Zoltan Haindrich 62664e36e7 fix windowtests 2024-11-13 14:16:48 +00:00
Zoltan Haindrich 88e57f972a fix some more 2024-11-13 14:12:25 +00:00
Zoltan Haindrich bd72b95777 fix some 2024-11-13 14:01:45 +00:00
Zoltan Haindrich 0d07cf137e dbl1/etc 2024-11-13 13:55:38 +00:00
Akshat Jain 390c2d68c8
Remove `intellij-inspections` check from CI (#17469) 2024-11-13 18:58:17 +05:30
Zoltan Haindrich 7777e48b77 dbl1/etc 2024-11-13 13:11:10 +00:00
Kiran Gadhave 1dbd005df6
updated docs with behavior for empty collections in pod template selector config (#17464) 2024-11-12 13:21:27 -08:00
zachjsh 1f3b1f85f9
Add documentation for Druids catalog extension (#17459)
* SQL syntax error should target USER persona

* * revert change to queryHandler and related tests, based on review comments

* * add test

* Add documentation for druid-catalog extension

* * fix error

* * fix error

* Apply suggestions from code review

Co-authored-by: Andreas Maechler <amaechler@gmail.com>

* * fix spelling error

* * fix spelling

---------

Co-authored-by: Andreas Maechler <amaechler@gmail.com>
2024-11-12 14:50:55 -05:00
Zoltan Haindrich d4b4c94a3c Reapply "rename: d1/dbl1"
This reverts commit 0d03cc5587.
2024-11-12 15:04:41 +00:00
Zoltan Haindrich 0d03cc5587 Revert "rename: d1/dbl1"
This reverts commit c49fc2796b.
2024-11-12 15:04:14 +00:00
Zoltan Haindrich c49fc2796b rename: d1/dbl1 2024-11-12 15:04:11 +00:00
Zoltan Haindrich 04ab28f4cd one way 2024-11-12 14:28:25 +00:00
Zoltan Haindrich e8f06bf90d make helper method 2024-11-12 14:19:29 +00:00
Zoltan Haindrich 322ec81fb0 make some assert function and fail with that 2024-11-12 14:12:30 +00:00
Zoltan Haindrich d38215fd5a ok-but alon trim should have affected native query - right? 2024-11-12 11:46:30 +00:00
Zoltan Haindrich f296102f05
ScanQuery should not ignore columnTypes in equals/hashCode (#17463)
* ScanQuery: equals/hashCode/toString
* DruidQuery: changes of Align ScanQuery column order with its desired signature #17457
* ScanQueryTest: add equalsverifer test
2024-11-12 14:26:59 +05:30
Akshat Jain c571e6905d
Refactor WindowOperatorQueryKit to use WindowStage class for representing different window stages (#17158) 2024-11-12 14:18:16 +05:30
Virushade 8278a1f7df
Fix Javadocs in ColumnCapablities.java (#17462) 2024-11-12 11:30:33 +05:30
Akshat Jain 3f56b57c7e
MSQ WF: Pass a flag from broker to determine operator chain transformation (#17443) 2024-11-12 09:28:28 +05:30
Zoltan Haindrich b45b3ba495 run trimmer 2024-11-11 13:56:06 +00:00
Zoltan Haindrich e7061ad04a more wood 2024-11-11 13:10:52 +00:00
Zoltan Haindrich 4e0a37a59b more wood 2024-11-11 12:39:30 +00:00
Zoltan Haindrich 2ceefdc079 try-unnestfiled-instead-rowtype
(cherry picked from commit c4eb2cf1b82c7a332d57701e430ac582a986593f)
2024-11-11 11:15:40 +00:00
Zoltan Haindrich 314d3f7a17 some updates 2024-11-11 10:28:11 +00:00
Shekhar Prasad Rajak ae049a4bab
AWS Glue Catalog for Iceberg ingest extension (#17392)
* iceberg glue catalog dependencies added

* GlueIcebergCatalog added in druid module

* default version of iceberg glue catalog implementation - basics

* basic tests added

* removed dependecy iceberg-aws-bundle

* glue catalog support - docs update for iceberg

* Update IcebergDruidModule.java

* Update IcebergDruidModule.java

* updates in dependencies and warehousePath must be under catalogProp

* removed some dependencies - which not required

* only glue sdk added

* update license

* avro exclusion removed

* doc update

* doc update

* set the type to glue

* minor change

* minor change

* fixing codestyle

* checkstyle fixes

* checkstyle fixes

* checkstyle fixes

* dependency check fixes

* update pom for ignore warning for glue catalog

* compile scope needed - iceberg-aws and awssdk

* updates pom with comment

* minor change

* mvn dependency check in iceberg extension

* revert pom.xml changes

* aws sdk sts and s3 for gluecatalog initialize

* dependency check - ignore aws sdk s3 and sts

---------

Co-authored-by: SHEKHAR PRASAD RAJAK <shekhar_rajak@apple.com>
2024-11-10 18:43:55 -08:00
Zoltan Haindrich 00837c56ed clenaup 2024-11-09 16:54:13 +00:00
jtuglu-netflix f906d0d446
Fix query failed metric double count bug (#17454) 2024-11-08 23:15:03 -08:00
Vivek Dhiman 0dcc2bc469
Fixed NPE in `array_overlap` and `array_contains`. (#17465) 2024-11-08 20:39:14 -08:00
Zoltan Haindrich 54b057e81d x 2024-11-08 17:53:59 +00:00
Zoltan Haindrich 7021b3f42c working? 2024-11-08 17:33:31 +00:00
George Shiqi Wu 5764183d4e
k8s-based-ingestion: Wait for task lifecycles to enter RUNNING state before returning from KubernetesTaskRunner.start (#17446)
* Add a wait on start() for task lifecycle to go into running

* handle exceptions

* Fix logging messages

* Don't pass in the settable future as a arg

* add some unit tests
2024-11-08 11:13:35 -05:00
Gian Merlino d8162163c8
Run JDK 21 workflows with 21.0.4. (#17458)
* Run JDK 21 workflows with 21.0.4.

To work around #17429, run our JDK 21 workflows with
version 21.0.4. It does not appear to have this problem.

* Undo changes in standard-its.yml

* Add comments.

---------

Co-authored-by: Zoltan Haindrich <kirk@rxd.hu>
2024-11-07 10:53:52 -08:00
Nandini Anagondi 32394e55f9
Upgrading org.codehaus to com.fasterxml (#17371) 2024-11-07 10:55:47 +01:00
Akshat Jain 73cbce9109
WindowOperatorQueryFrameProcessorFactory: Pass QueryContext instead of WindowOperatorQuery to WindowOperatorQueryFrameProcessor (#17405)
* WindowOperatorQueryKit: Pass QueryContext instead of WindowOperatorQuery to subsequent layers

* Add serializer for QueryContext class

* Revert changes of WindowOperatorQueryFrameProcessorFactory json param

* Fix checkstyle

* Address review comment: Remove older method in favor of calling new method inline
2024-11-07 11:29:49 +05:30
Gian Merlino 9c25226e06
QueryableIndexSegment: Re-use time boundary inspector. (#17397)
This patch re-uses timeBoundaryInspector for each cursor holder, which
enables caching of minDataTimestamp and maxDataTimestamp.

Fixes a performance regression introduced in #16533, where these fields
stopped being cached across cursors. Prior to that patch, they were
cached in the QueryableIndexStorageAdapter.
2024-11-06 09:27:59 -08:00
Abhishek Radhakrishnan d8e4be654f
ManageLifecycle DropwizardEmitter instantiation. (#17451) 2024-11-05 18:57:22 -08:00
George Shiqi Wu 8850023811
Fix error where communication failures to k8s can lead to stuck tasks (#17431)
* Fix save logs error

* Update extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* make things final

* fix merge conflicts

---------

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2024-11-05 09:58:30 -08:00
Zoltan Haindrich 2eac8318f8
Support Union in Decoupled planning (#17354)
* introduces `UnionQuery`
* some changes to enable a `UnionQuery` to have multiple input datasources
* `UnionQuery` execution is driven by the `QueryLogic` - which could later enable to reduce some complexity in `ClientQuerySegmentWalker`
* to run the subqueries of `UnionQuery` there was a need to access the `conglomerate` from the `Runner`; to enable that some refactors were done
* renamed `UnionQueryRunner` to `UnionDataSourceQueryRunner`
* `QueryRunnerFactoryConglomerate` have taken the place of `QueryToolChestWarehouse` which shaves of some unnecessary things here and there
* small cleanup/refactors
2024-11-05 16:58:57 +01:00
Virushade ba76264244
Update build documentation (#17444)
Add build instructions for developers
Follow up from issue #17375, add instructions solely for distribution profile. Note that this build command is mostly used by me, everyone is welcome to add further optimizations for a faster distribution build.

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* Update docs/development/build.md

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* Update docs/development/build.md

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

---------

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2024-11-04 18:31:46 -08:00
Tom e4cdbca23c
make planner errors be user persona (#17437)
Change the persona for errors within the planner from Admin to User. The ADMIN persona is meant to be "a persona who is interacting with admin APIs and understands Druid query concepts". This isn't an admin API, it's a query API. Low quality error messages being returned to the correct audience is better than hiding all error messages.

The errors that can be returned back can be user solvable, and other times requires a druid expert. But the errors do not leak information that should only be seen by more expert/privileged personas.

The original ADMIN persona showed some reticence to tag low-quality error messages with a USER persona. but it really does seem user-directed to me so USER to me would make sense.
2024-11-04 10:48:35 -08:00
Kiran Gadhave 5fcf4205e4
Handle empty values for task and datasource conditions in pod template selector (#17400)
* handling empty sets for dataSourceCondition and taskTypeCondition

* using new HashSet<>() to fix forbidden api error in testCheck

* fixing style issues
2024-10-30 20:18:20 -07:00
Vadim Ogievetsky 4b7902e74a
Web console: Improve workbench view with resizable side panels (#17387) 2024-10-30 11:50:52 -07:00
Ashwin Tumma d5bb7de5cf
Fix Map Lookup Introspection Endpoints and update doc for Globally Cached Lookups (#17436)
Map Lookup Introspection API endpoints /keys and /values no longer return an invalid JSON object.
Also, update documentation to clarify the version returned by the /version introspection endpoint.

---------

Co-authored-by: Ashwin Tumma <ashwin.tumma@salesforce.com>
2024-10-30 08:23:22 -07:00
Akshat Jain 21e7e5cddd
Add benchmark suite for MSQ window functions (#17377)
* Add benchmark suite for MSQ window functions

* Fix inspection checks

* Address review comment: Rename method
2024-10-30 11:32:28 +05:30
Akshat Jain 63c91ad813
Fix backward compatibility issues in WindowOperatorQueryFrameProcessorFactory and WindowOperatorQueryFrameProcessor (#17433) 2024-10-30 11:32:02 +05:30
George Shiqi Wu 66eb365e4d
Revert "always set taskLocation (#17350)" (#17417)
This reverts commit a664fc8be3.
2024-10-29 13:31:01 -07:00