95fb813b6d
Fix bug with while adding DistinctSketches ( #17184 )
...
* Fix bug with checking the incorrect key
* Add tests
2024-09-30 11:35:32 +05:30
ab361747a8
Migrated commons-lang usages to commons-lang3 ( #17156 )
2024-09-28 10:28:11 +02:00
661614129e
Window Functions : Context Parameter to Enable Transfer of RACs over wire ( #17150 )
2024-09-28 08:04:22 +02:00
f8a72b987a
read metadata in SimpleQueryableIndex if available to compute segment ordering ( #17181 )
2024-09-27 19:39:03 -07:00
7417ead3af
Web console: Fixed sampling for delta source in classic data loader and MSQ ( #17160 )
...
Added the delta tile to MSQ and made sure it works there and classic data loader. It is special because it always assumes a parquet format so it has to skip some steps in the code. Now the framework is there to add more tiles that force a specific format
2024-09-27 07:51:24 -07:00
dc223f22db
SQL: Use regular filters for time filtering in subqueries. ( #17173 )
...
* SQL: Use regular filters for time filtering in subqueries.
Using the "intervals" feature on subqueries, or any non-table, should be
avoided because it isn't a meaningful optimization in those cases, and
it's simpler for runtime implementations if they can assume all filters
are located in the regular filter object.
Two changes:
1) Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended
to return false for QueryDataSource, but actually returned true.
2) Add a validation to ScanQueryFrameProcessor to ensure that when running
on an input channel (which would include any subquery), the query has
"intervals" set to ONLY_ETERNITY.
Prior to this patch, the new test case in testTimeFilterOnSubquery would
throw a "Can only handle a single interval" error in the native engine,
and "QueryNotSupported" in the MSQ engine.
* Mark new case as having extra columns in decoupled mode.
* Adjust test.
2024-09-27 10:32:30 +05:30
157fe1bc1f
fix a mistake in CursorGranularizer to check doneness after advance ( #17175 )
...
Fixes a mistake introduced in #16533 which can result in CursorGranularizer incorrectly trying to get values from a selector after calling cursor.advance because of a missing check for cursor.isDone
2024-09-27 09:36:05 +05:30
6ee9e42f03
fix issue with ScanQueryFrameProcessor cursor build not adjusting intervals ( #17168 )
...
* fix issue with ScanQueryFrameProcessor cursor build not adjusting intervals
* all hail the robot overlords
* style bot
2024-09-26 21:09:14 +05:30
ca96629207
Add kttm-nested dataset and some usage sessions of it ( #16867 )
...
* Add kttm-nested dataset and some usage sessions of it
Squashed commit of the following:
commit 3075269bdefbbfeeb3c7958751d9c44dd017944f
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:48:19 2024 +0000
rename files
commit ad8d37a235a89cdf92ea3c1e9fdda9dff87fa7ce
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:44:09 2024 +0000
rename
commit 8fc4efe19eba10e7d6a82042aedec0ec7c04c77c
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:43:31 2024 +0000
cleanup kttm loader
commit 113a4b6f4c0a450581600aa7483a5163ee00e937
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:38:10 2024 +0000
weed out infoschema stuff
commit 1eb9b5070c16a42a19ab9774d2b0378e090097df
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:17:27 2024 +0000
reduce crap
commit 1745ececc77cae93ed9130fd6e9e83fddd44471a
Merge: 9dd2bff6cc 1f8473d82b
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 09:49:52 2024 +0000
Merge branch 'quidem-msq' into quidem-qa
commit 9dd2bff6ccdb30c927a256ff7ab6ebf491508008
Merge: 7120f84e36 fda0d63e444e5d7c9c216cd8c6be224e5d7c9c216cef3eb6e873904b520dad5768e512039d9c88f87a6ba55fa25bc32d56785bc32d56788a92ec6b2edbe2674971ebea34a814dbe267497123c6f3bd9677408329953e2b59f80834651e70e534651e70e5449a7f3a7369a39a42da2700557a552700557a5542cc5d62a8d5b82af4a921016a28a23f507d864819fe5867b1941c39aae500ee182d7591d5d14bf129b2b559d96d38e8f075c3aa033e146d38e8f075f867db774a61c3b16b17130252bb5e3b77784e6e22d8a4b87234ab9113995e5c94d6d893457c6b3e12cfde805ec8f91478106d339d1706181458c8734c722f271f10639487491063948749c40474285cf4af51ef7fbc70443c7f841ab462dd929e68c11afda0d63e44fda0d63e44bb23ace51826e3c44f4b929e68c11a436ba1881570e46eadb9090f937d585e81a026e9bb23ace518bb23ace518e6add9ea84a03fb49f4fc7eacd079e5e81a026e95f6290eb54a03fb49f4fa03fb49f4f5f6290eb54de207c7295b1ab252b314157a8f105ca121aa083ca121aa083eb2a047e4b57980066b3954aaafe0c57980066b37f99ee24d7b345dd9d03df422456859ac26e3a8978b75d3e8ef6cc540368ce667eeb5e4157a8f105ed9ef1f6358bb38a04a5d705c2759ba489e19242d010b488a77428da51de0be1f81d7e7cfbfdc3eee60a200d95a9dcb2da4631e97324ced227029b6bd227029b6bf7247e1bb7b38935a45031e97324cee2a54b5758361149b097bc7174cb6a9cf723adae7a34b6e092d216b934fc76ff3f26e1e388140b2a76ff3f26e1eb4fd9f66ce388140b2a47aeb016df70ff2a3e9706b68b6c8906b68b6c8982436df585b53c26f5c570ff2a3e978b26e490e9c59f1adcc895ca0a9f5db100e982a40811d801fb97c32ca3de6790f9cf8b51d465df6d0eaf4c61b9f3cf77811542b30865128ada2ff23882436df5852a590eb3ae20c543dfc201d67ae54320c543dfc2c559bc3422cc4e0adcbfb8992434a9c559bc34229a4d04a818e7141e20801b7dd8fd3c1b7dd8fd3c1a5faf1afb3dfe5c4a056c02cbdf4d0d76a73c4c9a4d04a81826a16fb4fe0af3b910f1cefdf96a26724212381c26a16fb4fe82f24e61f282f24e61f2eda48497e2d329686d5c6e48cb86d5ebb27cf462604910ceadb2be5abdd5eda48497e24bd80397151a0ab2c3b1fdf115940b169a8dbd1afdf115940b61dd670784dffa33101864a971f893d14b7374ba74c488679b475f5b5830e725df71109df6f3341a036619255475c578ae33e81591499c6dd62fb7c6f5720ce97d7eccf5b518e06e54631ea9952292f247696a61087a65938fd66c6053eee0002c3056d4298a4707132fbd7fe47b0bcfec4c53d75dcea4dc90c88aeab0fb4b32e5b7d1b4587eae88da8d9659ff7abeb0f1295c09a03c07e05a664dff31c14dbaa713b663b06c264f2977ee195719b7a1b7f981fbb20ee993719ae80f05deec2ecde235206363c255ec2ecde235206363c255e4ca09d7564e1de50e30e4ca09d756ad64784a2944ea4e1c514595b0c128ba09e7d1de08b73d196952598d3bca52598d3bcaad64784a29ecdcd0f621453c11f9f5c948201507453c11f9f5652cc2059794f93865e8b7b73fa7fe435b58f101e7e119b559fc9a6c7740cabf2a31c31d2a79f5be1fb9fac15976ffbfb7cfe2986ae612f4c73e149959be71c068607ef174c53658ab24c3bec1f38a0e688611eab393892b6524b63a80e5b7118eb6193928ea884e1927735f2621cab3d945bec9638b78367e10df1ffa4a47b0229e5f552a2997074161dfde55b2051f9d8ee41f58d0d4b052a57973011267afa16f98269943fd8af63c3fbb3cb8533fbb3cb853b7b73fa7fe95789536783132c1278118a4722d113132c12781e36c46a85ae13d560b6edd1905d2ef4d4de51f4fe366192fb5
2024-09-26 15:24:23 +05:30
d77637344d
log.warn anytime a column is relying on ArrayIngestMode.MVD ( #17164 )
...
* log.warn anytime a column is relying on ArrayIngestMode.MVD
2024-09-26 13:44:37 +05:30
a2b011cdcd
Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. ( #17125 )
...
changes:
* filter index processing is now automatically ordered based on estimated 'cost', which is approximated based on how many expected bitmap operations are required to construct the bitmap used for the 'offset'
* cursorAutoArrangeFilters context flag now defaults to true, but can be set to false to disable cost based filter index sorting
2024-09-25 23:11:26 -07:00
1b5b61ef7f
add multi-value string object vector matcher and expression vector object selectors ( #17162 )
2024-09-25 22:57:29 -07:00
2e531bb724
ScanQueryFrameProcessor: Close CursorHolders as we go along. ( #17152 )
...
* ScanQueryFrameProcessor: Close CursorHolders as we go along.
The change in #16533 added CursorHolders to the processor-level Closer.
This is problematic when running on an input channel: it means we started
keeping around all CursorHolders for all frames we process and closing them
when the channel is complete, rather than closing them as we go along.
This patch restores the prior behavior, where resources are closed as we go.
* Fix other call sites.
* Fix reference.
* Improvements.
2024-09-25 15:51:52 -07:00
203d6345af
docs: Separate section on ingesting MVDs in migration guide ( #17109 )
2024-09-25 14:45:25 -07:00
6f7e8ca74a
Decoupled planning: improve join predicate handling ( #17105 )
...
* enforces to only allow supported predicates in join conditions
* fixed a recursive query building issue by caching the `source` in `DruidQueryGenerator`
* moved `DruidAggregateRemoveRedundancyRule.instance` higher up; as if `CoreRules.AGGREGATE_EXPAND_DISTINCT_AGGREGATES` runs earlier the resulting `GROUPING` might become invalid
2024-09-25 16:00:25 +02:00
9132a65a48
Add `StreamSupervisor` interface ( #17151 )
...
Follow up to #17137 .
Instead of moving the streaming-only methods to the SeekableStreamSupervisor abstract class, this patch moves them to a separate StreamSupervisor interface. The reason is that the SeekableStreamSupervisor abstract class also has many other abstract methods. The StreamSupervisor interface on the other hand provides a minimal set of functions offering a good middle ground for any custom concrete implementation that doesn't require all the goodies from SeekableStreamSupervisor.
2024-09-25 14:52:39 +05:30
446ffc325c
MSQ window functions: Fix boost column not being written to the frame in window stage ( #17141 )
...
In WindowOperatorQueryFrameProcessor, currently we were not adding the partition boost column to the ColumnSelectorFactory used to create the FrameWriter. Because of this, the boost column was not being written to the frame.
2024-09-25 09:50:42 +05:30
464b153850
Make sketch encoding configurable ( #17086 )
...
Makes sketch encoding in MSQ configurable by the user. This would allow a user to configure the sketch encoding method for a specific query.
The default is octet stream encoding.
2024-09-25 09:15:32 +05:30
0ec7eb3ae5
use CastToObjectVectorProcessor for cast to string ( #17148 )
2024-09-24 15:21:33 -07:00
83299e9882
Miscellaneous cleanup in the supervisor API flow. ( #17144 )
...
Extracting a few miscellaneous non-functional changes from the batch supervisor branch:
- Replace anonymous inner classes with lambda expressions in the SQL supervisor manager layer
- Add explicit @Nullable annotations in DynamicConfigProviderUtils to make IDE happy
- Small variable renames (copy-paste error perhaps) and fix typos
- Add table name for this exception message: Delete the supervisor from the table[%s] in the database...
- Prefer CollectionUtils.isEmptyOrNull() over list == null || list.size() > 0. We can change the Precondition checks to throwing DruidException separately for a batch of APIs at a time.
2024-09-24 13:06:23 -07:00
d1bfabbf4d
inter-Extension dependency support ( #16973 )
...
* update docs for kafka lookup extension to specify correct extension ordering
* fix first line
* test with extension dependencies
* save work on dependency management
* working dependency graph
* working pull
* fix style
* fix style
* remove name
* load extension dependencies recursively
* generate depenencies on classloader creation
* add check for circular dependencies
* fix style
* revert style changes
* remove mutable class loader
* clean up class heirarchy
* extensions loader test working
* add unit tests
* pr comments
* fix unit tests
2024-09-24 14:17:33 -04:00
5c862f6ed9
Refactor: Move streaming supervisor methods to `SeekableStreamSupervisor` ( #17137 )
...
The current Supervisor interface is primarily focused on streaming use cases. However, as we introduce supervisors for non-streaming use cases, such as the recently added CompactionSupervisor (and the upcoming BatchSupervisor), certain operations like resetting offsets, checkpointing, task group handoff, etc., are not really applicable to non-streaming use cases.
So the methods are split between:
1. Supervisor: common methods that are applicable to both streaming and non-streaming use cases
2. SeekableStreamSupervisor: Supervisor + streaming-only operations. The existing streaming-only overrides exist along with the new abstract method public abstract LagStats computeLagStats(), for which custom implementations already exist in the concrete types
This PR is primarily a refactoring change with minimal functional adjustments (e.g., throwing an exception in a few places in SupervisorManager when the supervisor isn't the expected SeekableStreamSupervisor type).
2024-09-24 10:46:37 -07:00
4872f0457a
switch to using arrays by default ( #17133 )
2024-09-24 09:20:04 -07:00
1a13bd5485
Make the tooltip better and improve structure ( #17132 )
2024-09-24 09:19:52 -07:00
9670305669
Cleanup Coordinator logs, add duty status API ( #16959 )
...
Description
-----------
Coordinator logs are fairly noisy and don't give much useful information (see example below).
Even when the Coordinator misbehaves, these logs are not very useful.
Main changes
------------
- Add API `GET /druid/coordinator/v1/duties` that returns a status list of all duty groups currently running on the Coordinator
- Emit metrics `segment/poll/time`, `segment/pollWithSchema/time`, `segment/buildSnapshot/time`
- Remove redundant logs that indicate normal operation of well-tested aspects of the Coordinator
Refactors
---------
- Move some logic from `DutiesRunnable` to `CoordinatorDutyGroup`
- Move stats collection from `CollectSegmentAndServerStats` to `PrepareBalancerAndLoadQueues`
- Minor cleanup of class `DruidCoordinator`
- Clean up class `DruidCoordinatorRuntimeParams`
- Remove field `coordinatorStartTime`. Maintain start time in `MarkOvershadowedSegmentsAsUnused` instead.
- Remove field `MetadataRuleManager`. Pass supplier to constructor of applicable duties instead.
- Make `usedSegmentsNewestFirst` and `datasourcesSnapshot` as non-nullable as they are always required.
2024-09-24 19:46:22 +05:30
f576e299db
Allow MSQ engine only for compaction supervisors ( #17033 )
...
#16768 added the functionality to run compaction as a supervisor on the overlord.
This patch builds on top of that to restrict MSQ engine to compaction in the supervisor-mode only.
With these changes, users can no longer add MSQ engine as part of datasource compaction config,
or as the default cluster-level compaction engine, on the Coordinator.
The patch also adds an Overlord runtime property `druid.supervisor.compaction.engine=<msq/native>`
to specify the default engine for compaction supervisors.
Since these updates require major changes to existing MSQ compaction integration tests,
this patch disables MSQ-specific compaction integration tests -- they will be taken up in a follow-up PR.
Key changed/added classes in this patch:
* CompactionSupervisor
* CompactionSupervisorSpec
* CoordinatorCompactionConfigsResource
* OverlordCompactionScheduler
2024-09-24 17:19:16 +05:30
77a362c555
various fixes and improvements to vectorization fallback ( #17098 )
...
changes:
* add `ApplyFunction` support to vectorization fallback, allowing many of the remaining expressions to be vectorized
* add `CastToObjectVectorProcessor` so that vector engine can correctly cast any type
* add support for array and complex vector constants
* reduce number of cases which can block vectorization in expression planner to be unknown inputs (such as unknown multi-valuedness)
* fix array constructor expression, apply map expression to make actual evaluated type match the output type inference
* fix bug in array_contains where something like array_contains([null], 'hello') would return true if the array was a numeric array since the non-null string value would cast to a null numeric
* fix isNull/isNotNull to correctly handle any type of input argument
2024-09-24 04:29:08 -07:00
c1f8ae25b5
Support Iceberg ingestion from REST based catalogs ( #17124 )
...
Adds support to the iceberg input source to read from Iceberg REST Catalogs.
2024-09-23 22:13:24 -07:00
ed33dbb76d
Web console: add stage graph ( #17135 )
...
* add stage graph
* update snapshot
2024-09-23 21:19:13 -07:00
36dfff4b1a
Adding extra debug logs for the checkpoint logic ( #16321 )
...
Logging to understand checkpointing better in streaming ingestion
2024-09-24 09:38:46 +05:30
8eaac2c051
cgroup monitors: Add mem/disk/cpu usage metrics for V2 ( #16905 )
...
* cgroup monitors: Add mem/disk/cpu usage metrics for V2
* intellij inspection
* docs and checks
* fix-dos
* add comments
* comments
2024-09-23 20:32:01 -07:00
ba8245f114
Properly url encode schema and table names when syncing catalog ( #17131 )
...
* SQL syntax error should target USER persona
* * revert change to queryHandler and related tests, based on review comments
* * add test
* Properly encode schema and table names when syncing catalog
Previously table names and schema names were not being properly url
encododed when requested. Now they are.
2024-09-23 23:23:26 -04:00
37a2a12d79
rerwrite node so dynamic parameter applies to ingest node as well. ( #17126 )
2024-09-23 12:49:46 -07:00
67d361c9bf
Window Functions : Remove enable windowing flag ( #17087 )
2024-09-23 08:24:26 +02:00
40414cfe78
MSQ window functions: Reject MVDs during window processing ( #17036 )
...
* MSQ window functions: Reject MVDs during window processing
* MSQ window functions: Reject MVDs during window processing
* Remove parameterization from MSQWindowTest
2024-09-23 11:39:35 +05:30
df680bab05
Introduced `includeTrailerHeader` to enable `TrailerHeaders` in response ( #16672 )
...
Introduced includeTrailerHeader to enable TrailerHeaders in response
If enabled, a header X-Error-Message will be added to indicate reasons for partial results.
2024-09-21 14:29:37 +05:30
a4c971c373
Fix Multiple GC declared error when running Druid Cluster ( #17078 )
...
* Remove UseG1GC in all jvm.config to prevent multiple GC declared error
2024-09-20 11:06:13 +02:00
2eee470f6e
Support explain in decoupled planning and log native plan consistently with DruidHook ( #17101 )
...
* enables to use DruidHook for native plan logging
* qudiem tests doesn't necessarily need to run the query to get an explain - this helps during development as if there is a runtime issue it could still be explained in the test
2024-09-20 10:53:43 +02:00
635e418131
Support to parse numbers in text-based input formats ( #17082 )
...
Text-based input formats like csv and tsv currently parse inputs only as strings, following the RFC4180Parser spec).
To workaround this, the web-console and other tools need to further inspect the sample data returned to sample data returned by the Druid sampler API to parse them as numbers.
This patch introduces a new optional config, tryParseNumbers, for the csv and tsv input formats. If enabled, any numbers present in the input will be parsed in the following manner -- long data type for integer types and double for floating-point numbers, and if parsing fails for whatever reason, the input is treated as a string. By default, this configuration is set to false, so numeric strings will be treated as strings.
2024-09-19 13:21:18 -07:00
4f137d2700
hard-code compaction tasks to use ARRAY for multi-value handling to preserve order ( #17110 )
2024-09-19 11:56:12 -07:00
40572360c5
fix small README typo ( #17114 )
2024-09-19 10:01:42 -07:00
3d45f9829c
Use the whole frame when writing rows. ( #17094 )
...
* Use the whole frame when writing rows.
This patch makes the following adjustments to enable writing larger
single rows to frames:
1) RowBasedFrameWriter: Max out allocation size on the final doubling.
i.e., if the final allocation "naturally" would be 1 MiB but the
max frame size is 900 KiB, use 900 KiB rather than failing the 1 MiB
allocation.
2) AppendableMemory: In reserveAdditional, release the last block if it
is empty. This eliminates waste when a frame writer uses a
successive-doubling approach to find the right allocation size.
3) ArenaMemoryAllocator: Reclaim memory from the last allocation when
the last allocation is closed.
Prior to these changes, a single row could be much smaller than the
frame size and still fail to be added to the frame.
* Style.
* Fix test.
2024-09-19 00:42:03 -07:00
b9a4c73e52
Window Functions : Improve performance by comparing Strings in frame bytes without converting them ( #17091 )
2024-09-19 09:36:28 +02:00
8d1e596740
PostJoinCursor should never advance without interruption ( #17099 )
2024-09-19 09:10:59 +02:00
953fe11e31
gRPC query extension ( #15982 )
...
Revives #14024 and additionally supports,
Native queries
gRPC health check endpoint
This PR doesn't have the shaded module for packaging gRPC and Guava libraries since grpc-query module uses the same Guava version as that of Druid.
The response is gRPC-specific. It provides the result schema along with the results as a binary "blob". Results can be in CSV, JSON array lines or as an array of Protobuf objects. If using Protobuf, the corresponding class must be installed along with the gRPC query extension so it is available to the Broker at runtime.
2024-09-19 12:16:32 +05:30
d1bd6a8156
Update doc for allowedHeaders ( #17045 )
...
Update doc for allowedHeaders and make allowedHeaders more restrictive
2024-09-19 08:37:39 +05:30
2d2882cdfe
Add test for exceptions in FutureUtils.transformAsync. ( #17106 )
...
Adds an additional test case to FutureUtilsTest.
2024-09-18 16:08:47 -07:00
ca0cb64ee8
MSQ: Fix calculation of suggested memory in WorkerMemoryParameters. ( #17108 )
...
The "suggested server memory" figure needs to take into account
maxConcurrentStages. The fix here does not affect the main memory
calculations, but it does affect the accuracy of error messages.
2024-09-18 16:08:35 -07:00
39723e5401
Update note about `sys.tasks` table ( #17096 )
...
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-09-18 11:02:45 -07:00
43d790fdb7
Normalize schema fingerprint for column permutations ( #17044 )
...
Parent issue: #14989
It is possible for the order of columns to vary across segments especially during realtime ingestion.
Since, the schema fingerprint is sensitive to column order this leads to creation of a large number of segment schema in the metadata database for essentially the same set of columns.
This is wasteful, this patch fixes this problem by computing schema fingerprint on lexicographically sorted columns. This would result in creation of a single schema in the metadata database with the first observed column order for a given signature.
2024-09-18 11:37:06 +05:30
Adarsh Sanjeev
Shivam Garg
Sree Charan Manamala
Clint Wylie
Vadim Ogievetsky
Gian Merlino
Zoltan Haindrich
Cece Mei
Victoria Lim
Abhishek Radhakrishnan
Akshat Jain
George Shiqi Wu
Kashif Faraz
Vishesh Garg
Atul Mohan
PANKAJ KUMAR
Adithya Chakilam
zachjsh
Vivek Dhiman
Virushade
Ben Smithgall
Abhishek Agarwal
Rishabh Singh
Pranav