9932f2e70a
docs: concurrent append and replace is gA ( #17269 )
2024-10-08 07:55:55 +05:30
f42ecc9f25
Fail concurrent replace tasks with finer segment granularity than append ( #17265 )
2024-10-08 07:35:13 +05:30
5d7c7a87ec
Add maximumCapacity to taskRunner ( #17107 )
...
* Add maximumCapacity to taskRunner
* fix tests
* pr comments
2024-10-07 15:03:51 -04:00
ff97c67945
Fix batch segment allocation failure with replicas ( #17262 )
...
Fixes #16587
Streaming ingestion tasks operate by allocating segments before ingesting rows.
These allocations happen across replicas which may send different requests but
must get the same segment id for a given (datasource, interval, version, sequenceName)
across replicas.
This patch fixes the bug by ignoring the previousSegmentId when skipLineageCheck is true.
2024-10-07 19:52:38 +05:30
6a4352f466
When removeNullBytes is set, length calculations did not take into account null bytes. ( #17232 )
...
* When replaceNullBytes is set, length calculations did not take into account null bytes.
2024-10-07 18:02:52 +05:30
c9201ad658
Minor refactors to processing module ( #17136 )
...
Refactors a few things.
- Adds SemanticUtils maps to columns.
- Add some addAll functions to reduce duplication, and for future reuse.
- Refactor VariantColumnAndIndexSupplier to only take a SmooshedFileMapper instead.
- Refactor LongColumnSerializerV2 to have separate functions for serializing a value and null.
2024-10-07 13:18:35 +05:30
7e35e50052
Fix issues with MSQ Compaction ( #17250 )
...
The patch makes the following changes:
1. Fixes a bug causing compaction to fail on array, complex, and other non-primitive-type columns
2. Updates compaction status check to be conscious of partition dimensions when comparing dimension ordering.
3. Ensures only string columns are specified as partition dimensions
4. Ensures `rollup` is true if and only if metricsSpec is non-empty
5. Ensures disjoint intervals aren't submitted for compaction
6. Adds `compactionReason` to compaction task context.
2024-10-06 21:48:26 +05:30
7d9e6d36fd
Upgraded Protobuf to 3.25.5 ( #17249 )
...
* Bump com.google.protobuf:protobuf-java from 3.24.0 to 3.25.5
Bumps [com.google.protobuf:protobuf-java](https://github.com/protocolbuffers/protobuf ) from 3.24.0 to 3.25.5.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases )
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl )
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.24.0...v3.25.5 )
---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* Updated the license
* Updated licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-06 12:34:02 +05:30
0bd13bcd51
Projections prototype ( #17214 )
2024-10-05 04:38:57 -07:00
04fe56835d
add druid.expressions.allowVectorizeFallback and default to false ( #17248 )
...
changes:
adds ExpressionProcessing.allowVectorizeFallback() and ExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false until few remaining bugs can be fixed (mostly complex types and some odd interactions with mixed types)
add cannotVectorizeUnlessFallback functions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future
2024-10-05 12:42:42 +05:30
d1709a329f
Dart: Skip final getCounters, postFinish to idle historicals. ( #17255 )
...
In a Dart query, all Historicals are given worker IDs, but not all of them
are going to actually be started or receive work orders.
Attempting to send a getCounters or postFinish command to a worker that
never received a work order is not only wasteful, but it causes errors due
to the workers not knowing about that query ID.
2024-10-04 23:05:21 -07:00
babf7f2ef6
Web console: don't assume that activeTasks is an array ( #17254 )
2024-10-04 16:01:13 -07:00
2ffe7b177c
Explore view fix spin when applying defaults ( #17252 )
2024-10-04 13:02:15 -07:00
93b5a8326b
Upgrade commons-io to 2.17.0 ( #17227 )
2024-10-04 09:56:56 +02:00
e41648f5bf
Simplify release script find-missing-backports.py ( #17218 )
2024-10-04 10:37:04 +05:30
d1bc369f06
Web console: Final explore QA pass ( #17240 )
...
* cleanup
* remove redundancy
* aggregate works for multiple queries
2024-10-03 19:52:17 -07:00
b9634a8613
SuperSorter: Don't set allDone if it's already set. ( #17238 )
...
This fixes a race where, if there is no output at all, setAllDoneIfPossible
could be called twice (once when the output partitions future resolves, and
once when the batcher finishes). If the calls happen in that order, it would
try to create nil output channels both times, resulting in a "Channel already set"
error.
2024-10-04 06:41:16 +05:30
acd973273f
Docs: adds MSQ examples to front coded dict. migration ( #17236 )
...
* add msq example
* adjust json formatting
2024-10-03 16:33:34 -07:00
fb94428a58
Web console: Explore view QA with live data ( #17234 )
...
* Explore view QA with live data
* update snapshots
* add t for preview also
* use pulse icon consistently
2024-10-03 13:50:08 -07:00
fc00664760
KafkaInputFormat: Fix handling of CSV/TSV keyFormat. ( #17226 )
...
* KafkaInputFormat: Fix handling of CSV/TSV keyFormat.
Follow-up to #16630 , which fixed a similar issue for the valueFormat.
* Simplify.
2024-10-03 13:05:09 -07:00
db7cc4634c
Dart: Smoother handling of stage early-exit. ( #17228 )
...
Stages can be instructed to exit before they finish, especially when a
downstream stage includes a "LIMIT". This patch has improvements related
to early-exiting stages.
Bug fix:
- WorkerStageKernel: Don't allow fail() to set an exception if the stage is
already in a terminal state (FINISHED or FAILED). If fail() is called while
in a terminal state, log the exception, then throw it away. If it's a
cancellation exception, don't even log it. This fixes a bug where a stage
that exited early could transition to FINISHED and then to FAILED, causing
the overall query to fail.
Performance:
- DartWorkerManager previously sent stopWorker commands to workers
even when "interrupt" was false. Now it only sends those commands when
"interrupt" is true. The method javadoc already claimed this is what the
method did, but the implementation did not match the javadoc. This reduces
the number of RPCs by 1 per worker per query.
Quieter logging:
- In ReadableByteChunksFrameChannel, skip logging exception from setError if
the channel has been closed. Channels are closed when readers are done with
them, so at that point, we wouldn't be interested in the errors.
- In RunWorkOrder, skip calling notifyListener on failure of the main work,
in the case when stop() has already been called. The stop() method will
set its own error using CanceledFault. This enables callers to detect
when a stage was canceled vs. failed for some other reason.
- In WorkerStageKernel, skip logging cancellation errors in fail(). This is
made possible by the previous change in RunWorkOrder.
2024-10-03 20:09:02 +05:30
421aae39ad
Upgrade avro - minor version ( #17230 )
2024-10-03 18:02:11 +05:30
316f8c81d2
RunWorkOrder: Account for two simultaneous statistics collectors. ( #17216 )
...
* RunWorkOrder: Account for two simultaneous statistics collectors.
As a follow up to #17057 , divide the amount of partitionStatsMemory
by two, to account for the fact that there are at some times going to
be two copies of the full collector. First there will be one for processors
and one for the accumulated collector. Then, after the processor ones are
GCed, a snapshot of the accumulated collector will be created.
Also includes an optimization to "addAll" for the two KeyCollectors,
for the case where we're adding into an empty collector. This is always
going to happen once per stage due to the "withAccumulation" call.
* Fix missing variable.
* Don't divide by numProcessingThreads twice.
* Fix test.
2024-10-03 16:25:01 +05:30
edc235cfe1
WindowOperatorQueryFrameProcessor: Avoid unnecessary re-runs of runIncrementally() ( #17211 )
2024-10-03 15:33:50 +05:30
8c4db8aeed
explore QA ( #17225 )
2024-10-02 23:05:19 -07:00
135ca8f6a7
WindowOperatorQueryFrameProcessor: Fix frame writer capacity issues + adhere to FrameProcessor's contract ( #17209 )
...
This PR fixes the above issue by maintaining the state of last rowId flushed to output channel, and triggering another iteration of runIncrementally() method if frame writer has rows pending flush to the output channel.
The above is done keeping in mind FrameProcessor's contract which enforces that we should write only a single frame to each output channel in any given iteration of runIncrementally().
2024-10-03 10:39:22 +05:30
fbc1221837
DartTableInputSpecSlicer: Fix for TLS workers. ( #17224 )
...
We should use getHost(), which returns TLS if configured or plaintext
otherwise. getHostAndPort() returns plaintext only.
2024-10-03 11:01:11 +08:00
715ae5ece0
Web console: misc fixes to the Explore view ( #17213 )
...
* make record table able to hide column
* stickyness
* refactor query log
* fix measure drag
* start nested column dialog
* nested expand
* fix filtering on Measures
* use output name
* fix scrolling
* select all / none
* use ARRAY_CONCAT_AGG
* no need to limit if aggregating
* remove magic number
* better search
* update arg list
* add, don't replace
2024-10-02 08:52:08 -07:00
e5d027ee1c
Skip generating task context reports for sub tasks ( #17219 )
...
* Skip task context for sub tasks
* DRY a little + skip context for live report
2024-10-02 09:32:50 -04:00
65277b17a9
Decoupled planning: add support for unnest ( #17177 )
...
* adds support for `UNNEST` expressions
* introduces `LogicalUnnestRule` to transform a `Correlate` doing UNNEST into a `LogicalUnnest`
* `UnnestInputCleanupRule` could move the final unnested expr into the `LogicalUnnest` itself (usually its an `mv_to_array` expression)
* enhanced source unwrapping to utilize `FilteredDataSource` if it looks right
2024-10-02 08:54:56 +02:00
c8529294eb
Web console: add support for Dart engine ( #17147 )
...
* add console support for Dart engine
This reverts commit 6e46edf15dd55e5c51a1a4068e83deba4f22529b.
* feedback fixes
* surface new fields
* prioratize error over results
* better metadata refresh
* feedback fixes
2024-10-01 17:53:36 -07:00
1fc82a96bd
docs: update future development blurbs ( #16939 )
...
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-10-01 15:02:05 -07:00
878adff9aa
MSQ profile for Brokers and Historicals. ( #17140 )
...
This patch adds a profile of MSQ named "Dart" that runs on Brokers and
Historicals, and which is compatible with the standard SQL query API.
For more high-level description, and notes on future work, refer to #17139 .
This patch contains the following changes, grouped into packages.
Controller (org.apache.druid.msq.dart.controller):
The controller runs on Brokers. Main classes are,
- DartSqlResource, which serves /druid/v2/sql/dart/.
- DartSqlEngine and DartQueryMaker, the entry points from SQL that actually
run the MSQ controller code.
- DartControllerContext, which configures the MSQ controller.
- DartMessageRelays, which sets up relays (see "message relays" below) to read
messages from workers' DartControllerClients.
- DartTableInputSpecSlicer, which assigns work based on a TimelineServerView.
Worker (org.apache.druid.msq.dart.worker)
The worker runs on Historicals. Main classes are,
- DartWorkerResource, which supplies the regular MSQ WorkerResource, plus
Dart-specific APIs.
- DartWorkerRunner, which runs MSQ worker code.
- DartWorkerContext, which configures the MSQ worker.
- DartProcessingBuffersProvider, which provides processing buffers from
sliced-up merge buffers.
- DartDataSegmentProvider, which provides segments from the Historical's
local cache.
Message relays (org.apache.druid.messages):
To avoid the need for Historicals to contact Brokers during a query, which
would create opportunities for queries to get stuck, all connections are
opened from Broker to Historical. This is made possible by a message relay
system, where the relay server (worker) has an outbox of messages.
The relay client (controller) connects to the outbox and retrieves messages.
Code for this system lives in the "server" package to keep it separate from
the MSQ extension and make it easier to maintain. The worker-to-controller
ControllerClient is implemented using message relays.
Other changes:
- Controller: Added the method "hasWorker". Used by the ControllerMessageListener
to notify the appropriate controllers when a worker fails.
- WorkerResource: No longer tries to respond more than once in the
"httpGetChannelData" API. This comes up when a response due to resolved future
is ready at about the same time as a timeout occurs.
- MSQTaskQueryMaker: Refactor to separate out some useful functions for reuse
in DartQueryMaker.
- SqlEngine: Add "queryContext" to "resultTypeForSelect" and "resultTypeForInsert".
This allows the DartSqlEngine to modify result format based on whether a "fullReport"
context parameter is set.
- LimitedOutputStream: New utility class. Used when in "fullReport" mode.
- TimelineServerView: Add getDruidServerMetadata as a performance optimization.
- CliHistorical: Add SegmentWrangler, so it can query inline data, lookups, etc.
- ServiceLocation: Add "fromUri" method, relocating some code from ServiceClientImpl.
- FixedServiceLocator: New locator for a fixed set of service locations. Useful for
URI locations.
2024-10-01 14:38:55 -07:00
3d56fa6f56
Improve logging to include taskId in segment handoff notifier thread ( #17185 )
2024-10-01 15:34:39 +05:30
f33f60b32e
fix input step typo ( #17202 )
2024-09-30 21:26:22 -07:00
5ad6ed0b73
Refactor pod template logic to make it easier to test ( #17178 )
...
* Refactoring of pod template logic
* fix javadoc
* Fix intellij
* remove unneeded throw
* PR comments
* fix style
* Fix unit tests
2024-09-30 15:34:13 -07:00
28fead58b4
MSQ: Use task context flag useConcurrentLocks to determine task lock type ( #17193 )
2024-09-30 21:15:25 +05:30
15987f51f1
Update Delta Kernel to 3.2.1 ( #17179 )
...
Updated Delta Kernel from 3.2.0 to 3.2.1. This upstream version bump contains fixes to reading long columns, class loader and better retry mechanism when reading checkpoint files.
2024-09-30 07:22:49 -07:00
d982727a29
Web console: revamp the experimental explore view ( #17180 )
...
* explore revamp
* remove ToDo
* fix CodeQL
* add tooltips
* show issue on echart chars
* fix: browser back does not refresh chart
* fix maxRows 0
* be more resiliant to missing __time
2024-09-29 23:15:21 -07:00
95fb813b6d
Fix bug with while adding DistinctSketches ( #17184 )
...
* Fix bug with checking the incorrect key
* Add tests
2024-09-30 11:35:32 +05:30
ab361747a8
Migrated commons-lang usages to commons-lang3 ( #17156 )
2024-09-28 10:28:11 +02:00
661614129e
Window Functions : Context Parameter to Enable Transfer of RACs over wire ( #17150 )
2024-09-28 08:04:22 +02:00
f8a72b987a
read metadata in SimpleQueryableIndex if available to compute segment ordering ( #17181 )
2024-09-27 19:39:03 -07:00
7417ead3af
Web console: Fixed sampling for delta source in classic data loader and MSQ ( #17160 )
...
Added the delta tile to MSQ and made sure it works there and classic data loader. It is special because it always assumes a parquet format so it has to skip some steps in the code. Now the framework is there to add more tiles that force a specific format
2024-09-27 07:51:24 -07:00
dc223f22db
SQL: Use regular filters for time filtering in subqueries. ( #17173 )
...
* SQL: Use regular filters for time filtering in subqueries.
Using the "intervals" feature on subqueries, or any non-table, should be
avoided because it isn't a meaningful optimization in those cases, and
it's simpler for runtime implementations if they can assume all filters
are located in the regular filter object.
Two changes:
1) Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended
to return false for QueryDataSource, but actually returned true.
2) Add a validation to ScanQueryFrameProcessor to ensure that when running
on an input channel (which would include any subquery), the query has
"intervals" set to ONLY_ETERNITY.
Prior to this patch, the new test case in testTimeFilterOnSubquery would
throw a "Can only handle a single interval" error in the native engine,
and "QueryNotSupported" in the MSQ engine.
* Mark new case as having extra columns in decoupled mode.
* Adjust test.
2024-09-27 10:32:30 +05:30
157fe1bc1f
fix a mistake in CursorGranularizer to check doneness after advance ( #17175 )
...
Fixes a mistake introduced in #16533 which can result in CursorGranularizer incorrectly trying to get values from a selector after calling cursor.advance because of a missing check for cursor.isDone
2024-09-27 09:36:05 +05:30
6ee9e42f03
fix issue with ScanQueryFrameProcessor cursor build not adjusting intervals ( #17168 )
...
* fix issue with ScanQueryFrameProcessor cursor build not adjusting intervals
* all hail the robot overlords
* style bot
2024-09-26 21:09:14 +05:30
ca96629207
Add kttm-nested dataset and some usage sessions of it ( #16867 )
...
* Add kttm-nested dataset and some usage sessions of it
Squashed commit of the following:
commit 3075269bdefbbfeeb3c7958751d9c44dd017944f
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:48:19 2024 +0000
rename files
commit ad8d37a235a89cdf92ea3c1e9fdda9dff87fa7ce
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:44:09 2024 +0000
rename
commit 8fc4efe19eba10e7d6a82042aedec0ec7c04c77c
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:43:31 2024 +0000
cleanup kttm loader
commit 113a4b6f4c0a450581600aa7483a5163ee00e937
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:38:10 2024 +0000
weed out infoschema stuff
commit 1eb9b5070c16a42a19ab9774d2b0378e090097df
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 13:17:27 2024 +0000
reduce crap
commit 1745ececc77cae93ed9130fd6e9e83fddd44471a
Merge: 9dd2bff6cc 1f8473d82b
Author: Zoltan Haindrich <kirk@rxd.hu>
Date: Thu Aug 8 09:49:52 2024 +0000
Merge branch 'quidem-msq' into quidem-qa
commit 9dd2bff6ccdb30c927a256ff7ab6ebf491508008
Merge: 7120f84e36 fda0d63e444e5d7c9c216cd8c6be224e5d7c9c216cef3eb6e873904b520dad5768e512039d9c88f87a6ba55fa25bc32d56785bc32d56788a92ec6b2edbe2674971ebea34a814dbe267497123c6f3bd9677408329953e2b59f80834651e70e534651e70e5449a7f3a7369a39a42da2700557a552700557a5542cc5d62a8d5b82af4a921016a28a23f507d864819fe5867b1941c39aae500ee182d7591d5d14bf129b2b559d96d38e8f075c3aa033e146d38e8f075f867db774a61c3b16b17130252bb5e3b77784e6e22d8a4b87234ab9113995e5c94d6d893457c6b3e12cfde805ec8f91478106d339d1706181458c8734c722f271f10639487491063948749c40474285cf4af51ef7fbc70443c7f841ab462dd929e68c11afda0d63e44fda0d63e44bb23ace51826e3c44f4b929e68c11a436ba1881570e46eadb9090f937d585e81a026e9bb23ace518bb23ace518e6add9ea84a03fb49f4fc7eacd079e5e81a026e95f6290eb54a03fb49f4fa03fb49f4f5f6290eb54de207c7295b1ab252b314157a8f105ca121aa083ca121aa083eb2a047e4b57980066b3954aaafe0c57980066b37f99ee24d7b345dd9d03df422456859ac26e3a8978b75d3e8ef6cc540368ce667eeb5e4157a8f105ed9ef1f6358bb38a04a5d705c2759ba489e19242d010b488a77428da51de0be1f81d7e7cfbfdc3eee60a200d95a9dcb2da4631e97324ced227029b6bd227029b6bf7247e1bb7b38935a45031e97324cee2a54b5758361149b097bc7174cb6a9cf723adae7a34b6e092d216b934fc76ff3f26e1e388140b2a76ff3f26e1eb4fd9f66ce388140b2a47aeb016df70ff2a3e9706b68b6c8906b68b6c8982436df585b53c26f5c570ff2a3e978b26e490e9c59f1adcc895ca0a9f5db100e982a40811d801fb97c32ca3de6790f9cf8b51d465df6d0eaf4c61b9f3cf77811542b30865128ada2ff23882436df5852a590eb3ae20c543dfc201d67ae54320c543dfc2c559bc3422cc4e0adcbfb8992434a9c559bc34229a4d04a818e7141e20801b7dd8fd3c1b7dd8fd3c1a5faf1afb3dfe5c4a056c02cbdf4d0d76a73c4c9a4d04a81826a16fb4fe0af3b910f1cefdf96a26724212381c26a16fb4fe82f24e61f282f24e61f2eda48497e2d329686d5c6e48cb86d5ebb27cf462604910ceadb2be5abdd5eda48497e24bd80397151a0ab2c3b1fdf115940b169a8dbd1afdf115940b61dd670784dffa33101864a971f893d14b7374ba74c488679b475f5b5830e725df71109df6f3341a036619255475c578ae33e81591499c6dd62fb7c6f5720ce97d7eccf5b518e06e54631ea9952292f247696a61087a65938fd66c6053eee0002c3056d4298a4707132fbd7fe47b0bcfec4c53d75dcea4dc90c88aeab0fb4b32e5b7d1b4587eae88da8d9659ff7abeb0f1295c09a03c07e05a664dff31c14dbaa713b663b06c264f2977ee195719b7a1b7f981fbb20ee993719ae80f05deec2ecde235206363c255ec2ecde235206363c255e4ca09d7564e1de50e30e4ca09d756ad64784a2944ea4e1c514595b0c128ba09e7d1de08b73d196952598d3bca52598d3bcaad64784a29ecdcd0f621453c11f9f5c948201507453c11f9f5652cc2059794f93865e8b7b73fa7fe435b58f101e7e119b559fc9a6c7740cabf2a31c31d2a79f5be1fb9fac15976ffbfb7cfe2986ae612f4c73e149959be71c068607ef174c53658ab24c3bec1f38a0e688611eab393892b6524b63a80e5b7118eb6193928ea884e1927735f2621cab3d945bec9638b78367e10df1ffa4a47b0229e5f552a2997074161dfde55b2051f9d8ee41f58d0d4b052a57973011267afa16f98269943fd8af63c3fbb3cb8533fbb3cb853b7b73fa7fe95789536783132c1278118a4722d113132c12781e36c46a85ae13d560b6edd1905d2ef4d4de51f4fe366192fb5
2024-09-26 15:24:23 +05:30
d77637344d
log.warn anytime a column is relying on ArrayIngestMode.MVD ( #17164 )
...
* log.warn anytime a column is relying on ArrayIngestMode.MVD
2024-09-26 13:44:37 +05:30
a2b011cdcd
Incorporate `estimatedComputeCost` into all `BitmapColumnIndex` classes. ( #17125 )
...
changes:
* filter index processing is now automatically ordered based on estimated 'cost', which is approximated based on how many expected bitmap operations are required to construct the bitmap used for the 'offset'
* cursorAutoArrangeFilters context flag now defaults to true, but can be set to false to disable cost based filter index sorting
2024-09-25 23:11:26 -07:00
317brian
AmatyaAvadhanula
George Shiqi Wu
Karan Kumar
Adarsh Sanjeev
Vishesh Garg
Shivam Garg
Clint Wylie
Gian Merlino
Vadim Ogievetsky
Kashif Faraz
Charles Smith
Abhishek Agarwal
Akshat Jain
Arun Ramani
Zoltan Haindrich
Hardik Bajaj
Abhishek Radhakrishnan
Sree Charan Manamala
Cece Mei