14799 Commits

Author SHA1 Message Date
Rishabh Singh
74422b58f5
Emit disk spill and merge buffer utilisation metrics for GroupBy queries (#17360)
This change is to emit following metrics as part of GroupByStatsMonitor monitor,
mergeBuffer/used -> Number of merge buffers used.
mergeBuffer/acquisitionTimeNs -> Total time required to acquire merge buffer.
mergeBuffer/acquisition -> Number of queries that acquired a batch of merge buffers.
groupBy/spilledQueries -> Number of queries that spilled onto the disk.
groupBy/spilledBytes-> Spilled bytes on the disk.
groupBy/mergeDictionarySize -> Size of the merging dictionary.
2024-11-22 14:22:03 +05:30
Adarsh Sanjeev
df649c0bbd
Refactors (#17498)
Follow-up PR to #17493 to address pending unaddressed comments.
2024-11-22 09:22:38 +05:30
Katya Macedo
bd93d0046d
Docs: update text and example (#17480)
* Docs: update text and example

* Update after review

* Update the spelling file

* Update text for clarity

* Update after review
2024-11-21 08:40:41 -08:00
Vivek Dhiman
bb44f85bb6
Updated error response to hide error stack in case of JsonMappingException (#16821)
Added flag druid.server.http.showDetailedJsonMappingError similar druid.server.http.showDetailedJettyError to configure error message detail.
2024-11-21 19:11:48 +05:30
Zoltan Haindrich
5ca3f14813 more sophisiticated filter 2024-11-21 13:29:25 +00:00
Adarsh Sanjeev
2726c6f388
Minor refactors to processing
Some refactors across druid to clean up the code and add utility functions where required.
2024-11-21 15:37:55 +05:30
Akshat Jain
17215cd677
Remove support for Java 8 (#17466)
All JDK 8 based CI checks have been removed.
    Images used in Dockerfile(s) have been updated to Java 17 based images.
    Documentation has been updated accordingly.
2024-11-21 15:33:08 +05:30
Zoltan Haindrich
bdfcef8220 add note; conditional 2024-11-21 08:12:49 +00:00
Zoltan Haindrich
0c6b40c138 update 2024-11-20 16:30:24 +00:00
Zoltan Haindrich
d4ebca947c enable trimmer to create blinding project 2024-11-20 16:27:15 +00:00
Zoltan Haindrich
93bae231f1 Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-20 14:41:09 +00:00
Zoltan Haindrich
4cc00777e3 Merge remote-tracking branch 'apache/master' into rename-d1-dbl1 2024-11-20 14:07:12 +00:00
Adithya Chakilam
c1d6328249
StreamingTaskRunner: Close the rejection period updater executor service (#17490) 2024-11-19 12:49:20 -08:00
zachjsh
8853c7e5c6
Add ingest/notices/queueSize and ingest/pause/time to statsd emitter (#17487)
* SQL syntax error should target USER persona

* * revert change to queryHandler and related tests, based on review comments

* * add test

* * add `ingest/notices/queueSize` and `ingest/pause/time` to statsd emitter

* * add taskStatus dimension to `service/heartbeat` metric

* Revert "* add taskStatus dimension to `service/heartbeat` metric"

This reverts commit cfb02a2813b24243ea0703def7a0dddd948bee00.
2024-11-18 20:58:00 -05:00
Adithya Chakilam
6f436301be
supervisor: make rejection periods work with stopTasksCount (#17442)
* kafka-indexing: Report consumer io time

* commit

* backward

* tests

* remove unwanted changes

* comments

* comments

* coverage

* change name

* fixes

* fixes

* comments
2024-11-18 13:12:24 -08:00
Clint Wylie
24a1fafaa7
projection segment merge fixes (#17460)
changes:
* fix issue when merging projections from multiple-incremental persists which was hoping that some 'dim conversion' buffers were not closed, but they already were (by the merging iterator). fix involves selectively persisting these conversion buffers to temp files in the segment write out directory and mapping them and tying them to the segment level closer so that they are available after the lifetime of the parent merger
* modify auto column serializers to use segment write out directory for temp files instead of java.io.tmpdir
* fix queryable index projection to not put the time-like column as a dimension, instead only adding it as __time
* use smoosh for temp files so can safely write any Serializer to a temp smoosh
2024-11-15 16:46:04 -08:00
Zoltan Haindrich
79c489cb88 ;stuf 2024-11-15 15:46:49 +00:00
Zoltan Haindrich
400c05e0ca last trim possibly nbot needed 2024-11-15 15:29:11 +00:00
Zoltan Haindrich
0378e65216 trimmed 2024-11-15 15:22:07 +00:00
Zoltan Haindrich
afad58ce80 Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-15 14:41:05 +00:00
Zoltan Haindrich
d2e97da900 add back dummyproject 2024-11-15 14:40:13 +00:00
Zoltan Haindrich
7e46880aac cleanup 2024-11-15 14:23:56 +00:00
Zoltan Haindrich
8800763484 correct indent 2024-11-15 14:20:17 +00:00
Zoltan Haindrich
797f35de1e celanup 2024-11-15 14:19:26 +00:00
Zoltan Haindrich
3aff161271 fix fail 2024-11-15 14:12:39 +00:00
Zoltan Haindrich
f9823b0384 Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-15 11:23:15 +00:00
Zoltan Haindrich
f89cc82fbb update 2 more tests 2024-11-15 07:58:14 +00:00
Rishabh Singh
7f335ff486
Resolve CVEs: Upgrade jetty version and suppress azure cve (#17385) 2024-11-15 10:55:02 +05:30
Zoltan Haindrich
351615a3c9 Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-14 17:09:26 +00:00
Zoltan Haindrich
44523a8ce3 update 2024-11-14 17:09:08 +00:00
Zoltan Haindrich
acc9eb6ef9 Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-14 12:03:01 +00:00
Zoltan Haindrich
1e5c37b66f inline const back 2024-11-14 11:57:57 +00:00
Zoltan Haindrich
6ee980e8ce update unnest.iq 2024-11-14 11:56:48 +00:00
Zoltan Haindrich
12373a877e fix one 2024-11-14 11:55:21 +00:00
Zoltan Haindrich
a075cb9c12 removal of some projects 2024-11-14 11:51:43 +00:00
Zoltan Haindrich
23e4dab3df trim-trim 2024-11-14 11:50:17 +00:00
Zoltan Haindrich
dcdd7d4a92 fix final test 2024-11-14 11:44:17 +00:00
Zoltan Haindrich
9afdfb2dcf updates 2024-11-14 10:03:28 +00:00
Zoltan Haindrich
9448ed3825 enable trim for unnest 2024-11-14 09:53:17 +00:00
Zoltan Haindrich
fd564fda72 cleanup/enhance rule order to simplify 2024-11-14 09:07:44 +00:00
Zoltan Haindrich
f4d7ec2695 cleanup/enhance rule order to simplify 2024-11-14 07:51:22 +00:00
Katya Macedo
75d9ece665
Docs: update descriptions and default values (#17473) 2024-11-13 16:29:27 -08:00
Zoltan Haindrich
fbfed8a07f update 2024-11-13 18:09:50 +00:00
Zoltan Haindrich
7d9693bd91 aa 2024-11-13 18:00:52 +00:00
zachjsh
b0c73d7c2a
Add 'ingest/notices/time' metric to statsd emitter (#17468)
* SQL syntax error should target USER persona

* * revert change to queryHandler and related tests, based on review comments

* * add test

* Add 'ingest/notices/time' metric to statsd emitter

This metric gives the milliseconds taken to process a notice by the supervisor.
2024-11-13 12:17:01 -05:00
Zoltan Haindrich
96f572e679 updates/etc 2024-11-13 17:03:06 +00:00
Zoltan Haindrich
0d09c18cab undo trials; re-introduce expr if needed (will be trimmed in most cases) 2024-11-13 16:48:31 +00:00
Zoltan Haindrich
8dfd57f922 update test 2024-11-13 16:30:08 +00:00
Zoltan Haindrich
1379c9aa2c Merge branch 'rename-d1-dbl1' into unnest-relfieldtrimmer-unnestfieldtype 2024-11-13 16:24:35 +00:00
Zoltan Haindrich
247ee17f17 fixups after merge 2024-11-13 15:35:00 +00:00