Commit Graph

14265 Commits

Author SHA1 Message Date
Clint Wylie 82ac48786b
document arrayContainsElement filter (#15455) 2023-12-07 00:14:00 -08:00
Clint Wylie 557f3f6f57
add array column type support to EXTEND operator (#15458) 2023-12-06 23:21:35 -08:00
Benjamin Hopp fea53c7084
Re-arranging sections for append and replace docs. (#15497) 2023-12-06 13:13:05 -08:00
Rishabh Singh 6a64f72c67
Lookup on incomplete partition set in SegmentMetadataQuerySegmentWalker (#15496)
Description
With CentralizedDatasourceSchema (#14989) feature enabled, metadata for appended segments was not being refreshed. This caused numRows to be 0 for the new segments and would probably cause the datasource schema to not include columns from the new segments.

Analysis
The problem turned out in the new QuerySegmentWalker implementation in the Coordinator. It first finds the segment to be queried in the Coordinator timeline. Then it creates a new timeline of the segments present in the timeline.
The problem was that it is looking up complete partition set in the new timeline. Since the appended segments by themselves do not make a complete partition set, no SegmentMetadataQuery were executed.
2023-12-06 15:25:28 +05:30
Gian Merlino 6f51155ccb
Fix NullFilter getDimensionRangeSet. (#15500)
It wasn't checking the column name, so it would return a domain regardless
of the input column. This means that null filters on data sources with range
partitioning would lead to excessive pruning of segments, and therefore
missing results.
2023-12-06 15:09:59 +05:30
Abhishek Radhakrishnan f4949afdd7
clarify and fixup typos related to unused segments in docs and javadocs. (#15498) 2023-12-05 22:30:32 -08:00
Xavier Léauté ae6893edc3
unpin guava related dependabot dependencies (#15494)
Several dependabot ignore directives are no longer relevant. Unpin them
to ensure we get again get timely updates via dependabot.

* support for Hadoop 2 was dropped as part of #14763
* Guava was upgraded to 31 as part of #14767
* Calcite was upgraded to 1.35 as part of #14510
2023-12-05 16:04:39 -08:00
Vadim Ogievetsky 0b41b05aa0
Web console: Update and prune dependancies (#15487)
* update the basics
* remove babel
2023-12-05 14:25:07 -08:00
Vadim Ogievetsky aa696b0310
Web console: Log out any request errors in e2e tests for better CI debugging (#15483) 2023-12-05 14:23:47 -08:00
Pranav 82e3c61514
Update lookup model in console (#15472)
* Update lookup model in console

* ran prettify

* move Defaults to info

* setting defaultValue and removing placeholder
2023-12-05 13:22:22 -08:00
Jan Werner ff0e838d30
add gson to dependencyManagement (#15488)
This change completes the change introduced in #15461
and unifies the version of gson dependency used between all the modules.

gson is used by kubernetes-extension, avro-extensions, ranger-security,
and as a test dependency in several core modules.

---------

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2023-12-05 11:50:32 -08:00
Jill Osborne 0e14a2c77f
Update retention rules doc (#15439) 2023-12-05 09:53:17 -08:00
Jan Werner f4856bc1c1
ranger-security: exclude jackson-jaxrs from + fix outdated documentation (#15481)
* Excluding jackson-jaxrs dependency from ranger-plugin-common to address CVE regression introduced by ranger-upgrade: CVE-2019-10202, CVE-2019-10172
* remove the reference to outdated ranger 2.0 from the docs

---------

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2023-12-05 08:24:37 -08:00
Rishabh Singh 77b929f494
Fix CentralizedDatasourceSchema IT (#15493) 2023-12-05 20:05:13 +05:30
Vishesh Garg 326b7b731d
Upgrade zookeeper from 3.5.10 to 3.8.3 (#15477)
Upgrade zookeeper from 3.5.10 to 3.8.3
2023-12-05 18:57:56 +05:30
Rishabh Singh d968bb3f43
Rename config for enabling CentralizedDatasourceSchema feature (#15476)
* Rename property to druid.centralizedDatasourceSchema.enabled
* Update config name in docker-compose
2023-12-05 16:57:25 +05:30
Jan Werner a469c53c0c
cleanup already resolved CVEs (#15447)
Remove the crud from the dependency-check suppression file
2023-12-05 10:30:35 +05:30
Jan Werner b66d995e6f
remove licenses of removed libraries, update the license checker (#15446)
- Licenses file contains several licenses for outdated libraries. In this PR we remove licenses for no longer used components. 
  This change is purely cosmetic / cleans up the license database. 
  The candidates were designated by reviewing the output of the license check script and comparing it against the depdency tree.

 - Minor fix to license check tool to fail more gracefully when the license of used dependency is not listed as known, as well as fix not to fail on multi licensed components when at least one of the licenses is accepted. 

---------

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2023-12-04 13:20:40 -08:00
Jan Werner 8cc256b079
update guava to 32.0.1-jre to address CVEs (#15482)
Update guava to 32.0.1-jre to address two CVEs: CVE-2020-8908, CVE-2023-2976
This change requires a minor test change to remove assumptions about ordering.

---------

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2023-12-04 13:18:42 -08:00
Jan Werner 3d3d23c53f
run npm audit fix to update JS packages (#15466) 2023-12-04 13:17:24 -08:00
Adarsh Sanjeev ddd2299272
Add null check for VarianceAggregatorCollector 2023-12-04 22:26:44 +05:30
Jan Werner ddeb55fac1
update few minor dependencies to resolve CVEs (#15464)
Update multiple dependencies to clear CVEs
Update dropwizard-metrics to 4.2.22 to address GHSA-mm8h-8587-p46h in com.rabbitmq:amqp-client
Update ant to 1.10.14 to resolve GHSA-f62v-xpxf-3v68 GHSA-4p6w-m9wc-c9c9 GHSA-q5r4-cfpx-h6fh GHSA-5v34-g2px-j4fw
Update comomons-compress to resolve GHSA-cgwf-w82q-5jrr
Update jose4j to 0.9.3 to resolve GHSA-7g24-qg88-p43q GHSA-jgvc-jfgh-rjvv
Update kotlin-stdlib to 1.6.0 to resolve GHSA-cqj8-47ch-rvvq and CVE-2022-24329
2023-12-04 08:49:51 +05:30
Zoltan Haindrich a1aa4340d0
Changing the queryFrameWork in Calcite*Tests may have sideeffects (#15428)
changes how its configured a bit to use an annotation instead of methods
2023-12-04 00:38:01 +05:30
Jan Werner b854058491
remove unnecessary elasticsearch dependencies to fix CVE regressions (#15443)
Recent upgrade of ranger introduced CVE regressions due to outdated elasticsearch components.
Druid-ranger-plugin does not elasticsearch components , and they have been explicitly removed.

Update woodstox-core to 6.4.0 to address GHSA-3f7h-mf4q-vrm4
2023-12-03 20:56:40 +05:30
AmatyaAvadhanula 4a594bb9f6
Use task actions to fetch used segments in MSQ (#15284)
* Use task actions to fetch used segments in MSQ

* Fix tests

* Fixing tests.

* Revert "Fix tests"

This reverts commit 95ab6494

* Removing conditional check in tests.

* Pulling in latest changes.

---------

Co-authored-by: cryptoe <karankumar1100@gmail.com>
2023-12-01 15:29:33 +05:30
Pranav 9f3b26676d
Log full stack when exception message is null (#15467) 2023-11-30 16:47:37 -08:00
Vadim Ogievetsky f8bd3b0b7e
Web console: fix tab duplication (#15457)
* fix duplication

* includeFuture defaults to true
2023-11-30 08:29:56 -08:00
Keerthana Srikanth 7467d2c00d
Upgrade Jackson and Google GSON to address CVEs (#15461)
Upgrade Jackson to version 2.12.7.1 to address CVE-2022-42003, CVE-2022-42004 which affects jackson-databind.
Upgrade com.google.code.gson:gson from 2.2.4 to the latest version (2.10.1) since 2.2.4 is affected by CVE-2022-25647.
2023-11-30 15:31:26 +05:30
Kashif Faraz 8ddb847658
Fix message when skipping compaction (#15460) 2023-11-30 14:57:13 +05:30
Vadim Ogievetsky 31fa63e789
Web console: better management proxy detection (#15453)
* better management proxy detection

* fix checkstyle issue

* add test

* test should read the body also

* use ObjectMapper

* assert read ammount
2023-11-29 21:43:42 -08:00
Pranav 74ab6024e1
Native doc update (#15456)
Updating the native docs for #15434
2023-11-30 10:37:23 +05:30
Clint Wylie 5ce4aab3b8
update ARRAY_OVERLAP to plan with ArrayContainsElement for ARRAY columns (#15451)
Updates ARRAY_OVERLAP to use the same ArrayContainsElement filter added in #15366 when filtering ARRAY typed columns so that it can also use indexes like ARRAY_CONTAINS.
2023-11-30 10:05:20 +05:30
Clint Wylie 0516d0dae4
simplify IncrementalIndex since group-by v1 has been removed (#15448) 2023-11-29 14:46:16 -08:00
Pranav 93cd638645
Enabling aggregateMultipleValues in all StringAnyAggregators (#15434)
* Enabling aggregateMultipleValues in all StringAnyAggregators

* Adding more tests

* More validation

* fix warning

* updating asserts in decoupled mode

* fix intellij inspection

* Addressing comments

* Addressing comments

* Adding early validations and make aggregate consistent across all

* fixing tests

* fixing tests

* Update docs/querying/sql-aggregations.md

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

* fixing static check

---------

Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2023-11-29 14:32:49 -08:00
Clint Wylie 64fcb32bcf
add native 'array contains element' filter (#15366)
* add native arrayContainsElement filter to use array column element indexes
2023-11-29 03:33:00 -08:00
Abhishek Agarwal 0a56c87e93
SQL: Plan non-equijoin conditions as cross join followed by filter (#15302)
This PR revives #14978 with a few more bells and whistles. Instead of an unconditional cross-join, we will now split the join condition such that some conditions are now evaluated post-join. To decide what sub-condition goes where, I have refactored DruidJoinRule class to extract unsupported sub-conditions. We build a postJoinFilter out of these unsupported sub-conditions and push to the join.
2023-11-29 13:46:11 +05:30
Jan Werner ee6ad36fab
update confluent's dependencies to common, supported version (#15441)
* update confluent's dependencies to common, supported version

  Update io.confluent.* dependencies to common, updated version 6.2.12
currently used versions are EOL

* move version definition to the top level pom
2023-11-28 21:35:22 -08:00
Clint Wylie 97623b408c
add optional 'castToType' parameter to 'auto' column schema (#15417)
* auto but.. with an expected type
2023-11-28 17:19:23 -08:00
George Shiqi Wu 3d1d26f824
Fix mmless ingestion and index tasks (#15372)
* Fix mmless ingestion and index tasks

* Move comment

* remove dup test
2023-11-28 10:06:07 -05:00
Zoltan Haindrich eb056e23b5
Fix dictionarySize overrides in tests (#15354)
I think this is a problem as it discards the false return value when the putToKeyBuffer can't store the value because of the limit

Not forwarding the return value at that point may lead to the normal continuation here regardless something was not added to the dictionary like here
2023-11-28 18:49:09 +05:30
Charles Smith a929b9f16e
clafiry DISTINCT is optional for COUNT() (#15394) 2023-11-28 16:52:16 +05:30
Vadim Ogievetsky bd6afb7056
time shift literals also (#15433) 2023-11-28 16:42:35 +05:30
317brian eb6b9cb4a9
Update publishing readme (#15406)
This updates the prod section of the release readme for the website steps. Previously, only staging was updated
2023-11-28 15:41:19 +05:30
Petrichor b102667695
[Docs] Add example connection parameters for Java APIs (#15345) 2023-11-28 15:09:41 +05:30
Keerthana Srikanth 7a8204db72
Upgrade Avro to 1.11.3 to address CVE-2023-39410 (#15419) 2023-11-28 14:09:48 +05:30
Zoltan Haindrich ca544e552c
Add option to compare results with relative error tolerance (#15429)
Adds a result comparision mode of EQUALS_RELATIVE_1000_ULPS ; which accepts floating point differences up-to 1000 units of least precision
2023-11-28 13:03:16 +05:30
Kashif Faraz 58a724c7e4
Use StubServiceEmitter in tests (#15426)
* Use StubServiceEmitter in tests
* Remove unthrown exception from declaration
2023-11-28 09:43:09 +05:30
Sébastien c8eb7adeb9
Improved web-console's time-chart brush and added auto-granularity (#14990)
* Improved time-chart brush and added auto-granularity

* prettier

* added highlight bubble to explore visualizations

* Added licenses and fixes from PR review

* added missing files...
2023-11-27 12:15:47 -08:00
Karan Kumar a0188192de
Fixing failing compaction/parallel index jobs during upgrade due to new actions being available on the overlord. (#15430)
* Fixing failing compaction/parallel index jobs during upgrade due to new actions not available on the overlord.

* Fixing build

* Removing extra space.

* Fixing json getter.

* Review comments.
2023-11-25 13:50:29 +05:30
Kashif Faraz 67c7b6248c
Fix log typos, clean up some kill messages in SeekableStreamSupervisor (#15424)
Changes:
- Fix log `Got end of partition marker for partition [%s] from task [%s] in discoverTasks`
by fixing order of args
- Simplify in-line classes by using lambda
- Update kill task message from `Task [%s] failed to respond to [set end offsets]
 in a timely manner, killing task` to `Failed to set end offsets, killing task`
- Clean up tests
2023-11-24 16:09:10 +05:30