Commit Graph

11982 Commits

Author SHA1 Message Date
Clint Wylie 16f5ac5bd5
json_value adjustments (#12968)
* json_value adjustments
changes:
* native json_value expression now has optional 3rd argument to specify type, which will cast all values to the specified type
* rework how JSON_VALUE is wired up in SQL. Now we are using a custom convertlet to translate JSON_VALUE(... RETURNING type) into dedicated JSON_VALUE_BIGINT, JSON_VALUE_DOUBLE, JSON_VALUE_VARCHAR, JSON_VALUE_ANY instead of using the calcite StandardConvertletTable that wraps JSON_VALUE_ANY in a CAST, so that we preserve the typing of JSON_VALUE to pass down to the native expression as the 3rd argument

* fix json_value_any to be usable by humans too, coverage

* fix bug

* checkstyle

* checkstyle

* review stuff

* validate that options to json_value are the supported options rather than ignore them

* remove more legacy undocumented functions
2022-08-27 07:15:47 -07:00
Alexander Saydakov 7e2371bbde
KLL sketch (#12498)
* KLL sketch

* added documentation

* direct static refs

* direct static refs

* fixed test

* addressed review points

* added KLL sketch related terms

* return a copy from get

* Copy unions when returning them from "get".

* Remove redundant "final".

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
Co-authored-by: Gian Merlino <gianmerlino@gmail.com>
2022-08-26 21:19:24 -07:00
Adam Peck 21b73bde20
Update Curator to 5.3.0 (#12939)
* Update Curator to 5.3.0

* Update licenses.yaml

* Fix inspections + add tests.

* Fix checkstyle

* Another intellij inspection fix

* Update curator exclusions

* Cleanup new exhibitor references

* Remove unused dep and checkstyle fix
2022-08-26 18:23:40 -07:00
Clint Wylie 4bdf9815c1
fix issue with SQL planner and null array constants (#12971) 2022-08-26 04:44:17 -07:00
Vadim Ogievetsky acb09ff18b
grab warnings from correct key + test (#12977) 2022-08-25 18:47:33 -07:00
Junge e476e75462
fix #12945 - type conversion exception occurs during the variance query (#12967)
Co-authored-by: gejun <gejun@tingyun.com>
2022-08-25 18:10:58 -07:00
Karan Kumar 275f834b2a
Race in Task report/log streamer (#12931)
* Fixing RACE in HTTP remote task Runner

* Changes in the interface

* Updating documentation

* Adding test cases to SwitchingTaskLogStreamer

* Adding more tests
2022-08-25 17:56:01 -07:00
Clint Wylie 72aba00e09
add json function support for paths with negative array indexes (#12972) 2022-08-25 17:11:28 -07:00
Lee Surprenant 4c61378ad1
fix broken link in web-console (#12976)
In 0.23.0, the info-button links to https://druid.apache.org/docs/0.23.0/ingestion/flatten-json.html which is a 404

If I got the spot right, this should now link to https://druid.apache.org/docs/[VERSION]/ingestion/data-formats.html#flattenspec
2022-08-25 16:47:41 -07:00
Clint Wylie ad62c0eb31
ignore licenses changes in check test script (#12964) 2022-08-25 12:31:28 -07:00
Jill Osborne 7a1e1f88bb
Remove experimental note from stable features (#12973)
* Removed experimental note for features that are no longer experimental

* Updated native batch doc
2022-08-25 09:26:46 -07:00
Vadim Ogievetsky fd6cfcb8fb
Web console: fix pagination, add error delimiters (#12969)
* fix pagination, add error delimiters

* reword debug message
2022-08-25 08:18:21 -07:00
Santosh Pingale 31dc9004bd
Auto-reload TLS certs for druid endpoints (#12933)
* #12064 Auto-reload tls certs for druid endpoints

* #12064 Add missing toString param

* #12064 Add tests and new jks
Co-authored-by: zemin-piao <pzm6391@gmail.com>

* #12064 Refine tests

* #12064 Add documentation

* Apply suggestions from code review

Co-authored-by: Frank Chen <frankchen@apache.org>

Co-authored-by: santosh <santosh.pingale@adyen.com>
Co-authored-by: Frank Chen <frankchen@apache.org>
2022-08-25 20:12:43 +08:00
abhagraw 9cc30ee120
Suppressing CVE-2022-25168 - hadoop-common-2.8.5.jar (#12970) 2022-08-25 16:02:17 +05:30
Clint Wylie 8ee8786d3c
add maxBytesInMemory and maxClientResponseBytes to SamplerConfig (#12947)
* add maxBytesInMemory and maxClientResponseBytes to SamplerConfig
2022-08-25 00:50:41 -07:00
Clint Wylie 82ad927087
tighten up array handling, fix bug with array_slice output type inference (#12914) 2022-08-25 00:48:49 -07:00
Clint Wylie 599bdde02a
Update asf-release-process-guide.md (#12966)
* Update asf-release-process-guide.md

* Update asf-release-process-guide.md
2022-08-25 13:15:06 +05:30
Karan Kumar 31db3beed8
Fixing json creator for s3 storage connector provider (#12948)
* Fixing json creator for s3 storage connector provider

* Adding guice tests
2022-08-25 11:08:57 +05:30
Vadim Ogievetsky 04ee7abeff
Web console: Multi-stage query support (#12919)
* MSQ web console

* fix typo in comments

* remove useless conditional

* wrap SQL_DATA_TYPES

* fixes sus regex

* rewrite regex

* remove problematic regex

* fix UTs

* convert PARTITIONED / CLUSTERED BY to ORDER BY for preview

* fix log

* updated to use shuffle

* Web console: Use Ace.Completion directly (#1405)

* Use Ace.Completion directly

* Another Ace.Completion

* better comment

* fix column ordering in e2e test

* add nested data example also

Co-authored-by: John Gozde <john.gozde@imply.io>
2022-08-24 16:17:12 -07:00
Victoria Lim 02914c17b9
Tutorial on ingesting and querying Theta sketches (#12723)
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2022-08-24 09:23:22 -07:00
Gian Merlino 35aaaa9573
Fix serialization in TaskReportFileWriters. (#12938)
* Fix serialization in TaskReportFileWriters.

For some reason, serializing a Map<String, TaskReport> would omit the
"type" field. Explicitly sending each value through the ObjectMapper
fixes this, because the type information does not get lost.

* Fixes for static analysis.
2022-08-24 08:11:01 -07:00
Vadim Ogievetsky c1a75fca3c
Docs: fix doc footer (#12943)
* fix doc footer

* move the logic to the fix-path script
2022-08-24 06:47:58 -07:00
Karan Kumar f7c6316992
Setting useNativeQueryExplain to true (#12936)
* Setting useNativeQueryExplain to true

* Update docs/querying/sql-query-context.md

Co-authored-by: Santosh Pingale <pingalesantosh@gmail.com>

* Fixing tests

* Fixing broken tests

Co-authored-by: Santosh Pingale <pingalesantosh@gmail.com>
2022-08-24 17:39:55 +05:30
Paul Rogers cfed036091
Add the new integration test framework (#12368)
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.

The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
Bartosz Mikulski 0bc9f9f303
#12912 Fix KafkaEmitter not emitting queryType for a native query (#12915)
Fixes KafkaEmitter not emitting queryType for a native query. The Event to JSON serialization was extracted to the external class: EventToJsonSerializer. This was done to simplify the testing logic for the serialization as well as extract the responsibility of serialization to the separate class.

The logic builds ObjectNode incrementally based on the event .toMap method. Parsing each entry individually ensures that the Jackson polymorphic annotations are respected. Not respecting these annotation caused the missing of the queryType from output event.
2022-08-24 14:07:00 +05:30
Adarsh Sanjeev 3b58a01c7c
Correct spelling in messages and variable names. (#12932) 2022-08-24 11:06:31 +05:30
Adam Peck 31eda58e9a
Fix misspelling in license.md (#12941) 2022-08-24 10:53:47 +08:00
Gian Merlino d7d15ba51f
Add druid-multi-stage-query extension. (#12918)
* Add druid-multi-stage-query extension.

* Adjustments from CI.

* Task ID validation.

* Various changes from code review.

* Remove unnecessary code.

* LGTM-related.
2022-08-23 18:44:01 -07:00
Abhishek Agarwal f0fc45028e
Update year in the notice file and the release process instructions (#12622)
* Update notice file

* Update release process instructions

* Better release instructions

* Update copyright year

* Web console: updated the doc link (#12619)

* updated the doc link

* update snapshots

Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
2022-08-23 18:17:18 +05:30
William Hyun a1c4eab522
Update ORC to 1.7.6 (#12928) 2022-08-23 01:09:38 -07:00
Clint Wylie 0c56b22a39
Update .spelling (#12940) 2022-08-22 18:47:40 -07:00
Clint Wylie 289e43281e
stricter behavior for parse_json, add try_parse_json, remove to_json (#12920) 2022-08-22 18:41:07 -07:00
Petar Petrov 6fec1d4c95
Add useNativeQueryExplain in sql query context documentation (#12924) (#12934)
Co-authored-by: Petar Petrov <petar.petrov@system73.com>
2022-08-22 16:31:15 +05:30
AmatyaAvadhanula 379df5f103
Kinesis docs and logs improvements (#12886)
Going ahead with the merge. CI is failing because of a code coverage change in the log line.
2022-08-22 14:49:42 +05:30
Rohan Garg a879d91a20
Remove misleading logging on router for JDBC queries (#12925) 2022-08-22 11:58:51 +05:30
Rohan Garg 3c129f6728
Add sql planning time metric (#12923) 2022-08-22 11:09:44 +05:30
Clint Wylie f8097ccfaa
basic docs for nested column query functions (#12922)
* basic docs for nested column query functions
2022-08-19 17:12:19 -07:00
Clint Wylie 69fe1f04e5
document virtualColumns in native query documentation, fix some redirects (#12917)
* document virtualColumns in native query documentation, fix some redirects

* after all that, forgot to run spellcheck locally

* review stuff
2022-08-18 20:49:23 -07:00
Paul Rogers eb902375a2
Light refactor of the heavily refactored statement classes (#12909)
Reflects lessons learned from working with consumers of
the new code.
2022-08-19 02:31:06 +05:30
Clint Wylie 7fb1153bba
add virtual columns to search query cache key (#12907)
* add virtual columns to search query cache key
2022-08-17 20:26:01 -07:00
Ian Roberts 770358dc34
Update tls-support.md (#12916)
Fixing " lists all possible values for the configs belong" in TLS section
2022-08-18 09:46:30 +08:00
imply-cheddar 536415b948
Stop leaking Avro objects from parser (#12828)
The Avro parsing code leaks some "object" representations.
We need to convert them into Maps/Lists so that other code
can understand and expect good things.  Previously, these
objects were handled with .toString(), but that's not a
good contract in terms of how to work with objects.
2022-08-18 03:16:20 +05:30
Xavier Léauté 752e42a312
fix running integration tests on macos aarch64 (#12913)
* add osx-aarch_64 netty-transport-native-kqueue native dependency
* align docker-java dependency versions using bom and update to 3.2.13
2022-08-17 18:03:24 +02:00
Karan Kumar a3a9c5f409
Fixing overlord issued too many redirects (#12908)
* Fixing race in overlord redirects where the node was redirecting to itself

* Fixing test cases
2022-08-17 18:27:39 +05:30
dependabot[bot] f70f7b4b89
Bump postgresql from 42.3.3 to 42.4.1 (#12871)
* Bump postgresql from 42.3.3 to 42.4.1

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.3 to 42.4.1.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.3.3...REL42.4.1)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-08-16 23:25:39 +02:00
Gian Merlino d3015d0f8e
DruidQuery: Return a copy from withScanSignatureIfNeeded, as promised. (#12906)
The method wasn't following its contract, leading to pollution of the
overall planner context, when really we just want to create a new
context for a specific query.
2022-08-16 13:23:14 -07:00
Peter Marshall f665a0c077
Docs - Add links to Basic Tuning guide in process pages (#12741)
Added link to the relevant section of the Basic Cluster Tuning page on each process page.

This is in order to improve access to this information, which is not easy to find through search or nav.
2022-08-16 18:42:44 +05:30
Abhishek Agarwal adbebc174a
Fix flaky tests in SeekableStreamSupervisorStateTest (#12875)
* Fix flaky test in SeekableStreamSupervisorStateTest

* Fix for flaky security IT Test

* fix tests

* retry queries if there is some flakiness
2022-08-16 18:38:03 +05:30
Adarsh Sanjeev 3755f30bc4
Add export parameters for Java 11 (#12859)
* Add exports for Java 11 parameters

* Add parameters for data sketches
2022-08-16 13:05:45 +05:30
Lucas Capistrant ec8bdeb9f6
Document missing property - druid.announcer.skipSegmentAnnouncementOnZk (#12891)
* document missing config related to segment announcement

* improve wording

* improve wording

* update docs
2022-08-16 12:32:56 +05:30