Commit Graph

13976 Commits

Author SHA1 Message Date
Adarsh Sanjeev 86a24012a6
Add security ITs for sending tasks to overlord (#16131)
* Add security ITs for sending tasks to overlord

* Add security ITs for sending tasks to overlord

* Resolve test flakiness
2024-03-18 09:33:40 +05:30
Kashif Faraz 1682d4570d
Increase delay to allow propagation of credentials (#16143) 2024-03-17 14:47:42 +05:30
Andreas Maechler 9b5571f84f
docs: Update `caniuse-lite` (#16137) 2024-03-15 15:58:16 -07:00
dependabot[bot] 7a0d6c53c8
Bump follow-redirects from 1.15.5 to 1.15.6 in /web-console (#16134)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.5...v1.15.6)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-15 15:30:41 -07:00
zachjsh f3d77fe684
Fix Cannot mark an unqueryable datasource's segments used / unused (#16127)
* * fix

* * address review comments

* * all remove the short-circuit for markUnused api

* * add test
2024-03-15 14:25:02 -07:00
Abhishek Radhakrishnan 3eefc47722
Refactor tests and code clean up (#16129)
* Add update() in TestDerbyConnectorRule

* use common function.

* fixup build.

* fixup indentations.

* Revert "fixup indentations."

This reverts commit a9d6b73e79.

* fixup indentataions.

* Remove Thread.sleep() by directly calling updateUsedStatusLastUpdated.

* another indentation slip.

* Move common segment initialization to setup().

* Fix for checkstyle.

* review comments: indentation fixes, type.

* Wrapper class for Segments table

* Add KillUnusedSegmentsTaskBuilder in test class

* Remove javadocs for self-explanatory methods.
2024-03-15 10:13:14 -07:00
Kashif Faraz 466057c61b
Remove deprecated DruidException, EntryExistsException (#14448)
Changes:
- Remove deprecated `DruidException` (old one) and `EntryExistsException`
- Use newly added comprehensive `DruidException` instead
- Update error message in `SqlMetadataStorageActionHandler` when max packet limit is violated.
- Factor out common code from several faults into `BaseFault`.
- Slightly update javadoc in `DruidException` to render it correctly
- Remove unused classes `SegmentToMove`, `SegmentToDrop`
- Move `ServletResourceUtils` from module `druid-processing` to `druid-server`
- Add utility method to build error Response from `DruidException`.
2024-03-15 21:29:11 +05:30
AlbericByte 33bb99cd0d
remove use log of log4j v1 (#15984) 2024-03-15 15:43:48 +05:30
Katya Macedo da6158c166
[Docs] Improve the "Update existing data" tutorial (#16081)
* Modify update data tutorial

* Update after review

* Add append topic

* Update after review

* Add upsert to spelling
2024-03-14 16:31:33 -07:00
Zoltan Haindrich d3e22c6e92
fix compile error: ARRAYS_DATASOURCE (#16120) 2024-03-14 18:15:43 +05:30
Karan Kumar 5e603ac5ff
Adding more logging for s3 RetryableS3OutputStream (#16117)
Adding more logging for s3 RetryableS3OutputStream which would help us determine if the chunk size needs to be adjusted.
2024-03-14 11:35:57 +05:30
Clint Wylie dd9bc3749a
fix issues with array_contains and array_overlap with null left side arguments (#15974)
changes:
* fix issues with array_contains and array_overlap with null left side arguments
* modify singleThreaded stuff to allow optimizing Function similar to how we do for ExprMacro - removed SingleThreadSpecializable in favor of default impl of asSingleThreaded on Expr with clear javadocs that most callers shouldn't be calling it directly and should be using Expr.singleThreaded static method which uses a shuttle and delegates to asSingleThreaded instead
* add optimized 'singleThreaded' versions of array_contains and array_overlap
* add mv_harmonize_nulls native expression to use with MV_CONTAINS and MV_OVERLAP to allow them to behave consistently with filter rewrites, coercing null and [] into [null]
* fix bug with casting rhs argument for native array_contains and array_overlap expressions
2024-03-13 18:16:10 -07:00
Sree Charan Manamala e9d2caccb6
Handling null operand in JSON_QUERY_ARRAY (#16118)
* fix return type inference for JSON_QUERY_ARRAY to be nullable
2024-03-13 18:06:27 -07:00
Gian Merlino 256160aba6
MSQ: Validate that strings and string arrays are not mixed. (#15920)
* MSQ: Validate that strings and string arrays are not mixed.

When multi-value strings and string arrays coexist in the same column,
it causes problems with "classic MVD" style queries such as:

  select * from wikipedia -- fails at runtime
  select count(*) from wikipedia where flags = 'B' -- fails at planning time
  select flags, count(*) from wikipedia group by 1 -- fails at runtime

To avoid these problems, this patch adds type verification for INSERT
and REPLACE. It is targeted: the only type changes that are blocked are
string-to-array and array-to-string. There is also a way to exclude
certain columns from the type checks, if the user really knows what
they're doing.

* Fixes.

* Tests and docs and error messages.

* More docs.

* Adjustments.

* Adjust message.

* Fix tests.

* Fix test in DV mode.
2024-03-13 15:37:27 -07:00
Vadim Ogievetsky ccae19a546
Web console: Make array ingest mode ux better (#15927)
* only set arrayIngestMode: array when needed (still do the queries the correct way)

* arrayIngestMode control

* update wording

* feedback fixes
2024-03-13 13:04:22 -07:00
317brian 03c191f701
docs: clarify description of uri/uriprefix (#16110)
* docs: clarify description of uri/uripath

* Apply suggestions from code review

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

---------

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2024-03-13 11:52:01 -07:00
Vadim Ogievetsky f7c0e425a9
fix URI wording (#16111) 2024-03-13 11:38:51 -07:00
Gian Merlino 910124d4de
MSQ: Plan without implicit sorting. (#16073)
* MSQ: Plan without implicit sorting.

This patch adds an EngineFeature "GROUPBY_IMPLICITLY_SORTS" and sets
it true for native, false for MSQ. It's useful for two reasons:

1) In the future we'll likely want MSQ to hash-partition for GROUP BY
   instead of using a global sort, which would mean MSQ would not
   implicitly ORDER BY when there is a GROUP BY.

2) When doing REPLACE with MSQ, CLUSTERED BY is transformed to ORDER BY.
   We should retain that ORDER BY, as it may be a subset of the GROUP BY,
   and it is important to remember which fields the user wanted to include in
   range shard specs.

* Fix tests.

* Fix tests for real.

* Fix test.
2024-03-13 08:27:39 -07:00
Zoltan Haindrich 818cc9eedf
Fix toString for SingleThreadSpecializable ConstantExprs (#16084) 2024-03-13 03:48:52 -07:00
Clint Wylie aa2959b2bd
reset keySerde when closing groupers to clear out heap dictionaries (#16114)
ConcurrentGrouper kind of misuses ThreadLocal to hold a SpillingGrouper, and never calls remove() on it, which can result in large amounts of heap being retained as weak references even after grouping is finished. This PR calls keySerde.reset() on all of the Grouper.close() implementations that have a KeySerde and should free up a bunch of space that is no longer needed.
2024-03-13 15:09:54 +05:30
Clint Wylie 795e342ba8
fix sql results mixed array and scalar values (#16105)
* fix sql results mixed array and scalar values

* simplify
2024-03-12 23:47:35 -07:00
Kashif Faraz 82fced571b
Remove deprecated UnknownSegmentIdsException (#16112)
Changes
- Replace usages of `UnknownSegmentIdsException` with `DruidException`
- Add method `SqlMetadataQuery.retrieveSegments`
- Add new field `used` to `DataSegmentPlus`
2024-03-13 11:07:37 +05:30
Abhishek Radhakrishnan fb7bb0953d
Kill segments by versions (#15994)
* Kill task version support.

Kill tasks by default kill all versions of unused segments in the specified
interval. Users wanting to delete specific versions (for example, data compliance
reasons) and keep rest of the versions can specify the optional version in the
kill task payload.

* Formatting changes.

* Multi version tests in RetrieveSegmentsActionsTest

Sort of like method-level parameterized tests.

* Address review feedback

* Accept a list of versions instead of a single version.

Support multiple versions.

* Tests for multiple versions.

* Update docs

* Cleanup

* Address review comments.

Retain the old interface method and make it default and route it to
the method with nullable versions variant. Update usages to use the
default method where versions doesn't matter.

* Remove versions from retreive used segments action.

* Some updates.

* Apply suggestions from code review

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* /s/actual/observed/g

* minor test cleanup

* WIP: Test fixes and updates. Also add test for kill by version with used load spec.

Checkpoint.

---------

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2024-03-13 09:37:30 +05:30
Karan Kumar 84c5098473
Fix data race in getting results from MSQ select tasks. (#16107)
* Fix data race in getting results from MSQ select tasks.

* Add better logging

* Handling number overflow.
2024-03-13 08:58:18 +05:30
Katya Macedo 6f6f86c325
Update `maxRowsInMemory` and `maxBytesInMemory` description (#16104) 2024-03-12 14:40:15 -07:00
Zoltan Haindrich 8252d72e2a
Pull up literals in InputAccessor (#16033)
* Pull up literals in InputAccessor

* pull up literals in `InputAccessor`
* remove the need to pass `constants` of `Window`  operator

Fixes #15353

* update test

* enable relax_nulls
2024-03-12 09:14:31 -07:00
Sree Charan Manamala ef9637eef1
Handling array with boolean literals (#16093)
Handling array with boolean literals like ARRAY[true, false]

Druid appears to be able to convert an array with boolean expressions like this array[added=deleted, added=delta] into a numeric array of 0 and 1: select array[added=deleted, added=delta] from wikipedia

However, select array[true, false] from wikipedia doesn't work.
This PR fixes this.
2024-03-12 12:28:16 +05:30
Abhishek Radhakrishnan 0a615f16de
Fix bug where numSegmentsKilled is reported incorrectly. Also, add a unit test. (#16103) 2024-03-12 10:02:54 +05:30
Vadim Ogievetsky 8ef3eebd30
Web console: upgrade axios and follow-redirects (#16087)
* upgrade axios

* upgrade jest
2024-03-11 18:57:00 -07:00
Soumyava 85ee775390
Handling latest_by and earliest_by on numeric columns correctly (#15939)
* Handling latest_by and earliest_by on numeric columns correctly

* Adding test
2024-03-11 13:49:21 -07:00
Clint Wylie 313da98879
decouple column serializer compression closers from SegmentWriteoutMedium to optionally allow serializers to release direct memory allocated for compression earlier than when segment is completed (#16076) 2024-03-11 12:28:04 -07:00
Abhishek Radhakrishnan 8084f2206b
Remove `@JsonIgnore` annotations for private members of `TaskAction` classes (#16099)
* Remove @JsonIgnore annotations for private members

* checkstyle fix - removed unused imports.
2024-03-12 00:12:36 +05:30
George Shiqi Wu 94d2a28465
Add deep storage segment metric (#16072)
* Add new metric for deepStorage segments

* Add docs

* change metric name
2024-03-11 10:24:46 -04:00
Vishesh Garg 2dd8b16467
Correct the API used to fetch the version for a GCS object (#16097)
Current API used to fetch the version for a GCS object is incorrect. This PR fixes that API.
2024-03-11 18:30:34 +05:30
Abhishek Radhakrishnan c7f1872bd1
Fixup KillUnusedSegmentsTest (#16094)
Changes:
- Use an actual SqlSegmentsMetadataManager instead of TestSqlSegmentsMetadataManager
- Simplify TestSegmentsMetadataManager
- Add a test for large interval segments.
2024-03-11 13:37:48 +05:30
sullis 148ad32e75
netty 4.1.107 (#16027)
* netty 4.1.107

* update licenses.yaml
2024-03-11 15:57:44 +08:00
Zoltan Haindrich 2eb7d7a89b
Calcite tests remove expected exception (#16046)
* Calcite tests remove expected exception

* update testcases using `expectedException` to utilize `assertThrows` instead
* remove `BaseCalciteQueryTest#expectedException`
* fixes `cannotVectorize` so it doesn't anymore stops further processing
* `msqIncompatible` is not anymore toggles a boolean - its an `Assume` instead

Fixes #15423

* cleanup

* move msqIncompat

* update test

* cleanup

* remove comment

* empty-commit

* empty-commit
2024-03-11 13:23:57 +05:30
Sensor 2d62b4f09b
docs refinement: json format (#16080)
* docs refinement: json format

* Update docs/api-reference/tasks-api.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

---------

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2024-03-11 15:49:14 +08:00
gzhao9 2d628cce84
Refactor AsyncQueryForwardingServletTest to reduce code duplication (#16092) 2024-03-10 17:32:43 +05:30
Sensor ba3d4daf45
Fix field names in PeonCommandContext (#16067) 2024-03-09 08:46:51 +05:30
Vishesh Garg b1c1937e94
Change last update timestamp granularity of GCS objects from seconds to milliseconds (#16083)
The previously used GCS API client library returned last update time for objects directly in milliseconds. The new library returns it in OffsetDateTime format which was being converted to seconds and stored against the object. This fix converts the time back to ms before storing it.
2024-03-09 07:54:33 +05:30
Parth Agrawal 3aec90563e
Fix Jest and Prettify Checks (#15544)
* Fix jest and prettify checks

* Remove defaultQueryContext and run jest again

---------

Co-authored-by: Ghazanfar-CFLT <mghazanfar@confluent.io>
2024-03-08 14:36:26 -08:00
Vadim Ogievetsky 2816121ef0
wait for the coordinator (and proxy service to start) (#16088) 2024-03-08 13:31:29 -08:00
George Shiqi Wu 40ebaf83c9
Fix bug with mmless ingestion and compaction tasks on azure (#16065)
* Update azure behavior to match s3

* Add test

* Cleanup logic

* fix checkstyle

* Add comment
2024-03-08 15:42:44 -05:00
Vadim Ogievetsky 19a8af866b
make detail archive opening more robust (#16071) 2024-03-08 10:55:31 -08:00
dependabot[bot] 775c1180ae
Bump redis.clients:jedis from 5.0.2 to 5.1.2 (#16074)
Bumps [redis.clients:jedis](https://github.com/redis/jedis) from 5.0.2 to 5.1.2.
- [Release notes](https://github.com/redis/jedis/releases)
- [Commits](https://github.com/redis/jedis/compare/v5.0.2...v5.1.2)

---
updated-dependencies:
- dependency-name: redis.clients:jedis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-08 07:40:37 -08:00
Jan Werner 834a0ad9f1
update jose4j and corresponding license file (#16078)
Update org.bitbucket.b_c:jose4j from 0.9.3 to 0.9.6. to resolve https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-51775

fixes #16075
2024-03-08 07:36:07 -08:00
Jan Werner a7b2747e56
remove aws-sdk from ranger-extension (#16011)
Fixes # size blowup regression introduced in https://github.com/apache/druid/pull/15443

This PR removes the transitive dependency of ranger-plugins-audit to reduce the size of the compiled artifacts

* add aws-logs-sdk to ensure that all the transitive dependencies are satisfied
* replace aws-bundle-sdk with aws-logs-sdk
* add additional guidance on ranger update, add dependency ignore to satisfy dependency analyzer
* add aws-sdk-logs to list of ignored dependencies to satisfy the maven plugin
* align aws-sdk versions
2024-03-08 07:35:29 -08:00
Zoltan Haindrich 60766495aa
Use dorny/paths-filter@v3.0.0 (#16082) 2024-03-08 13:35:26 +05:30
Abhishek Radhakrishnan daf03939a9
Upgrade GHA dependencies (#15954)
* Upgrade actions/checkout from v3 to v4.

* Upgrade actions/setup-java from v3 to v4.

* Upgrade dorny/paths-filter, actions/cdache/restore, actions/stale to v3, v4 and v9 respectively.

* Add a GHA label for .github/** and skip UT/IT on .github files.

* remove skipping UT/IT on .github/** changes.
2024-03-08 07:54:02 +05:30