Commit Graph

601 Commits

Author SHA1 Message Date
Sensor 6bc29534a7
[Web Console] Datasource page support search datasource by keyword (#16371)
* Frontend segment_timeline support filter by datasource

* add dependency

* fix eslint issues

* resolve code style issue, update snapshot

* fix comment

* update licence

* update package-lock.json

* update snapshot

* Update segment-timeline.tsx

* Update segment-timeline.tsx
2024-05-24 11:54:26 -07:00
Vadim Ogievetsky 10ea88e5bf
Web console: more robust durable storage setting detection (#16493)
* more robust durable storage setting

* add test
2024-05-22 15:47:20 -07:00
Vadim Ogievetsky 0ab3b34117
Web console: enable copy data as inline SQL (via VALUES) (#16458)
* copy as values

* address NULL issue

* add decription

* extend test

* fix json

* more types

* fix braces with nulls

* fix test

* update functions to scan
2024-05-22 08:33:07 -07:00
Vadim Ogievetsky 435b58f101
Web console: fix Druid doctor check to accept Java 17 (#16250)
* fix Druid doctor check

* fix doc link

* Update web-console/src/dialogs/doctor-dialog/doctor-checks.tsx

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

---------

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2024-05-15 20:37:15 -07:00
Vadim Ogievetsky c419ae5f73
use objectGlob (#16452)
Catching up to a change introduced in #13027
2024-05-15 15:11:11 +05:30
Vadim Ogievetsky 760e449875
Web console: Fix order-by-delta in explore view table (#16417)
* change to using measure name

* Implment order by delta

* less paring, stricter types

* safeDivide0

* fix no query

* new DTQ alows parsing JSON_VALUE(...RETURNING...)
2024-05-13 19:03:46 -07:00
Abhishek Radhakrishnan 2a638d77d9
Remove stale references to coordinator dynamic config killAllDataSources. (#16387)
This parameter has been removed for awhile now as of Druid 0.23.0
https://github.com/apache/druid/pull/12187.

The code was only used in tests to verify that serialization works.
Now remove all references to avoid any confusion.
2024-05-05 08:48:56 +05:30
Vadim Ogievetsky 4d62c4a917
Web console: concat data when doing a durable storage download (#16375)
* concat data

* fix silly console.error
2024-05-03 08:00:32 -07:00
Abhishek Radhakrishnan 3717554e16
Web console changes for https://github.com/apache/druid/pull/16288 (#16379)
Adds a text box for delta filter that can accept an optional json
object.
2024-05-02 15:50:17 -07:00
Vadim Ogievetsky 39ada8b9ad
Web console: surface more info on the supervisor view (#16318)
* add rate and stats

* better tabs

* detail

* add recent errors

* update tests

* don't let people hide the actions column because why

* don't sort on actions

* better way to agg

* add timeouts

* show error only once

* fix tests and Explain showing up

* only consider active tasks

* refresh

* fix tests

* better formatting
2024-05-02 08:50:27 -07:00
John Gozde 834b0eddeb
web-console: ACE editor refactoring (#16359)
* Move druid-sql completions to dsql mode

* Use font-size 12

* Convert ace-modes to typescript

* Move aceCompleters to class member

* Use namespace imports
2024-04-30 11:53:39 -07:00
Adarsh Sanjeev 9a2d7c28bc
Prepare master branch for 31.0.0 release (#16333) 2024-04-26 09:22:43 +05:30
Vadim Ogievetsky 3e42ebbaea
Web console: Fix the supervisor offset reset dialog. (#16298)
* Add host to query output

* Init fixes for reset offsets

* fix the supervisor offset reset dialog

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* reformat code

* &apos;

* fix conflict

---------

Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com>
2024-04-19 17:25:46 -07:00
Abhishek Radhakrishnan 041d0bff5e
Set default `KillUnusedSegments` duty to coordinator's indexing period & `killTaskSlotRatio` to 0.1 (#16247)
The default value for druid.coordinator.kill.period (if unspecified) has changed from P1D to the value of druid.coordinator.period.indexingPeriod. Operators can choose to override druid.coordinator.kill.period and that will take precedence over the default behavior.
The default value for the coordinator dynamic config killTaskSlotRatio is updated from 1.0 to 0.1. This ensures that that kill tasks take up only 1 task slot right out-of-the-box instead of taking up all the task slots.

* Remove stale comment and inline canDutyRun()

* druid.coordinator.kill.period defaults to druid.coordinator.period.indexingPeriod if not set.

- Remove the default P1D value for druid.coordinator.kill.period. Instead default
  druid.coordinator.kill.period to whatever value druid.coordinator.period.indexingPeriod is set
  to if the former config isn't specified.
- If druid.coordinator.kill.period is set, the value will take precedence over
  druid.coordinator.period.indexingPeriod

* Update server/src/test/java/org/apache/druid/server/coordinator/DruidCoordinatorConfigTest.java

* Fix checkstyle error

* Clarify comment

* Update server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinatorConfig.java

* Put back canDutyRun()

* Default killTaskSlotsRatio to 0.1 instead of 1.0 (all slots)

* Fix typo DEFAULT_MAX_COMPACTION_TASK_SLOTS

* Remove unused test method.

* Update default value of killTaskSlotsRatio in docs and web-console default mock

* Move initDuty() after params and config setup.
2024-04-14 18:56:17 -07:00
Vadim Ogievetsky 4ff7e2c6c9
Web console: Better manual capabilities detection indication (#16191)
* Better forced mode indication

* more robust

* Update web-console/src/components/header-bar/header-bar.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/header-bar.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/__snapshots__/restricted-mode.spec.tsx.snap

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/__snapshots__/restricted-mode.spec.tsx.snap

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/components/header-bar/restricted-mode/restricted-mode.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* reformat

* forced => manual capability detection

* typo

* typo2

---------

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2024-04-08 10:07:21 -07:00
Vadim Ogievetsky 3ba878f21b
don't send lookups to sampler (#16234) 2024-04-04 21:17:42 -07:00
Vadim Ogievetsky 9658e1ad7f
Web console: fix query timer issues (#16235)
* fix timer issues

* wording
2024-04-04 13:13:31 -07:00
Vadim Ogievetsky 06268bf060
only pick kafka input format by default when needed (#16180) 2024-04-01 13:47:49 -07:00
dependabot[bot] 27b4028782
Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /web-console (#16195)
Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.3 to 5.3.4.
- [Release notes](https://github.com/webpack/webpack-dev-middleware/releases)
- [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-middleware/compare/v5.3.3...v5.3.4)

---
updated-dependencies:
- dependency-name: webpack-dev-middleware
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-29 09:51:51 -07:00
dependabot[bot] bf88ddb2d7
Bump express from 4.18.2 to 4.19.2 in /web-console (#16204)
Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.18.2...4.19.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-29 09:50:53 -07:00
Vadim Ogievetsky 195221ca59
Web console: update druid-toolkit to get bug fixes (#16213)
* update druid-toolkit to get bug fixes

* update

* fix test
2024-03-29 08:31:35 -07:00
Vadim Ogievetsky e769ec7b96
Web console: fix Azure icon not rendering (#16173)
* do not lowercase the icon name

* missing icon

* update test
2024-03-20 13:37:39 -07:00
dependabot[bot] 7a0d6c53c8
Bump follow-redirects from 1.15.5 to 1.15.6 in /web-console (#16134)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.5 to 1.15.6.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.5...v1.15.6)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-15 15:30:41 -07:00
Vadim Ogievetsky ccae19a546
Web console: Make array ingest mode ux better (#15927)
* only set arrayIngestMode: array when needed (still do the queries the correct way)

* arrayIngestMode control

* update wording

* feedback fixes
2024-03-13 13:04:22 -07:00
Vadim Ogievetsky f7c0e425a9
fix URI wording (#16111) 2024-03-13 11:38:51 -07:00
Vadim Ogievetsky 8ef3eebd30
Web console: upgrade axios and follow-redirects (#16087)
* upgrade axios

* upgrade jest
2024-03-11 18:57:00 -07:00
Parth Agrawal 3aec90563e
Fix Jest and Prettify Checks (#15544)
* Fix jest and prettify checks

* Remove defaultQueryContext and run jest again

---------

Co-authored-by: Ghazanfar-CFLT <mghazanfar@confluent.io>
2024-03-08 14:36:26 -08:00
Vadim Ogievetsky 2816121ef0
wait for the coordinator (and proxy service to start) (#16088) 2024-03-08 13:31:29 -08:00
Vadim Ogievetsky 19a8af866b
make detail archive opening more robust (#16071) 2024-03-08 10:55:31 -08:00
Vadim Ogievetsky acb5124679
make double detection better (#15998) 2024-02-29 15:45:58 -08:00
Vadim Ogievetsky c5b032799c
Web console: add table and column search (#15990)
* Make a search

* fix snapshot

* added message when not found
2024-02-29 15:45:50 -08:00
Vadim Ogievetsky 6e222d47c8
fix crash when parsing data in data loader that can not be parsed (#15983) 2024-02-28 14:37:24 -08:00
Vadim Ogievetsky a81429746d
Web console: fix typos in Kinesis suggestions, add regions and groups (#15900)
* fix typo

* update regions

* add China

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: Benedict Jin <asdf2014@apache.org>

* add ,

---------

Co-authored-by: Benedict Jin <asdf2014@apache.org>
2024-02-27 10:00:02 +08:00
Vadim Ogievetsky bf3139562c
Web console: support for the export execution state (#15969)
* init

* add CSV keyword
2024-02-26 11:28:25 -08:00
Vadim Ogievetsky 28b3e117cf
Web console: Add input format props (#15950)
* fix typo

* add Protobuf

* better padding
2024-02-26 11:28:09 -08:00
Vadim Ogievetsky c52ddd0b86
make flattenSpec location adaptive (#15946) 2024-02-22 14:07:04 -08:00
Vadim Ogievetsky 66f54f2066
allow compaction config slots to drop to 0 (#15877) 2024-02-15 15:27:15 +08:00
Vadim Ogievetsky 26815d425b
Web console: add system fields UI (#15858)
This PR adds console support for configuring system fields in the batch data loader.
2024-02-08 11:08:55 +05:30
Vadim Ogievetsky f2b242b6e6
update console to core Druid changes (#15854) 2024-02-07 19:44:25 +05:30
George Shiqi Wu edb1ac1b71
Update azure console tile (#15820)
* Save web console changes

* Working new input type

* fix tests
2024-02-05 13:11:39 -08:00
Vadim Ogievetsky fcd65c9801
Web console: use arrayIngestMode: array (#15588)
* Adapt to new array mode

* Feedback fixes

* fixing type detection and highlighting

* goodies

* add docs

* feedback fixes

* finish array work

* update snapshots

* typo fix

* color fixes

* small fix

* make MVDs default for now

* better sqlStringifyArrays default

* fix spec converter

* fix tests
2024-01-31 20:19:29 -08:00
Vadim Ogievetsky 0089f6b905
Web console: Don't force waitUntilSegmentLoad to true (#15781)
* Don't force setting waitUntilSegmentsLoad

* delete irrelevant code
2024-01-31 16:16:36 +05:30
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
Vadim Ogievetsky 497e2123f0
Web console: Make table driven query modification actions work with slices. (#15779)
* Make table driven query modification actions work with slices.

* cleanup found query prefix

* fix regex complexity
2024-01-29 20:09:46 -08:00
Vadim Ogievetsky 45ad47cc66
allow segment table to sort on start and end when grouped (#15720) 2024-01-26 18:59:23 +08:00
Sensor 4e50a14d50
fix router page value inconsistent issue (#15742)
* fix router page value inconsistent issue

* make the fix more universal as suggested

* minor
2024-01-26 11:02:30 +08:00
Vadim Ogievetsky 55ed69f830
update mocks (#15745) 2024-01-24 11:41:52 -08:00
Karan Kumar c4990f56d6
Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
zachjsh 9d4e8053a4
Kinesis adaptive memory management (#15360)
### Description

Our Kinesis consumer works by using the [GetRecords API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) in some number of `fetchThreads`, each fetching some number of records (`recordsPerFetch`) and each inserting into a shared buffer that can hold a `recordBufferSize` number of records. The logic is described in our documentation at: https://druid.apache.org/docs/27.0.0/development/extensions-core/kinesis-ingestion/#determine-fetch-settings 

There is a problem with the logic that this pr fixes: the memory limits rely on a hard-coded “estimated record size” that is `10 KB` if `deaggregate: false` and `1 MB` if `deaggregate: true`. There have been cases where a supervisor had `deaggregate: true` set even though it wasn’t needed, leading to under-utilization of memory and poor ingestion performance.

Users don’t always know if their records are aggregated or not. Also, even if they could figure it out, it’s better to not have to. So we’d like to eliminate the `deaggregate` parameter, which means we need to do memory management more adaptively based on the actual record sizes.

We take advantage of the fact that GetRecords doesn’t return more than 10MB (https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html ):

This pr: 

eliminates `recordsPerFetch`, always use the max limit of 10000 records (the default limit if not set)

eliminate `deaggregate`, always have it true

cap `fetchThreads` to ensure that if each fetch returns the max (`10MB`) then we don't exceed our budget (`100MB` or `5% of heap`). In practice this means `fetchThreads` will never be more than `10`. Tasks usually don't have that many processors available to them anyway, so in practice I don't think this will change the number of threads for too many deployments

add `recordBufferSizeBytes` as a bytes-based limit rather than records-based limit for the shared queue. We do know the byte size of kinesis records by at this point. Default should be `100MB` or `10% of heap`, whichever is smaller.

add `maxBytesPerPoll` as a bytes-based limit for how much data we poll from shared buffer at a time. Default is `1000000` bytes.

deprecate `recordBufferSize`, use `recordBufferSizeBytes` instead. Warning is logged if `recordBufferSize` is specified

deprecate `maxRecordsPerPoll`, use `maxBytesPerPoll` instead. Warning is logged if maxRecordsPerPoll` is specified

Fixed issue that when the record buffer is full, the fetchRecords logic throws away the rest of the GetRecords result after `recordBufferOfferTimeout` and starts a new shard iterator. This seems excessively churny. Instead,  wait an unbounded amount of time for queue to stop being full. If the queue remains full, we’ll end up right back waiting for it after the restarted fetch.

There was also a call to `newQ::offer` without check in `filterBufferAndResetBackgroundFetch`, which seemed like it could cause data loss. Now checking return value here, and failing if false.

### Release Note

Kinesis ingestion memory tuning config has been greatly simplified, and a more adaptive approach is now taken for the configuration. Here is a summary of the changes made:

eliminates `recordsPerFetch`, always use the max limit of 10000 records (the default limit if not set)

eliminate `deaggregate`, always have it true

cap `fetchThreads` to ensure that if each fetch returns the max (`10MB`) then we don't exceed our budget (`100MB` or `5% of heap`). In practice this means `fetchThreads` will never be more than `10`. Tasks usually don't have that many processors available to them anyway, so in practice I don't think this will change the number of threads for too many deployments

add `recordBufferSizeBytes` as a bytes-based limit rather than records-based limit for the shared queue. We do know the byte size of kinesis records by at this point. Default should be `100MB` or `10% of heap`, whichever is smaller.

add `maxBytesPerPoll` as a bytes-based limit for how much data we poll from shared buffer at a time. Default is `1000000` bytes.

deprecate `recordBufferSize`, use `recordBufferSizeBytes` instead. Warning is logged if `recordBufferSize` is specified

deprecate `maxRecordsPerPoll`, use `maxBytesPerPoll` instead. Warning is logged if maxRecordsPerPoll` is specified
2024-01-19 14:30:21 -05:00
Vadim Ogievetsky c8c04f1032
fix worker disable rendering (#15712) 2024-01-18 13:21:32 -08:00