Commit Graph

620 Commits

Author SHA1 Message Date
Vadim Ogievetsky 6e222d47c8
fix crash when parsing data in data loader that can not be parsed (#15983) 2024-02-28 14:37:24 -08:00
Vadim Ogievetsky a81429746d
Web console: fix typos in Kinesis suggestions, add regions and groups (#15900)
* fix typo

* update regions

* add China

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: Benedict Jin <asdf2014@apache.org>

* add ,

---------

Co-authored-by: Benedict Jin <asdf2014@apache.org>
2024-02-27 10:00:02 +08:00
Vadim Ogievetsky bf3139562c
Web console: support for the export execution state (#15969)
* init

* add CSV keyword
2024-02-26 11:28:25 -08:00
Vadim Ogievetsky 28b3e117cf
Web console: Add input format props (#15950)
* fix typo

* add Protobuf

* better padding
2024-02-26 11:28:09 -08:00
Vadim Ogievetsky c52ddd0b86
make flattenSpec location adaptive (#15946) 2024-02-22 14:07:04 -08:00
Vadim Ogievetsky 66f54f2066
allow compaction config slots to drop to 0 (#15877) 2024-02-15 15:27:15 +08:00
Vadim Ogievetsky 26815d425b
Web console: add system fields UI (#15858)
This PR adds console support for configuring system fields in the batch data loader.
2024-02-08 11:08:55 +05:30
Vadim Ogievetsky f2b242b6e6
update console to core Druid changes (#15854) 2024-02-07 19:44:25 +05:30
George Shiqi Wu edb1ac1b71
Update azure console tile (#15820)
* Save web console changes

* Working new input type

* fix tests
2024-02-05 13:11:39 -08:00
Vadim Ogievetsky fcd65c9801
Web console: use arrayIngestMode: array (#15588)
* Adapt to new array mode

* Feedback fixes

* fixing type detection and highlighting

* goodies

* add docs

* feedback fixes

* finish array work

* update snapshots

* typo fix

* color fixes

* small fix

* make MVDs default for now

* better sqlStringifyArrays default

* fix spec converter

* fix tests
2024-01-31 20:19:29 -08:00
Vadim Ogievetsky 0089f6b905
Web console: Don't force waitUntilSegmentLoad to true (#15781)
* Don't force setting waitUntilSegmentsLoad

* delete irrelevant code
2024-01-31 16:16:36 +05:30
Abhishek Radhakrishnan 9f95a691f7
Extension to read and ingest Delta Lake tables (#15755)
* something

* test commit

* compilation fix

* more compilation fixes (fixme placeholders)

* Comment out druid-kereberos build since it conflicts with newly added transitive deps from delta-lake

Will need to sort out the dependencies later.

* checkpoint

* remove snapshot schema since we can get schema from the row

* iterator bug fix

* json json json

* sampler flow

* empty impls for read(InputStats) and sample()

* conversion?

* conversion, without timestamp

* Web console changes to show Delta Lake

* Asset bug fix and tile load

* Add missing pieces to input source info, etc.

* fix stuff

* Use a different delta lake asset

* Delta lake extension dependencies

* Cleanup

* Add InputSource, module init and helper code to process delta files.

* Test init

* Checkpoint changes

* Test resources and updates

* some fixes

* move to the correct package

* More tests

* Test cleanup

* TODOs

* Test updates

* requirements and javadocs

* Adjust dependencies

* Update readme

* Bump up version

* fixup typo in deps

* forbidden api and checkstyle checks

* Trim down dependencies

* new lines

* Fixup Intellij inspections.

* Add equals() and hashCode()

* chain splits, intellij inspections

* review comments and todo placeholder

* fix up some docs

* null table path and test dependencies. Fixup broken link.

* run prettify

* Different test; fixes

* Upgrade pyspark and delta-spark to latest (3.5.0 and 3.0.0) and regenerate tests

* yank the old test resource.

* add a couple of sad path tests

* Updates to readme based on latest.

* Version support

* Extract Delta DateTime converstions to DeltaTimeUtils class and add test

* More comprehensive split tests.

* Some test renames.

* Cleanup and update instructions.

* add pruneSchema() optimization for table scans.

* Oops, missed the parquet files.

* Update default table and rename schema constants.

* Test setup and misc changes.

* Add class loader logic as the context class loader is unaware about extension classes

* change some table client creation logic.

* Add hadoop-aws, hadoop-common and related exclusions.

* Remove org.apache.hadoop:hadoop-common

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Add entry to .spelling to fix docs static check

---------

Co-authored-by: abhishekagarwal87 <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Laksh Singla <lakshsingla@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2024-01-30 21:53:50 -08:00
Vadim Ogievetsky 497e2123f0
Web console: Make table driven query modification actions work with slices. (#15779)
* Make table driven query modification actions work with slices.

* cleanup found query prefix

* fix regex complexity
2024-01-29 20:09:46 -08:00
Vadim Ogievetsky 45ad47cc66
allow segment table to sort on start and end when grouped (#15720) 2024-01-26 18:59:23 +08:00
Sensor 4e50a14d50
fix router page value inconsistent issue (#15742)
* fix router page value inconsistent issue

* make the fix more universal as suggested

* minor
2024-01-26 11:02:30 +08:00
Vadim Ogievetsky 55ed69f830
update mocks (#15745) 2024-01-24 11:41:52 -08:00
Karan Kumar c4990f56d6
Prepare main branch for next 30.0.0 release. (#15707) 2024-01-23 15:55:54 +05:30
zachjsh 9d4e8053a4
Kinesis adaptive memory management (#15360)
### Description

Our Kinesis consumer works by using the [GetRecords API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) in some number of `fetchThreads`, each fetching some number of records (`recordsPerFetch`) and each inserting into a shared buffer that can hold a `recordBufferSize` number of records. The logic is described in our documentation at: https://druid.apache.org/docs/27.0.0/development/extensions-core/kinesis-ingestion/#determine-fetch-settings 

There is a problem with the logic that this pr fixes: the memory limits rely on a hard-coded “estimated record size” that is `10 KB` if `deaggregate: false` and `1 MB` if `deaggregate: true`. There have been cases where a supervisor had `deaggregate: true` set even though it wasn’t needed, leading to under-utilization of memory and poor ingestion performance.

Users don’t always know if their records are aggregated or not. Also, even if they could figure it out, it’s better to not have to. So we’d like to eliminate the `deaggregate` parameter, which means we need to do memory management more adaptively based on the actual record sizes.

We take advantage of the fact that GetRecords doesn’t return more than 10MB (https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html ):

This pr: 

eliminates `recordsPerFetch`, always use the max limit of 10000 records (the default limit if not set)

eliminate `deaggregate`, always have it true

cap `fetchThreads` to ensure that if each fetch returns the max (`10MB`) then we don't exceed our budget (`100MB` or `5% of heap`). In practice this means `fetchThreads` will never be more than `10`. Tasks usually don't have that many processors available to them anyway, so in practice I don't think this will change the number of threads for too many deployments

add `recordBufferSizeBytes` as a bytes-based limit rather than records-based limit for the shared queue. We do know the byte size of kinesis records by at this point. Default should be `100MB` or `10% of heap`, whichever is smaller.

add `maxBytesPerPoll` as a bytes-based limit for how much data we poll from shared buffer at a time. Default is `1000000` bytes.

deprecate `recordBufferSize`, use `recordBufferSizeBytes` instead. Warning is logged if `recordBufferSize` is specified

deprecate `maxRecordsPerPoll`, use `maxBytesPerPoll` instead. Warning is logged if maxRecordsPerPoll` is specified

Fixed issue that when the record buffer is full, the fetchRecords logic throws away the rest of the GetRecords result after `recordBufferOfferTimeout` and starts a new shard iterator. This seems excessively churny. Instead,  wait an unbounded amount of time for queue to stop being full. If the queue remains full, we’ll end up right back waiting for it after the restarted fetch.

There was also a call to `newQ::offer` without check in `filterBufferAndResetBackgroundFetch`, which seemed like it could cause data loss. Now checking return value here, and failing if false.

### Release Note

Kinesis ingestion memory tuning config has been greatly simplified, and a more adaptive approach is now taken for the configuration. Here is a summary of the changes made:

eliminates `recordsPerFetch`, always use the max limit of 10000 records (the default limit if not set)

eliminate `deaggregate`, always have it true

cap `fetchThreads` to ensure that if each fetch returns the max (`10MB`) then we don't exceed our budget (`100MB` or `5% of heap`). In practice this means `fetchThreads` will never be more than `10`. Tasks usually don't have that many processors available to them anyway, so in practice I don't think this will change the number of threads for too many deployments

add `recordBufferSizeBytes` as a bytes-based limit rather than records-based limit for the shared queue. We do know the byte size of kinesis records by at this point. Default should be `100MB` or `10% of heap`, whichever is smaller.

add `maxBytesPerPoll` as a bytes-based limit for how much data we poll from shared buffer at a time. Default is `1000000` bytes.

deprecate `recordBufferSize`, use `recordBufferSizeBytes` instead. Warning is logged if `recordBufferSize` is specified

deprecate `maxRecordsPerPoll`, use `maxBytesPerPoll` instead. Warning is logged if maxRecordsPerPoll` is specified
2024-01-19 14:30:21 -05:00
Vadim Ogievetsky c8c04f1032
fix worker disable rendering (#15712) 2024-01-18 13:21:32 -08:00
Vadim Ogievetsky 5b769a7d32
Update load query detail archive dialog for file input support (#15632)
* Update execution-submit-dialog for file input support

Modified the execution-submit-dialog to support file inputs instead of text inputs for better usability. Users can now submit their queries by selecting a JSON file directly or dragging the file into the dialog. Made appropriate UI adjustments to accommodate this change in execution-submit-dialog styles file.

* Update web-console/src/views/workbench-view/execution-submit-dialog/execution-submit-dialog.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/views/workbench-view/execution-submit-dialog/execution-submit-dialog.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/views/workbench-view/execution-submit-dialog/execution-submit-dialog.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update drag-and-drop instructions in execution-submit-dialog

* Add snapshot tests for ExecutionSubmitDialog

* prettify

---------

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2024-01-10 20:48:46 -08:00
Ankit Kothari 355c2f5da0
Add sql + ingestion compatibility for first/last on numeric values (#15607)
SQL compatibility for numeric last and first column types.
Ingestion UI now provides option for first and last aggregation as well.
2024-01-10 12:59:38 +05:30
Vadim Ogievetsky 85b8cf9f37
Web console: Fix concurrent tasks (#15649)
* Improve handling of concurrent tasks option

* Update snapshots
2024-01-09 16:09:42 -08:00
Vadim Ogievetsky 84adb9255e
Web console: Fix spec conversion, expose failOnEmptyInsert (#15627)
* Spec converter should dedupe the columns

* Add "Fail on empty insert" setting to QueryContext toggles
2024-01-08 12:05:12 -08:00
Abhishek Radhakrishnan da6b3cbc51
Detect EXPLAIN PLAN queries in web-console (#15570) 2023-12-15 12:12:03 -05:00
Pranav 81fe855b6f
Update com.github.eirslett to fix bad zip issue (#15556) 2023-12-13 17:22:54 -08:00
Vadim Ogievetsky f770eeb8be
Web console: Update webpack-dev-server v3 to v4 (#15555)
* init

* update usage

* revert licenses.yaml

* move the audience-annotations outside of the web console block
2023-12-13 16:16:54 -08:00
Vadim Ogievetsky 0b41b05aa0
Web console: Update and prune dependancies (#15487)
* update the basics
* remove babel
2023-12-05 14:25:07 -08:00
Vadim Ogievetsky aa696b0310
Web console: Log out any request errors in e2e tests for better CI debugging (#15483) 2023-12-05 14:23:47 -08:00
Pranav 82e3c61514
Update lookup model in console (#15472)
* Update lookup model in console

* ran prettify

* move Defaults to info

* setting defaultValue and removing placeholder
2023-12-05 13:22:22 -08:00
Jan Werner 3d3d23c53f
run npm audit fix to update JS packages (#15466) 2023-12-04 13:17:24 -08:00
Vadim Ogievetsky f8bd3b0b7e
Web console: fix tab duplication (#15457)
* fix duplication

* includeFuture defaults to true
2023-11-30 08:29:56 -08:00
Vadim Ogievetsky 31fa63e789
Web console: better management proxy detection (#15453)
* better management proxy detection

* fix checkstyle issue

* add test

* test should read the body also

* use ObjectMapper

* assert read ammount
2023-11-29 21:43:42 -08:00
Vadim Ogievetsky bd6afb7056
time shift literals also (#15433) 2023-11-28 16:42:35 +05:30
Sébastien c8eb7adeb9
Improved web-console's time-chart brush and added auto-granularity (#14990)
* Improved time-chart brush and added auto-granularity

* prettier

* added highlight bubble to explore visualizations

* Added licenses and fixes from PR review

* added missing files...
2023-11-27 12:15:47 -08:00
Vadim Ogievetsky ba1b6fa5a9
Web console: reset spec before looking for tile (#15396)
* reset spec before looking for tile

* improve logging

* log screenshots

* get and log jpeg

* other test tidy up
2023-11-18 09:57:29 -08:00
Vadim Ogievetsky 2cb74433fd
Web console: fix time shifting (#15359)
* fix time shifting
2023-11-14 15:33:52 -08:00
Vadim Ogievetsky fa48d4ea7d
use is not distinct from (#15349) 2023-11-08 18:02:42 -08:00
Vadim Ogievetsky d12f557492
fix ingest datasource detection falling over on paren (#15339) 2023-11-08 13:32:27 -08:00
Adarsh Sanjeev c5fa649ea5
Rename segment load wait parameter (#15251) 2023-10-25 18:08:37 +05:30
Sébastien 5752a1a383
Proper default for taskLockType in streaming ingestion (#15213)
* Proper value for taskLockType in streaming ingestion with concurrent compaction
2023-10-19 21:01:31 +05:30
Sébastien 9ca10c7bd7
Added concurrent compaction switches (#15114)
* Added concurrent compaction switches
2023-10-13 21:03:39 +05:30
Sébastien dba0246aca
Added UI support for waitTillSegmentsLoad (#15110)
This relies on the work done in #14322 and #15076. It allows the user to set waitTillSegmentsLoad in the query context (if they want, else it defaults to true) and shows the results in the UI :
2023-10-11 16:18:42 +05:30
Laksh Singla 5f86072456
Prepare master for Druid 29 (#15121)
Prepare master for Druid 29
2023-10-11 10:33:45 +05:30
Sam Wheating 73bab2f020
Add option to copy query results directly to clipboard (#14889)
* Add option to copy query results to clipboard

* Refactor, allow copying in all formats

---------

Co-authored-by: Sam Wheating <sam.wheating@reddit.com>
2023-09-19 10:25:39 -07:00
Kashif Faraz ec630e3671
Remove deprecated coordinator dynamic configs (#14923)
Changes:

[A] Remove config `decommissioningMaxPercentOfMaxSegmentsToMove`
- It is a complicated config 😅 , 
- It is always desirable to prioritize move from decommissioning servers so that
they can be terminated quickly, so this should always be 100%
- It is already handled by `smartSegmentLoading` (enabled by default)

[B] Remove config `maxNonPrimaryReplicantsToLoad`
This was added in #11135 to address two requirements:
- Prevent coordinator runs from getting stuck assigning too many segments to historicals
- Prevent load of replicas from competing with load of unavailable segments

Both of these requirements are now already met thanks to:
- Round-robin segment assignment
- Prioritization in the new coordinator
- Modifications to `replicationThrottleLimit`
- `smartSegmentLoading` (enabled by default)
2023-09-04 11:54:36 +05:30
Vadim Ogievetsky 680669fd3a
show execution dialog in task view (#14930) 2023-08-30 15:59:34 -07:00
Vadim Ogievetsky 04a1153d0f
line chart fix others not mapping correctly (#14931) 2023-08-30 15:59:26 -07:00
Sébastien 42cfb999cd
Added brush to time-chart (#14929) 2023-08-30 10:36:50 -07:00
Vadim Ogievetsky d295b9158f
Web console: dynamic query parameters UI (#14921)
* fix nvl in table

* add query parameter dialog

* pre-wrap in the tables

* fix typo
2023-08-29 23:14:25 -07:00
Vadim Ogievetsky 30c49c4cfc
Web console: misc fixes and SQL query re-formatting (#14906)
* better dialog formatting

* use CSS to render triangle

* can flatten in kafka also

* better formatting

* better format

* fill in empty values in line chart

* more fp

* add show others
2023-08-25 15:18:37 -07:00
George Shiqi Wu ad32f84586
Fix capacity response in mm-less ingestion (#14888)
Changes:
- Fix capacity response in mm-less ingestion.
- Add field usedClusterCapacity to the GET /totalWorkerCapacity response.
This API should be used to get the total ingestion capacity on the overlord.
- Remove method `isK8sTaskRunner` from interface `TaskRunner`
2023-08-25 08:17:38 +05:30
Katya Macedo 5f74ef56f1
Clean up Kafka supervisor topic (#14651)
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-08-21 11:55:38 -07:00
Vadim Ogievetsky 631dc3b589
add Kafka topic column controls (#14865) 2023-08-21 21:33:23 +05:30
Vadim Ogievetsky 7e147ee905
Web console: Reset to specific offsets dialog (#14863)
* add dialog

* copy changes
2023-08-17 15:38:56 -07:00
Vadim Ogievetsky 59415ba9b2
Web console: expose new coordinator properties in the dialog (#14791)
* expose new coordinator properties in the dialog

* escape
2023-08-17 15:37:23 -07:00
Vadim Ogievetsky dc2ae1e99c
Web console: improving the helper queries by allowing for running inline helper queries (#14801)
* remove helper queries

* fix tests

* take care of zero queries also

* switch to better place
2023-08-16 23:50:43 -07:00
Vadim Ogievetsky cb27d0d2ed
Web console: enable Kafka multi-topic ingestion from the data loader (#14833)
* multi topic ux

* updated to match new api
2023-08-17 09:57:34 +05:30
Vadim Ogievetsky 0b2563fea3
Web console: adding format notice for CSV and TSV (#14783)
* adding format notice for CSV and TSV

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* fix tests

---------

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
2023-08-15 15:35:50 -07:00
Vadim Ogievetsky ec28672d07
Web console: allow format picking for download (#14794)
* allow format picking for download

* better popover

* ux review tweaks
2023-08-11 09:43:29 -07:00
Vadim Ogievetsky b0c78ff295
Web console: make retention dialog clearer (#14793)
* make retention dialog clearer

* tweak

* another tweak

* Update web-console/src/dialogs/retention-dialog/retention-dialog.tsx

Co-authored-by: Suneet Saldanha <suneet@apache.org>

* update snapshot for copy

---------

Co-authored-by: Suneet Saldanha <suneet@apache.org>
2023-08-11 09:43:00 -07:00
Vadim Ogievetsky b1988b2f93
Web console: fix result count (#14786)
* fix result count

* fixes
2023-08-09 20:33:01 +05:30
Adarsh Sanjeev 6837a7be19
Add logging for downsampling sketches in MSQ (#14580)
* Add more logs for downsampling sketches

* Fix builds

* Lower log level

* Add new log message
2023-08-02 20:07:54 +05:30
Vadim Ogievetsky 4a31ae26f4
Web console: Page downloader, and fix JSON error resetting (#14712)
* fix error reset

* add page dialog logic

* add to detail archive

* update tests

* fix plurals

* use jsonl ext

* fix regex issue
2023-08-01 14:25:41 -07:00
Vadim Ogievetsky 153948198c
Web console: fix grouped filtering and add complex menu (#14668)
* fix filtering when grouped

* add complex menu

* complex aggs

* use ResizeObserverEntry

* add quantile and test

* fix style

* update snapshots
2023-08-01 10:41:44 -07:00
Gian Merlino 5387f1bac0
Remove chatAsync parameter, so chat is always async. (#14692)
* Remove chatAsync parameter, so chat is always async.

chatAsync has been made default in Druid 26. I have seen good
battle-testing of it in production, and am comfortable removing the
older sync client.

This was the last remaining usage of IndexTaskClient, so this patch
deletes all that stuff too.

* Remove unthrown exception.

* Remove unthrown exception.

* No more TimeoutException.
2023-07-31 19:42:51 -07:00
Vadim Ogievetsky 9e1650e327
Web console: add durable storage selector (#14669) 2023-07-31 05:33:24 +00:00
dependabot[bot] 915cea7586
Bump decode-uri-component from 0.2.0 to 0.2.2 in /web-console (#13481)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 09:07:09 -07:00
dependabot[bot] b2a07c5db1
Bump word-wrap from 1.2.3 to 1.2.4 in /web-console (#14613)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 06:27:06 -07:00
Vadim Ogievetsky f5784e66d3
Web console: add explore view (#14602)
This PR adds a simple, stateless, SQL backed, data exploration view to the web console. The idea is to let users explore data in Druid with point-and-click interaction and visualizations (instead of writing SQL and looking at a table). This can provide faster time-to-value for a user new to Druid and can allow a Druid veteran to quickly chart some data that they care about.
2023-07-21 11:19:23 +05:30
Vadim Ogievetsky 295653648b
Web console: make typing fun again (#14632)
* extract common function

* make typing fun again
2023-07-20 16:22:41 -07:00
Vadim Ogievetsky 01e9a39544
fix boolean segment filters (#14622) 2023-07-20 09:15:23 +05:30
Vadim Ogievetsky f7348d7389
add support for smartSegmentLoading (#14610) 2023-07-19 14:21:30 -07:00
Vadim Ogievetsky 0a8262edc8
make sure execution mode is set (#14606) 2023-07-18 11:54:30 -07:00
AmatyaAvadhanula 0412f40d36
Prepare master branch for next release, 28.0.0 (#14595)
* Prepare master branch for next release, 28.0.0
2023-07-18 09:22:30 +05:30
Vadim Ogievetsky 6becd8188e
Web console: better schema discovery copy (#14601)
* better-schema-discovery-copy

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* auto-format

---------

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-07-17 17:29:54 -07:00
Vadim Ogievetsky 6425ef4c3c
Web console: fix service view filtering on other bugs (#14597)
* fix service view filter

* MSQ choose best timeformat also
2023-07-17 13:57:37 -07:00
Vadim Ogievetsky d5f6749aa3
Web console: catchup to all the backend changes (#14540)
This PR catches the console up to all the backend changes for Druid 27

Specifically:

Add page information to SqlStatementResource API #14512
Allow empty tiered replicants map for load rules #14432
Adding Interactive API's for MSQ engine #14416
Add replication factor column to sys table #14403
Account for data format and compression in MSQ auto taskAssignment #14307
Errors take 3 #14004
2023-07-17 11:26:46 +05:30
Abhishek Radhakrishnan f4ee58eaa8
Add `aggregatorMergeStrategy` property in SegmentMetadata queries (#14560)
* Add aggregatorMergeStrategy property to SegmentMetadaQuery.

- Adds a new property aggregatorMergeStrategy to segmentMetadata query.
aggregatorMergeStrategy currently supports three types of merge strategies -
the legacy strict and lenient strategies, and the new latest strategy.
- The latest strategy considers the latest aggregator from the latest segment
by time order when there's a conflict when merging aggregators from different
segments.
- Deprecate lenientAggregatorMerge property; The API validates that both the new
and old properties are not set, and returns an exception.
- When merging segments as part of segmentMetadata query, the segments have a more
elaborate id -- <datasource>_<interval>_merged_<partition_number> format, similar to
the name format that segments usually contain. Previously it was simply "merged".
- Adjust unit tests to test the latest strategy, to assert the returned complete
SegmentAnalysis object instead of just the aggregators for completeness.

* Don't explicitly set strict strategy in tests

* Apply suggestions from code review

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/segmentmetadataquery.md

* Apply suggestions from code review

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

---------

Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com>
2023-07-13 12:37:36 -04:00
dependabot[bot] c91148c43b
Bump tough-cookie from 4.0.0 to 4.1.3 in /web-console (#14557)
Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.0.0 to 4.1.3.
- [Release notes](https://github.com/salesforce/tough-cookie/releases)
- [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/salesforce/tough-cookie/compare/v4.0.0...v4.1.3)

---
updated-dependencies:
- dependency-name: tough-cookie
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-11 08:53:42 -07:00
Kashif Faraz a6547febaf
Remove unused coordinator dynamic configs (#14524)
After #13197 , several coordinator configs are now redundant as they are not being
used anymore, neither with `smartSegmentLoading` nor otherwise.

Changes:
- Remove dynamic configs `emitBalancingStats`: balancer error stats are always
emitted, debug stats can be logged by using `debugDimensions`
- `useBatchedSegmentSampler`, `percentOfSegmentsToConsiderPerMove`:
batched segment sampling is always used
- Add test to verify deserialization with unknown properties
- Update `CoordinatorRunStats` to always track stats, this can be optimized later.
2023-07-06 12:11:10 +05:30
Vadim Ogievetsky 6fd28fc185
Web console: split the Ingestion view into two views: Supervisors and Tasks (#14395)
* init split

* don't crash if unable to get running tasks

* update snapshots

* push down state into call

* googies

* simplify

* update e2e tests

* feedback fixes

* update e2e tests

* better icons

* fix test

* adjust colors
2023-06-14 10:42:30 -07:00
John Gozde 4d146ca87d
Upgrades the React dependency to v18 (#14380)
* Use react 18

* Remove deprecated usage of Toaster

* Make AppToaster lazy

* Update testing-library, snapshots

* Licenses

* Document lazy-init, add license header
2023-06-09 12:09:13 -07:00
John Gozde cfc2a8d286
Switch to @blueprint/datetime2 (#14371)
* Bump blueprint packages

* Switch to datetime2 components

* Update licenses

* Update snapshots
2023-06-05 22:18:05 -07:00
John Gozde c14e54cf93
Remove context params from class component ctors (#14366) 2023-06-05 11:15:28 -07:00
Kashif Faraz 8091c6a547
Update default values in CoordinatorDynamicConfig (#14269)
The defaults of the following config values in the `CoordinatorDynamicConfig` are being updated.

1. `maxSegmentsInNodeLoadingQueue = 500` (previous = 100)
2. `replicationThrottleLimit = 500` (previous = 10)
Rationale: With round-robin segment assignment now being the default assignment technique,
the Coordinator can assign a large number of under-replicated/unavailable segments very quickly,
without getting stuck in `RunRules` duty due to very slow strategy-based cost computations.

3. `maxSegmentsToMove = 100` (previous = 5)
Rationale: A very low value (say 5) is ineffective in balancing especially if there are many segments
to balance. A very large value can cause excessive moves, which has these disadvantages:
- Load of moving segments competing with load of unavailable/under-replicated segments
- Unnecessary network costs due to constant download and delete of segments

These defaults will be revisited after #13197 is merged.
2023-05-30 08:51:33 +05:30
Vadim Ogievetsky 1873fca6c7
Web console: update DQT to latest version and fix bigint crash (#14318)
* update dqt

* don't crash on bigint values

* better submit experiance

* bump to an even version
2023-05-24 17:40:45 -07:00
Katya Macedo 269137c682
Update Ingestion section (#14023)
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
Co-authored-by: Victoria Lim <lim.t.victoria@gmail.com>
2023-05-19 09:42:27 -07:00
Vadim Ogievetsky 7f66fd049b
don't show merged stats until needed (#14311) 2023-05-18 20:32:58 -07:00
Vadim Ogievetsky 0a3889b192
account for auto allowing for leading and trailing spaces (#14224) 2023-05-08 13:18:31 -07:00
Vadim Ogievetsky 4c15e978f1
Web console: misc bug fixes (#14216)
* fixing little things

* clear edit columns when switching to SQL tab

* updated snapshots
2023-05-05 15:45:19 -07:00
Abhishek Agarwal edfd46ed45
Better actionable error message when druid services are not running (#14202)
We have seen that the first-time users often don't know the next steps if druid services are unresponsive for some reason. This PR makes some of those messages a bit more clear.
2023-05-04 18:03:59 +05:30
Vadim Ogievetsky ad93635e45
Web console: allow stringly schemas in the data loader (#14189)
* allow stringly schemas

* fix copy

* feedback fixes

* feedback

* fix copy

* add warning

* indicate submitting

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* feedback fix

* copy fix

---------

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2023-05-02 23:13:21 -07:00
Vadim Ogievetsky f976837eaa
allow marking segments as used when the whole datasoruce is unused (#14185) 2023-04-28 19:45:50 -07:00
Vadim Ogievetsky 98db960794
fix task query error decode (#14174) 2023-04-27 15:26:07 -07:00
Vadim Ogievetsky fceb505833
Web console: allow __time in MSQ (#14165)
* works in MSQ

* fix spec conversion
2023-04-27 09:02:22 -07:00
Vadim Ogievetsky e4d99c3e26
set count on rule history api (#14164) 2023-04-26 01:44:20 -07:00
Vadim Ogievetsky ee06137787
better native json error UX (#14155) 2023-04-25 10:30:42 -07:00
Vadim Ogievetsky b95708f389
quick fix the tier selector (#14143) 2023-04-21 17:21:00 -07:00
Vadim Ogievetsky e7ae825e0c
Web console: better end of (MSQ) query segment loading UX (#14120)
* better end of query segment loading UX

* fix snapshot

* handle case when MSQ query returns results directly

* add ip address column icon

* better icons

* add variance icon

* better summary
2023-04-20 10:26:43 -07:00
Vadim Ogievetsky 4fffee1776
Web console: better lookup 404 detection (#14108)
* better lookup 404 detection

* update snapshot
2023-04-19 09:27:55 -07:00