Commit Graph

647 Commits

Author SHA1 Message Date
Abhishek Radhakrishnan da6b3cbc51
Detect EXPLAIN PLAN queries in web-console (#15570) 2023-12-15 12:12:03 -05:00
Pranav 81fe855b6f
Update com.github.eirslett to fix bad zip issue (#15556) 2023-12-13 17:22:54 -08:00
Vadim Ogievetsky f770eeb8be
Web console: Update webpack-dev-server v3 to v4 (#15555)
* init

* update usage

* revert licenses.yaml

* move the audience-annotations outside of the web console block
2023-12-13 16:16:54 -08:00
Vadim Ogievetsky 0b41b05aa0
Web console: Update and prune dependancies (#15487)
* update the basics
* remove babel
2023-12-05 14:25:07 -08:00
Vadim Ogievetsky aa696b0310
Web console: Log out any request errors in e2e tests for better CI debugging (#15483) 2023-12-05 14:23:47 -08:00
Pranav 82e3c61514
Update lookup model in console (#15472)
* Update lookup model in console

* ran prettify

* move Defaults to info

* setting defaultValue and removing placeholder
2023-12-05 13:22:22 -08:00
Jan Werner 3d3d23c53f
run npm audit fix to update JS packages (#15466) 2023-12-04 13:17:24 -08:00
Vadim Ogievetsky f8bd3b0b7e
Web console: fix tab duplication (#15457)
* fix duplication

* includeFuture defaults to true
2023-11-30 08:29:56 -08:00
Vadim Ogievetsky 31fa63e789
Web console: better management proxy detection (#15453)
* better management proxy detection

* fix checkstyle issue

* add test

* test should read the body also

* use ObjectMapper

* assert read ammount
2023-11-29 21:43:42 -08:00
Vadim Ogievetsky bd6afb7056
time shift literals also (#15433) 2023-11-28 16:42:35 +05:30
Sébastien c8eb7adeb9
Improved web-console's time-chart brush and added auto-granularity (#14990)
* Improved time-chart brush and added auto-granularity

* prettier

* added highlight bubble to explore visualizations

* Added licenses and fixes from PR review

* added missing files...
2023-11-27 12:15:47 -08:00
Vadim Ogievetsky ba1b6fa5a9
Web console: reset spec before looking for tile (#15396)
* reset spec before looking for tile

* improve logging

* log screenshots

* get and log jpeg

* other test tidy up
2023-11-18 09:57:29 -08:00
Vadim Ogievetsky 2cb74433fd
Web console: fix time shifting (#15359)
* fix time shifting
2023-11-14 15:33:52 -08:00
Vadim Ogievetsky fa48d4ea7d
use is not distinct from (#15349) 2023-11-08 18:02:42 -08:00
Vadim Ogievetsky d12f557492
fix ingest datasource detection falling over on paren (#15339) 2023-11-08 13:32:27 -08:00
Adarsh Sanjeev c5fa649ea5
Rename segment load wait parameter (#15251) 2023-10-25 18:08:37 +05:30
Sébastien 5752a1a383
Proper default for taskLockType in streaming ingestion (#15213)
* Proper value for taskLockType in streaming ingestion with concurrent compaction
2023-10-19 21:01:31 +05:30
Sébastien 9ca10c7bd7
Added concurrent compaction switches (#15114)
* Added concurrent compaction switches
2023-10-13 21:03:39 +05:30
Sébastien dba0246aca
Added UI support for waitTillSegmentsLoad (#15110)
This relies on the work done in #14322 and #15076. It allows the user to set waitTillSegmentsLoad in the query context (if they want, else it defaults to true) and shows the results in the UI :
2023-10-11 16:18:42 +05:30
Laksh Singla 5f86072456
Prepare master for Druid 29 (#15121)
Prepare master for Druid 29
2023-10-11 10:33:45 +05:30
Sam Wheating 73bab2f020
Add option to copy query results directly to clipboard (#14889)
* Add option to copy query results to clipboard

* Refactor, allow copying in all formats

---------

Co-authored-by: Sam Wheating <sam.wheating@reddit.com>
2023-09-19 10:25:39 -07:00
Kashif Faraz ec630e3671
Remove deprecated coordinator dynamic configs (#14923)
Changes:

[A] Remove config `decommissioningMaxPercentOfMaxSegmentsToMove`
- It is a complicated config 😅 , 
- It is always desirable to prioritize move from decommissioning servers so that
they can be terminated quickly, so this should always be 100%
- It is already handled by `smartSegmentLoading` (enabled by default)

[B] Remove config `maxNonPrimaryReplicantsToLoad`
This was added in #11135 to address two requirements:
- Prevent coordinator runs from getting stuck assigning too many segments to historicals
- Prevent load of replicas from competing with load of unavailable segments

Both of these requirements are now already met thanks to:
- Round-robin segment assignment
- Prioritization in the new coordinator
- Modifications to `replicationThrottleLimit`
- `smartSegmentLoading` (enabled by default)
2023-09-04 11:54:36 +05:30
Vadim Ogievetsky 680669fd3a
show execution dialog in task view (#14930) 2023-08-30 15:59:34 -07:00
Vadim Ogievetsky 04a1153d0f
line chart fix others not mapping correctly (#14931) 2023-08-30 15:59:26 -07:00
Sébastien 42cfb999cd
Added brush to time-chart (#14929) 2023-08-30 10:36:50 -07:00
Vadim Ogievetsky d295b9158f
Web console: dynamic query parameters UI (#14921)
* fix nvl in table

* add query parameter dialog

* pre-wrap in the tables

* fix typo
2023-08-29 23:14:25 -07:00
Vadim Ogievetsky 30c49c4cfc
Web console: misc fixes and SQL query re-formatting (#14906)
* better dialog formatting

* use CSS to render triangle

* can flatten in kafka also

* better formatting

* better format

* fill in empty values in line chart

* more fp

* add show others
2023-08-25 15:18:37 -07:00
George Shiqi Wu ad32f84586
Fix capacity response in mm-less ingestion (#14888)
Changes:
- Fix capacity response in mm-less ingestion.
- Add field usedClusterCapacity to the GET /totalWorkerCapacity response.
This API should be used to get the total ingestion capacity on the overlord.
- Remove method `isK8sTaskRunner` from interface `TaskRunner`
2023-08-25 08:17:38 +05:30
Katya Macedo 5f74ef56f1
Clean up Kafka supervisor topic (#14651)
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-08-21 11:55:38 -07:00
Vadim Ogievetsky 631dc3b589
add Kafka topic column controls (#14865) 2023-08-21 21:33:23 +05:30
Vadim Ogievetsky 7e147ee905
Web console: Reset to specific offsets dialog (#14863)
* add dialog

* copy changes
2023-08-17 15:38:56 -07:00
Vadim Ogievetsky 59415ba9b2
Web console: expose new coordinator properties in the dialog (#14791)
* expose new coordinator properties in the dialog

* escape
2023-08-17 15:37:23 -07:00
Vadim Ogievetsky dc2ae1e99c
Web console: improving the helper queries by allowing for running inline helper queries (#14801)
* remove helper queries

* fix tests

* take care of zero queries also

* switch to better place
2023-08-16 23:50:43 -07:00
Vadim Ogievetsky cb27d0d2ed
Web console: enable Kafka multi-topic ingestion from the data loader (#14833)
* multi topic ux

* updated to match new api
2023-08-17 09:57:34 +05:30
Vadim Ogievetsky 0b2563fea3
Web console: adding format notice for CSV and TSV (#14783)
* adding format notice for CSV and TSV

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* fix tests

---------

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
2023-08-15 15:35:50 -07:00
Vadim Ogievetsky ec28672d07
Web console: allow format picking for download (#14794)
* allow format picking for download

* better popover

* ux review tweaks
2023-08-11 09:43:29 -07:00
Vadim Ogievetsky b0c78ff295
Web console: make retention dialog clearer (#14793)
* make retention dialog clearer

* tweak

* another tweak

* Update web-console/src/dialogs/retention-dialog/retention-dialog.tsx

Co-authored-by: Suneet Saldanha <suneet@apache.org>

* update snapshot for copy

---------

Co-authored-by: Suneet Saldanha <suneet@apache.org>
2023-08-11 09:43:00 -07:00
Vadim Ogievetsky b1988b2f93
Web console: fix result count (#14786)
* fix result count

* fixes
2023-08-09 20:33:01 +05:30
Adarsh Sanjeev 6837a7be19
Add logging for downsampling sketches in MSQ (#14580)
* Add more logs for downsampling sketches

* Fix builds

* Lower log level

* Add new log message
2023-08-02 20:07:54 +05:30
Vadim Ogievetsky 4a31ae26f4
Web console: Page downloader, and fix JSON error resetting (#14712)
* fix error reset

* add page dialog logic

* add to detail archive

* update tests

* fix plurals

* use jsonl ext

* fix regex issue
2023-08-01 14:25:41 -07:00
Vadim Ogievetsky 153948198c
Web console: fix grouped filtering and add complex menu (#14668)
* fix filtering when grouped

* add complex menu

* complex aggs

* use ResizeObserverEntry

* add quantile and test

* fix style

* update snapshots
2023-08-01 10:41:44 -07:00
Gian Merlino 5387f1bac0
Remove chatAsync parameter, so chat is always async. (#14692)
* Remove chatAsync parameter, so chat is always async.

chatAsync has been made default in Druid 26. I have seen good
battle-testing of it in production, and am comfortable removing the
older sync client.

This was the last remaining usage of IndexTaskClient, so this patch
deletes all that stuff too.

* Remove unthrown exception.

* Remove unthrown exception.

* No more TimeoutException.
2023-07-31 19:42:51 -07:00
Vadim Ogievetsky 9e1650e327
Web console: add durable storage selector (#14669) 2023-07-31 05:33:24 +00:00
dependabot[bot] 915cea7586
Bump decode-uri-component from 0.2.0 to 0.2.2 in /web-console (#13481)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 09:07:09 -07:00
dependabot[bot] b2a07c5db1
Bump word-wrap from 1.2.3 to 1.2.4 in /web-console (#14613)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 06:27:06 -07:00
Vadim Ogievetsky f5784e66d3
Web console: add explore view (#14602)
This PR adds a simple, stateless, SQL backed, data exploration view to the web console. The idea is to let users explore data in Druid with point-and-click interaction and visualizations (instead of writing SQL and looking at a table). This can provide faster time-to-value for a user new to Druid and can allow a Druid veteran to quickly chart some data that they care about.
2023-07-21 11:19:23 +05:30
Vadim Ogievetsky 295653648b
Web console: make typing fun again (#14632)
* extract common function

* make typing fun again
2023-07-20 16:22:41 -07:00
Vadim Ogievetsky 01e9a39544
fix boolean segment filters (#14622) 2023-07-20 09:15:23 +05:30
Vadim Ogievetsky f7348d7389
add support for smartSegmentLoading (#14610) 2023-07-19 14:21:30 -07:00
Vadim Ogievetsky 0a8262edc8
make sure execution mode is set (#14606) 2023-07-18 11:54:30 -07:00
AmatyaAvadhanula 0412f40d36
Prepare master branch for next release, 28.0.0 (#14595)
* Prepare master branch for next release, 28.0.0
2023-07-18 09:22:30 +05:30
Vadim Ogievetsky 6becd8188e
Web console: better schema discovery copy (#14601)
* better-schema-discovery-copy

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* auto-format

---------

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-07-17 17:29:54 -07:00
Vadim Ogievetsky 6425ef4c3c
Web console: fix service view filtering on other bugs (#14597)
* fix service view filter

* MSQ choose best timeformat also
2023-07-17 13:57:37 -07:00
Vadim Ogievetsky d5f6749aa3
Web console: catchup to all the backend changes (#14540)
This PR catches the console up to all the backend changes for Druid 27

Specifically:

Add page information to SqlStatementResource API #14512
Allow empty tiered replicants map for load rules #14432
Adding Interactive API's for MSQ engine #14416
Add replication factor column to sys table #14403
Account for data format and compression in MSQ auto taskAssignment #14307
Errors take 3 #14004
2023-07-17 11:26:46 +05:30
Abhishek Radhakrishnan f4ee58eaa8
Add `aggregatorMergeStrategy` property in SegmentMetadata queries (#14560)
* Add aggregatorMergeStrategy property to SegmentMetadaQuery.

- Adds a new property aggregatorMergeStrategy to segmentMetadata query.
aggregatorMergeStrategy currently supports three types of merge strategies -
the legacy strict and lenient strategies, and the new latest strategy.
- The latest strategy considers the latest aggregator from the latest segment
by time order when there's a conflict when merging aggregators from different
segments.
- Deprecate lenientAggregatorMerge property; The API validates that both the new
and old properties are not set, and returns an exception.
- When merging segments as part of segmentMetadata query, the segments have a more
elaborate id -- <datasource>_<interval>_merged_<partition_number> format, similar to
the name format that segments usually contain. Previously it was simply "merged".
- Adjust unit tests to test the latest strategy, to assert the returned complete
SegmentAnalysis object instead of just the aggregators for completeness.

* Don't explicitly set strict strategy in tests

* Apply suggestions from code review

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/segmentmetadataquery.md

* Apply suggestions from code review

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

---------

Co-authored-by: Katya Macedo <38017980+ektravel@users.noreply.github.com>
2023-07-13 12:37:36 -04:00
dependabot[bot] c91148c43b
Bump tough-cookie from 4.0.0 to 4.1.3 in /web-console (#14557)
Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.0.0 to 4.1.3.
- [Release notes](https://github.com/salesforce/tough-cookie/releases)
- [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/salesforce/tough-cookie/compare/v4.0.0...v4.1.3)

---
updated-dependencies:
- dependency-name: tough-cookie
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-11 08:53:42 -07:00
Kashif Faraz a6547febaf
Remove unused coordinator dynamic configs (#14524)
After #13197 , several coordinator configs are now redundant as they are not being
used anymore, neither with `smartSegmentLoading` nor otherwise.

Changes:
- Remove dynamic configs `emitBalancingStats`: balancer error stats are always
emitted, debug stats can be logged by using `debugDimensions`
- `useBatchedSegmentSampler`, `percentOfSegmentsToConsiderPerMove`:
batched segment sampling is always used
- Add test to verify deserialization with unknown properties
- Update `CoordinatorRunStats` to always track stats, this can be optimized later.
2023-07-06 12:11:10 +05:30
Vadim Ogievetsky 6fd28fc185
Web console: split the Ingestion view into two views: Supervisors and Tasks (#14395)
* init split

* don't crash if unable to get running tasks

* update snapshots

* push down state into call

* googies

* simplify

* update e2e tests

* feedback fixes

* update e2e tests

* better icons

* fix test

* adjust colors
2023-06-14 10:42:30 -07:00
John Gozde 4d146ca87d
Upgrades the React dependency to v18 (#14380)
* Use react 18

* Remove deprecated usage of Toaster

* Make AppToaster lazy

* Update testing-library, snapshots

* Licenses

* Document lazy-init, add license header
2023-06-09 12:09:13 -07:00
John Gozde cfc2a8d286
Switch to @blueprint/datetime2 (#14371)
* Bump blueprint packages

* Switch to datetime2 components

* Update licenses

* Update snapshots
2023-06-05 22:18:05 -07:00
John Gozde c14e54cf93
Remove context params from class component ctors (#14366) 2023-06-05 11:15:28 -07:00
Kashif Faraz 8091c6a547
Update default values in CoordinatorDynamicConfig (#14269)
The defaults of the following config values in the `CoordinatorDynamicConfig` are being updated.

1. `maxSegmentsInNodeLoadingQueue = 500` (previous = 100)
2. `replicationThrottleLimit = 500` (previous = 10)
Rationale: With round-robin segment assignment now being the default assignment technique,
the Coordinator can assign a large number of under-replicated/unavailable segments very quickly,
without getting stuck in `RunRules` duty due to very slow strategy-based cost computations.

3. `maxSegmentsToMove = 100` (previous = 5)
Rationale: A very low value (say 5) is ineffective in balancing especially if there are many segments
to balance. A very large value can cause excessive moves, which has these disadvantages:
- Load of moving segments competing with load of unavailable/under-replicated segments
- Unnecessary network costs due to constant download and delete of segments

These defaults will be revisited after #13197 is merged.
2023-05-30 08:51:33 +05:30
Vadim Ogievetsky 1873fca6c7
Web console: update DQT to latest version and fix bigint crash (#14318)
* update dqt

* don't crash on bigint values

* better submit experiance

* bump to an even version
2023-05-24 17:40:45 -07:00
Katya Macedo 269137c682
Update Ingestion section (#14023)
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
Co-authored-by: Victoria Lim <lim.t.victoria@gmail.com>
2023-05-19 09:42:27 -07:00
Vadim Ogievetsky 7f66fd049b
don't show merged stats until needed (#14311) 2023-05-18 20:32:58 -07:00
Vadim Ogievetsky 0a3889b192
account for auto allowing for leading and trailing spaces (#14224) 2023-05-08 13:18:31 -07:00
Vadim Ogievetsky 4c15e978f1
Web console: misc bug fixes (#14216)
* fixing little things

* clear edit columns when switching to SQL tab

* updated snapshots
2023-05-05 15:45:19 -07:00
Abhishek Agarwal edfd46ed45
Better actionable error message when druid services are not running (#14202)
We have seen that the first-time users often don't know the next steps if druid services are unresponsive for some reason. This PR makes some of those messages a bit more clear.
2023-05-04 18:03:59 +05:30
Vadim Ogievetsky ad93635e45
Web console: allow stringly schemas in the data loader (#14189)
* allow stringly schemas

* fix copy

* feedback fixes

* feedback

* fix copy

* add warning

* indicate submitting

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

* feedback fix

* copy fix

---------

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>
2023-05-02 23:13:21 -07:00
Vadim Ogievetsky f976837eaa
allow marking segments as used when the whole datasoruce is unused (#14185) 2023-04-28 19:45:50 -07:00
Vadim Ogievetsky 98db960794
fix task query error decode (#14174) 2023-04-27 15:26:07 -07:00
Vadim Ogievetsky fceb505833
Web console: allow __time in MSQ (#14165)
* works in MSQ

* fix spec conversion
2023-04-27 09:02:22 -07:00
Vadim Ogievetsky e4d99c3e26
set count on rule history api (#14164) 2023-04-26 01:44:20 -07:00
Vadim Ogievetsky ee06137787
better native json error UX (#14155) 2023-04-25 10:30:42 -07:00
Vadim Ogievetsky b95708f389
quick fix the tier selector (#14143) 2023-04-21 17:21:00 -07:00
Vadim Ogievetsky e7ae825e0c
Web console: better end of (MSQ) query segment loading UX (#14120)
* better end of query segment loading UX

* fix snapshot

* handle case when MSQ query returns results directly

* add ip address column icon

* better icons

* add variance icon

* better summary
2023-04-20 10:26:43 -07:00
Vadim Ogievetsky 4fffee1776
Web console: better lookup 404 detection (#14108)
* better lookup 404 detection

* update snapshot
2023-04-19 09:27:55 -07:00
Clint Wylie f6a0888bc0
document arrays in sql (#12549)
* document arrays in sql

* adjustments

* Update docs/querying/sql-array-functions.md

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/sql-data-types.md

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/sql-data-types.md

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/sql-array-functions.md

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update docs/querying/sql-array-functions.md

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>

* Update sql-array-functions.md

* fix stuff

* fix spelling

---------

Co-authored-by: Katya Macedo  <38017980+ektravel@users.noreply.github.com>
2023-04-17 19:08:46 -07:00
Abhishek Agarwal 8affcaa7eb
Increase timeout for call to get worker capacity (#14095) 2023-04-16 21:24:13 +05:30
Atul Mohan e3c160f2f2
Add start_time column to sys.servers (#13358)
Adds a new column start_time to sys.servers that captures the time at which the server was added to the cluster.
2023-04-14 15:23:34 +05:30
Clint Wylie 1aef72aa7e
Bump up the version in pom to 27.0.0 in preparation of release (#14051) 2023-04-10 14:56:59 +05:30
Vadim Ogievetsky 5ee4ecee62
Web console: use new sampler features (#14017)
* use new sampler features

* supprot kafka format

* update DQT, fix tests

* prefer non numeric formats

* fix input format step

* boost SQL data loader

* delete dimension in auto discover mode

* inline example specs

* feedback updates

* yeet the format into valueFormat when switching to kafka

* kafka format is now a toggle

* even better form layout

* rename
2023-04-07 06:28:29 -07:00
Vadim Ogievetsky 981662e9f4
Web console: add a nice UI for overlord dynamic configs and improve the docs (#13993)
* in progress

* better form

* doc updates

* doc changes

* add inline docs

* fix tests

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* final fixes

* fix case

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* fix overflow

* fix spelling

---------

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2023-03-31 10:12:25 -07:00
Vadim Ogievetsky abb7133153
Web console: use EXTEND syntax (#13985)
* use EXTEND syntax

* update licenses

* update demo queries

* updated snapshots

* add join algorithm selector

* dismiss
2023-03-29 16:19:49 -07:00
Vadim Ogievetsky 8d125b7c7f
Web console: segment writing progress indication (#13929)
* add segment writing progress indication

* update with more metrics

* add push metric
2023-03-22 16:34:38 -07:00
John Gozde 38adac4369
Dart sass (#13937)
* Run npx saas-migrator division

* Switch to dart sass

* Upgrade blueprint

* Remove deprecated import syntax

* Prettify

* Snapshots
2023-03-16 12:44:24 -07:00
Andreas Maechler 46766d245c
Replace deprecated substr with slice (#13822) 2023-03-15 03:57:06 -07:00
Gian Merlino fe9d0c46d5
Improve memory efficiency of WrappedRoaringBitmap. (#13889)
* Improve memory efficiency of WrappedRoaringBitmap.

Two changes:

1) Use an int[] for sizes 4 or below.
2) Remove the boolean compressRunOnSerialization. Doesn't save much
   space, but it does save a little, and it isn't adding a ton of value
   to have it be configurable. It was originally configurable in case
   anything broke when enabling it, but it's been a while and nothing
   has broken.

* Slight adjustment.

* Adjust for inspection.

* Updates.

* Update snaps.

* Update test.

* Adjust test.

* Fix snaps.
2023-03-09 15:48:02 -08:00
Vadim Ogievetsky ca4df85941
fix SQL in segment card (#13895) 2023-03-07 13:08:04 -08:00
Vadim Ogievetsky 38b6373bf7
Web console: Compaction history dialog (#13861)
* initial renames

* add comaction history diff

* final fixes

* update snapshots

* use maps

* update test
2023-03-06 11:52:25 -08:00
Elliott Freis d93fdb2632
Bump CycloneDX module to address POM errors (#13878)
* Bump CycloneDX module to address POM errors

* Including web-console in the PR

---------

Co-authored-by: Elliott Freis <elliottfreis@Elliott-Freis.earth.dynamic.blacklight.net>
2023-03-03 15:39:15 +05:30
Abhishek Radhakrishnan 775f89c75b
Include workaround for CycloneDX is causing POM build errors to web-console as well (#13874) 2023-03-02 00:06:49 -08:00
Vadim Ogievetsky 13721f5998
upgrade druid query toolkit (#13848) 2023-02-28 14:34:21 +05:30
Vadim Ogievetsky e4e6c7ed01
make completions smarter (#13830) 2023-02-23 10:17:10 -08:00
Vadim Ogievetsky 1ca0edb8c9
Web console: Fixes query cancel NPE and more (#13786)
* add null icon

* empty string table cell

* enable views only if they will work

* make sure method exists

* use SQL compatible nulls for e2e tests
2023-02-15 15:02:50 -08:00
John Gozde b33962cab7
Upgrade typescript and other dependencies (#13762)
* Bump zustand, licenses

* Bump TypeScript, Eslint, use type imports

* Switch to react-shallow-renderer from enzyme

* Update ts-loader
2023-02-06 23:12:54 -08:00
Vadim Ogievetsky 3b62d7929c
Web console: Data loader should allow for multiline JSON messages in kafka (#13709)
* stricter

* data loader should allow for mulit-line json

* add await

* kinesis also
2023-01-25 21:23:18 -08:00
Vadim Ogievetsky 4ee4d99b8d
better error reporting (#13636) 2023-01-05 20:00:33 -08:00
Vadim Ogievetsky fdc8aa2833
better show totals when grouping (#13631) 2023-01-05 18:06:32 -08:00
Vadim Ogievetsky 8773d619a2
Web console: tidy up stage UI (#13615)
* show the right info

* sort indicator

* nicer marker

* move error icon
2022-12-22 12:52:16 -08:00
Vadim Ogievetsky e23abc710a
Web console: default max workers to cluster capacity and simplify live reports (#13577)
* step

* better capacity

* start with capacity

* more compressed stats display

* better rule editor

* fix SQL data loader also

* update snapshots

* new line

* better formatting
2022-12-16 15:13:32 -08:00
Vadim Ogievetsky 639decdf2e
fix preview droping out of MSQ mode (#13586) 2022-12-16 15:13:07 -08:00
Rishabh Singh 4ebdfe226d
Druid automated quickstart (#13365)
* Druid automated quickstart

* remove conf/druid/single-server/quickstart/_common/historical/jvm.config

* Minor changes in python script

* Add lower bound memory for some services

* Additional runtime properties for services

* Update supervise script to accept command arguments, corresponding changes in druid-quickstart.py

* File end newline

* Limit the ability to start multiple instances of a service, documentation changes

* simplify script arguments

* restore changes in medium profile

* run-druid refactor

* compute and pass middle manager runtime properties to run-druid
supervise script changes to process java opts array
use argparse, leave free memory, logging

* Remove extra quotes from mm task javaopts array

* Update logic to compute minimum memory

* simplify run-druid

* remove debug options from run-druid

* resolve the config_path provided

* comment out service specific runtime properties which are computed in the code

* simplify run-druid

* clean up docs, naming changes

* Throw ValueError exception on illegal state

* update docs

* rename args, compute_only -> compute, run_zk -> zk

* update help documentation

* update help documentation

* move task memory computation into separate method

* Add validation checks

* remove print

* Add validations

* remove start-druid bash script, rename start-druid-main

* Include tasks in lower bound memory calculation

* Fix test

* 256m instead of 256g

* caffeine cache uses 5% of heap

* ensure min task count is 2, task count is monotonic

* update configs and documentation for runtime props in conf/druid/single-server/quickstart

* Update docs

* Specify memory argument for each profile in single-server.md

* Update middleManager runtime.properties

* Move quickstart configs to conf/druid/base, add bash launch script, support python2

* Update supervise script

* rename base config directory to auto

* rename python script, changes to pass repeated args to supervise

* remove exmaples/conf/druid/base dir

* add docs

* restore changes in conf dir

* update start-druid-auto

* remove hashref for commands in supervise script

* start-druid-main java_opts array is comma separated

* update entry point script name in python script

* Update help docs

* documentation changes

* docs changes

* update docs

* add support for running indexer

* update supported services list

* update help

* Update python.md

* remove dir

* update .spelling

* Remove dependency on psutil and pathlib

* update docs

* Update get_physical_memory method

* Update help docs

* update docs

* update method to get physical memory on python

* udpate spelling

* update .spelling

* minor change

* Minor change

* memory comptuation for indexer

* update start-druid

* Update python.md

* Update single-server.md

* Update python.md

* run python3 --version to check if python is installed

* Update supervise script

* start-druid: echo message if python not found

* update anchor text

* minor change

* Update condition in supervise script

* JVM not jvm in docs
2022-12-09 11:04:02 -08:00
Vadim Ogievetsky d8e27eaab4
update error anchors (#13527) 2022-12-08 13:18:35 -08:00
Vadim Ogievetsky d85fb8cc4e
Web console: improve compaction status display (#13523)
* improve compaction status display

* even more accurate

* fix snapshot
2022-12-07 21:03:59 -08:00
Vadim Ogievetsky 9679f6a9b5
Web console: add arrayOfDoublesSketch and other small fixes (#13486)
* add padding and keywords

* add arrayOfDoubles

* Update docs/development/extensions-core/datasketches-tuple.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update docs/development/extensions-core/datasketches-tuple.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update docs/development/extensions-core/datasketches-tuple.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update docs/development/extensions-core/datasketches-tuple.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update docs/development/extensions-core/datasketches-tuple.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* partiton int

* fix docs

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2022-12-06 21:21:49 -08:00
Vadim Ogievetsky 2fdcfffe40
don't render duration if aggregated (#13455) 2022-11-30 19:21:07 -08:00
Vadim Ogievetsky d8f4353c43
Web console: be more robust to aux queries failing and improve kill tasks (#13431)
* be more robust to aux queries failing

* feedback fixes

* remove empty block

* fix spelling

* remove killAllDataSources from the console
2022-11-28 16:50:38 -08:00
Vadim Ogievetsky a2d5e335f3
Web console: Index spec dialog (#13425)
* add index spec dialog

* add sanpshot
2022-11-28 11:40:45 -08:00
dependabot[bot] 16385c7101
Bump minimatch and replace in /web-console (#13396)
Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.0.5 and updates ancestor dependency [replace](https://github.com/ALMaclaine/replace). These dependencies need to be updated together.


Updates `minimatch` from 3.0.4 to 3.0.5
- [Release notes](https://github.com/isaacs/minimatch/releases)
- [Commits](https://github.com/isaacs/minimatch/compare/v3.0.4...v3.0.5)

Updates `replace` from 1.2.1 to 1.2.2
- [Release notes](https://github.com/ALMaclaine/replace/releases)
- [Commits](https://github.com/ALMaclaine/replace/commits)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-type: indirect
- dependency-name: replace
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-23 12:16:00 -08:00
Kashif Faraz 7cf761cee4
Prepare master branch for next release, 26.0.0 (#13401)
* Prepare master branch for next release, 26.0.0

* Use docker image for druid 24.0.1

* Fix version in druid-it-cases pom.xml
2022-11-22 15:31:01 +05:30
Vadim Ogievetsky fe34ecc5e3
add ability to make inputFormat part of the example datasets (#13402) 2022-11-21 12:50:44 -08:00
Kashif Faraz 133054bf27
Make batched segment sampling the default, minor cleanup of coordinator config (#13391)
The batch segment sampling performs significantly better than the older method
of sampling if there are a large number of used segments. It also avoids duplicates.

Changes:
- Make batch segment sampling the default
- Deprecate the property `useBatchedSegmentSampler`
- Remove unused coordinator config `druid.coordinator.loadqueuepeon.repeatDelay`
- Cleanup `KillUnusedSegments`
- Simplify `KillUnusedSegmentsTest`, add better tests, remove redundant tests
2022-11-21 20:31:46 +05:30
Vadim Ogievetsky 08fa0383b9
add supported indexSpec options (#13388) 2022-11-19 00:00:32 -08:00
Vadim Ogievetsky c628947c31
Web console: streaming json input format specifics (#13381)
* streaming json input format specifics

* goodies
2022-11-18 14:15:16 -08:00
Vadim Ogievetsky a3d45f6086
treat user cancelation seriously (#13376) 2022-11-18 14:04:16 -08:00
Kashif Faraz 71b133f3ff
Add `RoundRobinServerSelector` to speed up segment assignments (#13367)
Segment assignments can take very long due to the strategy cost computation
for a large number of segments. This commit allows segment assignments to be
done in a round-robin fashion within a tier. Only segment balancing takes cost-based
decisions to move segments around.

Changes
- Add dynamic config `useRoundRobinSegmentAssignment` with default value false
- Add `RoundRobinServerSelector`. This does not implement the `BalancerStrategy`
as it does not conform to that contract and may also be used in conjunction with a
strategy (round-robin for `RunRules` and a cost strategy for `BalanceSegments`)
- Drops are still cost-based even when round-robin assignment is enabled.
2022-11-16 20:05:17 +05:30
dependabot[bot] 61a7199f2b
Bump loader-utils from 1.4.0 to 1.4.2 in /web-console (#13372)
Bumps [loader-utils](https://github.com/webpack/loader-utils) from 1.4.0 to 1.4.2.
- [Release notes](https://github.com/webpack/loader-utils/releases)
- [Changelog](https://github.com/webpack/loader-utils/blob/v1.4.2/CHANGELOG.md)
- [Commits](https://github.com/webpack/loader-utils/compare/v1.4.0...v1.4.2)

---
updated-dependencies:
- dependency-name: loader-utils
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-16 20:00:33 +08:00
Vadim Ogievetsky f6aca21e82
Web console: update DQT to version 0.17 (#13323)
* update to DQT 17

* update licenses

* after npm i
2022-11-07 17:47:11 -08:00
Vadim Ogievetsky 176934e849
Web console: expose in the UI that a query is stuck waiting for task slots (#13291)
* add pending info

* update tests

* fixes after review
2022-11-01 09:40:55 -07:00
Margaret Brewster e40c7f2a67
Update data loader parse screen help text (#13241)
Co-authored-by: Margaret Brewster <margaretbrewster@Maggie-Brewster.local>
2022-10-29 11:51:37 +08:00
Vadim Ogievetsky 32020247d1
Web console: Update dqt to a version that adds quotes by default (#13243)
* update dqt

* auto quote by default
2022-10-28 01:06:45 -07:00
Vadim Ogievetsky 573e12c75f
Web console: making the cell filter menu more functional, removing the old query view, and updating d3 (#13169)
* remove old query view

* update tests

* add filter

* fix test

* bump d3 things to latest versions

* rent too far into the future with d3

* make config dialogs load

* goodies

* update snapshots

* only compute duration when running or pending
2022-10-07 12:44:40 -07:00
Vadim Ogievetsky ebfe1c0c90
Web console: fix DQT import (#13159)
* fix dqt import

* update licenses

* update tests
2022-09-30 09:31:06 -07:00
Vadim Ogievetsky a910764e41
better spec conversion with issues (#13136) 2022-09-22 10:46:57 -07:00
Vadim Ogievetsky 6c1dc6589e
initialize all counters for stages with input (#13137) 2022-09-22 08:10:50 -07:00
Vadim Ogievetsky f1d3728371
append to exisitng callout (#13130) 2022-09-21 19:39:28 -07:00
Vadim Ogievetsky b9edfe34a4
be consistent about referring to the web console by its name (#13118) 2022-09-19 15:02:17 -07:00
Vadim Ogievetsky de8f229bed
Web console: correctly escape path based flatten specs (#13105)
* fix path generation

* do escape

* fix replace

* fix replace for good
2022-09-17 14:02:42 -07:00
Vadim Ogievetsky c62a822121
support kafka lookups (#13098) 2022-09-16 15:25:25 -07:00
Vadim Ogievetsky 078b50ebe1
link to error docs (#13094) 2022-09-15 15:06:08 -07:00
Vadim Ogievetsky 08d6aca528
Web console: better detection for arrays containing objects (#13077)
* better detection for arrays containing objects

* include boolean also
2022-09-12 18:50:29 -07:00
Vadim Ogievetsky d978afc5b7
fix number of expected functions (#13050) 2022-09-09 13:42:01 -07:00
Vadim Ogievetsky 5cc5f7b60c
quote columns, datasources in auto complete if needed (#13060) 2022-09-09 11:22:40 -07:00
Kashif Faraz 7e20d70242
Fix web-console message in MSQ data loader (#12996)
* Fix typo in web-console message

* Prettify the changes
2022-09-07 13:34:10 -07:00
Vadim Ogievetsky 92789cfc4a
default to no compare (#13041) 2022-09-07 08:28:28 -07:00
Vadim Ogievetsky 2cf449386f
Web console: upgrade the console to use node 16 (#13017)
* upgrade the console to use node 16

* run npm audit fix
2022-09-06 11:15:23 -07:00
Vadim Ogievetsky 0ae515bd3c
Web console: don't crash if cookies are totally disabled (#13013)
* fix local storage detection

* fix numeric input dialog
2022-09-01 16:10:23 -07:00
Vadim Ogievetsky 5e850c6ea3
Make console e2e tests run in band so as to not hog task slots (#13004)
* increase e2e timeline

* get rid of pull deps

* increase post index task timeoout

* boost msq e2e timeout

* run in band
2022-08-30 21:55:53 -07:00
Vadim Ogievetsky 054688528f
don't show transform actions on * queries (#13005) 2022-08-30 21:54:18 -07:00
Abhishek Agarwal 618757352b
Bump up the version to 25.0.0 (#12975)
* Bump up the version to 25.0.0

* Fix the version in console
2022-08-29 11:27:38 +05:30
Vadim Ogievetsky acb09ff18b
grab warnings from correct key + test (#12977) 2022-08-25 18:47:33 -07:00
Lee Surprenant 4c61378ad1
fix broken link in web-console (#12976)
In 0.23.0, the info-button links to https://druid.apache.org/docs/0.23.0/ingestion/flatten-json.html which is a 404

If I got the spot right, this should now link to https://druid.apache.org/docs/[VERSION]/ingestion/data-formats.html#flattenspec
2022-08-25 16:47:41 -07:00
Vadim Ogievetsky fd6cfcb8fb
Web console: fix pagination, add error delimiters (#12969)
* fix pagination, add error delimiters

* reword debug message
2022-08-25 08:18:21 -07:00
Vadim Ogievetsky 04ee7abeff
Web console: Multi-stage query support (#12919)
* MSQ web console

* fix typo in comments

* remove useless conditional

* wrap SQL_DATA_TYPES

* fixes sus regex

* rewrite regex

* remove problematic regex

* fix UTs

* convert PARTITIONED / CLUSTERED BY to ORDER BY for preview

* fix log

* updated to use shuffle

* Web console: Use Ace.Completion directly (#1405)

* Use Ace.Completion directly

* Another Ace.Completion

* better comment

* fix column ordering in e2e test

* add nested data example also

Co-authored-by: John Gozde <john.gozde@imply.io>
2022-08-24 16:17:12 -07:00
Abhishek Agarwal f0fc45028e
Update year in the notice file and the release process instructions (#12622)
* Update notice file

* Update release process instructions

* Better release instructions

* Update copyright year

* Web console: updated the doc link (#12619)

* updated the doc link

* update snapshots

Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
2022-08-23 18:17:18 +05:30
Clint Wylie f8097ccfaa
basic docs for nested column query functions (#12922)
* basic docs for nested column query functions
2022-08-19 17:12:19 -07:00
AmatyaAvadhanula d294404924
Kinesis ingestion with empty shards (#12792)
Kinesis ingestion requires all shards to have at least 1 record at the required position in druid.
Even if this is satisified initially, resharding the stream can lead to empty intermediate shards. A significant delay in writing to newly created shards was also problematic.

Kinesis shard sequence numbers are big integers. Introduce two more custom sequence tokens UNREAD_TRIM_HORIZON and UNREAD_LATEST to indicate that a shard has not been read from and that it needs to be read from the start or the end respectively.
These values can be used to avoid the need to read at least one record to obtain a sequence number for ingesting a newly discovered shard.

If a record cannot be obtained immediately, use a marker to obtain the relevant shardIterator and use this shardIterator to obtain a valid sequence number. As long as a valid sequence number is not obtained, continue storing the token as the offset.

These tokens (UNREAD_TRIM_HORIZON and UNREAD_LATEST) are logically ordered to be earlier than any valid sequence number.

However, the ordering requires a few subtle changes to the existing mechanism for record sequence validation:

The sequence availability check ensures that the current offset is before the earliest available sequence in the shard. However, current token being an UNREAD token indicates that any sequence number in the shard is valid (despite the ordering)

Kinesis sequence numbers are inclusive i.e if current sequence == end sequence, there are more records left to read.
However, the equality check is exclusive when dealing with UNREAD tokens.
2022-08-05 22:38:58 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Vadim Ogievetsky f2a7970a6c
reindex flow should take order from Druid (#12790) 2022-07-14 20:03:33 -07:00