Commit Graph

409 Commits

Author SHA1 Message Date
Vadim Ogievetsky 5e850c6ea3
Make console e2e tests run in band so as to not hog task slots (#13004)
* increase e2e timeline

* get rid of pull deps

* increase post index task timeoout

* boost msq e2e timeout

* run in band
2022-08-30 21:55:53 -07:00
Vadim Ogievetsky 054688528f
don't show transform actions on * queries (#13005) 2022-08-30 21:54:18 -07:00
Abhishek Agarwal 618757352b
Bump up the version to 25.0.0 (#12975)
* Bump up the version to 25.0.0

* Fix the version in console
2022-08-29 11:27:38 +05:30
Vadim Ogievetsky acb09ff18b
grab warnings from correct key + test (#12977) 2022-08-25 18:47:33 -07:00
Lee Surprenant 4c61378ad1
fix broken link in web-console (#12976)
In 0.23.0, the info-button links to https://druid.apache.org/docs/0.23.0/ingestion/flatten-json.html which is a 404

If I got the spot right, this should now link to https://druid.apache.org/docs/[VERSION]/ingestion/data-formats.html#flattenspec
2022-08-25 16:47:41 -07:00
Vadim Ogievetsky fd6cfcb8fb
Web console: fix pagination, add error delimiters (#12969)
* fix pagination, add error delimiters

* reword debug message
2022-08-25 08:18:21 -07:00
Vadim Ogievetsky 04ee7abeff
Web console: Multi-stage query support (#12919)
* MSQ web console

* fix typo in comments

* remove useless conditional

* wrap SQL_DATA_TYPES

* fixes sus regex

* rewrite regex

* remove problematic regex

* fix UTs

* convert PARTITIONED / CLUSTERED BY to ORDER BY for preview

* fix log

* updated to use shuffle

* Web console: Use Ace.Completion directly (#1405)

* Use Ace.Completion directly

* Another Ace.Completion

* better comment

* fix column ordering in e2e test

* add nested data example also

Co-authored-by: John Gozde <john.gozde@imply.io>
2022-08-24 16:17:12 -07:00
Abhishek Agarwal f0fc45028e
Update year in the notice file and the release process instructions (#12622)
* Update notice file

* Update release process instructions

* Better release instructions

* Update copyright year

* Web console: updated the doc link (#12619)

* updated the doc link

* update snapshots

Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
2022-08-23 18:17:18 +05:30
Clint Wylie f8097ccfaa
basic docs for nested column query functions (#12922)
* basic docs for nested column query functions
2022-08-19 17:12:19 -07:00
AmatyaAvadhanula d294404924
Kinesis ingestion with empty shards (#12792)
Kinesis ingestion requires all shards to have at least 1 record at the required position in druid.
Even if this is satisified initially, resharding the stream can lead to empty intermediate shards. A significant delay in writing to newly created shards was also problematic.

Kinesis shard sequence numbers are big integers. Introduce two more custom sequence tokens UNREAD_TRIM_HORIZON and UNREAD_LATEST to indicate that a shard has not been read from and that it needs to be read from the start or the end respectively.
These values can be used to avoid the need to read at least one record to obtain a sequence number for ingesting a newly discovered shard.

If a record cannot be obtained immediately, use a marker to obtain the relevant shardIterator and use this shardIterator to obtain a valid sequence number. As long as a valid sequence number is not obtained, continue storing the token as the offset.

These tokens (UNREAD_TRIM_HORIZON and UNREAD_LATEST) are logically ordered to be earlier than any valid sequence number.

However, the ordering requires a few subtle changes to the existing mechanism for record sequence validation:

The sequence availability check ensures that the current offset is before the earliest available sequence in the shard. However, current token being an UNREAD token indicates that any sequence number in the shard is valid (despite the ordering)

Kinesis sequence numbers are inclusive i.e if current sequence == end sequence, there are more records left to read.
However, the equality check is exclusive when dealing with UNREAD tokens.
2022-08-05 22:38:58 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Vadim Ogievetsky f2a7970a6c
reindex flow should take order from Druid (#12790) 2022-07-14 20:03:33 -07:00
Yuanli Han 50f1f5840d
show json and add search box (#12784) 2022-07-14 17:01:30 -07:00
Yuanli Han 82315779ff
fix segment timeline bar chart (#12782) 2022-07-14 16:58:24 -07:00
Vadim Ogievetsky 14e5b8325c
make tick formatting more robust (#12788) 2022-07-14 16:56:53 -07:00
Vadim Ogievetsky c1c2104bd6
fix ordering in e2e test (#12775) 2022-07-13 15:08:00 -07:00
Rohan Garg c09b5a2294
Fix skipTests build flag (#12716)
* fix skipTests

* Skip console UTs with skipTests

* Use skipTests in skip-tests profile
2022-06-29 21:59:26 -07:00
Vadim Ogievetsky 6f7fa334fd
Web console: totalNumMergeTasks can be set on range also (#12648)
* totalNumMergeTasks can be set on range also

* fix formatting
2022-06-14 11:18:17 -07:00
Gian Merlino 1ace7336cd
Update node to 14.19.3. (#12632) 2022-06-10 10:18:12 -07:00
dependabot[bot] 4558b815e5
Bump eventsource from 1.1.0 to 1.1.1 in /web-console (#12595)
Bumps [eventsource](https://github.com/EventSource/eventsource) from 1.1.0 to 1.1.1.
- [Release notes](https://github.com/EventSource/eventsource/releases)
- [Changelog](https://github.com/EventSource/eventsource/blob/master/HISTORY.md)
- [Commits](https://github.com/EventSource/eventsource/compare/v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: eventsource
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-01 22:04:30 -07:00
Vadim Ogievetsky a235aca2b3
Web console: fix go to segments not working (#12541)
* use correct filter syntax

* fix tests
2022-05-19 14:34:03 -07:00
Atul Mohan eb6de94e1f
Add daily stats to console (#12329) 2022-05-05 15:31:21 -07:00
Vadim Ogievetsky 2d8eb117c0
Web console: add a button to get out of restricted mode, make capability detection more robust (#12503)
* allow unrestrict

* update tests
2022-05-05 15:06:59 -07:00
Vadim Ogievetsky fb08bac01a
Web console: Misc table fixes (#12489)
* Misc table fixes

* extract default className

* table spacing updates

* fix e2e action selector

* try more times

* make the web console exist again
2022-05-03 12:08:08 -07:00
Abhishek Agarwal 2fe053c5cb
Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
Vadim Ogievetsky a72cc28959
good stuff (#12435) 2022-04-14 00:23:06 -07:00
Vadim Ogievetsky a139cd22aa
Web console: Misc fixes and improvements (#12361)
* Misc fixes

* pad column numbers

* make shard_type filterable
2022-04-12 22:20:28 -07:00
dependabot[bot] 7d10e02463
Bump minimist from 1.2.5 to 1.2.6 in /web-console (#12401)
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-06 16:55:14 -07:00
Vadim Ogievetsky e8635df9e7
clean up some bp3 classes (#12403) 2022-04-06 15:27:44 -07:00
John Gozde 90680543d0
Blueprint 4 (#12391)
* Update blueprint dependencies & LICENSES

* Switch to bp4 namespace; use bp-ns variable in overrides

* Add webpack alias for colors.scss

* Snapshots

* Update selectors in e2e tests
2022-04-04 10:34:22 -07:00
Tejaswini Bandlamudi 984904779b
Increase default DatasourceCompactionConfig.inputSegmentSizeBytes to Long.MAX_VALUE (#12381)
The current default value of inputSegmentSizeBytes is 400MB, which is pretty
low for most compaction use cases. Thus most users are forced to override the
default.

The default value is now increased to Long.MAX_VALUE.
2022-04-04 16:28:53 +05:30
Gian Merlino 3b373114dc
Officially support Java 11. (#12232)
There aren't any changes in this patch that improve Java 11
compatibility; these changes have already been done separately. This
patch merely updates documentation and explicit Java version checks.

The log message adjustments in DruidProcessingConfig are there to make
things a little nicer when running in Java 11, where we can't measure
direct memory _directly_, and so we may auto-size processing buffers
incorrectly.
2022-03-04 14:15:45 -08:00
Victoria Lim c61b19d443
Refactor SQL docs (#12239)
* refactor and link fixes

* add sql docs to left nav

* code format for needle

* updated web console script

* link fixes

* update earliest/latest functions

* edits for grammar and style

* more link fixes

* another link

* update with #12226

* update .spelling file
2022-02-11 14:43:30 -08:00
John Gozde 0d23713862
Web console: update dev dependencies (#12240)
* Update dependencies

* Set "allowFunctions: true" for react/jsx-no-bind

* Prettify

* npm audit fix

* Bump playwright, set testEnvironment=node

* Bump node and npm

* Revert "Bump node and npm"

This reverts commit e93c8e00e7.

* Minor/patch bump node@14, npm@6
2022-02-08 16:37:36 -08:00
Vadim Ogievetsky 090c429c8c
Web console: make it possible to namespace local storage, auto flatten spec generator should deal better with bad data (#12238)
* improve computeFlattenExprsForData

* allow local storage namespacing

* add test
2022-02-07 18:52:41 -08:00
Vadim Ogievetsky fc76b014d1
Web console: fix supervisor stats table pagination (#12227)
* fixes #11627 supervisor stats table pagination

* use spread instead of assign
2022-02-03 00:09:21 -08:00
Vadim Ogievetsky bc408bacc8
Web console: Adding a shard detail column to the segments view (#12212)
* shard spec details

* improve pattern match

* refactor spec cleanup

* better format detection

* update JSONbig

* add multiline option to autoform
2022-02-02 18:46:17 -08:00
AshishKapoor 801d9e7f1b
[Web Console] fix deprecated keyboard event method "keyCode" with "key" (#11947)
* 11946 fix keyboard event keyCode method

* fix with key and respective cases

* e.which method required since it's anyway deprecated too.

* updated as per feedback
2022-02-02 18:26:21 -08:00
Vadim Ogievetsky fe8530dac4
Change link to Apache Druid Slack (#12206) 2022-01-27 21:10:40 -08:00
Vadim Ogievetsky 8ae5de5114
Web console: fix multi-value dimension column detection and tidy up (#12160)
* streamline services view query

* better column type detection

* fix query view page size bug

* fill out MetricSpec interface

* fix pagination in status dialog

* update tests

* adjust pagination

* better type guessing

* better test fixtures

* add Avg. row size to segments view
2022-01-25 15:46:29 -08:00
Vadim Ogievetsky 6ce14e6b17
allow W in durations (#12175) 2022-01-20 09:17:43 +05:30
aggarwalakshay eb4fafe08f
Upgrading follow-redirects to 1.14.7 (#12153)
* Upgrading follow-redirects to 1.14.7

* removed the existing follow-redirects i.e. 1.14.4 from package-lock.json
2022-01-13 14:01:36 -08:00
Vadim Ogievetsky 9cd52ed914
Web console: make range partitioning a first class citizen of the console (#12146)
* first class support for range partitioning

* update e2e tests
2022-01-12 03:50:10 -08:00
Vadim Ogievetsky 2a41b7bffa
Web console: correctly cancel JSON shaped SQL queries (#12134)
* misc fixes

* type typo
2022-01-10 14:24:05 -08:00
Vadim Ogievetsky 476d0bf4be
Web console: remove console.log (#12094)
* rm console.log

* force path-parse to 1.0.7
2021-12-22 19:31:23 -08:00
Vadim Ogievetsky 37112d24e2
Web console: new Ace, diff view, and cleanup. Decorating the console for the holidays 🎁 (#12085)
* Add diff view and upgrade AceEditor

* fix test

* function doc parsing fixes

* escape args

* allowKeys

* everyone gets a diff

* update snapshot
2021-12-22 16:31:17 -08:00
Vadim Ogievetsky 0cc998d8a1
improve spec upgrading (#12072) 2021-12-15 10:28:21 -08:00
Vadim Ogievetsky 6ac4e2dbb8
Web console: use query actions in query view (#12037)
* use query actions

* feedback fixes
2021-12-08 13:01:29 -08:00
Lucas Capistrant 150902b95c
clean up the balancing code around the batched vs deprecated way of sampling segments to balance (#11960)
* clean up the balancing code around the batched vs deprecated way of sampling segments to balance

* fix docs, clarify comments, add deprecated annotations to legacy code

* remove unused variable

* update dynamic config dialog in console to state percentOfSegmentsToConsiderPerMove deprecated

* fix dynamic config text for percentOfSegmentsToConsiderPerMove

* run prettier to cleanup coordinator-dynamic-config.tsx changes

* update jest snapshot

* update documentation per review feedback
2021-12-07 14:47:46 -08:00
Vadim Ogievetsky 1d3c8c187b
Web console: query view improvements and other fixes (#12031)
* don't copy commas

* use numeric type information

* add VALUES keyword

* propogate rollup config into spec

* fix

* cleanup

* understand range partitioning

* update snapshots

* better comp apis

* fix segment pages

* update snapshots
2021-12-07 10:16:16 -08:00