Commit Graph

570 Commits

Author SHA1 Message Date
Vadim Ogievetsky b9edfe34a4
be consistent about referring to the web console by its name (#13118) 2022-09-19 15:02:17 -07:00
Vadim Ogievetsky de8f229bed
Web console: correctly escape path based flatten specs (#13105)
* fix path generation

* do escape

* fix replace

* fix replace for good
2022-09-17 14:02:42 -07:00
Vadim Ogievetsky c62a822121
support kafka lookups (#13098) 2022-09-16 15:25:25 -07:00
Vadim Ogievetsky 078b50ebe1
link to error docs (#13094) 2022-09-15 15:06:08 -07:00
Vadim Ogievetsky 08d6aca528
Web console: better detection for arrays containing objects (#13077)
* better detection for arrays containing objects

* include boolean also
2022-09-12 18:50:29 -07:00
Vadim Ogievetsky d978afc5b7
fix number of expected functions (#13050) 2022-09-09 13:42:01 -07:00
Vadim Ogievetsky 5cc5f7b60c
quote columns, datasources in auto complete if needed (#13060) 2022-09-09 11:22:40 -07:00
Kashif Faraz 7e20d70242
Fix web-console message in MSQ data loader (#12996)
* Fix typo in web-console message

* Prettify the changes
2022-09-07 13:34:10 -07:00
Vadim Ogievetsky 92789cfc4a
default to no compare (#13041) 2022-09-07 08:28:28 -07:00
Vadim Ogievetsky 2cf449386f
Web console: upgrade the console to use node 16 (#13017)
* upgrade the console to use node 16

* run npm audit fix
2022-09-06 11:15:23 -07:00
Vadim Ogievetsky 0ae515bd3c
Web console: don't crash if cookies are totally disabled (#13013)
* fix local storage detection

* fix numeric input dialog
2022-09-01 16:10:23 -07:00
Vadim Ogievetsky 5e850c6ea3
Make console e2e tests run in band so as to not hog task slots (#13004)
* increase e2e timeline

* get rid of pull deps

* increase post index task timeoout

* boost msq e2e timeout

* run in band
2022-08-30 21:55:53 -07:00
Vadim Ogievetsky 054688528f
don't show transform actions on * queries (#13005) 2022-08-30 21:54:18 -07:00
Abhishek Agarwal 618757352b
Bump up the version to 25.0.0 (#12975)
* Bump up the version to 25.0.0

* Fix the version in console
2022-08-29 11:27:38 +05:30
Vadim Ogievetsky acb09ff18b
grab warnings from correct key + test (#12977) 2022-08-25 18:47:33 -07:00
Lee Surprenant 4c61378ad1
fix broken link in web-console (#12976)
In 0.23.0, the info-button links to https://druid.apache.org/docs/0.23.0/ingestion/flatten-json.html which is a 404

If I got the spot right, this should now link to https://druid.apache.org/docs/[VERSION]/ingestion/data-formats.html#flattenspec
2022-08-25 16:47:41 -07:00
Vadim Ogievetsky fd6cfcb8fb
Web console: fix pagination, add error delimiters (#12969)
* fix pagination, add error delimiters

* reword debug message
2022-08-25 08:18:21 -07:00
Vadim Ogievetsky 04ee7abeff
Web console: Multi-stage query support (#12919)
* MSQ web console

* fix typo in comments

* remove useless conditional

* wrap SQL_DATA_TYPES

* fixes sus regex

* rewrite regex

* remove problematic regex

* fix UTs

* convert PARTITIONED / CLUSTERED BY to ORDER BY for preview

* fix log

* updated to use shuffle

* Web console: Use Ace.Completion directly (#1405)

* Use Ace.Completion directly

* Another Ace.Completion

* better comment

* fix column ordering in e2e test

* add nested data example also

Co-authored-by: John Gozde <john.gozde@imply.io>
2022-08-24 16:17:12 -07:00
Abhishek Agarwal f0fc45028e
Update year in the notice file and the release process instructions (#12622)
* Update notice file

* Update release process instructions

* Better release instructions

* Update copyright year

* Web console: updated the doc link (#12619)

* updated the doc link

* update snapshots

Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
2022-08-23 18:17:18 +05:30
Clint Wylie f8097ccfaa
basic docs for nested column query functions (#12922)
* basic docs for nested column query functions
2022-08-19 17:12:19 -07:00
AmatyaAvadhanula d294404924
Kinesis ingestion with empty shards (#12792)
Kinesis ingestion requires all shards to have at least 1 record at the required position in druid.
Even if this is satisified initially, resharding the stream can lead to empty intermediate shards. A significant delay in writing to newly created shards was also problematic.

Kinesis shard sequence numbers are big integers. Introduce two more custom sequence tokens UNREAD_TRIM_HORIZON and UNREAD_LATEST to indicate that a shard has not been read from and that it needs to be read from the start or the end respectively.
These values can be used to avoid the need to read at least one record to obtain a sequence number for ingesting a newly discovered shard.

If a record cannot be obtained immediately, use a marker to obtain the relevant shardIterator and use this shardIterator to obtain a valid sequence number. As long as a valid sequence number is not obtained, continue storing the token as the offset.

These tokens (UNREAD_TRIM_HORIZON and UNREAD_LATEST) are logically ordered to be earlier than any valid sequence number.

However, the ordering requires a few subtle changes to the existing mechanism for record sequence validation:

The sequence availability check ensures that the current offset is before the earliest available sequence in the shard. However, current token being an UNREAD token indicates that any sequence number in the shard is valid (despite the ordering)

Kinesis sequence numbers are inclusive i.e if current sequence == end sequence, there are more records left to read.
However, the equality check is exclusive when dealing with UNREAD tokens.
2022-08-05 22:38:58 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Vadim Ogievetsky f2a7970a6c
reindex flow should take order from Druid (#12790) 2022-07-14 20:03:33 -07:00
Yuanli Han 50f1f5840d
show json and add search box (#12784) 2022-07-14 17:01:30 -07:00
Yuanli Han 82315779ff
fix segment timeline bar chart (#12782) 2022-07-14 16:58:24 -07:00
Vadim Ogievetsky 14e5b8325c
make tick formatting more robust (#12788) 2022-07-14 16:56:53 -07:00
Vadim Ogievetsky c1c2104bd6
fix ordering in e2e test (#12775) 2022-07-13 15:08:00 -07:00
Rohan Garg c09b5a2294
Fix skipTests build flag (#12716)
* fix skipTests

* Skip console UTs with skipTests

* Use skipTests in skip-tests profile
2022-06-29 21:59:26 -07:00
Vadim Ogievetsky 6f7fa334fd
Web console: totalNumMergeTasks can be set on range also (#12648)
* totalNumMergeTasks can be set on range also

* fix formatting
2022-06-14 11:18:17 -07:00
Gian Merlino 1ace7336cd
Update node to 14.19.3. (#12632) 2022-06-10 10:18:12 -07:00
dependabot[bot] 4558b815e5
Bump eventsource from 1.1.0 to 1.1.1 in /web-console (#12595)
Bumps [eventsource](https://github.com/EventSource/eventsource) from 1.1.0 to 1.1.1.
- [Release notes](https://github.com/EventSource/eventsource/releases)
- [Changelog](https://github.com/EventSource/eventsource/blob/master/HISTORY.md)
- [Commits](https://github.com/EventSource/eventsource/compare/v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: eventsource
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-01 22:04:30 -07:00
Vadim Ogievetsky a235aca2b3
Web console: fix go to segments not working (#12541)
* use correct filter syntax

* fix tests
2022-05-19 14:34:03 -07:00
Atul Mohan eb6de94e1f
Add daily stats to console (#12329) 2022-05-05 15:31:21 -07:00
Vadim Ogievetsky 2d8eb117c0
Web console: add a button to get out of restricted mode, make capability detection more robust (#12503)
* allow unrestrict

* update tests
2022-05-05 15:06:59 -07:00
Vadim Ogievetsky fb08bac01a
Web console: Misc table fixes (#12489)
* Misc table fixes

* extract default className

* table spacing updates

* fix e2e action selector

* try more times

* make the web console exist again
2022-05-03 12:08:08 -07:00
Abhishek Agarwal 2fe053c5cb
Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
Vadim Ogievetsky a72cc28959
good stuff (#12435) 2022-04-14 00:23:06 -07:00
Vadim Ogievetsky a139cd22aa
Web console: Misc fixes and improvements (#12361)
* Misc fixes

* pad column numbers

* make shard_type filterable
2022-04-12 22:20:28 -07:00
dependabot[bot] 7d10e02463
Bump minimist from 1.2.5 to 1.2.6 in /web-console (#12401)
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-06 16:55:14 -07:00
Vadim Ogievetsky e8635df9e7
clean up some bp3 classes (#12403) 2022-04-06 15:27:44 -07:00
John Gozde 90680543d0
Blueprint 4 (#12391)
* Update blueprint dependencies & LICENSES

* Switch to bp4 namespace; use bp-ns variable in overrides

* Add webpack alias for colors.scss

* Snapshots

* Update selectors in e2e tests
2022-04-04 10:34:22 -07:00
Tejaswini Bandlamudi 984904779b
Increase default DatasourceCompactionConfig.inputSegmentSizeBytes to Long.MAX_VALUE (#12381)
The current default value of inputSegmentSizeBytes is 400MB, which is pretty
low for most compaction use cases. Thus most users are forced to override the
default.

The default value is now increased to Long.MAX_VALUE.
2022-04-04 16:28:53 +05:30
Gian Merlino 3b373114dc
Officially support Java 11. (#12232)
There aren't any changes in this patch that improve Java 11
compatibility; these changes have already been done separately. This
patch merely updates documentation and explicit Java version checks.

The log message adjustments in DruidProcessingConfig are there to make
things a little nicer when running in Java 11, where we can't measure
direct memory _directly_, and so we may auto-size processing buffers
incorrectly.
2022-03-04 14:15:45 -08:00
Victoria Lim c61b19d443
Refactor SQL docs (#12239)
* refactor and link fixes

* add sql docs to left nav

* code format for needle

* updated web console script

* link fixes

* update earliest/latest functions

* edits for grammar and style

* more link fixes

* another link

* update with #12226

* update .spelling file
2022-02-11 14:43:30 -08:00
John Gozde 0d23713862
Web console: update dev dependencies (#12240)
* Update dependencies

* Set "allowFunctions: true" for react/jsx-no-bind

* Prettify

* npm audit fix

* Bump playwright, set testEnvironment=node

* Bump node and npm

* Revert "Bump node and npm"

This reverts commit e93c8e00e7.

* Minor/patch bump node@14, npm@6
2022-02-08 16:37:36 -08:00
Vadim Ogievetsky 090c429c8c
Web console: make it possible to namespace local storage, auto flatten spec generator should deal better with bad data (#12238)
* improve computeFlattenExprsForData

* allow local storage namespacing

* add test
2022-02-07 18:52:41 -08:00
Vadim Ogievetsky fc76b014d1
Web console: fix supervisor stats table pagination (#12227)
* fixes #11627 supervisor stats table pagination

* use spread instead of assign
2022-02-03 00:09:21 -08:00
Vadim Ogievetsky bc408bacc8
Web console: Adding a shard detail column to the segments view (#12212)
* shard spec details

* improve pattern match

* refactor spec cleanup

* better format detection

* update JSONbig

* add multiline option to autoform
2022-02-02 18:46:17 -08:00
AshishKapoor 801d9e7f1b
[Web Console] fix deprecated keyboard event method "keyCode" with "key" (#11947)
* 11946 fix keyboard event keyCode method

* fix with key and respective cases

* e.which method required since it's anyway deprecated too.

* updated as per feedback
2022-02-02 18:26:21 -08:00
Vadim Ogievetsky fe8530dac4
Change link to Apache Druid Slack (#12206) 2022-01-27 21:10:40 -08:00
Vadim Ogievetsky 8ae5de5114
Web console: fix multi-value dimension column detection and tidy up (#12160)
* streamline services view query

* better column type detection

* fix query view page size bug

* fill out MetricSpec interface

* fix pagination in status dialog

* update tests

* adjust pagination

* better type guessing

* better test fixtures

* add Avg. row size to segments view
2022-01-25 15:46:29 -08:00
Vadim Ogievetsky 6ce14e6b17
allow W in durations (#12175) 2022-01-20 09:17:43 +05:30
aggarwalakshay eb4fafe08f
Upgrading follow-redirects to 1.14.7 (#12153)
* Upgrading follow-redirects to 1.14.7

* removed the existing follow-redirects i.e. 1.14.4 from package-lock.json
2022-01-13 14:01:36 -08:00
Vadim Ogievetsky 9cd52ed914
Web console: make range partitioning a first class citizen of the console (#12146)
* first class support for range partitioning

* update e2e tests
2022-01-12 03:50:10 -08:00
Vadim Ogievetsky 2a41b7bffa
Web console: correctly cancel JSON shaped SQL queries (#12134)
* misc fixes

* type typo
2022-01-10 14:24:05 -08:00
Vadim Ogievetsky 476d0bf4be
Web console: remove console.log (#12094)
* rm console.log

* force path-parse to 1.0.7
2021-12-22 19:31:23 -08:00
Vadim Ogievetsky 37112d24e2
Web console: new Ace, diff view, and cleanup. Decorating the console for the holidays 🎁 (#12085)
* Add diff view and upgrade AceEditor

* fix test

* function doc parsing fixes

* escape args

* allowKeys

* everyone gets a diff

* update snapshot
2021-12-22 16:31:17 -08:00
Vadim Ogievetsky 0cc998d8a1
improve spec upgrading (#12072) 2021-12-15 10:28:21 -08:00
Vadim Ogievetsky 6ac4e2dbb8
Web console: use query actions in query view (#12037)
* use query actions

* feedback fixes
2021-12-08 13:01:29 -08:00
Lucas Capistrant 150902b95c
clean up the balancing code around the batched vs deprecated way of sampling segments to balance (#11960)
* clean up the balancing code around the batched vs deprecated way of sampling segments to balance

* fix docs, clarify comments, add deprecated annotations to legacy code

* remove unused variable

* update dynamic config dialog in console to state percentOfSegmentsToConsiderPerMove deprecated

* fix dynamic config text for percentOfSegmentsToConsiderPerMove

* run prettier to cleanup coordinator-dynamic-config.tsx changes

* update jest snapshot

* update documentation per review feedback
2021-12-07 14:47:46 -08:00
Vadim Ogievetsky 1d3c8c187b
Web console: query view improvements and other fixes (#12031)
* don't copy commas

* use numeric type information

* add VALUES keyword

* propogate rollup config into spec

* fix

* cleanup

* understand range partitioning

* update snapshots

* better comp apis

* fix segment pages

* update snapshots
2021-12-07 10:16:16 -08:00
Vadim Ogievetsky 1f95a42bb8
Web console: updated the explain dialog to use new explain output (#12009)
This is the UI followup to the work done in #11908

Updated the Explain dialog to use the new output format.
2021-12-02 00:18:11 +05:30
Abhishek Agarwal 652e1491e0
Update default values for tuning parameters in kinesis data loader (#11867) 2021-11-02 23:51:28 +05:30
andreacyc 88bbc8e9e1
Add info for compation config dialog (#11847)
* add-info-for-compation-config-dialog

* correct the info

* remove space typo

* Revert "remove space typo"

This reverts commit 28b28733ae.

* remove typo space

* update snapshots for jest-test
2021-11-02 10:03:29 -07:00
Vadim Ogievetsky 8ea9309168
Web console: update typescript 4.4 for faster build speeds (#11725)
* update typescript

* do not show pagination when there is only one page

* update snapshots

* fix pagination
2021-10-25 21:53:38 -07:00
Vadim Ogievetsky f2106d7621
Web console: Add segment size in bytes column and hide it by default (#11797)
* add segment size column

* allow hidden default column

* fix tests

* update e2e tests
2021-10-25 13:24:44 -07:00
Vadim Ogievetsky 4354e43983
Use existing queryId if it exists (#11834) 2021-10-23 19:02:39 -07:00
andreacyc adb2237628
Fix CVE-2021-3749 reported in security vulnerabilities job (#11786)
* Fix CVE-2021-3749 reported in security vulnerabilities job

* test why test fail

* update axios

* remove console log for testing
2021-10-08 23:02:58 -07:00
Vadim Ogievetsky 635490d568
don't throw local storage errors (#11752) 2021-10-05 18:49:16 -07:00
Vadim Ogievetsky c1e0e6825f
auto refresh in foreground only (#11750) 2021-10-05 18:48:23 -07:00
andreacyc f82baf174e
Support real query cancelling for web console (#11738)
* Support real query cancelling for web console

* use uuid for queryId, create isSql reuse variable, and add catch for rejectionhandled promise

* remove delete api promise.then() response

* slove conflicts

* update read me with debug

* add degub code to test why CI failed

* included a druid extension called druid-testing-tools and it is not build nor loaded by default

* remove unuse variable

* remove debug log
2021-10-05 10:28:49 -07:00
Vadim Ogievetsky b54d989cda
Web console: Account for segment granularity all (#11697)
* accont for segment granularity all

* reverse granularity list to align with ranking
2021-09-13 20:46:11 -07:00
Vadim Ogievetsky 779fe8e6ad
fix segment timeline in FireFox (#11701) 2021-09-13 20:45:18 -07:00
Clint Wylie fe1d8c206a
bump version to 0.23.0-SNAPSHOT (#11670) 2021-09-08 15:56:04 -07:00
Vadim Ogievetsky e4ec3527a4
Web console: Improve the lookup view UX (#11620)
* polish lookup view UX

* update snapshots

* add snapshot to git

* fixes

* update sanpshots

* restore column treatment

* update snapshot

* add gs
2021-08-30 14:36:23 -07:00
Yi Yuan ca5e55cc8c
Fix TRIM help broken in Console SQL Editor (#11628)
* delete escape

* bug fix

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-30 13:04:35 +08:00
Vadim Ogievetsky 257bc5c62f
Web console: fix count aggregation input in the data loader (#11485)
* add typeIs

* fix unused field in count metric

* better types

* typos

* work with readonly types

* factor out apply cancel buttons

* form editor

* selection type

* unsaved changes

* form editor spec

* tidy up sampler

* more menu controls

* update e2e test
2021-08-05 12:30:30 -07:00
Yuanli Han b83742179a
Reduce method invocation of reservoir sampling (#11257)
* reduce method invocation of reservoir sampling

* add a dynamic parameter and add benchmark

* rebase
2021-07-30 22:09:50 +08:00
John Gozde 280c08045f
Update awesome-code-style (#11503) 2021-07-28 09:25:18 -07:00
Vadim Ogievetsky aee2f2e24f
Web console: better handle BigInt math (#11450)
* better handle BigInt math

* correctly brace bigint

* feedback fixes and tests
2021-07-20 17:17:19 -07:00
Vadim Ogievetsky 377b5e708c
Web console: Data loading walkthrough fixes (#11416)
* fix quotes

* fix sql doc parsing

* prevent array-input from losing position while the user is typing

* make group filter click-to-filterable

* fix casing bug in exact table search

* do not sort columns in smaples

* can bypass transform step

* fixed string json parsing

* improve PartitionMessage

* better error messages

* feedback fixes

* tool to order dimensions in schema view
2021-07-10 07:56:50 -07:00
Vadim Ogievetsky 561cc71838
Web console: allow encoding of ASCII control chars (#10795)
* allow encoding of ascii control chars

* change to JSON

* make json escpaes work

* update snapshot

* break out component

* fix test

* update test script

* update formatter to be more chill
2021-06-26 18:54:41 -07:00
Vadim Ogievetsky f56a5b9ba2
Web console: Better hotkeys and library upgrades (#11365)
* improve hotkeys

* fix test name

* refactor explain dialog

* explain tests

* small fixes

* update to popover2

* use resize sensor 2

* misc cleanup
2021-06-17 18:24:29 -07:00
Vadim Ogievetsky 2e98d3c1ad
Web console: Make segment timeline work over all time intervals (#11359)
* tidy up

* add to segments view

* add unit tests for date

* better util export

* fix ds view

* fix tests

* fix test in time

* unset untermediate state
2021-06-17 16:10:06 -07:00
John Gozde ac9b1f81b4
Web console: Remove support for IE11 and other older browsers (#11357)
* Use common browserlist and update to drop IE11

* Change TypeScript target to ES2016

* Update browserslist for "supports es6" support

* Show a warning if accessed from an unsupported browser

* Inline browser-update styles; detect SyntaxErrors too

* Better wording

* Upgrade to the latest Blueprint

* Refactor RunButton to be FC, use useHotkeys

* Remove dead license

* Update snapshots

* Address feedback

* Wording

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>
2021-06-10 19:05:40 -07:00
Vadim Ogievetsky 0c5d1c9725
Web console: add more query fixing auto suggestions (#11203)
* add more query fixing auto suggestions

* update query gen

* update toolkit

* update licenses

* fix funky quotes

* funky => fancy

* revert engine change

* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
Vadim Ogievetsky 31c811d894
Web console: Fix maxRowsPerSegment validation in hashed compaction spec (#11308)
* allow defining of maxRowsPerSegment for now

* use common util

* update snapshots

* fix test

* fix e2e test
2021-05-27 16:36:42 -07:00
Vadim Ogievetsky d85e8b708b
Web console: fix SegmentTimeline query (#11204)
* fix SegmentTimeline query

* test query
2021-05-14 14:48:23 -07:00
Vadim Ogievetsky d11be88c4b
Fix required field treatment (#11228) 2021-05-11 17:58:37 -07:00
Vadim Ogievetsky 15de29a2c4
Handle not having default rules better (#11214) 2021-05-10 15:13:15 -04:00
Lucas Capistrant bb3c810b36
Create dynamic config that can limit number of non-primary replicants loaded per coordination cycle (#11135)
* lay the groundwork for throttling replicant loads per RunRules execution

* Add dynamic coordinator config to control new replicant threshold.

* remove redundant line

* add some unit tests

* fix checkstyle error

* add documentation for new dynamic config

* improve docs and logs

* Alter how null is handled for new config. If null, manually set as default
2021-05-05 07:39:36 -05:00
Clint Wylie cd1ddcd36b
Web Console: allow pollPeriod to be defined in JDBC lookup config (#11193)
* allow pollPeriod to be defined in JDBC lookup config

* add test

* save the whales
2021-05-04 00:11:26 -07:00
Vadim Ogievetsky 1b07d554a8
Misc QueryView UX improvments (#11158) 2021-04-27 10:26:01 -07:00
John Gozde 9745d9e1c3
Web console: Switch to ESLint (#11142)
* Initial eslint config

* I guess eslint sorts underscores differently

* Trim curlies (in jsx)

* Re-organize rules

* Use consistent quote props

* Restructure eslint rules as additions/overrides to recommended configs

* Fix the 'recommended' stuff

* Add prefer-readonly

* Add prefer-object-spread

* Prettify

* Add eslint-plugin-react-hooks

* Switch to eslint-plugin-simple-sort-order

So much better

* Add no-extraneous-dependencies

* ban-tslint-comment for funzies

* If we enabled no-shadow, we'd probably want this option

* Not prefer-for-of

* no-confusing-void-expression, no-confusing-non-null-assertion

* Add some no-unnecessary-* rules

* non-nullable-type-assertion-style!

* prefer-includes

* Reorganize

* prefer-things

* switch-exhaustiveness-check

* We don't need the jsdoc plugin, prettier has our backs

* Remove a useless rule

* Drop TSLint and (temporarily) awesome-code-style

* Removing Object.assign revealed a type issue

* Bring back awesome-code-style for sasslint config

* Disable react/jsx-no-target-blank

* Add prettify-check script

* Add license to eslint config

* Format readme

* Update README for eslint, IDE settings

* Add 'autofix' script

* Switch to @awesome-code-style
2021-04-22 19:33:03 -07:00
Vadim Ogievetsky 4caa221d72
Web console: Better inline docs (#11128)
* better highlight

* better highlighting

* add spec
2021-04-19 14:36:53 -07:00
John Gozde fdc3c2f362
Web console: update dev dependencies (#11119)
* Update some dev dependencies, prettify, tslint-fix

* Sort tsconfig keys for easy comparison

* Set noImplicitThis

* Slightly more accurate types

* Bump Jest and related

* Bump react to latest on v16

* Bump node-sass, sass-loader for node14 support

* Remove node-sass-chokidar (unused)

* More unused dependencies

* Fix blueprint imports

* Webpack 5

* Update webpack config for 'process' usage

* Update playwright-chromium

* Emit esnext modules for tree shaking

* Enable source maps in development

* Dedupe

* Bump babel and things

* npm audit fix

* Add .editorconfig file to match prettier settings

* Update licenses (tslib is 0BSD as of 1.11.2)

https://github.com/microsoft/tslib/pull/96

* Require node >= 10

* Use Node 10 to run e2e tests

* Use 'ws' transport mode for dev server (will be default in next version)

* Remove an 'any'

* No sourcemaps in prod

* Exclude .editorconfig from license checks

* Try nvm for setting node version
2021-04-16 20:15:19 -07:00
Yi Yuan d0a94a8c14
add avro stream input format (#11040)
* add avro stream input format

* bug fixed

* add document

* doc fix

* change doc

* add integretion test

* bug fixed

* bug fixed

* add string as binary getter

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-04-12 21:53:41 -07:00
Vadim Ogievetsky 8432d82c48
Web console: Do not put __time in the dimensions list (#11085)
* Do not make time dimensions

* update e2e test
2021-04-12 09:48:10 -07:00
Atul Mohan e2c4466fbd
Add missing config (#11020) 2021-03-26 04:45:53 -07:00
Gian Merlino bf20f9e979
DruidInputSource: Fix issues in column projection, timestamp handling. (#10267)
* DruidInputSource: Fix issues in column projection, timestamp handling.

DruidInputSource, DruidSegmentReader changes:

1) Remove "dimensions" and "metrics". They are not necessary, because we
   can compute which columns we need to read based on what is going to
   be used by the timestamp, transform, dimensions, and metrics.
2) Start using ColumnsFilter (see below) to decide which columns we need
   to read.
3) Actually respect the "timestampSpec". Previously, it was ignored, and
   the timestamp of the returned InputRows was set to the `__time` column
   of the input datasource.

(1) and (2) together fix a bug in which the DruidInputSource would not
properly read columns that are used as inputs to a transformSpec.

(3) fixes a bug where the timestampSpec would be ignored if you attempted
to set the column to something other than `__time`.

(1) and (3) are breaking changes.

Web console changes:

1) Remove "Dimensions" and "Metrics" from the Druid input source.
2) Set timestampSpec to `{"column": "__time", "format": "millis"}` for
   compatibility with the new behavior.

Other changes:

1) Add ColumnsFilter, a new class that allows input readers to determine
   which columns they need to read. Currently, it's only used by the
   DruidInputSource, but it could be used by other columnar input sources
   in the future.
2) Add a ColumnsFilter to InputRowSchema.
3) Remove the metric names from InputRowSchema (they were unused).
4) Add InputRowSchemas.fromDataSchema method that computes the proper
   ColumnsFilter for given timestamp, dimensions, transform, and metrics.
5) Add "getRequiredColumns" method to TransformSpec to support the above.

* Various fixups.

* Uncomment incorrectly commented lines.

* Move TransformSpecTest to the proper module.

* Add druid.indexer.task.ignoreTimestampSpecForDruidInputSource setting.

* Fix.

* Fix build.

* Checkstyle.

* Misc fixes.

* Fix test.

* Move config.

* Fix imports.

* Fixup.

* Fix ShuffleResourceTest.

* Add import.

* Smarter exclusions.

* Fixes based on tests.

Also, add TIME_COLUMN constant in the web console.

* Adjustments for tests.

* Reorder test data.

* Update docs.

* Update docs to say Druid 0.22.0 instead of 0.21.0.

* Fix test.

* Fix ITAutoCompactionTest.

* Changes from review & from merging.
2021-03-25 10:32:21 -07:00
Vadim Ogievetsky 4897731e37
Make web console fast around sys.segments (#10909)
* do not load all the segments

* fix filtering

* update datasource view

* updated tests

* remove trimmedSegments

* Availability detail

* be smart about when showing smart modes

* fix tests

* add coordinator overlord mode
2021-03-10 19:59:50 -08:00
Vadim Ogievetsky c0fb326788
Web console: fix service view actions when grouping (#10898)
* fix service view actions when grouping

* fix test
2021-03-09 21:38:56 -08:00
frank chen 80ec28578a
show leader in Services Tab (#10951)
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-03-09 08:03:56 -08:00
Vadim Ogievetsky 1a4c43f9fd
Web console: remove namespace prop that does not exist from JDBC lookup (#10888)
* remove namespace prop that does not exist from JDBC lookup

* remove namespace from tests
2021-02-17 17:07:32 -08:00
Vadim Ogievetsky 2a1e47afc3
Web console: Remove first / last suggestions (#10794)
* Remove first / last suggestions

* remove commened out code
2021-01-28 13:37:10 -08:00
Suneet Saldanha a7542652ff
Fix dependabot warnings (#10796)
* Bump http-proxy from 1.18.0 to 1.18.1 in /web-console (#7)

Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump elliptic from 6.5.1 to 6.5.3 in /web-console (#6)

Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.1 to 6.5.3.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](https://github.com/indutny/elliptic/compare/v6.5.1...v6.5.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dot-prop from 4.2.0 to 4.2.1 in /web-console (#5)

Bumps [dot-prop](https://github.com/sindresorhus/dot-prop) from 4.2.0 to 4.2.1.
- [Release notes](https://github.com/sindresorhus/dot-prop/releases)
- [Commits](https://github.com/sindresorhus/dot-prop/compare/v4.2.0...v4.2.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump bl from 1.2.2 to 1.2.3 in /website (#4)

Bumps [bl](https://github.com/rvagg/bl) from 1.2.2 to 1.2.3.
- [Release notes](https://github.com/rvagg/bl/releases)
- [Commits](https://github.com/rvagg/bl/compare/v1.2.2...v1.2.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump prismjs from 1.20.0 to 1.23.0 in /website (#3)

Bumps [prismjs](https://github.com/PrismJS/prism) from 1.20.0 to 1.23.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](https://github.com/PrismJS/prism/compare/v1.20.0...v1.23.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-01-26 23:11:26 -08:00
Vadim Ogievetsky 8c227bc566
use new example manifest (#10787) 2021-01-24 12:38:13 -08:00
Jihoon Son 95065bdf1a
Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
Vadim Ogievetsky e52db19823
treat null as not defined (#10751) 2021-01-14 18:22:59 -08:00
Vadim Ogievetsky 2fc2938b01
Web console: fix bad results if there is not native bigint (#10741)
* fix bigint when it does not exist

* add test
2021-01-12 16:32:23 -08:00
Yi Yuan 3624acbcf8
fix web-console show json bug (#10710)
* fix web-console show json bug

* replace all JSON.stringify

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-01-08 14:55:55 -08:00
Jonathan Wei c7f2d3fbb5
Update deps for CVE-2020-28168 and CVE-2020-28052 (#10733)
* Update deps for CVE-2020-28168 and CVE-2020-28052

* Make BC runtime scope
2021-01-07 20:31:44 -08:00
Lucas Capistrant 58ce2e55d8
Add dynamic coordinator config that allows control over how many segments are considered when picking a segment to move. (#10284)
* dynamic coord config adding more balancing control

add new dynamic coordinator config, maxSegmentsToConsiderPerMove. This
config caps the number of segments that are iterated over when selecting
a segment to move. The default value combined with current balancing
strategies will still iterate over all provided segments. However,
setting this value to something > 0 will cap the number of segments
visited. This could make sense in cases where a cluster has a very large
number of segments and the admins prefer less iterations vs a thorough
consideration of all segments provided.

* fix checkstyle failure

* Make doc more detailed for admin to understand when/why to use new config

* refactor PR to use a % of segments instead of raw number

* update the docs

* remove bad doc line

* fix typo in name of new dynamic config

* update RservoirSegmentSampler to gracefully deal with values > 100%

* add handler for <= 0 in ReservoirSegmentSampler

* fixup CoordinatorDynamicConfigTest naming and argument ordering

* fix items in docs after spellcheck flags

* Fix lgtm flag on missing space in string literal

* improve documentation for new config

* Add default value to config docs and add advice in cluster tuning doc

* Add percentOfSegmentsToConsiderPerMove to web console coord config dialog

* update jest snapshot after console change

* fix spell checker errors

* Improve debug logging in getRandomSegmentBalancerHolder to cover all bad inputs for % of segments to consider

* add new config back to web console module after merge with master

* fix ReservoirSegmentSamplerTest

* fix line breaks in coordinator console dialog

* Add a test that helps ensure not regressions for percentOfSegmentsToConsiderPerMove

* Make improvements based off of feedback in review

* additional cleanup coming from review

* Add a warning log if limit on segments to consider for move can't be calcluated

* remove unused import

* fix tests for CoordinatorDynamicConfig

* remove precondition test that is redundant in CoordinatorDynamicConfig Builder class
2020-12-22 08:27:55 -08:00
Vadim Ogievetsky 55b8cc428a
remove extra word (#10682) 2020-12-15 23:02:33 -08:00
Vadim Ogievetsky 577cd66002
Web console: reflect the changes to interval requirement in the data loader flow (#10647)
* no need for intervals

* don't set redundant fields

* fix tests

* better filter control

* work with not

* wrap callout with form group

* update snapshot

* add split hint

* highlight issues with spec

* fixes

* fix default value

* move intervals back to partition step

* work with all sorts of chars

* fix enabled view
2020-12-09 10:18:42 -08:00
Vadim Ogievetsky e3f7217546
Web console: Improve the handling of extreme data (funky datasources, longs) (#10641)
* better API escape

* fix escaping issue, bigints

* update licenses

* fix align

* do not show Query with SQL if no SQL

* add prettify script

* update dev readme

* add ordering to the datasource list

* add ordering to supervisor table
2020-12-08 09:25:14 -08:00
Vadim Ogievetsky 5b06c7a3a9
Web console: improve how code is imported, use API instance (#10597)
* fix imports

* clean up imports

* update DQT to fix escaping
2020-12-01 13:16:14 -08:00
zhangyue19921010 abb03d38b0
double confirm before delete all the ununsed segmensts (#10581)
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-11-19 10:07:52 -06:00
Vadim Ogievetsky 9964dd4cb2
Web console: fix data loader schema table column ordering bug and other polish (#10588)
* remove unused fields

* keep tables live

* advanced

* fix schema view

* better indication

* tests pass

* Show more instead of show advanced

* fix tests

* extract dynamic configs

* update snapshots

* fix issues

* update snapshot

* reword without >
2020-11-17 13:25:03 -08:00
Lucas Capistrant 83667172ce
Add missing coordinator dynamic config to the web-console dialog for dynamic coordinator config (#10545)
* Add missing coordinator dynamic config to the web-console dialog for dynamic coordinator config

* update jest snapshots after making console change
2020-11-09 13:00:52 -06:00
Vadim Ogievetsky ee61a165e3
Web console: improve make type preservation in ingestion configs in the data loader (#10533)
* improve validation

* move to druid modals

* adjust specs

* oneOf

* move transform

* segment grans

* tidy up webpack

* add Partitioning

* break out druid models

* tidy up

* rename to Expression

* move druid time

* cleanup

* update format

* better detail

* fix e2e test

* remove forceGuaranteedRollup from e2e tests

* clean up imports

* shardSpec -> shard_spec

* fix css

* adjust snapshot

* add granularity to segments view

* rename to Time span

* use OFFSET in segments view query

* update doc links

* simplify require

* more oneOf
2020-10-30 19:02:44 -07:00
Vadim Ogievetsky f391e89653
Web console: refresh and tighten up the console styles 💅💫 (#10515)
* no hardcoded dark mode

* update blueprint

* style pass

* add open sans font

* fix public path

* update licenses

* add comment

* allow zero bsd

* add new lines

* update tests

* remove default mods

* use variables

* fix background

* fix query init style

* pad fracional part

* paginate braces

* update to latest blueprint, fix column tree

* Smart query limit -> Auto limit

* adjust gutter color for dark mode

* fix spelling

* update snapshot

Co-authored-by: asherbitter <ofir@imply.io>
2020-10-20 22:11:29 -07:00
Vadim Ogievetsky e8c5893c34
Web console: show segment sizes in rows not bytes (#10496)
* added query error suggestions

* simplify the SQLs

* change segment size display to rows

* suggestion tests

* update snapshot

* make error detection more robust

* remove errant console log

* fix imports

* put suggestion on top

* better error rendering

* format as millions

* add .druid.pid to gitignore

* rename segment_size to segment_rows, fix visability, fix divide by zero

* update snapshots
2020-10-13 13:19:39 -07:00
Suneet Saldanha b45a56f989
Web console: targetRowsPerSegment for hashed partionin (#10500)
* Web console: targetRowsPerSegment for hashed partionin

Added `targetRowsPerSegment` to the web console for hashed partition for both
the auto compaction view and as part of the ingestion workflow.

The help text was also updated to indicate when a user should care about
updating these fields

* code review

* update test snapshots

* oops
2020-10-11 16:55:28 -07:00
Ashish Kapoor 4c78b514c9
Update README.md (#10357)
Compile scss files before npm start.
2020-10-09 20:21:59 +05:30
Joseph Glanville 7ce9ac4548
Fix Avro support in Web Console (#10232)
* Fix Avro OCF detection prefix and run formation detection on raw input

* Support Avro Fixed and Enum types correctly

* Check Avro version byte in format detection

* Add test for AvroOCFReader.sample

Ensures that the Sampler doesn't receive raw input that it can't
serialize into JSON.

* Document Avro type handling

* Add TS unit tests for guessInputFormat
2020-10-07 21:08:22 -07:00
Vadim Ogievetsky 2e50ada407
Web console: fix compaction status when no compaction config, and small cleanup (#10483)
* move timed button to icons

* cleanup redundant logic

* fix compaction status text

* remove extra style
2020-10-07 14:54:08 -07:00
Gian Merlino d78fedd13c
Web console: Don't include realtime segments in size calculations. (#10482)
It's always zero, and so it messes up averages, mins, and counts.
2020-10-06 18:56:54 -07:00
Chi Cao Minh 1c77491da6
Test UI to trigger auto compaction (#10469)
In the web console E2E tests, Use the new UI to trigger auto compaction
instead of calling the REST API directly so that the UI is covered by
tests.
2020-10-04 00:06:07 -07:00
Jonathan Wei 65c0d64676
Update version to 0.21.0-SNAPSHOT (#10450)
* [maven-release-plugin] prepare release druid-0.21.0

* [maven-release-plugin] prepare for next development iteration

* Update web-console versions
2020-10-03 16:08:34 -07:00
Vadim Ogievetsky f77c16bc6c
Web console: fix lookup edit dialog version setting (#10461)
* fix lookup edit dialog

* update snapshots

* clean up test
2020-10-03 08:35:20 -07:00
Chi Cao Minh d11537b5f7
Improve UI E2E test usability (#10466)
- Update playwright to latest version
- Provide environment variable to disable/enable headless mode
- Allow running E2E tests against any druid cluster running on standard
  ports (tutorial-batch.spec.ts now uses an absolute instead of relative
  path for the input data)
- Provide environment variable to change target web console port
- Druid setup does not need to download zookeeper
2020-10-03 08:21:44 -07:00
Chi Cao Minh ede25f1b45
Fix UI datasources view edit action compaction (#10459)
Restore the web console's ability to view a datasource's compaction
configuration via the "action" menu. Refactoring done in
https://github.com/apache/druid/pull/10438 introduced a regression that
always caused the default compaction configuration to be shown via the
"action" menu instead.

Regression test is added in e2e-tests/auto-compaction.spec.ts.
2020-10-01 23:59:21 -07:00
Chi Cao Minh 7385af0272
Web console reindexing E2E test (#10453)
Add an E2E test for the web console workflow of reindexing a Druid
datasource to change the secondary partitioning type.  The new test
changes dynamic to single dim partitions since the autocompaction test
already does dynamic to hashed partitions.

Also, run the web console E2E tests in parallel to reduce CI time and
change naming convention for test datasources to make it easier to map
them to the corresponding test run.

Main changes:

1) web-consolee2e-tests/reindexing.spec.ts
   - new E2E test

2) web-console/e2e-tests/component/load-data/data-connector/reindex.ts
   - new data loader connector for druid input source

3) web-console/e2e-tests/component/load-data/config/partition.ts
   - move partition spec definitions from compaction.ts
   - add new single dim partition spec definition
2020-10-01 15:14:41 -07:00
Vadim Ogievetsky d09fd8b035
Web console: switch to switches instead of checkboxes (#10454)
* switch to switches

* add img alt

* add relative

* change icons

* update snapshot
2020-09-30 15:13:22 -07:00
Vadim Ogievetsky 729bcba7ac
Web console: Display compaction status (#10438)
* init compaction status

* % compacted

* final UI tweaks

* extracted utils, added tests

* add tests to general foramt functions
2020-09-28 22:19:28 -07:00
Chi Cao Minh cbe2b44e29
Compaction config UI optional numShards (#10446)
* Compaction config UI optional numShards

Specifying `numShards` for hashed partitions is no longer required after
https://github.com/apache/druid/pull/10419. Update the UI to make
`numShards` an optional field for hash partitions.

* Update snapshot
2020-09-28 17:15:48 -07:00
Chi Cao Minh d16c78ba98
Add intent for web console IntervalInput (#10447)
When using the web console to load data by reindexing from Druid, the
`Datasource` and `Interval` inputs are required during the `Connect`
step. Unlike the `Datasource` input, the `Interval` input did not have a
blue outline to indicate that it was required as the `IntervalInput`
component did not support an `intent` property.
2020-09-28 17:13:07 -07:00
Chi Cao Minh cbd9ac8592
Web console autocompaction E2E test (#10425)
Add an E2E test for the common case web console workflow of setting up
autocompaction that changes the partitions from dynamic to hashed.

Also fix an issue with the async test setup to properly wait for the web
console to be ready.
2020-09-25 18:28:25 -07:00
Vadim Ogievetsky 89160c2f9b
better query view initial state (#10431) 2020-09-24 09:49:58 -07:00
Vadim Ogievetsky a60d034d01
Web console: compaction dialog update (#10417)
* compaction dialog update

* fix test snapshot

* Update web-console/src/dialogs/compaction-dialog/compaction-dialog.tsx

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>

* Update web-console/src/dialogs/compaction-dialog/compaction-dialog.tsx

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>

* feedback changes

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>
2020-09-23 09:34:19 -07:00
Vadim Ogievetsky ac0a45471e
Web console: add sort to tiers list (#10416)
* add sort to tiers list

* update snapshot
2020-09-22 19:00:55 -07:00
Vadim Ogievetsky 7cc0a7be68
Web console: clean up styling imports (#10410)
* fix styling for importing

* fix quotes
2020-09-21 17:30:25 -07:00
Vadim Ogievetsky 6c5c86d800
Web console: fix lookup edit dialog, allow column renaming (#10406)
* column rename

* update licenses file

* remove empty file

* update license file

* move comment
2020-09-20 14:10:05 -07:00
Vadim Ogievetsky e465f05717
Web console: Improve number alignment in tables (#10389)
* Improve tables

* removed unused state interfaces

* better copy

* one more functional component

* updated e2e tests

* extract braced text correctly
2020-09-14 19:53:38 -07:00
Vadim Ogievetsky 3c8eacb2d4
Web console: improve query manager (convert to React hook) (#10360)
* Better query running

* update licenses

* update tests

* updated tests v2

* fade in cancel

* add exemplary tests

* update mkcomp

* fix inconsistent state update

* remove lastParsedQuery

* work if not a valid literal

* remove unused params

* fix licenses

* better state update

* get error message

* isEmpty tidy

* add tests around error message highlighting

* pull live query selector into a component

* add LiveQueryModeSelector tests

* update snapshots
2020-09-11 19:42:50 -07:00
Vadim Ogievetsky e81a9df507
Web console: add tile for Azure Event Hubs (via Kafka API) (#10317)
* Add Azure Event Hubs

* better note

* update icon
2020-08-31 20:58:52 -07:00
Vadim Ogievetsky 748a83cb78
Web console: fix json input (#10271)
* fix json input

* tidy up

* add error extraction test
2020-08-13 12:20:58 -07:00
Suneet Saldanha 6baea0b4d5
Fix broken sampler for re-indexing (#10196)
* Fix broken sampler for re-indexer

When re-indexing a Druid datasource, the web-console would generate an
invalid inputFormat since the type is not specified.

* code review
2020-08-11 19:22:26 -07:00
Vadim Ogievetsky 1fae0e15c9
improve JSON paste (#10256) 2020-08-07 17:01:06 -07:00
Vadim Ogievetsky 9a29496b6c
Web console: Improve retention rules dialog in all sorts of ways (#10226)
* improve ret rules

* tidy up tests
2020-08-04 16:43:31 -07:00
keefe roedersheimer 4bb198eb4c
change search filter to includes (#10141) 2020-08-04 16:37:07 -07:00
Vadim Ogievetsky 6d8799f2df
Update QueryView to use latest DruidQueryToolkit (#10201)
* Update to latest DruidQueryToolkit

* add THEN keyword

* do not crash on invalid JSON
2020-07-23 22:45:01 -07:00
Suneet Saldanha eeb9012743
Change color of Run button for native queries (#10170)
* Change color of Run button for native queries

When a user tries to run a native query, change the color of the button to
Druid's secondary color to indicate that the user is not running a SQL query.

Before this change, the web-console would indicate this by changing the text
of the button from Run (SQL queries) to Rune (native queries). Rune could be
confusing to users as this appears to be a typo.

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* code review
2020-07-15 15:46:02 -07:00
Clint Wylie 0a8fd62f22
split web-console e2e-tests from unit tests (#10173)
* split web-console e2e-test from unit test

* fix stuff

* smaller change

* oops
2020-07-12 18:51:08 -07:00
Egor Riashin d54a5e009f
ui: fix missing columns during Transform step (#10086)
Co-authored-by: egor-ryashin <egor.ryashin@metamarkets.com>
2020-07-07 15:16:52 -07:00
Clint Wylie c86e7ce30b
bump version to 0.20.0-SNAPSHOT (#10124) 2020-07-06 15:08:32 -07:00
Clint Wylie 477335abb4
update links datasketches.github.io to datasketches.apache.org (#10107)
* update links datasketches.github.io to datasketches.apache.org

* now with more apache

* oops

* oops
2020-07-01 14:56:17 -07:00
Vadim Ogievetsky c01fd56182
Web console: allow link overrides for docs, and more (#10100)
* link overrides

* change doc version

* fix snapshots
2020-06-30 12:46:50 -07:00
Yuanli Han fc555980e8
Remove payload field from table sys.segment (#9883)
* remove payload field from table sys.segments

* update doc

* fix test

* fix CI failure

* add necessary fields

* fix doc

* fix comment
2020-06-29 22:20:23 -07:00
Suneet Saldanha 15a0b4ffe2
Filter http requests by http method (#10085)
* Filter http requests by http method

Add a config that allows a user which http methods to allow against their
Druid server.

Druid will only accept http requests with the method: GET, PUT, POST, DELETE
and OPTIONS.
If a Druid admin wants to allow other methods, they can do so by using the
ServerConfig#allowedHttpMethods config.

If a Druid user would like to disallow OPTIONS, this can be done by changing
the AuthConfig#allowUnauthenticatedHttpOptions config

* Exclude OPTIONS from always supported HTTP methods

Add HEAD as an allowed method for web console e2e tests

* fix docs

* fix security IT

* Actually fix the web console e2e tests

* Ignore icode coverage for nitialization classes

* code review
2020-06-29 16:59:31 -07:00
Will Xu 35c7c0ec25
Segment timeline doesn't show results older than 3 months (#9956)
* Segment timeline doesn't show results older than 3 months

* Adoption testing patch for web segment timeline view and also refactoring default time config
2020-06-28 01:45:05 -07:00
Maytas Monsereenusorn 5d35f3e080
Remove colocated datasources from web console for broadcast indexed tables (#10018) 2020-06-11 14:08:03 -10:00
Xavier Léauté 65280a6953
update kafka client version to 2.5.0 (#9902)
- remove dependency on deprecated internal Kafka classes
- keep LZ4 version in line with the version shipped with Kafka
2020-05-27 13:20:32 -07:00
Joseph Glanville 132a1c9fe7
Re-order and document format detection in web console (#9887)
Motivation for this change is to not inadvertently identify binary
formats that contain uncompressed string data as TSV or CSV.

Moving detection of magic byte headers before heuristics should be more
robust in general.
2020-05-21 16:29:39 -07:00
Vadim Ogievetsky 63baa29ad1
Fix web console query view crashing on simple query (#9897)
* only parse full queries

* upgraded sql parser
2020-05-21 12:57:07 -07:00
Maytas Monsereenusorn f470bcd11f
Fix deleting a data node tier causes load rules to display incorrectly (#9891)
* Fix Deleting a data node tier causes load rules to malfunction & display incorrectly

* add tests

* fix style
2020-05-20 16:49:28 -07:00
Joseph Glanville 793f386d6a
Add support for Avro OCF using InputFormat (#9671)
* Add AvroOCFInputFormat

* Support supplying a reader schema in AvroOCFInputFormat

* Add docs for Avro OCF input format

* Address review comments

* Address second round of review
2020-05-16 14:09:12 -07:00
Chi Cao Minh 41cf826928
Console E2E test docs (#9864) 2020-05-13 16:41:04 -07:00
mcbrewster 28be107a1c
add flag to flattenSpec to keep null columns (#9814)
* add flag to flattenSpec to keep null columns

* remove changes to inputFormat interface

* add comment

* change comment message

* update web console e2e test

* move keepNullColmns to JSONParseSpec

* fix merge conflicts

* fix tests

* set keepNullColumns to false by default

* fix lgtm

* change Boolean to boolean, add keepNullColumns to hash, add tests for keepKeepNullColumns false + true with no nuulul columns

* Add equals verifier tests
2020-05-08 21:53:39 -07:00
Jian Wang 85dfbb64cb
Update documention for metricCompression (#9811) 2020-05-03 12:56:48 -07:00
Clint Wylie 28f56978ab
web-console clean coverage report on build clean (#9718) 2020-04-21 17:02:05 -07:00
mcbrewster b7fdb29423
add joins to column tree menu (#9705)
* add joins to column tree menu

* fix capitalization

* add keyword, keep columns if replaced

* actually fix capitalization

* add keywords
2020-04-16 17:51:59 -07:00
Chi Cao Minh 2262e33316
Fix flaky web console E2E test (#9685)
web-console/e2e-tests/tutorial-batch.spec.ts would occasionally timeout
between the transition from the data loader "configure schema" and
"partition" steps due to missing waits when toggling the rollup setting.

Also, fix shellcheck warnings for script/druid.
2020-04-14 15:27:16 -07:00
Chi Cao Minh 84c1c2505d
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.

The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
  abstractions for `load data`, `ingestion`, `datasources`, and `query`
  are implemented.
- `components/load-data/data-connector` contains abstractions for the
  different data source options available to the data loader's `Connect`
  step. Currently, only the `Local file` data source connector is
  implemented.
- `components/load-data/config` contains abstractions for the different
  configuration options available for each step of the data loader flow.
  Currently, the `Configure Schema`, `Partition`, and `Publish` steps
  have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
  contain abstractions of the web console.

Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
  different flavors for unit tests and for end-to-end tests. In
  particular, the latter adds a jest setup configuration to wait for the
  web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
  running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
  and stopping druid.

Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
  disable java tests into two new maven profiles. Since the same
  settings are used in several places (e.g., .travis.yml, Dockerfiles,
  etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 12:38:09 -07:00
mcbrewster 6f3d403491
Use auto-form for add an edit lookups (#9587)
* use auto form

* jest -u

* fix unreachable statment

* complete the owl

* jest -u

* remove changes to query-view

* fix permissions

* add test, fix info

* add cool highlights

* fix formatting

* fix capitalization

* add optional placeholder

* add space
2020-04-08 16:34:59 -07:00
mcbrewster 2b2b9efcd7
add new text to lookup action dialog (#9643) 2020-04-08 11:30:47 -07:00
mcbrewster 6e50d29b4e
fix global filter input (#9567)
* fix global filter input

* remove clear

* close global filters after clicking apply

* add restFilter
2020-04-07 13:31:19 -07:00
Jihoon Son 0da8ffc3ff
Bump up development version to 0.19.0-SNAPSHOT (#9586) 2020-03-30 16:24:04 -07:00
mcbrewster e1b201c279
Add view values to lookup actions menu (#9549)
* add test, add query

* jest -u

* add limit, explicitly get columns, remoove map
2020-03-24 09:57:33 -07:00
Clint Wylie bf85ea19b2
roaring bitmaps by default (#9548)
* it is finally time

* fix it

* more docs

* fix doc
2020-03-23 18:15:57 -07:00
Vadim Ogievetsky cdf4a26904
clean up spec before reopening in data loader (#9536) 2020-03-23 16:57:51 -07:00
Vadim Ogievetsky 3b536eea7f
Web console: expose props for S3 (#9432)
* expose props for S3

* added env inputs

* add scarry warning

* use .password

* put the warning front and center

* Update web-console/src/views/load-data-view/load-data-view.tsx

Co-Authored-By: Suneet Saldanha <44787917+suneet-s@users.noreply.github.com>

* let prettier rewrap the text

Co-authored-by: Suneet Saldanha <44787917+suneet-s@users.noreply.github.com>
2020-03-18 15:32:12 -07:00
Vadim Ogievetsky 7626be26ca
Web console: add config control for the query context (#9499)
* add default and mandetory query contexts

* added config docs
2020-03-16 14:34:19 -07:00
mcbrewster bcb9a632c7
Web console: update druid-query-toolkit to version 0.4.x (#9500)
* add support for new version of DQT

* update druid-query-toolkit

* fix direction css

* fix remove

* update package

* remove useless conditional

* bump package

* jest -u

Co-authored-by: Maggie Brewster <maggiebrewster@implydata20sMBP.attlocal.net>
2020-03-13 18:09:47 -07:00
Vadim Ogievetsky ddc6f87920
Web console: standardize the spec format (#9477)
* standerdize the spec format

* fix spec upgrade
2020-03-12 14:21:23 -07:00
mcbrewster 95406ca20a
[IMPLY-2285] fix maxRowsPerSegment tool tip (#9468) 2020-03-09 20:12:05 -07:00
mcbrewster 96ed7210d3
Fix history dialog overflow (#9471)
* [IMPLY-1661] fix history dialog overflow

* jest -u
2020-03-09 19:09:59 -07:00
mcbrewster a676d16226
[IMPLY-1767] fix popover direction (#9470) 2020-03-09 17:35:02 -07:00
mcbrewster da0ea627d0
Add disabled run button during loading state (#9474)
* [IMPLY-1782] add disabled run button during loading state

* jest -u
2020-03-09 17:10:35 -07:00
Clint Wylie 32cd47bc8e
Fix home view styling (#9444) 2020-03-04 19:39:36 -08:00
mcbrewster 99095c4ac5
Add Azure ingestion flow to web console (#9437)
* add support for azure

* change bucket to container

* add azure to input menu

* remove static-azure
2020-03-03 11:06:00 -08:00
Vadim Ogievetsky c294e0b7c6
Web console: Column counter (#9334)
* Column counter

* more general test
2020-02-27 12:04:27 -08:00
Aditya e506fc9fdf
fix cursor position after function autocomplete (#9396)
Closes #9395
2020-02-26 09:41:24 -08:00
Chi Cao Minh 4221573317
Update web-console dependencies (#9361)
Fixes security vulnerabilities as flagged by the "security
vulnerability" travis cron job.
2020-02-13 21:02:34 -08:00
Vadim Ogievetsky 7e53f23f07
Web console: make supervisor reset really scary in the UI (#9253)
* make supervisor reset really scary

* change warnings

* add text
2020-02-04 15:33:52 -08:00
Roman Leventov b9186f8f9f Reconcile terminology and method naming to 'used/unused segments'; Rename MetadataSegmentManager to MetadataSegmentsManager (#7306)
* Reconcile terminology and method naming to 'used/unused segments'; Don't use terms 'enable/disable data source'; Rename MetadataSegmentManager to MetadataSegments; Make REST API methods which mark segments as used/unused to return server error instead of an empty response in case of error

* Fix brace

* Import order

* Rename withKillDataSourceWhitelist to withSpecificDataSourcesToKill

* Fix tests

* Fix tests by adding proper methods without interval parameters to IndexerMetadataStorageCoordinator instead of hacking with Intervals.ETERNITY

* More aligned names of DruidCoordinatorHelpers, rename several CoordinatorDynamicConfig parameters

* Rename ClientCompactTaskQuery to ClientCompactionTaskQuery for consistency with CompactionTask; ClientCompactQueryTuningConfig to ClientCompactionTaskQueryTuningConfig

* More variable and method renames

* Rename MetadataSegments to SegmentsMetadata

* Javadoc update

* Simplify SegmentsMetadata.getUnusedSegmentIntervals(), more javadocs

* Update Javadoc of VersionedIntervalTimeline.iterateAllObjects()

* Reorder imports

* Rename SegmentsMetadata.tryMark... methods to mark... and make them to return boolean and the numbers of segments changed and relay exceptions to callers

* Complete merge

* Add CollectionUtils.newTreeSet(); Refactor DruidCoordinatorRuntimeParams creation in tests

* Remove MetadataSegmentManager

* Rename millisLagSinceCoordinatorBecomesLeaderBeforeCanMarkAsUnusedOvershadowedSegments to leadingTimeMillisBeforeCanMarkAsUnusedOvershadowedSegments

* Fix tests, refactor DruidCluster creation in tests into DruidClusterBuilder

* Fix inspections

* Fix SQLMetadataSegmentManagerEmptyTest and rename it to SqlSegmentsMetadataEmptyTest

* Rename SegmentsAndMetadata to SegmentsAndCommitMetadata to reduce the similarity with SegmentsMetadata; Rename some methods

* Rename DruidCoordinatorHelper to CoordinatorDuty, refactor DruidCoordinator

* Unused import

* Optimize imports

* Rename IndexerSQLMetadataStorageCoordinator.getDataSourceMetadata() to retrieveDataSourceMetadata()

* Unused import

* Update terminology in datasource-view.tsx

* Fix label in datasource-view.spec.tsx.snap

* Fix lint errors in datasource-view.tsx

* Doc improvements

* Another attempt to please TSLint

* Another attempt to please TSLint

* Style fixes

* Fix IndexerSQLMetadataStorageCoordinator.createUsedSegmentsSqlQueryForIntervals() (wrong merge)

* Try to fix docs build issue

* Javadoc and spelling fixes

* Rename SegmentsMetadata to SegmentsMetadataManager, address other comments

* Address more comments
2020-01-27 11:24:29 -08:00
Caroline1000 3daf0f8e12 Update ingestion-view.tsx (#9250)
Grammar and accuracy updates.
2020-01-24 01:21:55 -08:00
Vadim Ogievetsky ab2672514b allow empty values to be set in the auto form (#9198) 2020-01-16 21:06:51 -08:00
Vadim Ogievetsky 09efd20b42
fix refresh button (#9195) 2020-01-16 10:13:47 -08:00