Commit Graph

9157 Commits

Author SHA1 Message Date
Xue Yu 78fd5aff21 support radians and degrees in sql (#7336)
* support radians and degrees in sql

* update test case
2019-04-02 12:47:49 -07:00
Qi Shu 134f71d1b4 Add documentation for Druid native query in SQL view of web console (#7381)
* Add docmentation for Druid native query in SQL view of web console

* Edit sentence
2019-04-02 12:20:51 -07:00
Vadim Ogievetsky d2ff3eb199 Migrate the web console back to Blueprint v3 (#7398)
* migrate back to Blueprint v3

* rename filler file

* fix missing icon
2019-04-01 23:06:48 -07:00
Vadim Ogievetsky 1137bb2586 Popovers are no longer inline (#7389) 2019-04-01 13:10:48 -07:00
Jonathan Wei 2bf6fc353a Update scan benchmark for time ordering (#7385) 2019-03-30 11:36:25 -07:00
Qi Shu 2ad9544242 Support segment metadata in rune (#7371) 2019-03-29 15:46:09 -07:00
Michael Trelinski 347779b17a Zookeeper loss (#6740)
* Update init

Fix bin/init to source from proper directory.

* Fix for Proposal #6518: Shutdown druid processes upon complete loss of ZK connectivity

* Zookeeper Loss:

- Add feature documentation
- Cosmetic refactors
- Variable extractions
- Remove getter

* - Change config key name and reword documentation
- Switch from Function<Void,Void> to Runnable/Lambda
- try { … } finally { … }

* Fix line length too long

* - change to formatted string for logging
- use System.err.println after lifecycle stops

* commenting on makeEnsembleProvider()-created Zookeeper termination

* Add javadoc

* added java doc reference back to apache discussion thread.

* move comment to other class

* favor two-slash comments instead of multiline comments
2019-03-29 15:10:42 -07:00
Jihoon Son 62c3e89266 maxTotalRows should be checked in DataSourceCompactionConfig before setting targetCompactionSizeBytes (#7368)
* maxTotalRows should be checked in DataSourceCompactionConfig before setting targetCompactionSizeBytes

* remove unnecessary default values

* remove flacky test

* fix build

* Add comments
2019-03-28 20:25:10 -07:00
Qi Shu a09aa13ead Put all local storage keys and functions into one file in util (#7314)
* Add all local storage keys and functions into one file in util

* Use LocalStorageKey type instead of string

* Remove druid author key
2019-03-28 17:48:37 -07:00
Qi Shu be1d7f885b Add scss linter in web console (#7365)
* Add SCSS linter

* Fixed SCSS styles
2019-03-28 17:32:01 -07:00
Justin Borromeo ad7862c58a Time Ordering On Scans (#7133)
* Moved Scan Builder to Druids class and started on Scan Benchmark setup

* Need to form queries

* It runs.

* Stuff for time-ordered scan query

* Move ScanResultValue timestamp comparator to a separate class for testing

* Licensing stuff

* Change benchmark

* Remove todos

* Added TimestampComparator tests

* Change number of benchmark iterations

* Added time ordering to the scan benchmark

* Changed benchmark params

* More param changes

* Benchmark param change

* Made Jon's changes and removed TODOs

* Broke some long lines into two lines

* nit

* Decrease segment size for less memory usage

* Wrote tests for heapsort scan result values and fixed bug where iterator
wasn't returning elements in correct order

* Wrote more tests for scan result value sort

* Committing a param change to kick teamcity

* Fixed codestyle and forbidden API errors

* .

* Improved conciseness

* nit

* Created an error message for when someone tries to time order a result
set > threshold limit

* Set to spaces over tabs

* Fixing tests WIP

* Fixed failing calcite tests

* Kicking travis with change to benchmark param

* added all query types to scan benchmark

* Fixed benchmark queries

* Renamed sort function

* Added javadoc on ScanResultValueTimestampComparator

* Unused import

* Added more javadoc

* improved doc

* Removed unused import to satisfy PMD check

* Small changes

* Changes based on Gian's comments

* Fixed failing test due to null resultFormat

* Added config and get # of segments

* Set up time ordering strategy decision tree

* Refactor and pQueue works

* Cleanup

* Ordering is correct on n-way merge -> still need to batch events into
ScanResultValues

* WIP

* Sequence stuff is so dirty :(

* Fixed bug introduced by replacing deque with list

* Wrote docs

* Multi-historical setup works

* WIP

* Change so batching only occurs on broker for time-ordered scans

Restricted batching to broker for time-ordered queries and adjusted
tests

Formatting

Cleanup

* Fixed mistakes in merge

* Fixed failing tests

* Reset config

* Wrote tests and added Javadoc

* Nit-change on javadoc

* Checkstyle fix

* Improved test and appeased TeamCity

* Sorry, checkstyle

* Applied Jon's recommended changes

* Checkstyle fix

* Optimization

* Fixed tests

* Updated error message

* Added error message for UOE

* Renaming

* Finish rename

* Smarter limiting for pQueue method

* Optimized n-way merge strategy

* Rename segment limit -> segment partitions limit

* Added a bit of docs

* More comments

* Fix checkstyle and test

* Nit comment

* Fixed failing tests -> allow usage of all types of segment spec

* Fixed failing tests -> allow usage of all types of segment spec

* Revert "Fixed failing tests -> allow usage of all types of segment spec"

This reverts commit ec470288c7.

* Revert "Merge branch '6088-Time-Ordering-On-Scans-N-Way-Merge' of github.com:justinborromeo/incubator-druid into 6088-Time-Ordering-On-Scans-N-Way-Merge"

This reverts commit 57033f36df, reversing
changes made to 8f01d8dd16.

* Check type of segment spec before using for time ordering

* Fix bug in numRowsScanned

* Fix bug messing up count of rows

* Fix docs and flipped boolean in ScanQueryLimitRowIterator

* Refactor n-way merge

* Added test for n-way merge

* Refixed regression

* Checkstyle and doc update

* Modified sequence limit to accept longs and added test for long limits

* doc fix

* Implemented Clint's recommendations
2019-03-28 14:37:09 -07:00
Roman Leventov ffa95859c2 Remove [PROPOSAL] from Proposal issue template; Add 'Other' template for convenience (#7364)
* Remove [PROPOSAL] from Proposal issue template; Add 'Regular' template for convenience

* Regular -> Other
2019-03-28 18:40:43 +08:00
Surekha be318f4de3 Add column type to sys table docs (#7359)
* Add column type

* oops should be used=1
2019-03-27 20:21:57 -07:00
Clint Wylie db0125b709 this will fix it (#7358) 2019-03-27 19:42:58 -07:00
Qi Shu 18e5167245 Add history dialog in snitch dialog to allow viewing the editing history (#7321)
* Add history dialog in snitch dialog to allow viewing the editing history

* Improved CSS; better animation

* Use position: absolute instead of float: right to position element

* Removed author for history changes
2019-03-27 17:52:44 -07:00
Charles Allen eeb3dbe79d Move GCP to a core extension (#6953)
* Move GCP to a core extension

* Don't provide druid-core >.<

* Keep AWS and GCP modules separate

* Move AWSModule to its own module

* Add aws ec2 extension and more modules in more places

* Fix bad imports

* Fix test jackson module

* Include AWS and GCP core in server

* Add simple empty method comment

* Update version to 15

* One more 0.13.0-->0.15.0 change

* Fix multi-binding problem

* Grep for s3-extensions and update docs

* Update extensions.md
2019-03-27 09:00:43 -07:00
Jihoon Son 543324f8a9 Fix logging in IndexerSQLMetadataStorageCoordinator (#7349) 2019-03-26 20:36:19 -07:00
Jihoon Son 4d37edac1e Suppress stack trace in warning (#7348) 2019-03-26 17:27:29 -07:00
Roman Leventov b9354fbddb
Make 'field can be final' inspection a warning in IntelliJ config (#7301)
A small step towards #7227.
2019-03-26 20:45:03 -03:00
Justin Borromeo c7fea6ac8f Added better QueryInterruptedException error message for UnsupportedOperationException (#7248)
* Added error message for UOE

* Updated docs

* Doc change

* Doc change
2019-03-26 15:20:24 -07:00
Jihoon Son 5294277cb4
Fix exclusive start partitions for sequenceMetadata (#7339)
* Fix exclusvie start partitions for sequenceMetadata

* add empty check
2019-03-26 14:39:07 -07:00
Puneet Jaiswal 105b4fa237 Fixing a typo in logger initialization (#7341) 2019-03-27 00:38:28 +08:00
Roman Leventov bca40dcdaf
Fix some IntelliJ inspections (#7273)
Prepare TeamCity for IntelliJ 2018.3.1 upgrade. Mostly removed redundant exceptions declarations in `throws` clauses.
2019-03-25 21:11:01 -03:00
Jonathan Wei 8ca7cb4886 Fix rat check for source assembly after build (#7333) 2019-03-22 22:48:35 -07:00
Jihoon Son f410c28af6
Always convert start metadata to start (#7332) 2019-03-22 21:12:15 -07:00
Gian Merlino 4ca5fe0f60 SQL: Add PARSE_LONG function. (#7326)
* SQL: Add PARSE_LONG function.

* Fix test.
2019-03-22 15:40:10 -07:00
Surekha 01c021e6da Add Kafka integration test for transactional topics (#7295)
* Add integration test for transactional kafka

* Add true for transactions enabled for transactional test

* Add new test to travis_script_integration.sh, use version 0.2 of druid docker image

* Use different datasource name for ITKafkaIndexingServiceTest and ITKafkaIndexingServiceTransactionalTest

* use KafkaConsumerConfigs to get common consumer properties

* Remove double line breaks

* remove extra space
2019-03-22 13:39:05 -07:00
Jonathan Wei 7a57bc0dc3
Exclude git.version from rat check (#7322) 2019-03-21 20:54:27 -07:00
Vadim Ogievetsky 375668ef45 fix Supervisor terminate call (#7315)
* fix Supervisor terminate call

* fix refresh after submitting supervisor or task
2019-03-21 13:54:43 -07:00
Roman Leventov a4270da5f3
More TeamCity and Structural Search inspection instructions (#7275)
* More TeamCity and Structural Search inspection instructions

* the -> a

* an e-mail
2019-03-21 17:24:21 -03:00
Jihoon Son 0c5dcf5586 Fix exclusivity for start offset in kinesis indexing service & check exclusivity properly in IndexerSQLMetadataStorageCoordinator (#7291)
* Fix exclusivity for start offset in kinesis indexing service

* some adjustment

* Fix SeekableStreamDataSourceMetadata

* Add missing javadocs

* Add missing comments and unit test

* fix SeekableStreamStartSequenceNumbers.plus and add comments

* remove extra exclusivePartitions in KafkaIOConfig and fix downgrade issue

* Add javadocs

* fix compilation

* fix test

* remove unused variable
2019-03-21 13:12:22 -07:00
Surekha e170203876 Consolidate kafka consumer configs (#7249)
* Consolidate kafka consumer configs

* change the order of adding properties

* Add consumer properties to fix test

it seems kafka consumer does not reveive any message without these configs

* Use KafkaConsumerConfigs in integration test

* Update zookeeper and kafka versions in the setup.sh for the base druid image

*  use version 0.2 of base druid image

* Try to fix tests in KafkaRecordSupplierTest

* unused import

* Fix tests in KafkaSupervisorTest
2019-03-21 11:19:49 -07:00
Qi Shu 494c1a2ef8 Add overlord edit dialog to allow user to change overlord dynamic config on the fly (#7308)
* Add overlord edit dialog to allow user to edit overlord dynmamic config on the fly

* Set config to {} if druid return nothing for overlord config
2019-03-21 03:38:35 -07:00
Qi Shu b0271b4f4d Saving author to local storage in snitch dialog (#7307)
* Saving author to local storage

* Capitalized const
2019-03-21 03:01:10 -07:00
Vadim Ogievetsky e4f2dcacf2 Druid console docs (#7300)
* console docs

* fix typo
2019-03-21 00:37:33 -07:00
Qi Shu 30e646308a Add table column selection in druid console to allow hiding/showing of columns (#7292)
* Add table column selections to all tables to allow user to hide/show columns

* Small change for re-rendering

* Use column selection handler class to process all column hiding/showing

* dereference table handler function at the start; use more specific file name for table.tsx
2019-03-20 13:11:00 -07:00
Don Bowman 2814729d32 Remove mysql database connector from container image (#7296)
* Remove mysql database connector from container image

See discussion on `dev` mailing list from 2019-03
around [docker
build](http://mail-archives.apache.org/mod_mbox/druid-dev/201903.mbox/browser)

Remove the mysql database connector from the container
image in favour of solely the postgresql connector.

Instructions are present in distribution/docker/README.md
for someone to re-add if they need.

Signed-off-by: Don Bowman <don@agilicus.com>

* Add sample Dockerfile for mysql using pre-built upstream
2019-03-20 08:33:28 -07:00
Justin Borromeo ff94bd16e6 Fix conflicting information in configuration doc (#7299)
* Doc fix

* Fix typo
2019-03-19 14:55:58 -07:00
Roman Leventov dfd27e00c0
Avoid many unnecessary materializations of collections of 'all segments in cluster' cardinality (#7185)
* Avoid many  unnecessary materializations of collections of 'all segments in cluster' cardinality

* Fix DruidCoordinatorTest; Renamed DruidCoordinator.getReplicationStatus() to computeUnderReplicationCountsPerDataSourcePerTier()

* More Javadocs, typos, refactor DruidCoordinatorRuntimeParams.createAvailableSegmentsSet()

* Style

* typo

* Disable StaticPseudoFunctionalStyleMethod inspection because of too much false positives

* Fixes
2019-03-19 18:22:56 -03:00
Qi Shu e58f541047 Small tweak to override Blueprint's CSS rule for lookup edit dialog (#7283)
* Small tweak to override Blueprint's CSS rule

* Follow the same CSS style by using &
2019-03-18 14:24:50 -07:00
Christian Senet b2a940addd Fix flattening Avro Maps with Utf8 keys (#7258)
* #5884 Avro Maps can't be flattened in Kafka Indexing on master

* correct typo and reduce code size
2019-03-18 14:23:53 -07:00
Jonathan Wei c17f4bbad8 Fix 0.14.0 rc1 voting issues (#7284) 2019-03-18 14:16:21 -07:00
Jihoon Son e18d5d96d9 Ignore bad JSON entries in SQLMetadataSupervisorManager.getAll() (#7278) 2019-03-18 14:28:11 +08:00
Qi Shu 1f4ad518d8 Add compaction dialog in druid console which allows users to add/edit data source compaction configuration (#7242)
* Add compaction dialog in druid console which allows users to add/edit data source compaction configuration

* Addressed naming issues; changed json input validating process
2019-03-17 19:21:23 -07:00
Vadim Ogievetsky b6b1e6160c Added tslint to web console (#7280)
* added tslint to web console

* added react linting and made rules stricter

* order imports

* update package-lock
2019-03-17 09:23:17 -07:00
Qi Shu 1b6b40e511 Add lookups view to allow adding/editing/deleting of druid lookups (#7259)
* Add lookups view to allow adding/editing/deleting of druid lookups

* Remove unused bp3 class

* Make lookup editor dialog wider
2019-03-16 01:46:43 -07:00
Qi Shu 5406aaa49d Add SQL auto complete in druid console (#7244)
* Add SQL auto complete in druid console

* Add comment in sql.md to alert user to change create-sql-function-doc if sql.md format gets changed
2019-03-16 01:45:53 -07:00
Jihoon Son 892d1d35d6
Deprecate NoneShardSpec and drop support for automatic segment merge (#6883)
* Deprecate noneShardSpec

* clean up noneShardSpec constructor

* revert unnecessary change

* Deprecate mergeTask

* add more doc

* remove convert from indexMerger

* Remove mergeTask

* remove HadoopDruidConverterConfig

* fix build

* fix build

* fix teamcity

* fix teamcity

* fix ServerModule

* fix compilation

* fix compilation
2019-03-15 23:29:25 -07:00
Qi Shu fdbbf82cda Fixed bugs in retention cell and retention dialog (#7276)
* Fixed a bug when there are so many retention rules that it becomes unclickable; fixed a bug when retention dialog is at bottom and becomes unreachable

* Using class for CSS instead of inline style
2019-03-15 17:30:47 -07:00
Qi Shu a337f6d4cc Fixed height issue in dynamic config dialog where user cannot scroll (#7268) 2019-03-15 15:15:12 -07:00