Commit Graph

10175 Commits

Author SHA1 Message Date
Sashidhar Thallam 51a7235ebc Making optimal usage of multiple segment cache locations (#8038)
* #7641 - Changing segment distribution algorithm to distribute segments to multiple segment cache locations

* Fixing indentation

* WIP

* Adding interface for location strategy selection, least bytes used strategy impl, round-robin strategy impl, locationSelectorStrategy config with least bytes used strategy as the default strategy

* fixing code style

* Fixing test

* Adding a method visible only for testing, fixing tests

* 1. Changing the method contract to return an iterator of locations instead of a single best location. 2. Check style fixes

* fixing the conditional statement

* Added testSegmentDistributionUsingLeastBytesUsedStrategy, fixed testSegmentDistributionUsingRoundRobinStrategy

* to trigger CI build

* Add documentation for the selection strategy configuration

* to re trigger CI build

* updated docs as per review comments, made LeastBytesUsedStorageLocationSelectorStrategy.getLocations a synchronzied method, other minor fixes

* In checkLocationConfigForNull method, using getLocations() to check for null instead of directly referring to the locations variable so that tests overriding getLocations() method do not fail

* Implementing review comments. Added tests for StorageLocationSelectorStrategy

* Checkstyle fixes

* Adding java doc comments for StorageLocationSelectorStrategy interface

* checkstyle

* empty commit to retrigger build

* Empty commit

* Adding suppressions for words leastBytesUsed and roundRobin of ../docs/configuration/index.md file

* Impl review comments including updating docs as suggested

* Removing checkLocationConfigForNull(), @NotEmpty annotation serves the purpose

* Round robin iterator to keep track of the no. of iterations, impl review comments, added tests for round robin strategy

* Fixing the round robin iterator

* Removed numLocationsToTry, updated java docs

* changing property attribute value from tier to type

* Fixing assert messages
2019-09-28 00:17:44 -06:00
Evan Ren 17d9d7daed Increase column size for taskID and createdTime, and decrease Type and Duration (#8594) 2019-09-27 14:19:04 -07:00
Faxian Zhao e1b4a3ab71 bug fix for lookup leak when we remove the last lookup from lookup tier (#8598)
* bug fix for lookup leak when we remove the last lookup from lookup tier

* warnings about lookups that will never be loaded

* fix unit test
2019-09-27 03:55:02 -07:00
Fokko Driesprong a2363b6b61 Remove commons-httpclient (#8407) 2019-09-27 02:14:58 -07:00
Fokko Driesprong 99c3e0bb3f Bump HttpClient to 4.5.10 (#8404)
* Bump HttpClient to 4.5.9

* Remove Licenses file

* Revert license

* Remove duplicate dependency

* Bump HttpClient to 4.5.10
2019-09-27 02:14:36 -07:00
Jonathan Wei 5f83cd879e Fix download link in header links (#8600) 2019-09-26 18:06:12 -07:00
Clint Wylie 7781820dea JsonParserIterator.init future timeout (#8550)
* add timeout support for JsonParserIterator init future

* add queryId

* should be less than 1

* fix

* fix npe

* fix lgtm

* adjust exception, nullable

* fix test

* refactor

* revert queryId change

* add log.warn to tie exception to json parser iterator
2019-09-27 09:13:37 +09:00
elloooooo 7f2b6577ef get active task by datasource when supervisor discover tasks (#8450)
* get active task by datasource when supervisor discover tasks

* fix ut

* fix ut

* fix ut

* remove unnecessary condition check

* fix ut

* remove stream in hot loop
2019-09-26 16:15:24 -07:00
Fangyuan Deng a280c5dc03 fix queuedSize not decrease in HttpLoadQueuePeon when load failed (#8596) 2019-09-26 08:48:00 -07:00
Himanshu 9f1f5e115c
doubleMean aggregator to be used at query time (#8459)
* doubleMean aggregator for computing mean

* make docs

* build fixes

* address review comment: handle null args
2019-09-26 08:04:33 -07:00
Vadim Ogievetsky f4605f45be Website: stricter replace (#8593)
* stricter replace

* better fix
2019-09-25 17:25:39 -07:00
Nishant Bangarwa a75ddaad9e Add TrustedDomain Authenticator (#8248)
* Add TrustedDomain Authenticator

update javadoc

Add nullable annotations

Add cautionary note

fix travis failure

* add IP to spell checker
2019-09-25 11:25:03 -07:00
Vadim Ogievetsky 563718c5b2 add druid version correctly (#8586) 2019-09-24 18:19:06 -07:00
Evan Ren 0467cce7a0 Web console: Add frontend buttons to remove group by (#8537)
* Add frontend buttons to remove group by

* Change icon for remove group by

* Update web console to use latest toolkit

* Add test cases to verify that remove group by buttons are rendered

* Correct mistake of using incorrect components

* Update tests for two cases

* Put remove button after group by
2019-09-24 16:32:02 -07:00
Clint Wylie eabddffd6e fix http firehose factory leaky connection in constructor (#8576)
* fix http firehose factory leaky connection in constructor

* stylin
2019-09-24 17:08:43 -06:00
Vadim Ogievetsky 7c14fa08f8 Web console: Expand filter UI (#8579)
* add controls to the filter UI

* fix double base
2019-09-24 12:27:46 -07:00
Himanshu b6a16b5eb6 make it possible to not emit cache metrics and disable by default (#8561) 2019-09-24 22:12:09 +08:00
Rye f2a444321b Added live reports for Kafka and Native batch task (#8557)
* Added live reports for Kafka and Native batch task

* Removed unused local variables

* Added the missing unit test

* Refine unit test logic, add implementation for HttpRemoteTaskRunner

* checksytle fixes

* Update doc descriptions for updated API

* remove unnecessary files

* Fix spellcheck complaints

* More details for api descriptions
2019-09-23 21:08:36 -07:00
Vadim Ogievetsky 52f3f2c229 fix docs version interpolation (#8568) 2019-09-22 17:38:55 -07:00
Vadim Ogievetsky 2104cee79b Web console: prevent extra trim in auto complete (#8543)
* prevent extra trim in auto complete

* add unit test
2019-09-22 15:18:51 -07:00
Vadim Ogievetsky 94298f7809 Update Kafka loading docs to use the streaming data loader (#8544)
* fix redirects

* remove useless page

* fix Single server reference configurations formatting

* update batch data loading

* update Kafka docs

* fix typos and tests

* add more links

* fix spelling
2019-09-22 15:00:52 -07:00
Roman Leventov e7cc968749
Add new items to concurrency code review checklist (#8493) 2019-09-22 14:19:37 +03:00
SandishKumarHN ade8d1922d #8156 : StructuralSearchInspection, Prohibit check on Thread.ge… (#8394)
* StructuralSearchInspection, Prohibit check on Thread.getState()

* review changes - 1

* review changes 2

* review changes 3

* test fix

* review changes-2

* review changes-3
2019-09-22 14:12:05 +03:00
Vadim Ogievetsky 868bb42301 clean up default values and add infos (#8567) 2019-09-21 21:00:21 -07:00
Chi Cao Minh aeac0d4fd3 Adjust defaults for hashed partitioning (#8565)
* Adjust defaults for hashed partitioning

If neither the partition size nor the number of shards are specified,
default to partitions of 5,000,000 rows (similar to the behavior of
dynamic partitions). Previously, both could be null and cause incorrect
behavior.

Specifying both a partition size and a number of shards now results in
an error instead of ignoring the partition size in favor of using the
number of shards. This is a behavior change that makes it more apparent
to the user that only one of the two properties will be honored
(previously, a message was just logged when the specified partition size
was ignored).

* Fix test

* Handle -1 as null

* Add -1 as null tests for single dim partitioning

* Simplify logic to handle -1 as null

* Address review comments
2019-09-21 20:57:40 -07:00
Chi Cao Minh 99b6eedab5 Rename partition spec fields (#8507)
* Rename partition spec fields

Rename partition spec fields to be consistent across the various types
(hashed, single_dim, dynamic). Specifically, use targetNumRowsPerSegment
and maxRowsPerSegment in favor of targetPartitionSize and
maxSegmentSize. Consistent and clearer names are easier for users to
understand and use.

Also fix various IntelliJ inspection warnings and doc spelling mistakes.

* Fix test

* Improve docs

* Add targetRowsPerSegment to HashedPartitionsSpec
2019-09-20 14:59:18 -06:00
Chi Cao Minh 187b507b3d Fix CuratorModule flaky test (#8562)
For CuratorModuleTest. exitsJvmWhenMaxRetriesExceeded(), the expected
log message is intermittently not the first one in the list of captured
log messages. For example, it is the second one in
https://travis-ci.org/apache/incubator-druid/jobs/586792178#L754.
2019-09-20 13:36:22 -07:00
Himanshu 62afbca7b9
update HRTR to account for task known to be running on a worker when it shows up (#8427) 2019-09-19 10:19:17 -07:00
Xavier Léauté e184d24a74
add support for dogstatsd events in statsd-emitter (#8546)
* add support for dogstatsd events in statsd-emitter
* add option to turn on alert events (off by default)
* updated docs
2019-09-19 08:12:30 -07:00
Vadim Ogievetsky 36a6365d9f Web console: polish the data loader (#8554)
* rearrange ioConfig prop ordering in Tune step

* make useEarliestOffset mandatory

* required intent

* make segmentGranularity requred for batch ingestion

* update tests
2019-09-18 20:53:18 -07:00
Evan Ren 8650ee9fd0 Web console: Druid status displayed in a table (#8484)
* Retrieved data from endpoint and displayed on table

* Added view raw button, removed json, and fixed formatting for table

* Remove error var

* Fixed snapshot for updated status dialog

* Made changes based on PR review

* Added version tag

* Updated snapshot to match changes

* Made more changes based on review

* Fix filter and formatting

* Fix filter and add unit test

* fix styling of dialog

* Fix footer height

* Fixed testing of filtering
2019-09-17 15:26:51 -06:00
Chi Cao Minh 7dcbaca658 Spellcheck docs (#8548)
* Spellcheck docs

Fix spelling mistakes in docs and add CI job for running spellcheck on
docs.

* Add missing license header
2019-09-17 12:47:30 -07:00
Gian Merlino d96ca9bd61 Fix serde of FilterTuning maxCardinalityToUseBitmapIndex. (#8551) 2019-09-17 12:46:46 -07:00
Chi Cao Minh baec3a06e9 Fix IntelliJ inspection error (#8553)
Change by #8535 causes TeamCity inspection error in CI (although it does
not show the error in the local IDE).
2019-09-17 12:45:25 -07:00
Benedict Jin c6f4f09557 Fix missing space in string literal and spurious Javadoc @param tags from LGTM (#8491)
* Fix missing space in string literal

* Fix spurious Javadoc @param tags
2019-09-16 14:37:47 +05:30
Chi Cao Minh c254261e73 Make running test in Intellij work with JaCoCo (#8526)
* Make running test in Intellij work with JaCoCo

After the jacoco-maven-plugin was added, running test in IntelliJ fails
with the error "Could not find or load main class @{jacocoArgLine}".
Add a default IntelliJ configuration file so that running tests in the
IDE works with the jacoco-maven-plugin configuration and the
out-of-the-box developer experience is better. Other solutions described
in https://stackoverflow.com/q/24115142 (e.g., setting an empty
"jacocoArgLine" maven property) did not work.

* Add comment with link to PR
2019-09-14 11:56:09 -07:00
Chi Cao Minh 5f39ee21ff Add diags for flaky CuratorModuleTest (#8528)
Add diagnostic messages to debug
CuratorModuleTest.exitsJvmWhenMaxRetriesExceeded() intermittent test
failures.
2019-09-14 11:55:26 -07:00
Clint Wylie df14e5d696
fix caching bug with multi-column group-by (#8535)
* fix caching bug with multi-column group-by

* review
2019-09-13 17:41:23 -07:00
Chi Cao Minh 8b7b412d9d Add 'edit' and 'copy code' buttons to docs (#8483)
Having an 'edit' button will help encourage doc contributions from the
community.

Having 'copy code' buttons on the code blocks is especially convenient
for the quickstart tutorials in the docs.
2019-09-13 16:56:50 -07:00
Vadim Ogievetsky 6c7f36d364 fix spec dialog highlighting (#8533) 2019-09-13 15:24:08 -07:00
Vadim Ogievetsky 0490909ab3 Web console: Update web console docs for 0.16.0 (#8530)
* Update webconsole docs

* home view

* fix annotation typo
2019-09-13 09:09:36 -07:00
Abhishek Radhakrishnan 66341cdade Fix incorrect build from source path in README.md and druid repo url. (#8531) 2019-09-12 19:48:01 -07:00
Kamal Gurala 61761bd0b1 kafka version update (#8525) 2019-09-12 18:56:47 -07:00
Vadim Ogievetsky ad72f36360 fix forceGuaranteedRollup tooltip (#8529) 2019-09-12 17:24:29 -07:00
Clint Wylie 75978e5b98 move google ext docs from contrib to core (#8512)
* move google ext docs from contrib to core

* fix links

* revert unintended change

* more links, add note to example ext doc that it was removed, unlink from sidebar
2019-09-12 09:40:39 -07:00
Vadim Ogievetsky a6eca5e935 Web console: Force intervals config (#8514)
* make sure intervals are required

* all truncated values everywhere

* continue to spec when going from tasks table

* remove unused thigns

* fix alert
2019-09-12 00:35:04 -07:00
Jonathan Wei 0145642d8b Move router/indexer config/API docs to main pages (#8510)
* Move router/indexer config/API docs to main pages

* Restore missing properties, fix typo

* Use sentence casing

* Fix broken link
2019-09-11 21:42:58 -07:00
Clint Wylie b00dd84fa2 clarify error messaging for parallel indexing task when when missing numShards or intervals (#8513) 2019-09-11 20:47:27 -07:00
Vadim Ogievetsky 1498250204 Fix snitch dialog final step (#8506) 2019-09-11 12:44:16 -07:00
Clint Wylie fb078eea1e
fix web-console build in src distribution, fix kafka doc minimum version (#8502) 2019-09-10 21:01:07 -07:00