8091 Commits

Author SHA1 Message Date
Jihoon Son
c0eecc0faa Merge branch 'master' of https://github.com/druid-io/druid into streaming-grouper 2017-09-08 13:44:43 +09:00
Jihoon Son
dee9633b59 Address comments 2017-09-08 13:31:37 +09:00
Himanshu
d43687d578 increase test timeout to 15sec as leader election jitter is random sleep between 1 to 5 secs (#4756) 2017-09-06 09:56:25 -07:00
Gian Merlino
dc5c6f13b1 Use trusty for travis jobs. (#4755)
* Use trusty for travis jobs.

The distro was set to "precise" in #4572 due to memory issues on trusty,
but we've been seeing performance issues on "precise" recently so let's
see how trusty is working these days.

* Less quiet.

* Adjust memory settings.

* Add back -q option.

* Tweak memory again.

* Adjustments.

* Try squeezing memory a bit more.
2017-09-06 18:51:06 +09:00
Gian Merlino
33c0928bed Collapse worker select strategies, change default, add strong affinity. (#4534)
* Collapse worker select strategies, change default, add strong affinity.

- Change default worker select strategy to equalDistribution. It is
  more generally useful than fillCapacity.
- Collapse the *WithAffinity strategies into the regular ones. The
  *WithAffinity strategies are retained for backwards compatibility.
- Change WorkerSelectStrategy to return nullable instead of Optional.
- Fix a couple of errors in the docs.

* Fix test.

* Review adjustments.

* Remove unused imports.

* Switch to DateTimes.nowUtc.

* Simplify code.

* Fix tests (worker assignment started off on a different foot)
2017-09-04 14:40:55 -07:00
Roman Leventov
c7b8116b3a Remove HadoopIOPeon (#4742) 2017-09-03 13:36:38 -07:00
Kevin Conaway
83e94e8b31 Expose version property for CustomVersioningPolicy so that it may serialize/deserialize correctly (#4747)
Expose version property for  so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly

Expose version property for CustomVersioningPolicy so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly
2017-09-03 13:36:18 -07:00
Jihoon Son
3f14db44e5 use AbstractPrioritizedCallable 2017-09-02 20:28:10 +09:00
Jihoon Son
2a619b946a Handle InterruptedException 2017-09-02 20:15:58 +09:00
Jihoon Son
3cdce754f2 Add ParallelCombinerTest 2017-09-02 20:02:19 +09:00
Jihoon Son
f31f6d203b Add ParallelCombiner 2017-09-02 19:15:52 +09:00
Jihoon Son
5c4b846cdf Refactor iterator 2017-09-02 18:43:38 +09:00
Jihoon Son
7040ffb2b8 Refactor buildCombineTree 2017-09-02 16:49:33 +09:00
Jihoon Son
a888db0884 Add comments for combining tree 2017-09-02 15:49:03 +09:00
Jihoon Son
92675a95bd Refactor Groupers.mergeIterators() 2017-09-02 14:13:01 +09:00
Jihoon Son
48dc2cdd80 Fix some typos and bugs 2017-09-02 14:08:34 +09:00
Himanshu
06ac6678e6 DruidLeaderSelector interface for leader election and Curator based impl. (#4699)
* DruidLeaderSelector interface for leader election and Curator based impl. DruidCoordinator/TaskMaster are updated to use the new interface.

* add fake DruidNode binding in integration-tests module

* add docs on DruidLeaderSelector interface

* remove start/stop and keep register/unregister Listener in DruidLeaderSelector interface

* updated comments on DruidLeaderSelector

* cache the listener executor in CuratorDruidLeaderSelector

* use same latch owner name that was used before

* remove stuff related to druid.zk.paths.indexer.leaderLatchPath config

* randomize the delay when giving up leadership and restarting leader latch
2017-09-01 09:49:04 -07:00
Gian Merlino
34a03b8e6c SQL: EXPLAIN improvements. (#4733)
* SQL: EXPLAIN improvements.

- Include query JSON in explain output.
- Fix a bug where semi-joins and nested groupBys were not fully explained.
- Fix a bug where limits were not included in "select" query explanations.

* Fix compile error.

* Fix compile error.

* Fix tests.
2017-09-01 09:35:13 -07:00
Himanshu
4c04083926 kafkaIndexTask unannounce service in final block (#4736) 2017-09-01 09:31:15 -07:00
Charles Allen
bdfc6fe25e Move common TypeReference into JacksonUtils (#4738) 2017-08-31 13:40:16 -07:00
Gian Merlino
9078925cab Docs for finalizingFieldAccess post-aggregator. (#4737) 2017-08-31 11:45:49 -07:00
T R Kyaw
d6179126ed Allow index job to utilize hadoop cluster information from job config. (#4626)
* Allow ndex job to utilize hadoop cluster information from job config.

* Add new method that inject system configuration and then job configuration.

* Make changes to use HadoopDruidIndexerConfig.addJobProperties method.

* refactor code for overloaded addJobProperties.
2017-08-30 16:44:33 -05:00
Bartosz Ługowski
8dddccc687 Graphite emitter - add plaintext protocol (#4265)
* Graphite emitter - add plaintext protocol. Configurable option of replacing slash to dot in metric name.

* Graphite emitter - fix misspelling in docs.

* Graphite emitter - extend docs.

* Graphite emitter - fix code style.
2017-08-29 06:23:06 -07:00
Jihoon Son
29a7dfe4fc Remove unused config 2017-08-29 20:47:44 +09:00
Niketh Sabbineni
beecb9e210 Fix failing build, remove unused import (#4726)
LGTM
2017-08-29 14:46:38 +09:00
Roman Leventov
4d109a358a Refactoring of Storage Adapters (#4710)
* Factor QueryableIndexColumnSelectorFactory and IncrementalIndexColumnSelectorFactory out of QueryableIndexStorageAdapter and IncrementalIndexStorageAdapter; Add Offset.getBaseReadableOffset(); Remove OffsetHolder interface; Replace Cursor extends ColumnSelectorFactory with composition; Reduce indirection in ColumnValueSelectors created by QueryableIndexColumnSelectorFactory

* Don't override clone() in FilteredOffset (the prev. implementation was broken); Some warnings fixed

* Simplify Cursors in QueryableIndexStorageAdapter

* Address comments

* Remove unused and unimplemented methods from GenericColumn interface

* Comments
2017-08-28 18:07:31 -07:00
Parag Jain
594a66f3c0 add scheme to AsyncQueryForwardingServlet (#4688)
* add scheme to AsyncQueryForwardingServlet

* add sslContext binding for Router
2017-08-28 15:03:43 -07:00
Gian Merlino
daf3c5f927 Add "round" option to cardinality and hyperUnique aggregators. (#4720)
* Add "round" option to cardinality and hyperUnique aggregators.

Also turn it on by default in SQL, to make math on distinct counts
work more as expected.

* Fix some compile errors.

* Fix test.

* Formatting.
2017-08-28 14:52:11 -07:00
Gian Merlino
9fbfc1be32 Add @ExtensionPoint and @PublicApi annotations. (#4433)
* Add @ExtensionPoint and @PublicApi annotations.

* Clean up wording.

* Remove unused import.

* Remove unused imports.

* Only types can be extension points.

* Adjust annotations some more.

* Remove unused import.

* Make ServletFilterHolder an extension point.

* Add a couple extension points, and update docs.
2017-08-28 14:50:58 -07:00
zhangxinyu1
b04261e7a2 In indexing service flow chart, it should be middlemanager who writes task status to zookeeper (#4654) 2017-08-27 10:17:15 -07:00
Jihoon Son
68ccece4ae More cleanup 2017-08-26 10:03:04 +09:00
Gian Merlino
43488df975 Fix dimension selectors with extractionFns on missing columns. (#4717)
* Fix dimension selectors with extractionFns on missing columns.

This patch properly applies the requested extractionFn to missing columns.
It's important when the extractionFn maps null to something other than null.

* Extract helper method.

* Change contracts of VirtualColumns and VirtualColumn methods based on review comments.

* Remove unused import.

* Remove unused method.

* Adjust helper function.

* Adjustments
2017-08-25 18:34:42 -05:00
dgolitsyn
fd0f349c68 Mark segments that are going to be dropped from the server and use it in DruidBalancer (#4486)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Mark segments that are going to be dropped from server and use this information in CostBalancerStrategy

* Fix tests
2017-08-25 18:25:56 -05:00
Roman Leventov
598cc46bae Replace HashMap with Obj2IntMap in StringDimensionIndexer; Small optimization in StringDimensionMergerV9 (#4721) 2017-08-25 12:30:39 -07:00
Himanshu
74538c3288 update internal-discovery Listener for node list and use same at router and coordinator (#4697)
* update LookupCoordinatorManager to use internal discovery to discover lookup nodes

* router:use internal-discovery to discover brokers

* minor [Curator]DruidDiscoveryProvider refactoring

* add initialized() method to DruidNodeDiscovery.Listener

* update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously

* Revert "update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously"

This reverts commit f796e441221fe8b0e9df87fdec6c9f47bcedd890.

* Revert "add initialized() method to DruidNodeDiscovery.Listener"

This reverts commit f0661541d073683f28fce2dd4f30ec37db90deb0.

* minor refactoring removing synchronized from DruidNodeDiscoveryProvider

* updated DruidNodeDiscovery.Listener contract to take List of nodes and first call marks initialization

* update HttpServerInventoryView to handle new contract and handle initialization

* router update to handle updated listener contract

* document DruidNodeDiscovery.Listener contract

* fix forbidden-api error

* change log level to info for unknown path children cache events in CuratorDruidNodeDiscoveryProvider

* announce broker only after segment inventory is initialized
2017-08-25 09:28:15 -07:00
Jihoon Son
701adb10cc Removed unused code 2017-08-25 20:58:50 +09:00
Jihoon Son
21d42ceee2 Revert GroupByBenchmark 2017-08-25 20:55:02 +09:00
Jihoon Son
c79959f51c Use a single dictionary while combining 2017-08-25 20:53:21 +09:00
hzy001
4f61dc66a9 Remove the deprecated variable localChildren (#4357)
Signed-off-by: Hao Ziyu <haoziyu@qiyi.com>
2017-08-24 15:27:34 -05:00
Parag Jain
421e02203f fix null sslPort issue (#4696)
* fix null sslPort issue

* add tests
2017-08-24 15:19:07 -05:00
QiuMM
59a48a560a Redis cache extension doc (#4702)
* Redis cache extension doc

* link redis cache doc in extensions.md
2017-08-24 09:53:51 -05:00
QiuMM
765f46af5b git ignore dependency-reduced-pom.xml (#4711) 2017-08-23 10:10:50 -07:00
Akash Dwivedi
b43720c46d Correction in indexing-service configuration doc. (#4700) 2017-08-22 23:21:34 -05:00
Roman Leventov
326a85a9a4 Add Offset.reset() and remove unused Offset implementations (#4706)
* Add Offset.reset() and remove unused Offset implementations

* Fix BitmapOffset

* Address comments
2017-08-22 17:43:29 -07:00
Himanshu
92bcfdc200 consider docker ip be 127.0.0.1 by default (#4707) 2017-08-23 00:55:46 +05:30
Roman Leventov
cacf63b007 Add AggregateCombiners (#4676)
* Add MetricCombiners

* Rename MetricCombiner to AggregateCombiner

* Spelling

* Fix TimestampAggregatorFactory.combine() and add makeAggregateCombiner() implementation

* Rename AggregateCombiner.combine() to fold()
2017-08-21 16:45:29 -07:00
Roman Leventov
cbd1902db8 Add forbidden-apis plugin; prohibit using system time zone (#4611)
* Forbidden APIs WIP

* Remove some tests

* Restore io.druid.math.expr.Function

* Integration tests fix

* Add comments

* Fix in SimpleWorkerProvisioningStrategy

* Formatting

* Replace String.format() with StringUtils.format() in RemoteTaskRunnerTest

* Address comments

* Fix GroupByMultiSegmentTest
2017-08-21 13:02:42 -07:00
Roman Leventov
fa87eaa6e8 Remove IndexedInts.fill() (#4705) 2017-08-21 13:01:34 -07:00
Jihoon Son
30c0038853 Fix doc 2017-08-19 12:04:28 +09:00
Jihoon Son
0e68c08d9d Add steaming grouper 2017-08-19 11:46:53 +09:00