Commit Graph

8159 Commits

Author SHA1 Message Date
Parag Jain 594a66f3c0 add scheme to AsyncQueryForwardingServlet (#4688)
* add scheme to AsyncQueryForwardingServlet

* add sslContext binding for Router
2017-08-28 15:03:43 -07:00
Gian Merlino daf3c5f927 Add "round" option to cardinality and hyperUnique aggregators. (#4720)
* Add "round" option to cardinality and hyperUnique aggregators.

Also turn it on by default in SQL, to make math on distinct counts
work more as expected.

* Fix some compile errors.

* Fix test.

* Formatting.
2017-08-28 14:52:11 -07:00
Gian Merlino 9fbfc1be32 Add @ExtensionPoint and @PublicApi annotations. (#4433)
* Add @ExtensionPoint and @PublicApi annotations.

* Clean up wording.

* Remove unused import.

* Remove unused imports.

* Only types can be extension points.

* Adjust annotations some more.

* Remove unused import.

* Make ServletFilterHolder an extension point.

* Add a couple extension points, and update docs.
2017-08-28 14:50:58 -07:00
zhangxinyu1 b04261e7a2 In indexing service flow chart, it should be middlemanager who writes task status to zookeeper (#4654) 2017-08-27 10:17:15 -07:00
Gian Merlino 43488df975 Fix dimension selectors with extractionFns on missing columns. (#4717)
* Fix dimension selectors with extractionFns on missing columns.

This patch properly applies the requested extractionFn to missing columns.
It's important when the extractionFn maps null to something other than null.

* Extract helper method.

* Change contracts of VirtualColumns and VirtualColumn methods based on review comments.

* Remove unused import.

* Remove unused method.

* Adjust helper function.

* Adjustments
2017-08-25 18:34:42 -05:00
dgolitsyn fd0f349c68 Mark segments that are going to be dropped from the server and use it in DruidBalancer (#4486)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Mark segments that are going to be dropped from server and use this information in CostBalancerStrategy

* Fix tests
2017-08-25 18:25:56 -05:00
Roman Leventov 598cc46bae Replace HashMap with Obj2IntMap in StringDimensionIndexer; Small optimization in StringDimensionMergerV9 (#4721) 2017-08-25 12:30:39 -07:00
Himanshu 74538c3288 update internal-discovery Listener for node list and use same at router and coordinator (#4697)
* update LookupCoordinatorManager to use internal discovery to discover lookup nodes

* router:use internal-discovery to discover brokers

* minor [Curator]DruidDiscoveryProvider refactoring

* add initialized() method to DruidNodeDiscovery.Listener

* update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously

* Revert "update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously"

This reverts commit f796e441221fe8b0e9df87fdec6c9f47bcedd890.

* Revert "add initialized() method to DruidNodeDiscovery.Listener"

This reverts commit f0661541d073683f28fce2dd4f30ec37db90deb0.

* minor refactoring removing synchronized from DruidNodeDiscoveryProvider

* updated DruidNodeDiscovery.Listener contract to take List of nodes and first call marks initialization

* update HttpServerInventoryView to handle new contract and handle initialization

* router update to handle updated listener contract

* document DruidNodeDiscovery.Listener contract

* fix forbidden-api error

* change log level to info for unknown path children cache events in CuratorDruidNodeDiscoveryProvider

* announce broker only after segment inventory is initialized
2017-08-25 09:28:15 -07:00
hzy001 4f61dc66a9 Remove the deprecated variable localChildren (#4357)
Signed-off-by: Hao Ziyu <haoziyu@qiyi.com>
2017-08-24 15:27:34 -05:00
Parag Jain 421e02203f fix null sslPort issue (#4696)
* fix null sslPort issue

* add tests
2017-08-24 15:19:07 -05:00
QiuMM 59a48a560a Redis cache extension doc (#4702)
* Redis cache extension doc

* link redis cache doc in extensions.md
2017-08-24 09:53:51 -05:00
QiuMM 765f46af5b git ignore dependency-reduced-pom.xml (#4711) 2017-08-23 10:10:50 -07:00
Akash Dwivedi b43720c46d Correction in indexing-service configuration doc. (#4700) 2017-08-22 23:21:34 -05:00
Roman Leventov 326a85a9a4 Add Offset.reset() and remove unused Offset implementations (#4706)
* Add Offset.reset() and remove unused Offset implementations

* Fix BitmapOffset

* Address comments
2017-08-22 17:43:29 -07:00
Himanshu 92bcfdc200 consider docker ip be 127.0.0.1 by default (#4707) 2017-08-23 00:55:46 +05:30
Roman Leventov cacf63b007 Add AggregateCombiners (#4676)
* Add MetricCombiners

* Rename MetricCombiner to AggregateCombiner

* Spelling

* Fix TimestampAggregatorFactory.combine() and add makeAggregateCombiner() implementation

* Rename AggregateCombiner.combine() to fold()
2017-08-21 16:45:29 -07:00
Roman Leventov cbd1902db8 Add forbidden-apis plugin; prohibit using system time zone (#4611)
* Forbidden APIs WIP

* Remove some tests

* Restore io.druid.math.expr.Function

* Integration tests fix

* Add comments

* Fix in SimpleWorkerProvisioningStrategy

* Formatting

* Replace String.format() with StringUtils.format() in RemoteTaskRunnerTest

* Address comments

* Fix GroupByMultiSegmentTest
2017-08-21 13:02:42 -07:00
Roman Leventov fa87eaa6e8 Remove IndexedInts.fill() (#4705) 2017-08-21 13:01:34 -07:00
Asif Mansoor Amanullah 37f85b08d2 move row up/down for null metric ordering (#4681)
* move row up/down for null metric ordering

* addressed comments

* addressed changes
2017-08-17 11:36:19 -05:00
Akash Dwivedi 786e7815c2 Fix issue https://github.com/druid-io/druid/issues/4690 (#4691) 2017-08-17 09:45:33 -05:00
Himanshu 74a64c88ab internal-discovery: interfaces for announcement/discovery, curator based impls (#4634)
* internal-discovery: interfaces for announcement/discovery, curator impls

* more tests

* address some review comments

* more fixes

* address more review comments

* simplify ObjectMapper setup in CuratorDruidNodeAnnouncerAndDiscoveryTest

* fix KafkaIndexTaskTest

* make lookupTier overridable via RealtimeIndexTask and KafkaIndexTask context

* make teamcity build happy
2017-08-16 13:07:16 -07:00
Niketh Sabbineni dba7c7d3cd Reduce excessive logging (#4680)
* Reduce excessive logging

* Refactoring code as per comments
2017-08-15 10:14:20 -07:00
Jonathan Wei ab28dc3b97 free() dictionary merging buffers in IndexMerger (#4684)
* free() dictionary merging buffers in IndexMerger

* Use close() for dictionary merge iterators

* Add comments on buffer free
2017-08-15 10:11:29 -07:00
Parag Jain 725a144096 add localhost as advertised hostname (#4689)
* add localhost as advertised hostname

* set advertised.host.name to localhost for test kafka broker
2017-08-14 16:59:26 -07:00
Jonathan Wei e91d4d1b80 Remove makeObjectColumnSelector() from DimensionIndexer (#4679) 2017-08-11 14:39:00 -07:00
Gian Merlino d775347b06 TimestampSpec: Have "auto" detect timestamps in almost-iso format. (#4682)
Fixes #4082.
2017-08-11 13:02:42 -07:00
Jonathan Wei 1bddfc089c Additional docs/log for direct memory usage (#4631)
* Additional docs/log for direct memory usage

* Tweak docs

* Doc rewording
2017-08-10 23:33:20 -07:00
Niketh Sabbineni eb0deba54a Fix NPE when locations are empty (#4667)
* Fix NPE when locations are empty

* Addressing comments
2017-08-10 23:31:28 -07:00
Gian Merlino 5ff8c52f16 SQL: Fix race with metadata caching. (#4674)
If DruidSchema started too long after the BrokerServerView, its
initialization callback would never get called, and it would sit
there not knowing about any tables.

This moves the registration of the callback into the constructor,
where it belongs.
2017-08-10 18:27:10 -07:00
Roman Leventov bf28d0775b Remove QueryRunner.run(Query, responseContext) and related legacy methods (#4482)
* Remove QueryRunner.run(Query, responseContext) and related legacy methods

* Remove local var
2017-08-11 09:12:38 +09:00
Yuewen Wang c821bc9a5a Implement "earlyMessageRejectionPeriod" config discussed in issue #4599 (#4607)
* Implement "earlyMessageRejectionPeriod" config discussed in issue #4599
    * implement the logics of this param
    * Added doc of this config
    * Added unit tests of it

* Update KafkaSupervisor.java

ameliorate comment

* fix format

* fix bug when rebasing
2017-08-11 09:12:08 +09:00
Jihoon Son 65c1d6c797 Add IntGrouper to avoid unnecessary boxing/unboxing in array-based aggregation (#4668)
* Add IntGrouper

* Fix build

* Address comments

* Add a benchmark query
2017-08-10 07:41:39 -07:00
solimant de9ba97d54 Move equals() from Float[Sum|Min|Max]AggregatorFactory to SimpleFloat... (#4675)
Addresses #4671
2017-08-10 07:22:22 -07:00
Gian Merlino 7c89e12ca9 Replace Guava Enum.getIfPresent with builtin version. (#4659)
* Replace Guava Enum.getIfPresent with builtin version.

This is useful for running in Hadoop environments that use Guava 11. Some
code is also simplified.

* Code review
2017-08-09 17:20:00 -07:00
Jihoon Son fe3421032b Parallel sort for ConcurrentGrouper (#4660)
* Multi-thread sort

* Address comments
2017-08-09 16:24:36 -07:00
Peter Cunningham ede7cf9eef Added support for where clauses to JDBC lookups. (#4643)
* Added support for where clauses to filter lookup values on ingestion.

Added a filter field to the JDBC lookups that is used to generate a
where clause so that only rows matching the filter value will be
brought into Druid. Example being filter="SOMECOLUMN=1"

* Required changes based on code review.

* Required changes based on code review.

* Added support for where clauses to filter lookup values on ingestion.

Added a filter field to the JDBC lookups that is used to generate a
where clause so that only rows matching the filter value will be
brought into Druid. Example being filter="SOMECOLUMN=1"

* Updates based on code review, mainly formatting and small refactor of
the buildLookupQuery method.

* Fixed broken buildLookupQuery method

* Removed empty line.

* Updates per review comments
2017-08-09 10:47:46 -07:00
Goh Wei Xiang 42569e65e2 Minor fix in ExpressionSelectors to avoid potential NPE. (#4669) 2017-08-09 10:13:31 -07:00
Roman Leventov 7454fd86a0 Polymorphic numeric getters for ColumnValueSelector (#4623)
* Add methods getFloat(), getDouble() and getLong() to ColumnValueSelector

* Fix copy-paste mistake in docs

* Spelling
2017-08-08 18:38:06 -07:00
Jihoon Son d5606bc558 Passing lockTimeout as a parameter for TaskLockbox.lock() (#4549)
* Passing lockTimeout as a parameter for TaskLockbox.lock()

* Remove TIME_UNIT

* Fix tc fail

* Add taskLockTimeout to TaskContext

* Add caution
2017-08-08 18:21:07 -07:00
Roman Leventov f5d4171459 Prohibit for loops which could be foreach with IntelliJ (#4653)
* Replace for with foreach

* Replace for with for-each in GroupByQueryEngineV2

* Remove io.druid.collections.IntList
2017-08-08 18:05:33 -07:00
QiuMM f18cc5df97 Redis cache extension (#4615)
* Redis cache extension

* Fix some trival and optimize code

* Add Override annotation in RedisCacheTest
2017-08-08 10:11:45 -07:00
dgolitsyn 4dd1e2b59e Do not remove segments from currentlyMovingSegments in DruidBalancer if move is impossible or not needed (#4472)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Replace putIfAbsent with computeIfAbsent in DruidBalancer

* Refactoring
2017-08-08 17:22:59 +03:00
Charles Allen bbe7fb8c46 Better logging for S3DataSegmentPuller `getVersion` (#4657)
* Eventual consistency of S3 means a `404` can be thrown. It helps to know the URI that was attempted.
2017-08-08 16:21:22 +03:00
Roman Leventov 59a2507268 Fix spacing in KafkaEmitterConfig (#4663) 2017-08-08 15:12:01 +03:00
Yuewen Wang f8dcb05fd1 Fix a NPE using kafka emitter extension (#4608)
* Fix a NPE using kafka emitter extension

* fix format

* Add @Nullable annotation on kafkaProducerConfig
2017-08-08 12:21:24 +09:00
Roman Leventov 486b7a2347 TaskMaster deadlock fix (#4548)
* Stop RemoteTaskRunner's cleanupExec using TaskMaster's lifecycle, not global injected lifecycle

* Prohibit starting Lifecycle twice; Make Lifecycle to reject addMaybeStartHandler() attempts in the process of stopping rather than entering deadlock

* Fix Lifecycle.addMaybeStartHandler()

* Remove RemoteTaskRunnerFactoryTest

* Add docs

* Language

* Address comments

* Fix RemoteTaskRunnerTestUtils
2017-08-07 14:28:43 -07:00
Roman Leventov aa7e4ae5e4 Enforce correct spacing with Checkstyle (#4651) 2017-08-05 10:18:25 -07:00
Jonathan Wei aa8d75004c More informative QueryInterruptedException toString() (#4642)
* More informative QueryInterruptedException toString()

* Use StringUtils.format
2017-08-04 16:00:20 -07:00
Jonathan Wei 9650d80f3e Don't use limit push down with having spec (#4630)
* Don't use limit push down with having spec

* Throw exception when forcing limit push down with having

* Tests for having and limit push down

* Fix pool sizes in unit test
2017-08-04 15:13:29 -07:00
Roman Leventov 7a005088d9 Add HistoricalCursor.getReadableOffset() to access unwrapped offset in selectors (#4633)
* Add HistoricalCursor.getReadableOffset() to access unwrapped offset in selectors, when the 'main' offset if FilteredOffset (fixes #4628)

* Stack overflow test
2017-08-04 12:51:48 -07:00