Commit Graph

7815 Commits

Author SHA1 Message Date
Parag Jain f9a61ea2ba Kafka lag emitter - Kafka Indexing Service (#4194)
* Kafka lag emitter

* enforce minimum emit period to a minute

* fixed comment
2017-05-02 17:30:07 -06:00
Roman Leventov 5e85fcc0f5 Restore BaseQuery.computeOverridenContext() for compatibility (#4241) 2017-05-02 10:22:02 -07:00
Roman Leventov 0bc18e7906 Make UpdateCounter proof to update count overflow (#4138)
* Make UpdateCounter proof to update count overflow.

* Fix
2017-05-01 09:59:49 -07:00
hzy001 0c464f4a84 Fix docs (#4225)
* Fix one typo

Signed-off-by: Hao Ziyu <haoziyu@qiyi.com>

* Fix deprecated links

Signed-off-by: Hao Ziyu <haoziyu@qiyi.com>
2017-05-01 09:55:43 -07:00
Jihoon Son 7411b18df9 Add BroadcastDistributionRule (#4077)
* Add BroadcastDistributionRule

* Add missing null check

* Rename variable 'colocateDataSource' to 'colocatedDatasource'

* Address comments

* Document for broadcast rules

* Drop segments which are not co-located anymore

* Remove duplicated segment loading and dropping

* Add caveat

* address comments
2017-05-01 09:55:17 -07:00
Himanshu 5a5a2749cd improvements to coordinator lookups management (#3855)
* coordinator lookups mgmt improvements

* revert replaces removal, deprecate it instead

* convert and use older specs stored in db

* more tests and updates

* review comments

* add behavior for 0.10.0 to 0.9.2 downgrade

* incorporating more review comments

* remove explicit lock and use LifecycleLock in LookupReferencesManager. use LifecycleLock in LookupCoordinatorManager as well

* wip on LookupCoordinatorManager

* lifecycle lock

* refactor thread creation into utility method

* more review comments addressed

* support smooth roll back of lookup snapshots from 0.10.0 to 0.9.2

* correctly use LifecycleLock in LookupCoordinatorManager and remove synchronization from start/stop

* run lookup mgmt on leader coordinator only

* wip: changes to do multiple start() and stop() on LookupCoordinatorManager

* lifecycleLock fix usage in LookupReferencesManagerTest

* add LifecycleLock back

* fix license hdr

* some fixes

* make LookupReferencesManager.getAllLookupsState() consistent while still being lockless

* address review comments

* addressing leventov's comments

* address charle's comments

* add IOE.java

* for safety in LookupReferencesManager mainThread check for lifecycle started state on each loop in addition to interrupt

* move thread creation utility method to Execs

* fix names

* add tests for LookupCoordinatorManager.lookupManagementLoop()

* add further tests for figuring out toBeLoaded and toBeDropped on LookupCoordinatorManager

* address leventov comments

* remove LookupsStateWithMap and parameterize LookupsState

* address review comments

* address more review comments

* misc fixes
2017-04-28 08:41:38 -05:00
Roman Leventov b9fd30e90a Add Checkstyle check to prohibit IntelliJ-style commented code lines (#4220)
* Add Checkstyle check to prohibit IntelliJ-style commented code lines

* Address comment

* Restore issue link
2017-04-27 18:11:25 -07:00
Gian Merlino 631068b099 Fix broken DataSketches link. (#4221)
* Fix broken DataSketches link.

* Better fixed link.
2017-04-27 17:37:12 -07:00
kaijianding c47cfed0ec Significantly improve LongEncodingStrategy.AUTO build performance (#4215)
* Significantly improve LongEncodingStrategy.AUTO build performance

* use numInserted instead of tempIn.available

* fix bug
2017-04-27 15:11:07 +03:00
Fokko Driesprong 13143f9376 Update to Parquet 1.8.2 (#4210)
Hi guys,

Since Spark 2.x uses Parquet 1.8.2, we would like to update Druid's parquet library from 1.8.1 to 1.8.2 as well. It includes a lot of patches, performance improvements and better compatibility:
4aba4da...c652278

Cheers, Fokko
2017-04-27 15:34:30 +09:00
Himanshu 40057570f3 doc update on overlord console url when coordinator is acting as overlord (#4213) 2017-04-26 15:03:54 -07:00
Himanshu 9b9e1cfecb coordinator dynamic config POST to update only explicitly specified fields (#4141)
* coordinator dynamic config POST to update only explicitly specified fields instead of resetting everything else to zeros

* address review comments
2017-04-26 14:59:20 -07:00
Bas van Schaik 54463941b9 Fix two alerts from lgtm.com: comparing two boxed primitive values using (#4212)
the == or != operator compares object identity, which may not be
intended

Details:
013566ade9/files/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchEstimatePostAggregator.java (V144)
013566ade9/files/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchMergeAggregatorFactory.java (V164)
2017-04-26 14:56:25 -07:00
David Lim 52f7bb091d suppress warn message if metricsSpec is absent when using no-rollup ingestion (#4211) 2017-04-25 22:52:49 -06:00
Roman Leventov ee9b5a619a Fix bugs in query builders and in TimeBoundaryQuery.getFilter() (#4131)
* Add queryMetrics property to Query interface; Fix bugs and removed unused code in Druids

* Fix a bug in TimeBoundaryQuery.getFilter() and remove TimeBoundaryQuery.getDimensionsFilter()

* Don't reassign query's queryMetrics if already present in CPUTimeMetricQueryRunner and MetricsEmittingQueryRunner

* Add compatibility constructor to BaseQuery

* Remove Query.queryMetrics property

* Move nullToNoopLimitSpec() method to LimitSpec interface

* Rename GroupByQuery.applyLimit() to postProcess(); Fix inconsistencies in GroupByQuery.Builder
2017-04-25 16:32:02 -05:00
Akash Dwivedi a2419654ea Allow hadoop configurations using runtime properties. (#4189) 2017-04-26 00:05:27 +05:30
Gian Merlino 3b92220015 Reduce log spam from Avro decoders. (#4205)
These objects get constructed semi-frequently (any time a parser is
deserialized) and so info logs are spammy. They'll still appear in
task logs at least once, since they're part of the task definition and
will get logged due to that.
2017-04-25 23:59:59 +05:30
kaijianding 336089563d skip rows which are added after cursor created (#4049)
* fix can't get dim value via IncrementalIndexStorageAdapter cursor

* address the comment

* add ut

* address ut comments

* fix bug and fix ut
2017-04-26 03:26:46 +09:00
Himanshu 4d3745d6c9 log the exception on failure to send query response (#4179) 2017-04-25 10:27:20 -07:00
Gian Merlino 97ddb38d75 DatasourceInputSplit: Serialize with write instead of writeUTF. (#4195)
writeUTF has a limit of 64KB, making it difficult to write out splits
that read a large number of descriptors for small segments.
2017-04-25 10:26:44 -07:00
asrayousuf e4fbc2bc5b Updating the description of useCache (#4200)
Updating the description of useCache

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment
2017-04-25 10:26:15 -07:00
Gian Merlino 809112cd5f DirectDruidClient: Fix division by zero. (#4206)
* DirectDruidClient: Fix division by zero.

Introduced in #3954 when some floating math was changed to
integer math. This patch restores the old math.

* Added comment.
2017-04-25 13:03:00 +09:00
Benedict Jin de815da942 Some code refactor for better performance of `Avro-Extension` (#4092)
* 1. Collections.singletonList instand of Arrays.asList; 2. close FSDataInputStream/ByteBufferInputStream for releasing resource; 3. convert com.google.common.base.Function into java.util.function.Function; 4. others code refactor

* Put each param on its own line for  code style

* Revert GenericRecordAsMap back about `Function`
2017-04-25 12:46:32 +09:00
satishbhor d51097c809 Fix lz4 library incompatibility in kafka-indexing-service extension (#4115)
* Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Bumped Kafka version to 0.10.2.0 for : Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Replaced Lists.newArrayList() with Collections.singletonList() For Fix lz4 library incompatibility in kafka-indexing-service extension #4115
2017-04-25 12:23:51 +09:00
Jonathan Wei 723a855ab9 Fix nested groupBys with outer exfns on inner numeric columns (#4182) 2017-04-21 19:47:46 -07:00
Gian Merlino 2ca7b00346 Update versions to 0.10.1-SNAPSHOT. (#4191) 2017-04-20 18:12:28 -07:00
Gian Merlino 60caa641f3 Restore backwards compatibility of Query. (#4185) 2017-04-19 19:47:50 +03:00
Jihoon Son 5b69f2eff2 Make timeout behavior consistent to document (#4134)
* Make timeout behavior consistent to document

* Refactoring BlockingPool and add more methods to QueryContexts

* remove unused imports

* Addressed comments

* Address comments

* remove unused method

* Make default query timeout configurable

* Fix test failure

* Change timeout from period to millis
2017-04-19 09:47:53 +09:00
kaijianding db656c5a88 fix kafka8 unparsable message halt job issue (#4164) 2017-04-18 11:23:02 -07:00
Jerry Chung 0bcfd9354c Fix S3 deep storage push and s3 insert-segment-to-db (#4174)
* Fix S3 deep storage push and s3 insert-segment-to-db

* Less verbose checks in S3DataSegmentFinder
2017-04-14 19:42:10 -07:00
Himanshu 8432138a07 log query in requestLog when client has disconnected and response write fails (#4169) 2017-04-15 04:04:52 +05:30
Himanshu 5c9198347b fix over counting of bytes in ByteCountingLRUMap (#4168) 2017-04-14 07:06:34 +09:00
Gian Merlino b2954d5fea Better groupBy error messages and docs around resource limits. (#4162)
* Better groupBy error messages and docs around resource limits.

* Fix BufferGrouper test from datasketches.

* Further clarify.
2017-04-13 10:38:53 -07:00
Ram iyer 2e9589215e removing unused var (#4163) 2017-04-13 04:03:41 +09:00
kaijianding 676af79044 don't do postAgg in TimeseriesQueryQueryToolChest when not necessary (#4155)
* don't do postAgg in TimeseriesQueryQueryToolChest when not necessary

* set postAggs to empty list in TimeseriesQueryQueryToolChest instead of checking finalizing fn

* fix ut

* fix ut again
2017-04-12 15:46:46 +05:30
Gian Merlino 9f4266fba1 Fix SortCollapseRule when inner order is DESC. (#4157)
* Fix SortCollapseRule when inner order is DESC.

* Remove unused import.
2017-04-12 15:39:45 +05:30
Roman Leventov 15f3a94474 Copy closer into Druid codebase (fixes #3652) (#4153) 2017-04-10 09:38:45 +09:00
Parag Jain 7e0d4c9555 secure supervisor endpoints (#3985) 2017-04-05 16:42:32 -07:00
Xiuming Chen 7e4e5510e0 Outdated property names (#4146)
Outdated property names?
2017-04-05 16:37:38 -07:00
Dongkyu Hwangbo 0d2e91ed50 Adding Kafka-emitter (#3860)
* Initial commit

* Apply another config: clustername

* Rename variable

* Fix bug

* Add retry logic

* Edit retry logic

* Upgrade kafka-clients version to the most recent release

* Make callback single object

* Write documentation

* Rewrite error message and emit logic

* Handling AlertEvent

* Override toString()

* make clusterName more optional

* bump up druid version

* add producer.config option which make user can apply another optional config value of kafka producer

* remove potential blocking in emit()

* using MemoryBoundLinkedBlockingQueue

* Fixing coding convention

* Remove logging every exception and just increment counting

* refactoring

* trivial modification

* logging when callback has exception

* Replace kafka-clients 0.10.1.1 with 0.10.2.0

* Resolve the problem related of classloader

* adopt try statement

* code reformatting

* make variables final

* rewrite toString
2017-04-04 14:07:43 -07:00
Jihoon Son b166e13d2b Make DruidCoordinatorTest fail fast (#4142) 2017-04-04 13:58:32 -07:00
Roman Leventov 81ba0224c5 Report unsuccessful queries in AsyncQueryForwardingServlet (#4143) 2017-04-04 13:53:07 -07:00
Nishant Bangarwa 801ea5efa4 Fix: Broker fails throws OOME with conf-quickstart (#4127)
when running the the packaged conf-quickstart druid broker fails to
start and throws OOME. increasing the direct memory to get around this.
2017-03-29 11:43:58 -07:00
Gian Merlino bbb61e638b SQL: Support for another form of filtered aggregator. (#4109)
* SQL: Support for another form of filtered aggregator.

* Fix comment, add test for MAX too.
2017-03-27 15:22:36 -07:00
Roman Leventov 73d9b31664 GenericIndexed minor bug fixes, optimizations and refactoring (#3951)
* Minor bug fixes in GenericIndexed; Refactor and optimize GenericIndexed; Remove some unnecessary ByteBuffer duplications in some deserialization paths; Add ZeroCopyByteArrayOutputStream

* Fixes

* Move GenericIndexedWriter.writeLongValueToOutputStream() and writeIntValueToOutputStream() to SerializerUtils

* Move constructors

* Add GenericIndexedBenchmark

* Comments

* Typo

* Note in Javadoc that IntermediateLongSupplierSerializer, LongColumnSerializer and LongMetricColumnSerializer are thread-unsafe

* Use primitive collections in IntermediateLongSupplierSerializer instead of BiMap

* Optimize TableLongEncodingWriter

* Add checks to SerializerUtils methods

* Don't restrict byte order in SerializerUtils.writeLongToOutputStream() and writeIntToOutputStream()

* Update GenericIndexedBenchmark

* SerializerUtils.writeIntToOutputStream() and writeLongToOutputStream() separate for big-endian and native-endian

* Add GenericIndexedBenchmark.indexOf()

* More checks in methods in SerializerUtils

* Use helperBuffer.arrayOffset()

* Optimizations in SerializerUtils
2017-03-27 14:17:31 -05:00
Jason Banich 117c698c59 Update StatsDEmitter.java (#4111)
This was mentioned in the original pull (https://github.com/druid-io/druid/pull/2410) by @sascha-coenen, and the original author (@michaelschiff) agreed that it seemed reasonable

This commit fixes issue https://github.com/druid-io/druid/issues/3960
2017-03-27 10:32:21 -07:00
JackyWoo a0f2cf05d5 Add EqualDistributionWithAffinityWorkerSelectStrategy which balance w… (#3998)
* Add EqualDistributionWithAffinityWorkerSelectStrategy which balance work load within affinity workers.

* add docs to equalDistributionWithAffinity
2017-03-25 19:15:49 -07:00
Gian Merlino 90f9932bd3 SQL: Rule to collapse sort chains. (#4085)
Useful for queries like `SELECT * FROM (...) LIMIT X`, where the inner query
has an order by or limit in it.
2017-03-24 19:20:01 -07:00
Gian Merlino 76c4b6446e SQL: Fix handling of CURRENT_TIMESTAMP and friends in non-UTC timezones. (#4114) 2017-03-24 18:45:23 -07:00
Gian Merlino dd6c0ab509 Add SQL REGEXP_EXTRACT function; add "index" to "regex" extractionFn. (#4055)
* Add SQL REGEXP_EXTRACT function; add "index" to "regex" extractionFn.

* Fix tests.
2017-03-24 17:38:36 -07:00