7805 Commits

Author SHA1 Message Date
Himanshu
40057570f3 doc update on overlord console url when coordinator is acting as overlord (#4213) 2017-04-26 15:03:54 -07:00
Himanshu
9b9e1cfecb coordinator dynamic config POST to update only explicitly specified fields (#4141)
* coordinator dynamic config POST to update only explicitly specified fields instead of resetting everything else to zeros

* address review comments
2017-04-26 14:59:20 -07:00
Bas van Schaik
54463941b9 Fix two alerts from lgtm.com: comparing two boxed primitive values using (#4212)
the == or != operator compares object identity, which may not be
intended

Details:
013566ade9/files/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchEstimatePostAggregator.java (V144)
013566ade9/files/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchMergeAggregatorFactory.java (V164)
2017-04-26 14:56:25 -07:00
David Lim
52f7bb091d suppress warn message if metricsSpec is absent when using no-rollup ingestion (#4211) 2017-04-25 22:52:49 -06:00
Roman Leventov
ee9b5a619a Fix bugs in query builders and in TimeBoundaryQuery.getFilter() (#4131)
* Add queryMetrics property to Query interface; Fix bugs and removed unused code in Druids

* Fix a bug in TimeBoundaryQuery.getFilter() and remove TimeBoundaryQuery.getDimensionsFilter()

* Don't reassign query's queryMetrics if already present in CPUTimeMetricQueryRunner and MetricsEmittingQueryRunner

* Add compatibility constructor to BaseQuery

* Remove Query.queryMetrics property

* Move nullToNoopLimitSpec() method to LimitSpec interface

* Rename GroupByQuery.applyLimit() to postProcess(); Fix inconsistencies in GroupByQuery.Builder
2017-04-25 16:32:02 -05:00
Akash Dwivedi
a2419654ea Allow hadoop configurations using runtime properties. (#4189) 2017-04-26 00:05:27 +05:30
Gian Merlino
3b92220015 Reduce log spam from Avro decoders. (#4205)
These objects get constructed semi-frequently (any time a parser is
deserialized) and so info logs are spammy. They'll still appear in
task logs at least once, since they're part of the task definition and
will get logged due to that.
2017-04-25 23:59:59 +05:30
kaijianding
336089563d skip rows which are added after cursor created (#4049)
* fix can't get dim value via IncrementalIndexStorageAdapter cursor

* address the comment

* add ut

* address ut comments

* fix bug and fix ut
2017-04-26 03:26:46 +09:00
Himanshu
4d3745d6c9 log the exception on failure to send query response (#4179) 2017-04-25 10:27:20 -07:00
Gian Merlino
97ddb38d75 DatasourceInputSplit: Serialize with write instead of writeUTF. (#4195)
writeUTF has a limit of 64KB, making it difficult to write out splits
that read a large number of descriptors for small segments.
2017-04-25 10:26:44 -07:00
asrayousuf
e4fbc2bc5b Updating the description of useCache (#4200)
Updating the description of useCache

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment

Updating query-context doc based on Gian's comment
2017-04-25 10:26:15 -07:00
Gian Merlino
809112cd5f DirectDruidClient: Fix division by zero. (#4206)
* DirectDruidClient: Fix division by zero.

Introduced in #3954 when some floating math was changed to
integer math. This patch restores the old math.

* Added comment.
2017-04-25 13:03:00 +09:00
Benedict Jin
de815da942 Some code refactor for better performance of Avro-Extension (#4092)
* 1. Collections.singletonList instand of Arrays.asList; 2. close FSDataInputStream/ByteBufferInputStream for releasing resource; 3. convert com.google.common.base.Function into java.util.function.Function; 4. others code refactor

* Put each param on its own line for  code style

* Revert GenericRecordAsMap back about `Function`
2017-04-25 12:46:32 +09:00
satishbhor
d51097c809 Fix lz4 library incompatibility in kafka-indexing-service extension (#4115)
* Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Bumped Kafka version to 0.10.2.0 for : Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Replaced Lists.newArrayList() with Collections.singletonList() For Fix lz4 library incompatibility in kafka-indexing-service extension #4115
2017-04-25 12:23:51 +09:00
Jonathan Wei
723a855ab9 Fix nested groupBys with outer exfns on inner numeric columns (#4182) 2017-04-21 19:47:46 -07:00
Gian Merlino
2ca7b00346 Update versions to 0.10.1-SNAPSHOT. (#4191) 2017-04-20 18:12:28 -07:00
Gian Merlino
60caa641f3 Restore backwards compatibility of Query. (#4185) 2017-04-19 19:47:50 +03:00
Jihoon Son
5b69f2eff2 Make timeout behavior consistent to document (#4134)
* Make timeout behavior consistent to document

* Refactoring BlockingPool and add more methods to QueryContexts

* remove unused imports

* Addressed comments

* Address comments

* remove unused method

* Make default query timeout configurable

* Fix test failure

* Change timeout from period to millis
2017-04-19 09:47:53 +09:00
kaijianding
db656c5a88 fix kafka8 unparsable message halt job issue (#4164) 2017-04-18 11:23:02 -07:00
Jerry Chung
0bcfd9354c Fix S3 deep storage push and s3 insert-segment-to-db (#4174)
* Fix S3 deep storage push and s3 insert-segment-to-db

* Less verbose checks in S3DataSegmentFinder
2017-04-14 19:42:10 -07:00
Himanshu
8432138a07 log query in requestLog when client has disconnected and response write fails (#4169) 2017-04-15 04:04:52 +05:30
Himanshu
5c9198347b fix over counting of bytes in ByteCountingLRUMap (#4168) 2017-04-14 07:06:34 +09:00
Gian Merlino
b2954d5fea Better groupBy error messages and docs around resource limits. (#4162)
* Better groupBy error messages and docs around resource limits.

* Fix BufferGrouper test from datasketches.

* Further clarify.
2017-04-13 10:38:53 -07:00
Ram iyer
2e9589215e removing unused var (#4163) 2017-04-13 04:03:41 +09:00
kaijianding
676af79044 don't do postAgg in TimeseriesQueryQueryToolChest when not necessary (#4155)
* don't do postAgg in TimeseriesQueryQueryToolChest when not necessary

* set postAggs to empty list in TimeseriesQueryQueryToolChest instead of checking finalizing fn

* fix ut

* fix ut again
2017-04-12 15:46:46 +05:30
Gian Merlino
9f4266fba1 Fix SortCollapseRule when inner order is DESC. (#4157)
* Fix SortCollapseRule when inner order is DESC.

* Remove unused import.
2017-04-12 15:39:45 +05:30
Roman Leventov
15f3a94474 Copy closer into Druid codebase (fixes #3652) (#4153) 2017-04-10 09:38:45 +09:00
Parag Jain
7e0d4c9555 secure supervisor endpoints (#3985) 2017-04-05 16:42:32 -07:00
Xiuming Chen
7e4e5510e0 Outdated property names (#4146)
Outdated property names?
2017-04-05 16:37:38 -07:00
Dongkyu Hwangbo
0d2e91ed50 Adding Kafka-emitter (#3860)
* Initial commit

* Apply another config: clustername

* Rename variable

* Fix bug

* Add retry logic

* Edit retry logic

* Upgrade kafka-clients version to the most recent release

* Make callback single object

* Write documentation

* Rewrite error message and emit logic

* Handling AlertEvent

* Override toString()

* make clusterName more optional

* bump up druid version

* add producer.config option which make user can apply another optional config value of kafka producer

* remove potential blocking in emit()

* using MemoryBoundLinkedBlockingQueue

* Fixing coding convention

* Remove logging every exception and just increment counting

* refactoring

* trivial modification

* logging when callback has exception

* Replace kafka-clients 0.10.1.1 with 0.10.2.0

* Resolve the problem related of classloader

* adopt try statement

* code reformatting

* make variables final

* rewrite toString
2017-04-04 14:07:43 -07:00
Jihoon Son
b166e13d2b Make DruidCoordinatorTest fail fast (#4142) 2017-04-04 13:58:32 -07:00
Roman Leventov
81ba0224c5 Report unsuccessful queries in AsyncQueryForwardingServlet (#4143) 2017-04-04 13:53:07 -07:00
Nishant Bangarwa
801ea5efa4 Fix: Broker fails throws OOME with conf-quickstart (#4127)
when running the the packaged conf-quickstart druid broker fails to
start and throws OOME. increasing the direct memory to get around this.
2017-03-29 11:43:58 -07:00
Gian Merlino
bbb61e638b SQL: Support for another form of filtered aggregator. (#4109)
* SQL: Support for another form of filtered aggregator.

* Fix comment, add test for MAX too.
2017-03-27 15:22:36 -07:00
Roman Leventov
73d9b31664 GenericIndexed minor bug fixes, optimizations and refactoring (#3951)
* Minor bug fixes in GenericIndexed; Refactor and optimize GenericIndexed; Remove some unnecessary ByteBuffer duplications in some deserialization paths; Add ZeroCopyByteArrayOutputStream

* Fixes

* Move GenericIndexedWriter.writeLongValueToOutputStream() and writeIntValueToOutputStream() to SerializerUtils

* Move constructors

* Add GenericIndexedBenchmark

* Comments

* Typo

* Note in Javadoc that IntermediateLongSupplierSerializer, LongColumnSerializer and LongMetricColumnSerializer are thread-unsafe

* Use primitive collections in IntermediateLongSupplierSerializer instead of BiMap

* Optimize TableLongEncodingWriter

* Add checks to SerializerUtils methods

* Don't restrict byte order in SerializerUtils.writeLongToOutputStream() and writeIntToOutputStream()

* Update GenericIndexedBenchmark

* SerializerUtils.writeIntToOutputStream() and writeLongToOutputStream() separate for big-endian and native-endian

* Add GenericIndexedBenchmark.indexOf()

* More checks in methods in SerializerUtils

* Use helperBuffer.arrayOffset()

* Optimizations in SerializerUtils
2017-03-27 14:17:31 -05:00
Jason Banich
117c698c59 Update StatsDEmitter.java (#4111)
This was mentioned in the original pull (https://github.com/druid-io/druid/pull/2410) by @sascha-coenen, and the original author (@michaelschiff) agreed that it seemed reasonable

This commit fixes issue https://github.com/druid-io/druid/issues/3960
2017-03-27 10:32:21 -07:00
JackyWoo
a0f2cf05d5 Add EqualDistributionWithAffinityWorkerSelectStrategy which balance w… (#3998)
* Add EqualDistributionWithAffinityWorkerSelectStrategy which balance work load within affinity workers.

* add docs to equalDistributionWithAffinity
2017-03-25 19:15:49 -07:00
Gian Merlino
90f9932bd3 SQL: Rule to collapse sort chains. (#4085)
Useful for queries like `SELECT * FROM (...) LIMIT X`, where the inner query
has an order by or limit in it.
2017-03-24 19:20:01 -07:00
Gian Merlino
76c4b6446e SQL: Fix handling of CURRENT_TIMESTAMP and friends in non-UTC timezones. (#4114) 2017-03-24 18:45:23 -07:00
Gian Merlino
dd6c0ab509 Add SQL REGEXP_EXTRACT function; add "index" to "regex" extractionFn. (#4055)
* Add SQL REGEXP_EXTRACT function; add "index" to "regex" extractionFn.

* Fix tests.
2017-03-24 17:38:36 -07:00
Himanshu
de081c711b RealtimeIndexTask to support alertTimeout in context (#4089)
* RealtimeIndexTask to support alertTimeout in context and raise alert if task process exists after the timeout

* move alertTimeout config to tuningConfig and document
2017-03-24 12:48:12 -07:00
Gian Merlino
b4289c0004 Remove "granularity" from IngestSegmentFirehose. (#4110)
It wasn't doing anything useful (the sequences were being concatted, and
cursor.getTime() wasn't being called) and it defaulted to Granularities.NONE.
Changing it to Granularities.ALL gave me a 700x+ performance boost on a
small dataset I was reindexing (2m27s to 365ms). Most of that was from avoiding
making a lot of unnecessary column selectors.
2017-03-24 10:28:54 -07:00
Benedict Jin
23f77ebd20 Explain Avro's unnecessary EOFException (#4098) (#4100)
* Explain Avro's unnecessary EOFException (#4098)

* add jira link into log message
2017-03-24 10:45:45 -05:00
Erik Dubbelboer
2cbc4764f8 Comparing dimensions to each other in a filter (#3928)
Comparing dimensions to each other using a select filter
2017-03-23 18:23:46 -07:00
Roman Leventov
4b5ae31207 QueryMetrics: abstraction layer of query metrics emitting (part of #3798) (#3954)
* QueryMetrics: abstraction layer of query metrics emitting

* Minor fixes

* QueryMetrics.emit() for bulk emit and improve Javadoc

* Fixes

* Fix

* Javadoc fixes

* Typo

* Use DefaultObjectMapper

* Add tests

* Address PR comments

* Remove QueryMetrics.userDimensions(); Rename QueryMetric.register() to report()

* Dedicated TopNQueryMetricsFactory, GroupByQueryMetricsFactory and TimeseriesQueryMetricsFactory

* Typo

* More elaborate Javadoc of QueryMetrics

* Formatting

* Replace QueryMetric enum with lambdas

* Add comments and VisibleForTesting annotations
2017-03-23 17:23:59 -07:00
Himanshu
c9fc7d1709 fix failure message to mention version.bin instead of index.drd not exists msg (#4102) 2017-03-23 14:21:19 -07:00
Gian Merlino
4b9f975f50 Rename SketchAggregationWithSimpleDataTest. (#4105)
Tests that don't end in "Test" won't get run automatically by Maven.
2017-03-23 14:20:50 -07:00
Jonathan Wei
79f1a1d7f0 Allow float parameters for Bound/Selector/In filters on long columns (#4074)
* Allow float parameters for long filters

* Use BigDecimal intermediate form for string->long conversions

* PR comments

* PR comments
2017-03-23 14:18:05 -07:00
Gian Merlino
81d6b49d69 Downgrade Curator. (#4103)
Reverts #4060, fixes #4095, unfixes #4056, #3837. Better the devil you
know than the devil you don't, I always say.

See also https://issues.apache.org/jira/browse/CURATOR-394.
2017-03-23 13:44:00 -07:00
Akash Dwivedi
ff7f90b02d relocate method in BufferAggregator. (#4071)
*  relocate method in BufferAggregator.

* Unused import.

* Detailed javadoc.

* using Int2ObjectMap.

* batch relocate.

* Revert batch relocate.

* Unused import.

* code comments.

* code comment.
2017-03-23 13:07:59 -07:00