c2ae734848
KafkaIndexTask: Allow run thread to stop gracefully instead of interrupting ( #3534 )
...
* allow run thread to gracefully complete instead of interrupting when stopGracefully() is called
* add comments
2016-10-17 10:52:19 -04:00
c1d3b8a30c
Remove dropwizard-jdbc dependency from lookups-cached-single. ( #3573 )
...
Fixes #3548 .
2016-10-17 10:37:47 -04:00
0ce33bc95f
HdfsDataSegmentPusher: Properly include scheme, host in output path if necessary. ( #3577 )
...
Fixes #3576 .
2016-10-17 10:37:18 -04:00
472c409b99
KafkaLookupExtractorFactory: shutdown kafka consumer on close() ( #3539 )
...
* shutdown kafka consumer on close
* handle close() race condition
2016-10-15 09:55:51 -07:00
5dc95389f7
Add Checkstyle framework ( #3551 )
...
* Add Checkstyle framework
* Avoid star import
* Need braces for control flow statements
* Redundant imports
* Add NewLineAtEndOfFile check
2016-10-13 13:37:47 -07:00
6f21778364
Support finding segments in AWS S3. ( #3399 )
...
* support finding segments from a AWS S3 storage.
* add more Uts
* address comments and add a document for the feature.
* update docs indentation
* update docs indentation
* address comments.
1. add a Ut for json ser/deser for the config object.
2. more informant error message in a Ut.
* address comments.
1. use @Min to validate the configuration object
2. change updateDescriptor to a string as it does not take an argument otherwise
* fix a Ut failure - delete a Ut for testing default max length.
2016-10-10 17:27:09 -07:00
c255dd8b19
fix datasegment metadata ( #3555 )
2016-10-07 16:30:33 -05:00
76a60a007e
create parent dir on HDFS if it does not exist ( #3547 )
2016-10-06 16:14:00 -07:00
1523de08fb
SketchAggregatorFactory.combine(..) returns Union object now so that it can be reused across multiple combine(..) calls ( #3471 )
2016-10-05 08:40:14 -07:00
592903571a
add context to kafka supervisor for the kafka indexing task ( #3464 )
2016-10-04 20:08:43 -05:00
e419407eba
handle supervisor spec metadata failures ( #3456 )
...
close kafka consumer in case supervisor start fails
2016-10-04 10:15:28 -07:00
40f2fe7893
Bump versions to 0.9.3-SNAPSHOT ( #3524 )
2016-09-29 13:53:32 -07:00
15c9918c65
log exceptions while trying to pause task ( #3504 )
2016-09-23 16:53:23 -07:00
9226d4af3c
configurable shutdownTimeout for Kakfa supervisor ( #3497 )
...
* configurable shutdownTimeout
* cr change
2016-09-23 13:26:45 -06:00
ca9114b41b
add supervisor reset API ( #3484 )
...
* add supervisor reset API
* CR doc changes and kill running tasks / clear offsets from supervisor
2016-09-22 17:51:06 -07:00
6099d20303
[FIX] ReleaseException when the path is being written by multiple tasks ( #3494 )
...
* fix ReleaseException when the path is being written by multiple task
* Do not throw IOException if another replica wins the race for segment creation
fix if check
* handle logging comments
* fix test
2016-09-22 14:25:41 -05:00
74e1243c7e
Fix test fail of PollingLookupTest.testApplyAfterDataChange ( #3489 )
2016-09-22 08:33:59 -07:00
05ea88df5c
fix kafka-indexing-service pom to not reference specific version but parent version for druid core dependencies ( #3472 )
2016-09-20 15:18:21 -07:00
96fcca18ea
update KafkaSupervisor to make HTTP requests to tasks in parallel where possible ( #3452 )
2016-09-20 22:51:15 +05:30
3175e17a3b
Cached lookup module. first cut implementing JDBC cache ( #2819 )
2016-09-16 13:45:54 -07:00
95e08b38ea
[QTL] Reduced Locking Lookups ( #3071 )
...
* Lockless lookups
* Fix compile problem
* Make stack trace throw instead
* Remove non-germane change
* * Add better naming to cache keys. Makes logging nicer
* Fix #3459
* Move start/stop lock to non-interruptable for readability purposes
2016-09-16 11:54:23 -07:00
d981a2aa02
Avoid interrupting ZookeeperConsumerConnector.shutdown() #3346 ( #3403 )
2016-09-14 17:44:27 -07:00
a069257d37
avro-extension -- feature to specify multiple avro reader schemas inline ( #3368 )
...
* rename SimpleAvroBytesDecoder to InlineSchemaAvroBytesDecoder
* feature to specify multiple schemas inline in avro module
2016-09-13 14:54:31 -07:00
bcff08826b
KafkaIndexTask: Treat null values as unparseable. ( #3453 )
2016-09-13 10:56:38 -07:00
ba6ddf307e
Adding hadoop kerberos authentification. ( #3419 )
...
* adding kerberos authentication
* make the 2 functions identical
2016-09-13 10:42:50 -07:00
df766b2bbd
Add dimension handling interface for ingestion and segment creation ( #3217 )
...
* Add dimension handling interface for ingestion and segment creation
* update javadocs for DimensionHandler/DimensionIndexer
* Move IndexIO row validation into DimensionHandler
* Fix null column skipping in mergerV9
* Add deprecation note for 'numeric_dims' filename pattern in IndexIO v8->v9 conversion
* Fix java7 test failure
2016-09-12 12:54:02 -07:00
1a5042ca26
updated dependency on sketches-core ( #3443 )
...
* updated dependency on sketches-core to 0.7.0
* Use sketches-core-0.4.1, which is the latest version still compatible
with JDK7
2016-09-09 16:21:32 -07:00
146a17de48
KafkaIndexTask: allow pause to break out of retry loop ( #3401 )
2016-09-06 22:29:37 -06:00
5b1ae21bd1
retry calls to getStartTime ( #3429 )
2016-09-06 14:02:22 -07:00
48dce88aab
Add flag binaryAsString for parquet ingestion ( #3381 )
2016-08-30 17:30:50 -07:00
ed924bf214
allow registrants to opt out of announcing themselves when registering as a chat handler ( #3360 )
2016-08-16 10:51:28 +05:30
70d99fe3c6
Initialize ApproximateHistogram Module in ApproximateHistogramGroupByQueryTest ( #3363 )
...
or else the test fails if ran independently.
2016-08-15 10:19:33 -07:00
46da682231
avro-extensions -- feature to specify avro reader schema inline in the task json for all events ( #3249 )
2016-08-10 10:49:26 -07:00
890e3bdd3f
More informative query unit test names ( #3342 )
2016-08-09 22:24:48 -07:00
decefb7477
Add time interval dim filter and retention analysis example ( #3315 )
...
* Add time interval dim filter and retention analysis example
* Use closed-open matching for intervals, update cache key generation
* Fix time filtering tests for interval boundary change
2016-08-05 07:25:04 -07:00
5b3f0ccb1f
Support variance and standard deviation ( #2525 )
...
* Support variance and standard deviation
* addressed comments
2016-08-04 17:32:58 -07:00
33dbe0800c
Makes kafka lookup extraction factory's replace() behavior consistent with other lookup extraction factories ( #3326 )
2016-08-04 10:24:19 -07:00
8030f1cb67
Be more respectful of maxRowsInMemory. ( #3284 )
...
- Appenderator: Respect maxRowsInMemory across all sinks.
- KafkaIndexTask: Respect maxRowsInMemory across all partitions.
2016-07-26 15:02:35 -06:00
3f1681c16c
Caffeine cache extension ( #3028 )
...
* Initial commit of caffeine cache
* Address code comments
* Move and fixup README.md a bit
* Improve caffeine readme information
* Cleanup caffeine pom
* Address review comments
* Bump caffeine to 2.3.1
* Bump druid version to 0.9.2-SNAPSHOT
* Make test not fail randomly.
See https://github.com/ben-manes/caffeine/pull/93#issuecomment-227617998 for an explanation
* Fix distribution and documentation
* Add caffeine to extensions.md
* Fix links in extensions.md
* Lexicographic
2016-07-06 15:42:54 -07:00
bfa5c05aaa
Make global lookup cache introspector class public ( #3199 )
...
* Make global lookup cache introspector class public
* Fixes #3187
* Make KafkaLookupExtractorIntrospectionHandler a public static class
2016-07-01 15:50:57 -07:00
485e381387
remove datasource from hadoop output path ( #3196 )
...
fixes #2083 , follow-up to #1702
2016-06-29 08:53:45 -07:00
1d40df4bb7
fix kafka consumer concurrent access during shutdown ( #3193 )
2016-06-28 13:23:17 -07:00
45f553fc28
Replace the deprecated usage of NoneShardSpec ( #3166 )
2016-06-25 10:27:25 -07:00
4cc39b2ee7
Alternative groupBy strategy. ( #2998 )
...
This patch introduces a GroupByStrategy concept and two strategies: "v1"
is the current groupBy strategy and "v2" is a new one. It also introduces
a merge buffers concept in DruidProcessingModule, to try to better
manage memory used for merging.
Both of these are described in more detail in #2987 .
There are two goals of this patch:
1. Make it possible for historical/realtime nodes to return larger groupBy
result sets, faster, with better memory management.
2. Make it possible for brokers to merge streams when there are no order-by
columns, avoiding materialization.
This patch does not do anything to help with memory management on the broker
when there are order-by columns or when there are nested queries. That could
potentially be done in a future patch.
2016-06-24 18:06:09 -07:00
ebd654228b
fix: avro types exception in sketch ( #3167 )
2016-06-22 15:54:20 -05:00
674f94083e
Add more logging around failed S3DataSegmentMover DeleteExceptions ( #3104 )
...
* Add more logging around failed S3DataSegmentMover DeleteExceptions
* Fix test NPE
2016-06-16 14:58:33 -07:00
f7fa1d8c62
[QTL] Allow S3 version finder to search entire s3 object key ( #3139 )
...
* Allow S3 version finder to search entire s3 object key
* Previously only was able to search immediate "directory"
* Update method javadoc
* Expand docs a bit better
2016-06-13 21:02:28 -07:00
ebf890fe79
Update master version to 0.9.2-SNAPSHOT. ( #3133 )
2016-06-13 13:10:38 -07:00
4faa298977
update kafka client for kafka indexing service to 0.9.0.1 ( #3109 )
2016-06-08 06:51:03 -07:00
8cac710546
Async lookups-cached-global by default ( #3074 )
...
* Async lookups-cached-global by default
* Also better lookup docs
* Fix test timeouts
* Fix timing of deserialized test
* Fix problem with 0 wait failing immediately
2016-06-03 15:58:10 -05:00
David Lim
Gian Merlino
Roman Leventov
jaehong choi
Parag Jain
Himanshu
Nishant
Navis Ryu
Slim
Charles Allen
Gleb Smirnov
Jonathan Wei
Alexander Saydakov
Stéphane Derosiaux
Xavier Léauté
Hyukjin Kwon
du00cs