Commit Graph

7310 Commits

Author SHA1 Message Date
Charles Allen 5d9fd0a713 Migrate IndexerSQLMetadataStorageCoordinator.getUnusedSegmentsForInterval to streaming (#3043)
* Migrate IndexerSQLMetadataStorageCoordinator.getUnusedSegmentsForInterval to streaming
* Missed query from #2859

* Make inReadOnlyTransaction part of SQLMetadataConnector
2016-07-06 16:55:27 -07:00
Charles Allen 3f1681c16c Caffeine cache extension (#3028)
* Initial commit of caffeine cache

* Address code comments

* Move and fixup README.md a bit

* Improve caffeine readme information

* Cleanup caffeine pom

* Address review comments

* Bump caffeine to 2.3.1

* Bump druid version to 0.9.2-SNAPSHOT

* Make test not fail randomly.

See https://github.com/ben-manes/caffeine/pull/93#issuecomment-227617998 for an explanation

* Fix distribution and documentation

* Add caffeine to extensions.md

* Fix links in extensions.md

* Lexicographic
2016-07-06 15:42:54 -07:00
Gian Merlino b8a4f4ea7b DumpSegment: Add --dump bitmaps option. (#3221)
Also make --dump metadata respect --column.
2016-07-06 12:42:50 -07:00
Gian Merlino fdc7e88a7d Allow queries with no aggregators. (#3216)
This is actually reasonable for a groupBy or lexicographic topNs that is
being used to do a "COUNT DISTINCT" kind of query. No aggregators are
needed for that query, and including a dummy aggregator wastes 8 bytes
per row.

It's kind of silly for timeseries, but why not.
2016-07-06 20:38:54 +05:30
Charles Allen bfa5c05aaa Make global lookup cache introspector class public (#3199)
* Make global lookup cache introspector class public
* Fixes #3187

* Make KafkaLookupExtractorIntrospectionHandler a public static class
2016-07-01 15:50:57 -07:00
Himanshu e1313e4b90 add log msg when event recvr firehose buffer is full (#3209) 2016-07-01 17:35:30 -05:00
Fangjin Yang 8eeae2e844 remove bad docs on setting up clusters (#3188) 2016-07-01 15:41:40 -05:00
Parag Jain 99844dfeb5 remove need for tmp extensions dir (#3211)
correct lib path relative to base distribution dir
2016-07-01 12:55:57 -07:00
Bingkun Guo d2636d1a64 [pull-deps] If --clean flag is not set, skip creating root extension directories if they already exist. (#3130) 2016-07-01 11:18:57 -05:00
Charles Allen 8b7d9750ee Update extension docs for global lookup module (#3206) 2016-06-29 12:51:52 -07:00
Xavier Léauté 485e381387 remove datasource from hadoop output path (#3196)
fixes #2083, follow-up to #1702
2016-06-29 08:53:45 -07:00
Gian Merlino 4c9aeb7353 Revert "update druid console version (#3189)" (#3203)
This reverts commit 496b801bc3.
2016-06-29 08:29:57 -07:00
Jonathan Wei f3a3662133 Fix compile error in SearchBinaryFnTest (#3201) 2016-06-29 09:44:45 -05:00
David Lim b24425a280 update docs with new behavior (#3200) 2016-06-28 16:17:04 -07:00
jaehong choi efbcbf5315 Support alphanumeric sort in search query (#2593)
* support alphanumeric sort in search query

* address a comment about handling equals() and hashCode()

* address comments

* add Ut for string comparators

* address a comment about space indentations.
2016-06-28 15:06:18 -07:00
David Lim 1d40df4bb7 fix kafka consumer concurrent access during shutdown (#3193) 2016-06-28 13:23:17 -07:00
Xavier Léauté 496b801bc3 update druid console version (#3189) 2016-06-27 18:02:40 -07:00
du00cs bf53490d70 fix: no split file will throw IndexOutOfBounds Exception (#3179) 2016-06-26 12:50:18 -07:00
Hyukjin Kwon 45f553fc28 Replace the deprecated usage of NoneShardSpec (#3166) 2016-06-25 10:27:25 -07:00
Gian Merlino 4cc39b2ee7 Alternative groupBy strategy. (#2998)
This patch introduces a GroupByStrategy concept and two strategies: "v1"
is the current groupBy strategy and "v2" is a new one. It also introduces
a merge buffers concept in DruidProcessingModule, to try to better
manage memory used for merging.

Both of these are described in more detail in #2987.

There are two goals of this patch:

1. Make it possible for historical/realtime nodes to return larger groupBy
   result sets, faster, with better memory management.
2. Make it possible for brokers to merge streams when there are no order-by
   columns, avoiding materialization.

This patch does not do anything to help with memory management on the broker
when there are order-by columns or when there are nested queries. That could
potentially be done in a future patch.
2016-06-24 18:06:09 -07:00
Nishant 0aa7d71ca5 Add doc link to eclipse formatting settings as well (#3131) 2016-06-24 15:27:50 -07:00
Nishant 94b3c74cdc Druid launch script improvements (#3175)
* Add status command to launch scripts

* make druid init script to pick up config directories from environment variables

make druid init script to pick up config directories from environment
variables
2016-06-24 15:02:34 -07:00
Dave Li 8a08398977 Add segment pruning based on secondary partition dimension (#2982)
* add get dimension rangeset to filters

* add get domain to ShardSpec and added chunk filter in caching clustered client

* add null check and modified not filter, started with unit test

* add filter test with caching

* refactor and some comments

* extract filtershard to helper function

* fixup

* minor changes

* update javadoc
2016-06-24 14:52:19 -07:00
Charles Allen 6be18376c0 Make forking task runner have more informative thread names during the long-blocking part (#3172)
* Make forking task runner have more informative thread names during the long-blocking part

* Make string.format do the work
2016-06-24 08:56:01 -07:00
Charles Allen 15f833a861 Make extension classloader caching keyed on directory (#3165)
* Make extension classloaders keyed by extension directory
* Fixes #3163

* Add in same-directory-name unit test
2016-06-23 17:13:19 -07:00
michaelschiff 66d8ad36d7 adds new coordinator metrics 'segment/unavailable/count' and (#3176)
'segment/underReplicated/count' (#3173)
2016-06-23 14:53:15 -07:00
Gian Merlino da660bb592 DumpSegment tool. (#3182)
Fixes #2723.
2016-06-23 14:37:50 -07:00
Gian Merlino a437fb150b Fix SegmentMetadataQuery when queryGranularity is requested but not present. (#3181) 2016-06-23 14:30:50 -07:00
Nishant 2696b0c451 Retry for transient exceptions while doing cleanup for Hadoop Jobs (#3177)
* fix 1828

fixes https://github.com/druid-io/druid/issues/1828

* remove unused import

* Review comment
2016-06-23 13:38:47 -07:00
Nishant 6f330dc816 Better handling for parseExceptions for Batch Ingestion (#3171)
* Better handling for parseExceptions

* make parseException handling consistent with Realtime

* change combiner default val to true

* review comments

* review comments
2016-06-22 16:38:29 -07:00
Jonathan Wei 24860a1391 Two-stage filtering (#3018)
* Two-stage filtering

* PR comment
2016-06-22 16:08:21 -07:00
du00cs ebd654228b fix: avro types exception in sketch (#3167) 2016-06-22 15:54:20 -05:00
Nishant f46ad9a4cb support Union Segment metadata queries (#3132)
* support Union Segment metadata queries

fix 3128

* remove extraneous sys out
2016-06-21 10:30:50 -07:00
Dave Li 12be1c0a4b Add bucket extraction function (#3033)
* add bucket extraction function

* add doc and header

* updated doc and test
2016-06-17 09:24:27 -07:00
linbo.jin 8c76fe7b97 docs: change OR to AND inside query docs about multi-value dims (#3162)
* docs: replace OR by AND inside topnquery docs about multi value dimensions

* docs: replace OR by AND inside groupby docs about multi value dimensions
2016-06-17 08:54:18 -07:00
Charles Allen 8b21036833 Make integration test dependency copy run in batch mode (#3152) 2016-06-16 18:44:15 -07:00
Fangjin Yang 07288c8fc0 update compares some more (#3158)
* update compares some more

* fix

* fix

* fix
2016-06-16 18:34:43 -07:00
Gian Merlino c12712e8b8 Move "libraries.md" out of docs, onto the main site. (#3159) 2016-06-16 18:14:35 -07:00
Charles Allen 674f94083e Add more logging around failed S3DataSegmentMover DeleteExceptions (#3104)
* Add more logging around failed S3DataSegmentMover DeleteExceptions

* Fix test NPE
2016-06-16 14:58:33 -07:00
Fangjin Yang 6c2fd75e46 update vs spark doc (#3116)
* update vs spark doc

* update based on comments
2016-06-15 10:30:19 -07:00
Gian Merlino 7da4a283a9 Add missing layout: toc to TOC. (#3144) 2016-06-14 10:48:05 -07:00
Charles Allen f7fa1d8c62 [QTL] Allow S3 version finder to search entire s3 object key (#3139)
* Allow S3 version finder to search entire s3 object key
* Previously only was able to search immediate "directory"

* Update method javadoc

* Expand docs a bit better
2016-06-13 21:02:28 -07:00
Charles Allen e7ac2e8ba2 Set a non-zero exit code when error incommand parsing (#3137) 2016-06-13 20:38:37 -07:00
Gian Merlino dc2bf9efa5 Update absolute TOC links. (#3138)
See druid-io/druid-io.github.io#286.
2016-06-13 17:57:52 -07:00
Gian Merlino ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
Charles Allen aa2982ee31 Update bytebuffer-collections to 0.2.5 (#3117) 2016-06-13 08:41:20 -07:00
Gian Merlino 3b3e772748 Add --no-default-remote-repositories flag to pull-deps. (#3120) 2016-06-13 17:01:18 +05:30
Fangjin Yang 53886a677c include avro in the druid tarball (#3123) 2016-06-13 16:58:21 +05:30
Charles Allen cf90508793 Disable cobertura travis portion (#3122) 2016-06-13 12:27:35 +05:30
michaelschiff 7294ea87c3 link to statsd metrics emitter docs from development/extensions.html doc page (#3125) 2016-06-10 16:27:16 -07:00