fjy
47f5c1bd0a
fix retry interval is stupid
2014-11-20 12:50:56 -08:00
fjy
3d9d989a9f
A set of fixes to retry the query for missing intervals in the timeline
2014-11-20 12:04:37 -08:00
nishantmonu51
0ab34f86da
Revert "fix filtered Aggregator"
...
This reverts commit 6fd37ce023
.
2014-11-20 10:17:01 +05:30
nishantmonu51
6fd37ce023
fix filtered Aggregator
...
fix filtered Aggregator
remove unused name parameter for filtered aggregator
add tests
2014-11-20 09:29:26 +05:30
fjy
a49e673122
put back another missing test
2014-11-19 16:55:20 -08:00
fjy
14668846aa
add back some tests
2014-11-19 14:35:26 -08:00
fjy
fdeab0c6af
make Druid case sensitive
2014-11-19 14:27:31 -08:00
Fangjin Yang
590d31799e
Merge pull request #876 from metamx/remove-backwards-compatible
...
Remove backwards compatible
2014-11-19 14:33:14 -07:00
Charles Allen
18f44beee9
CompressedObjectStrategy improvements
...
* Added more unit tests
* Now properly uses safe / fast decompressor for LZ4
* Now chooses fastest lz4 instance instead of only looking at Java implmentations
* Encapsulate ResourceHolder in try-with-resources to make sure they close correctly
2014-11-19 11:10:59 -08:00
Charles Allen
ccc757dc64
Merge remote-tracking branch 'origin/master' into druid-0.7.x-compressionstrategy
2014-11-19 09:39:35 -08:00
Charles Allen
1bbc8fcbe5
Allow Smile to fall back to text
...
* Modify SmileFactory to set the delegate to text option.
* This option only occurs when a Reader type object is passed in to the deserialization stuff
* This is needed by the X-Druid-Response-Context header return value, which is JSON
2014-11-18 15:16:14 -08:00
Charles Allen
42517f5d37
Merge pull request #884 from metamx/optimize-topN-pruning
...
optimise pruning of aggs
2014-11-18 14:19:30 -08:00
nishantmonu51
6023d602e6
optimise pruning of aggs
...
optimise pruning of aggregators for topN
2014-11-19 00:17:25 +05:30
Xavier Léauté
d914afe1cd
make defaultVersion configurable for non-jar testing
2014-11-17 13:54:32 -08:00
nishantmonu51
0c2d06475d
merge from master
2014-11-17 19:19:18 +05:30
nishantmonu51
cbffe3c648
merge from master and resolve conflicts
2014-11-17 18:07:08 +05:30
fjy
d5c4282766
fix broken ut
2014-11-14 13:45:42 -08:00
xvrl
e1e171ad20
Merge pull request #865 from metamx/fix-retry-qr
...
Fix a couple of bugs with retry query runner after testing it locally
2014-11-14 13:33:51 -08:00
fjy
df1ad95075
remove useless config
2014-11-14 13:32:19 -08:00
fjy
d641d41f9e
address another cr
2014-11-14 13:29:59 -08:00
fjy
7736c3fc27
address cr
2014-11-14 13:28:32 -08:00
Fangjin Yang
6ee8029462
Merge pull request #866 from metamx/mutableBitmapBenchmark
...
Add benchmarking for bitmaps
2014-11-14 14:16:21 -07:00
xvrl
a4fc64ca3f
Merge pull request #856 from metamx/druid-845
...
Fix query by segment
2014-11-14 13:10:54 -08:00
Charles Allen
4b7ab23289
Remove getIntervalString from BySegmentResultValue
2014-11-14 13:03:48 -08:00
fjy
bbc079b880
fix retry to actually return correct sequences
2014-11-14 12:10:04 -08:00
Charles Allen
648759e9f6
Add deserialization benchmark to BitmapCreationBenchmark
2014-11-13 13:43:14 -08:00
Charles Allen
483b2c7be0
Add copyright notice to BitmapCreationBenchmark
2014-11-13 12:55:02 -08:00
Charles Allen
228fb0cf40
Add benchmarking for bitmaps
...
Here are the results on my laptop:
BitmapCreationBenchmark.testRandomAddition[0]: [measured 10 out of 20 rounds, threads: 1 (sequential)]
round: 0.49 [+- 0.07], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 16, GC.time: 0.01, time.total: 9.91, time.warmup: 5.06, time.bench: 4.86
BitmapCreationBenchmark.testLinearAdditionDescending[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.01 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 74, GC.time: 0.03, time.total: 5.82, time.warmup: 0.06, time.bench: 5.76
BitmapCreationBenchmark.testToImmutableByteArray[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 1.80, time.warmup: 0.02, time.bench: 1.78
BitmapCreationBenchmark.testRandomAddition[1]: [measured 10 out of 20 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 0.12, time.warmup: 0.08, time.bench: 0.04
BitmapCreationBenchmark.testLinearAdditionDescending[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 10, GC.time: 0.01, time.total: 4.26, time.warmup: 0.04, time.bench: 4.22
BitmapCreationBenchmark.testToImmutableByteArray[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.01 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 27, GC.time: 0.01, time.total: 5.11, time.warmup: 0.05, time.bench: 5.06
BitmapCreationBenchmark.testLinearAddition[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 3.48, time.warmup: 0.04, time.bench: 3.45
BitmapCreationBenchmark.testLinearAddition[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 9, GC.time: 0.00, time.total: 2.95, time.warmup: 0.03, time.bench: 2.92
2014-11-13 12:47:23,995 INFO [main] io.druid.segment.data.BitmapCreationBenchmark - Entry [0] is io.druid.segment.data.ConciseBitmapSerdeFactory
2014-11-13 12:47:23,995 INFO [main] io.druid.segment.data.BitmapCreationBenchmark - Entry [1] is io.druid.segment.data.RoaringBitmapSerdeFactory
2014-11-13 12:48:04 -08:00
fjy
6cdd6a6af7
change default settings for retry
2014-11-13 12:43:30 -08:00
fjy
2b0ed30344
add test
2014-11-13 12:38:35 -08:00
fjy
0d6816a037
Fix a couple of bugs with retry query runner after testing it locally
2014-11-13 11:53:29 -08:00
Fangjin Yang
4a3c0fc5c4
Merge pull request #848 from metamx/druid-0.7.x-fastTopN-rebase
...
TopN performance improvements
2014-11-13 11:56:20 -07:00
Charles Allen
34c3464bc9
Added more explanatory comments in PooledTopNAlgorithm
2014-11-13 10:47:56 -08:00
Charles Allen
9a9238a801
Merge remote-tracking branch 'origin/master' into druid-845
2014-11-13 10:04:56 -08:00
Xavier Léauté
19a37f773f
test for groupBy order
2014-11-12 22:48:16 -08:00
Charles Allen
31fed7d329
Fix query by segment
...
* Changed topN queries to use joda Interval instead of string values
* topN by segment now implements BySegmentResultValue<Result<TopNResultValue>> instead of BySegmentResultValue<TopNResultValue>
* Added a unit test which failed uner the prior implementation.
2014-11-12 21:20:59 -08:00
Xavier Léauté
4ac1aaf90e
Merge remote-tracking branch 'origin/master' into druid-0.7.x
...
Conflicts:
processing/src/main/java/io/druid/segment/QueryableIndexStorageAdapter.java
2014-11-12 14:08:51 -08:00
Fangjin Yang
4b4f1c7d38
Merge pull request #846 from vikramakrishnan/cacheKeyFix
...
Include origin when creating the cacheKey for period Grans
2014-11-12 11:15:21 -07:00
Fangjin Yang
7ef19009db
Merge pull request #834 from metamx/optimize-timestampchecking
...
skip timestamp checking if not required, remove duplicate code
2014-11-12 09:45:34 -07:00
Vikram Ramakrishnan
8151d14786
Include origin when creating the cacheKey for period Grans
2014-11-12 21:36:46 +05:30
Charles Allen
581e6830d5
Merge pull request #840 from metamx/powers-of-2-buffers
...
make buffer size a power of 2 and optimize buffer lookup
2014-11-11 19:54:00 -08:00
Xavier Léauté
60e98c35d7
workaround for annotations requiring class literals
2014-11-11 18:48:49 -08:00
Xavier Léauté
b580269f6d
Distinguish between default and legacy bitmaps
2014-11-11 18:35:45 -08:00
fjy
bc5c56e441
fix default impl
2014-11-11 18:00:46 -08:00
fjy
63ca2375a3
remove dead code and cleanup some defaults
2014-11-11 17:57:24 -08:00
fjy
5629307548
address a few more comments on roaring
2014-11-11 17:50:03 -08:00
Charles Allen
a89b539b4f
Merge pull request #823 from metamx/roaring
...
Configurable bitmap indexes: roaring and concise
2014-11-11 17:26:38 -08:00
Xavier Léauté
007e57f876
make buffer size a power of 2 and optimize
2014-11-11 16:24:37 -08:00
fjy
1cc162727b
address code review
2014-11-11 14:05:37 -08:00
Xavier Léauté
3f5449d40a
loop unrolling provides no benefit for timeseries
2014-11-11 10:58:37 -08:00
Xavier Léauté
e817db8b6c
unroll timeseries aggregations + naming
2014-11-11 10:09:37 -08:00
fjy
e6b7b03b5b
fix conversion
2014-11-10 17:13:28 -08:00
fjy
eb457c280e
revert change
2014-11-10 17:00:53 -08:00
fjy
336c73bdc2
cleanup dead code
2014-11-10 16:53:13 -08:00
fjy
df886fac1b
fix
2014-11-10 16:49:27 -08:00
fjy
d68bc3bdea
cleanup unused imports
2014-11-10 16:15:28 -08:00
Charles Allen
92e71be864
Change TopNQueryRunnerBenchmark to use a ByteBuffer as per OffheapBufferPool
2014-11-10 15:40:54 -08:00
Charles Allen
a093f3728a
Reformat on TopNNumericResultBuilder
2014-11-10 15:26:09 -08:00
Charles Allen
2b0f4534bf
Modify formatting in TopNQueryRunnerBenchmark
2014-11-10 15:17:26 -08:00
Charles Allen
fc78f215c4
Aggressive dimValue unrolling in PooledTopNAlgorithm
2014-11-10 15:14:45 -08:00
fjy
6188315293
Merge branch 'master' into druid-0.7.x
...
Conflicts:
processing/src/test/java/io/druid/query/search/SearchQueryRunnerTest.java
2014-11-10 14:52:10 -08:00
Xavier Léauté
49e878cf1a
unroll multi-value dimensions
2014-11-10 14:21:56 -08:00
fjy
df9be030db
remove more legacy code
2014-11-10 14:09:00 -08:00
Charles Allen
dbb6401fbe
Aggressive unrolling in PooledTopNAlgorithm
2014-11-10 14:04:37 -08:00
fjy
72c355e8ae
remove crazy handler code
2014-11-10 13:58:23 -08:00
fjy
fc34858e95
fix things for real-time ingestion
2014-11-10 13:48:21 -08:00
fjy
c5cc826998
make things actually work with roaring
2014-11-10 13:42:26 -08:00
fjy
358b2add17
make things actually work with roaring
2014-11-10 13:42:06 -08:00
Charles Allen
9ac8589143
Merge remote-tracking branch 'origin/druid-0.7.x-fastTopN-rebase' into druid-0.7.x-fastTopN-rebase
2014-11-10 12:55:08 -08:00
Xavier Léauté
70468400bf
stupid mistake
2014-11-10 12:54:48 -08:00
Charles Allen
bfc9d9f283
Merge remote-tracking branch 'origin/druid-0.7.x-fastTopN-rebase' into druid-0.7.x-fastTopN-rebase
2014-11-10 12:52:36 -08:00
Xavier Léauté
17d32a2561
try to benefit from out of order execution
2014-11-10 12:50:10 -08:00
Charles Allen
5aad149b24
Make some variables in PooledTopNAlgorithm final
2014-11-10 12:19:04 -08:00
nishantmonu51
0ca60e0334
skip timestamp checking, remove duplicate code
...
1) No need compare all timestamps when all the values are within range
2) remove redundant code.
2014-11-10 22:36:07 +05:30
fjy
3c21f62afd
a whole bunch of fixes and some ugly conversion code
2014-11-07 16:03:22 -08:00
Charles Allen
7131cde710
Code reformat in CompressedObjectStrategy
2014-11-07 11:19:32 -08:00
Charles Allen
8bd6cf0d07
Updates to CompressedObjectStrategy to support more compression types
...
* Compression types are not yet dynamically configurable.
* Added a benchmarking system for topN to test the compression
* Updated pom.xml to include junit benchmarking
* added an Uncompressed option
2014-11-07 11:14:46 -08:00
Charles Allen
75c79e7562
Added comments in TopNNumericResultBuilder
2014-11-07 11:01:46 -08:00
Charles Allen
b4893b4490
Pre allocate LinkedHashMap in TopNNumericResultBuilder
2014-11-07 10:56:49 -08:00
Charles Allen
bc92de233c
Update dim name comparator in TopNNumericResultBuilder to better handle nulls
2014-11-07 10:54:10 -08:00
Charles Allen
0a562ebd77
Removed CompressedObjectStrategy from this pull request. will submit a new one later
2014-11-07 10:42:28 -08:00
Charles Allen
228614ddb5
Changed TopNNumericResultBuilder imports to not use package.*, but instead pulls all in explicitly
2014-11-07 10:34:28 -08:00
Charles Allen
956bdf80dd
Removed negating comparator in TopNNumericResultBuilder
2014-11-07 10:29:06 -08:00
nishantmonu51
fd8eb7742b
handle union query on realtime node
2014-11-07 23:27:50 +05:30
nishantmonu51
8bebb24fd5
Union Queries move merge to historical
2014-11-07 18:05:16 +05:30
Charles Allen
ee019872f7
TopN performance improvements
...
Re-factor scanAndAggregate in PooledTopN
* Loops are now a little bit tighter when looping over aggregates. This will hopefully assist in loop execution optimization.
* Pre-calculated the aggregate offsets instead of shifting them during runtime.
* Cursor loop could use some TLC, but would require a massive refactoring on how TopN queries are executed.
* Any potential modifications to query workflow need to account for Stream vs Batch data, and that not all data will be array backed that comes in.
Change data storage type in TopNNumericResultBuilder.
* Use PriorityQueue to store
* Checks to see if should even bother adding to Queue before adding.
* Re-orders Queue on build() call.
* Ideally the order would be directly preserved on build(), but this is a close second.
Updates to CompressedObjectStrategy to support more compression types
* Compression types are not yet dynamically configurable.
* Added a benchmarking system for topN to test the compression
* Updated pom.xml to include junit benchmarking
* added an Uncompressed option
2014-11-05 11:10:37 -08:00
fjy
253009d208
initial commit for roaring bitmaps
2014-11-04 17:04:40 -08:00
fjy
3e158f00ef
fix random lowercasing in test
2014-10-30 10:56:39 -07:00
fjy
75da039396
fix search with same value in multiple columns
2014-10-30 10:43:42 -07:00
fjy
41ebdd2bae
Merge branch 'druid-0.7.x' into fix-test
...
Conflicts:
processing/pom.xml
processing/src/main/java/io/druid/segment/IndexMerger.java
processing/src/main/java/io/druid/segment/MetricHolder.java
processing/src/main/java/io/druid/segment/data/CompressedObjectStrategy.java
processing/src/test/java/io/druid/query/groupby/GroupByQueryRunnerTest.java
processing/src/test/java/io/druid/segment/SchemalessTestFull.java
server/src/test/java/io/druid/client/CachingClusteredClientTest.java
services/src/main/java/io/druid/cli/CliHistorical.java
2014-10-24 11:07:52 -07:00
fjy
bef74104d9
merge with 0.7.x and resolve any conflicts
2014-10-23 17:24:06 -07:00
fjy
f790a05bd7
Merge branch 'master' into fix-test
...
Conflicts:
pom.xml
2014-10-23 14:39:51 -07:00
fjy
1b3921faad
fix for long columns
2014-10-23 14:33:25 -07:00
fjy
94da2b6ad3
more logging for potential NPE because of segment unmapping
2014-10-22 16:16:10 -07:00
Xavier Léauté
4ec1e6ef13
simplify code a bit
2014-10-22 14:25:59 -07:00
nishantmonu51
0785baf54c
review comments
2014-10-22 21:28:31 +05:30
nishantmonu51
6cb6ec39cf
add support for not filter and add test
2014-10-22 10:25:24 +05:30
Xavier Léauté
e24a6d222f
we even have tests now
2014-10-21 17:46:24 -07:00
Xavier Léauté
23a6ab19b0
cleanup and add missing methods
2014-10-21 17:00:39 -07:00
nishantmonu51
56912adccb
avoid typecasts
2014-10-21 16:47:50 -07:00
nishantmonu51
10db94101f
skip value lookup
...
compare ids instead of values
2014-10-21 16:47:49 -07:00
Xavier Léauté
71831e49d9
rudimentary filtered aggregator
2014-10-21 16:47:49 -07:00
fjy
bb90432862
Updated druid-api version to always sort dimension names, these tests needed fixing
2014-10-15 12:27:21 -04:00
Xavier Léauté
5b194ee5fe
fix groupBy ordering casing bug
2014-10-15 12:16:17 -04:00
nishantmonu51
bce388fb27
merge changes from 0.7.x branch
2014-10-14 18:46:02 +05:30
nishantmonu51
454acd3f5a
remove backwards compatible code
...
1) remove backwards compatible and deprecated code
2) make hashed partitions spec default
2014-10-13 19:30:44 +05:30
Xavier Léauté
592611f2ee
fix cardinality aggregator serde
2014-10-09 17:06:29 -07:00
Xavier Léauté
290107ce7b
remove unused jsonspec content type
2014-10-09 10:26:47 -07:00
Xavier Léauté
30183d3b4d
update druid-api for json parsing improvements
2014-10-07 13:25:02 -07:00
fjy
fde510bbfc
Merge pull request #779 from metamx/fix-histo
...
Fix groupBy with histograms
2014-10-07 11:53:24 -06:00
fjy
d4217f1588
Merge pull request #777 from metamx/initialized-endpoint
...
Historical endpoints accessible while loading + new loadstatus endpoint
2014-10-06 17:35:58 -06:00
fjy
c7b4d5b7b4
Merge branch 'master' into druid-0.7.x
...
Conflicts:
processing/src/test/java/io/druid/segment/filter/SpatialFilterTest.java
2014-10-02 18:12:10 -07:00
fjy
816986f914
cleanup according to cr
2014-10-02 17:51:33 -07:00
fjy
1a67d6cb2e
some tests for broken approx histo
2014-10-01 15:10:23 -07:00
nishantmonu51
09c41afb5b
license header
2014-10-01 17:52:57 +05:30
nishantmonu51
ad75a21040
separate offheapIncrementalIndex implementation
2014-10-01 13:58:51 +05:30
fjy
267e861162
fix search
2014-09-30 13:36:55 -07:00
fjy
0f96a09939
use multi-val dims for metrics
2014-09-30 13:36:01 -07:00
nishantmonu51
358ff915bb
fix merge conflicts
2014-09-30 22:19:18 +05:30
nishantmonu51
63fcd11864
fix compilation
2014-09-30 22:12:06 +05:30
nishantmonu51
2789536bed
merge changes from druid-0.7.x
2014-09-30 22:05:49 +05:30
nishantmonu51
7ba914d379
review comments
2014-09-30 18:26:43 +05:30
nishantmonu51
61c7fd2e6e
make ingestOffheap tuneable
2014-09-30 15:30:02 +05:30
Gian Merlino
a3a2c8b86f
Merge pull request #765 from metamx/fix-spatial
...
fix spatial indexing with multiple spatial dims
2014-09-29 17:23:29 -07:00
fjy
e1c1e8997a
address cr
2014-09-29 17:22:58 -07:00
fjy
b57ac513bd
Fixes #761
2014-09-29 14:56:43 -07:00
fjy
2b5ce89554
fix spatial indexing with multi spatial dims
2014-09-29 12:07:27 -07:00
nishantmonu51
3f66d3c167
review comments + mapdb deadlock fix
2014-09-29 21:49:12 +05:30
nishantmonu51
adb4a65e0a
Merge branch 'offheap-incremental-index' into mapdb-branch
2014-09-29 12:38:31 +05:30
fjy
eaf0a48b92
Merge branch 'master' into druid-0.7.x
...
Conflicts:
cassandra-storage/pom.xml
common/pom.xml
examples/pom.xml
hdfs-storage/pom.xml
histogram/pom.xml
indexing-hadoop/pom.xml
indexing-service/pom.xml
kafka-eight/pom.xml
kafka-seven/pom.xml
pom.xml
processing/pom.xml
processing/src/main/java/io/druid/guice/PropertiesModule.java
rabbitmq/pom.xml
s3-extensions/pom.xml
server/pom.xml
services/pom.xml
2014-09-26 11:39:24 -07:00
Xavier Léauté
77c9afe92f
try to make javadoc happy
2014-09-23 18:53:29 -07:00
fjy
0187c7f1c3
Merge pull request #748 from metamx/fix-columnschema-access
...
fix column selector on varying incr index schema
2014-09-23 11:25:46 -06:00
fjy
01bb7de3aa
Merge pull request #756 from metamx/dimdim-comments
...
add some comments about string interning
2014-09-23 11:24:11 -06:00
nishantmonu51
f51ab84386
merge changes from druid-0.7.x
2014-09-22 23:48:45 +05:30
nishantmonu51
443e5788fb
make OffheapIncrementalIndex tuneable
2014-09-22 19:26:10 +05:30
nishantmonu51
59aa8dc368
Merge branch 'offheap-incremental-index' into mapdb-branch
2014-09-19 23:09:35 +05:30
nishantmonu51
8c16377e9e
add custom serializer
...
add custom serializers
2014-09-19 23:09:31 +05:30
nishantmonu51
8eb6466487
revert buffer size and add back rowFlushBoundary
2014-09-19 23:06:04 +05:30
Xavier Léauté
e5a13544a0
fix column selector on varying incr index schema
2014-09-17 10:36:05 -07:00
fjy
cada14586a
fix tests
2014-09-16 14:16:49 -07:00
fjy
8b5c2fcd82
fix compile error
2014-09-16 14:10:02 -07:00
xvrl
ff3163516f
Merge pull request #744 from metamx/fix-select
...
fix select for non existant dimensions and metrics
2014-09-16 14:03:51 -07:00
fjy
9e0a2fd487
fix select for non existant dimensions and metrics
2014-09-16 14:01:21 -07:00
Xavier Léauté
0c35fe2cf4
properly close reader
2014-09-15 13:00:06 -07:00
Xavier Léauté
c8b8e3f6e9
negating compare is bad
2014-09-15 13:00:06 -07:00
Xavier Léauté
137ad50bf1
classes that should be static
2014-09-15 13:00:06 -07:00
Xavier Léauté
e57e2d97ba
make constants final
2014-09-15 12:53:40 -07:00
Xavier Léauté
d4795ce927
fix missing charsets
2014-09-15 12:53:40 -07:00
Xavier Léauté
cfa92e8217
fix incorrect nullable annotations
2014-09-15 12:13:52 -07:00
fjy
257fb3f18e
fix retry logic
2014-09-12 15:47:41 -07:00