1110 Commits

Author SHA1 Message Date
Charles Allen
6230ac90ae Use IndexMerger for conversion 2015-06-10 11:34:58 -07:00
Xavier Léauté
395ba79f8b Merge pull request #1403 from metamx/mergerMakerTests
Improvements around resource handling in IndexMerger / IndexIO / QueryableIndex
2015-06-04 15:59:10 -07:00
Charles Allen
ed8eb5c991 Improvements around resource handling in IndexMerger / IndexIO / QueryableIndex
* Fix resource leak in `io.druid.segment.IndexIO.DefaultIndexIOHandler#validateTwoSegments(java.io.File, java.io.File)`
* Un-deprecate `close()` in `QueryableIndex` and make it inherit `Closeable`
* Fix resource leaks in various unit tests
* Add `CloserRule` for closing out resources
2015-06-04 14:18:27 -07:00
Himanshu
50ad0e6474 Merge pull request #1412 from pjain1/alphaNumericTopN_NPE_fix
NPE fix for TopN query with alphaNumericTopN metric spec
2015-06-04 09:49:31 -05:00
Parag Jain
a7b09e857c NPE fix for alphaNumericTopN when pervious stop is not specified 2015-06-04 09:30:31 -05:00
Xavier Léauté
35e2fde18e Merge pull request #1386 from himanshug/aggregation_testing1
General class for testing any Aggregation Implementation
2015-06-03 23:43:36 -07:00
Xavier Léauté
92d7316ed8 Merge pull request #1414 from metamx/timeout2TIMEOUT
Replace "timeout" with QueryContextKeys.TIMEOUT
2015-06-02 17:11:09 -07:00
Charles Allen
1c4d42bc15 Replace "timeout" with QueryContextKeys.TIMEOUT 2015-06-02 14:49:21 -07:00
Charles Allen
f48db09e35 Add optimizations for ExtractionFn by enabling MANY_TO_ONE vs ONE_TO_ONE codepaths
* Also adds LookupExtractionFn and MapLookupExtractor which takes in an explicit mapping of renames
* Add injective to javascript extraction fn
2015-06-02 12:22:56 -07:00
Himanshu Gupta
215c1ab01e UTs for hyperUnique aggregation 2015-06-01 12:52:40 -05:00
Himanshu Gupta
160d5fe6b7 a general class for testing any [complex] aggregation implementation 2015-06-01 12:52:40 -05:00
Charles Allen
55292bba13 Add more IndexMergerTests 2015-05-28 18:18:20 -07:00
Charles Allen
1ebe622c7d Add checkin GroupByQuery for null DimensionSpec in dimension list 2015-05-28 14:55:34 -07:00
Xavier Léauté
f9c624c7db Merge pull request #1361 from mrijke/groupby-limithavingorder-unittest
GroupBy Query with Having/Limit/Orderingspec inconsistencies (UnitTest)
2015-05-27 14:49:18 -07:00
Xavier Léauté
1a3f04f0ed Merge pull request #1354 from metamx/multi-valued-dimension-compression
Enabling compression for multiValued dimension
2015-05-26 23:43:53 -07:00
Charles Allen
fd64c24e43 Fix roaring extraction filter on empty values 2015-05-26 13:54:18 -07:00
nishant
81415282aa Enabling compression for multiValued dimension
Add test and refactoring

Add benchmark tests
2015-05-27 00:09:14 +05:30
Charles Allen
e97d22a10a Fix Extraction Filter cast problems for empty results 2015-05-22 15:20:11 -07:00
Charles Allen
e1399b7ce4 Add unit test to show breaking Dimension Extraction Filter 2015-05-22 15:02:11 -07:00
Xavier Léauté
75c092ccb1 Merge pull request #1375 from metamx/MetricManipulatorFnInstances
Modify MetricManipulatorFns to use instanced classes
2015-05-22 15:56:47 -04:00
Charles Allen
042653ebcb Modify MetricManipulatorFns to use instanced classes 2015-05-22 12:38:38 -07:00
Himanshu Gupta
723df735e9 force eagerness of processing of SegmentMetadata queries on the processing executor by converting the Sequence into List 2015-05-22 13:46:26 -05:00
Himanshu Gupta
5852b64852 adding UT for SegmentMetadata bySegment query which catches following regression caused by commit 55ebf0cfdf5c290a462f188d71c806b595eec689
it fails when we issue the SegmentMetadataQuery by setting {"bySegment" : true} in context with exception -
java.lang.ClassCastException: io.druid.query.Result cannot be cast to io.druid.query.metadata.metadata.SegmentAnalysis
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$4.compare(SegmentMetadataQueryQueryToolChest.java:222) ~[druid-processing-0.7.3-SNAPSHOT.jar:0.7.3-SNAPSHOT]
at com.google.common.collect.NullsFirstOrdering.compare(NullsFirstOrdering.java:44) ~[guava-16.0.1.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:46) ~[java-util-0.27.0.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:42) ~[java-util-0.27.0.jar:?]
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649) ~[?:1.7.0_80]
2015-05-22 13:45:54 -05:00
Himanshu Gupta
da0cc32bc8 Revert commit 55ebf0cfdf5c290a462f188d71c806b595eec689
which caused following regression
 it fails when we issue the SegmentMetadataQuery by setting {"bySegment" : true} in context with exception -
java.lang.ClassCastException: io.druid.query.Result cannot be cast to io.druid.query.metadata.metadata.SegmentAnalysis
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$4.compare(SegmentMetadataQueryQueryToolChest.java:222) ~[druid-processing-0.7.3-SNAPSHOT.jar:0.7.3-SNAPSHOT]
at com.google.common.collect.NullsFirstOrdering.compare(NullsFirstOrdering.java:44) ~[guava-16.0.1.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:46) ~[java-util-0.27.0.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:42) ~[java-util-0.27.0.jar:?]
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649) ~[?:1.7.0_80]
2015-05-22 13:39:34 -05:00
Maarten Rijke
82da479464 Fix for GroupBy with Having+Limit+Orderspec
* Inverted function arguments to compose postProcFn for GroupBy queries
    with havingspec + limitspec.
  * Replaced query.getLimitSpec() with null in GroupByQueryToolChest's
    mergeGroupByResults
  * Added unittest to verify functionality
2015-05-19 18:35:48 +02:00
Himanshu Gupta
2fd3e9e8e5 return size = 0 in ColumnAnalysis if its unknown
that is if complex agg did not implement inputSizeFn() so
that segment metadata query shows atleast some information.
also instead of COMPLEX, return type of data stored.
2015-05-15 20:11:56 -05:00
Xavier Léauté
3c3db7229c Merge pull request #1355 from himanshug/long_max_min_aggregators
Long max/min aggregators
2015-05-13 12:08:11 -07:00
Himanshu Gupta
cebb550796 additional UTs for [DoubleMax/DoubleMin] aggregation 2015-05-13 09:25:41 -05:00
Himanshu Gupta
d0ec945129 adding aliases doubleMax and doubleMin for max and min respectively
renamed all [Max/Min]*.java to [DoubleMax/DoubleMin]*.java and created [Max/Min]AggregatorFactory.java which can be removed when we dont need the min/max aggregator type backward compatibility
2015-05-13 09:25:41 -05:00
Himanshu Gupta
2de38f7d29 UTs for long[Max/Min] aggregation 2015-05-13 09:25:22 -05:00
Himanshu Gupta
00436f93e2 long max/min aggregators implementation 2015-05-13 09:25:22 -05:00
fjy
7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
Xavier Léauté
33265d63e1 Merge pull request #1262 from metamx/fix-null-dimension
fix handling of dimension having only null values
2015-05-06 13:51:26 -07:00
nishant
34be1e96fa fix NPE
review comments

Add test

fix test for java8
2015-05-05 23:11:13 +05:30
Neo
8f8400e24e fix handling of dimension having only null values
fixes #1211

fix value matcher

more improvements

more fixes for partial null column

fix handling of dimension having only null values

fixes #1211

fix value matcher

more improvements

more fixes for partial null column

review comment

IndexMaker speedups
* About 15% speedup

Conflicts:
	processing/src/main/java/io/druid/segment/IndexMaker.java

fix handling of dimension having only null values

fixes #1211

fix value matcher

more improvements

more fixes for partial null column

fix handling of dimension having only null values

fixes #1211

fix value matcher

more improvements

more fixes for partial null column

review comment

review comments

review comment

fix failing tests

review comment

fix compilation
2015-05-04 22:07:45 +05:30
nishant
50158357ff fixes #1330
fixes #1330,
Avoid creating Period instance as creating a Period from Long.MAX_VALUE
throws arithmetic exception.
After this query metric will emit duration in seconds instead of
minutes.
2015-05-04 20:34:28 +05:30
Xavier Léauté
721505c017 Merge pull request #1208 from druid-io/rework-metrics
Schemaless metrics + additional metrics for things we care about
2015-04-27 15:04:54 -07:00
fjy
963e5765bf Schemaless metrics + additional metrics for things we care about 2015-04-27 13:39:40 -07:00
Charles Allen
27016c0289 Fix IndexIO segment validator to account for timestamp mismatches. 2015-04-27 12:42:16 -07:00
Charles Allen
633fdb029e Add option to ConvertSegmentTask to skip validation
* Validation is enabled by default
2015-04-27 08:37:55 -07:00
Charles Allen
303727e6a9 IndexMaker speedups
* About 15% speedup

Conflicts:
	processing/src/main/java/io/druid/segment/IndexMaker.java
2015-04-23 13:19:21 -07:00
Charles Allen
f2300430d1 Cleanup some code in index creation.
* Add some unit tests
* Add io.druid.segment.IndexMerger.reprocess for quick re-indexing of data
* Add dim-value validation to validation checker (instead of ONLY index #)
* General code refactoring to make things a little easier to read
2015-04-23 12:41:42 -07:00
Xavier Léauté
7939f43681 Merge pull request #1296 from druid-io/limit-test
Add test for order by metric and limit across multiple days
2015-04-22 11:28:06 -07:00
fjy
97d87a06d0 Add another test for limit across multiple days 2015-04-22 11:27:37 -07:00
Fangjin Yang
28f69d6bd3 Merge pull request #1299 from metamx/improve-filter-datasource-metadata
Improve filtering of segments for dataSourceMetadataQuery
2015-04-22 11:07:35 -07:00
Xavier Léauté
a0a28de551 fix serde issue when pulling timestamps from cache 2015-04-22 11:03:26 -07:00
Xavier Léauté
2b4406671e Merge pull request #1301 from druid-io/fix-type
fix count agg factory type
2015-04-21 09:24:20 -07:00
fjy
7805357ab1 fix count agg factory type 2015-04-21 09:23:04 -07:00
nishant
bb8c0cb50b Improve filtering of segments for dataSourceMetadataQuery
dataSourceMetadataQuery only needs to be executed on latest segments at
present, modify filterSegments and add test.
2015-04-21 09:31:13 +05:30
Xavier Léauté
f73f14ab91 Merge pull request #1297 from metamx/versionConverterTaskUpdates
Update VersionConverterTask for IndexSpec and allowing Forced updates
2015-04-20 16:44:35 -07:00