Commit Graph

1489 Commits

Author SHA1 Message Date
Fangjin Yang 3bdd757024 Merge pull request #1773 from b-slim/log_details
Adding downstream source when throwing QueryInterruptedException
2016-02-22 10:16:07 -08:00
Slim Bouguerra 77925cc061 adding downstream source of QueryInterruptedException 2016-02-20 13:05:14 -06:00
Fangjin Yang 8ee81947cd Merge pull request #2494 from himanshug/fix_timeseries
do not drop post-aggs in TimeseriesQueryToolChest.makePreComputeManipulatorFn
2016-02-20 10:37:32 -08:00
Gian Merlino d25c46cb9f Add comparator to HyperUniquesFinalizingPostAggregator.
This makes it possible to do groupBys with clauses like "HAVING uniques > 10".
Beforehand you couldn't do it with either an aggregator (because it returns
an HLLV1 which the havingSpec can't understand) or a finalized postaggregator
(because it didn't have a comparator).

Now you can at least do it with a finalizing postaggregator. Trying it with
the aggregator alone still doesn't work.

Added some topN and groupBy tests verifying the comparator, and added an
@Ignore test that should pass if havingSpecs are made work on the aggregator
directly.
2016-02-19 08:36:08 -08:00
Himanshu Gupta 11b0117422 do not drop post-aggs in timeseries query tool chest makePreComputeManipulatorFn like other query types 2016-02-17 20:51:35 -06:00
Jaehong Choi 32b9d57b23 handle a failing UT in GroupByQueryRunnerTest after merging into the master 2016-02-16 16:56:57 +09:00
Jaehong Choi b25bca85bc Merge branch 'master' of https://github.com/druid-io/druid into support-alphanumeric-dimensional-sort-in-gropu-by 2016-02-16 16:42:05 +09:00
Jaehong Choi e89afc901b delete System.out.println() in test code 2016-02-16 15:26:37 +09:00
Navis Ryu cd315627c9 Merge pull request #2393 from CHOIJAEHONG1/support-alphanumeric-dimensional-sort-in-gropu-by
support alphanumeric sorting for dimensional columns in groupby (#2393)
2016-02-16 14:11:30 +09:00
Slim 16092eb5e2 Merge pull request #2464 from gianm/print-properties
Make startup properties logging optional.
2016-02-14 15:11:35 -06:00
Gian Merlino e0c049c0b0 Make startup properties logging optional.
Off by default, but enabled in the example config files. See also #2452.
2016-02-12 14:12:16 -08:00
Himanshu Gupta da5fcd0124 before facts get it , indexAndOffsets should already know about it 2016-02-12 13:32:06 -06:00
Jonathan Wei d63eec65a1 Merge pull request #2208 from navis/metadataquery-minmax
Support min/max values for metadata query
2016-02-11 17:28:07 -08:00
Jonathan Wei e1b022eac9 Merge pull request #2349 from navis/dimensionspec-for-selectquery
Support dimension spec for select query
2016-02-11 16:38:16 -08:00
navis.ryu dd2375477a Support min/max values for metadata query (#2208) 2016-02-12 09:35:58 +09:00
Gian Merlino 2d037ef05e Merge pull request #2453 from DreamLab/fix/topn_sorting_anomaly
Fix for unstable behavior of HyperLogLog comparator
2016-02-11 16:05:34 -08:00
navis.ryu 4d63196535 Support dimension spec for select query 2016-02-12 08:54:28 +09:00
Himanshu 47d48e1e67 Merge pull request #2452 from gianm/print-properties
PropertiesModule: Print properties, processors, totalMemory on startup.
2016-02-11 16:49:34 -06:00
turu f277a54a5c removed unsafe heuristics from hll compareTo and provided unit test for regression 2016-02-11 23:46:24 +01:00
Slim 368988d187 Merge pull request #2291 from druid-io/lookupManager
Promoting LookupExtractor state and LookupExtractorFactory to be a first class druid state object.
2016-02-11 16:07:27 -06:00
Gian Merlino 29f7758e74 PropertiesModule: Print properties, processors, totalMemory on startup. 2016-02-11 13:51:08 -08:00
Slim Bouguerra 4e119b7a24 Adding lookup ref manager and lookup dimension spec impl 2016-02-11 12:11:51 -06:00
Jaehong Choi 2f2e2ff5b9 support alphanumeric sorting for dimensional columns in groupby 2016-02-11 17:31:28 +09:00
Keuntae Park 05a144e39a fix crash with filtered aggregator at ingestion time
- only for selector filter because extraction filter is not supported as
  cardinality is not fixed at ingestion time
2016-02-11 11:25:33 +09:00
Fangjin Yang b1673ee90e Merge pull request #2409 from gianm/smq-merged-thing
SegmentMetadataQuery: Retain segment id when merging, if possible.
2016-02-08 15:43:39 -08:00
Fangjin Yang c9c20bb7f3 Merge pull request #2395 from metamx/fixExtractionDimFilterNullTest
Actually check cache key null checking in ExtractionDimFilterTest
2016-02-08 14:10:52 -08:00
Gian Merlino bd9c04244f SegmentMetadataQuery: Retain segment id when merging, if possible.
This is helpful on realtime nodes, where two analyses from two different hydrants
are merged together but they are actually from the same segment.
2016-02-08 13:07:02 -08:00
Himanshu Gupta 9fe1b28ee5 provide configuration to enable usage of Off heap merging for groupBy query 2016-02-05 14:18:06 -06:00
Himanshu Gupta b40c342cd1 make Global stupid pool cache size configurable 2016-02-05 14:18:06 -06:00
Himanshu Gupta 72a1e730a2 OffheapIncrementalIndex updates to do the aggregation merging off-heap 2016-02-05 14:17:05 -06:00
Himanshu Gupta 907dd77483 OffheapIncrementalIndex a copy/paste of OnheapIncrementalIndex 2016-02-05 14:02:31 -06:00
Charles Allen aac5f9b2c9 Actually check cache key null checking in ExtractionDimFilterTest 2016-02-04 09:44:13 -08:00
fjy 1aa363cea7 new quickstart 2016-02-04 09:37:38 -08:00
Fangjin Yang da77591129 Merge pull request #2392 from metamx/fix2391
Allow ExtractionDimFilter value to be null
2016-02-03 17:47:14 -08:00
Charles Allen d4f00096ff Allow ExtractionDimFilter value to be null
* Fixes #2391
2016-02-03 15:51:47 -08:00
Himanshu Gupta 6e7d90cf56 UTs for DefaultLimitSpec 2016-02-03 15:59:12 -06:00
Himanshu Gupta 29e0d7f971 lazily create comparators for row columns when needed 2016-02-03 13:38:20 -06:00
navis.ryu 1d602be0f9 Replace string[] with int[] for dimensions 2016-02-03 15:03:22 +09:00
binlijin a5ef30ff84 optimize topn on particular situation 2016-02-02 14:20:09 +08:00
Himanshu 93c50d8538 Merge pull request #2094 from navis/simplify-index-merge
Simplifying dimension merging
2016-01-29 11:23:14 -06:00
navis.ryu 55a888ea2f time-descending result of select queries 2016-01-29 10:06:05 +09:00
navis.ryu dd774ef4dd one-pass merging of dictionary & index 2016-01-29 10:03:53 +09:00
Himanshu edd7ce58aa Merge pull request #2348 from AlexanderSaydakov/fix-aggregator-test-helper
fixed createIndex
2016-01-28 16:01:36 -06:00
saydakov e0860661b1 fixed createIndex 2016-01-28 13:20:50 -08:00
Nishant 99017f4518 Merge pull request #2326 from navis/use-reverse-iterator
use reverse-iterator if possible
2016-01-28 19:48:38 +05:30
Nishant 3880f54b87 Merge pull request #2332 from himanshug/configurable_partial
make populateUncoveredIntervals a configuration in query context
2016-01-28 10:34:35 +05:30
navis.ryu 7324ece8f9 use reverse-iterator if possible 2016-01-28 09:04:55 +09:00
Xavier Léauté 5a3642bb93 Merge pull request #2247 from metamx/pedanticBuild
Enable strict building in travis
2016-01-27 10:27:03 -08:00
Xavier Léauté 2e5004095a Merge pull request #2341 from gianm/smq-test
SegmentMetadataQuery: Fix merging of ColumnAnalysis errors.
2016-01-27 09:37:06 -08:00
Charles Allen 508734c8b0 Long constant reformatting in tests `l` --> `L` 2016-01-27 08:59:19 -08:00
Gian Merlino b1e6c01762 Make LookupExtractor abstract methods public, they have to work across classloaders. 2016-01-26 23:08:03 -08:00
Gian Merlino 795343f7ef SegmentMetadataQuery: Fix merging of ColumnAnalysis errors.
Also add tests for:
- ColumnAnalysis folding
- Mixed mmap/incremental merging
2016-01-26 17:16:26 -08:00
Himanshu Gupta 3719b6e3c8 make populateUncoveredIntervals a configuration in query context 2016-01-26 15:13:45 -06:00
Himanshu 3844658fb5 Merge pull request #2323 from druid-io/update-druidapi
Update druid-api to 0.3.16
2016-01-26 13:02:10 -06:00
Himanshu Gupta 09d3678667 adding single threaded indexing and querying test for IncrementalIndex 2016-01-23 00:17:14 -06:00
Charles Allen 0000b9fc62 Remove sorting in ProtoBufInputRowParserTest
Due to processing/src/test/java/io/druid/data/input/ProtoBufInputRowParserTest.java
2016-01-22 16:02:25 -08:00
Himanshu Gupta 2f7f5119cf older segments might not have field bitmapSerdeFactory for dimension columns and we must use appropriate default 2016-01-22 13:28:25 -06:00
binlijin 1d1f4d996d Merge pull request #2111 from binlijin/optimize-create-inverted-indexes
optimize create inverted indexes
2016-01-22 11:36:27 +08:00
binlijin 55f7dd4629 optimize create inverted indexes 2016-01-22 10:40:09 +08:00
Gian Merlino d416279c14 SegmentMetadataQuery support for returning aggregators. 2016-01-21 17:27:25 -08:00
Fangjin Yang 5a9cd89059 Merge pull request #2305 from gianm/segment-metadata-query-multivalues
Add StorageAdapter#getColumnTypeName, and various SegmentMetadataQuery adjustments
2016-01-21 17:22:34 -08:00
Gian Merlino e5913be90e Merge pull request #2257 from tubemogul/index-merge-bug
Adds support for empty merge metrics. fixes #2256
2016-01-21 16:38:00 -08:00
Gian Merlino 87c8046c6c Add StorageAdapter#getColumnTypeName, and various SegmentMetadataQuery adjustments.
SegmentMetadataQuery stuff:

- Simplify implementation of SegmentAnalyzer.
- Fix type names for realtime complex columns; this used to try to merge a nice type
  name (like "hyperUnique") from mmapped segments with the word "COMPLEX" from incremental
  index segments, leading to a merge failure. Now it always uses the nice name.
- Add hasMultipleValues to ColumnAnalysis.
- Add tests for both mmapped and incremental index segments.
- Update docs to include errorMessage.
2016-01-21 15:50:33 -08:00
Fangjin Yang 3f998117a6 Merge pull request #2306 from jon-wei/inherit2
More specific null/empty str handling in IndexMerger
2016-01-21 14:36:09 -08:00
Michael Schiff 1e44445f06 Adds support for empty merge metrics. fixes #2256 2016-01-21 13:21:37 -08:00
jon-wei 459a236067 More specific null/empty str handling in IndexMerger 2016-01-21 12:24:38 -08:00
Slim 201539260c Merge pull request #2076 from b-slim/issue_2010_upper_lower_extractionFN
adding lower and upper extraction fn
2016-01-21 09:58:07 -06:00
Slim Bouguerra 78feb3a13e adding lower and upper extraction fn 2016-01-21 08:59:05 -06:00
Gian Merlino 5a932d28c1 Merge pull request #2288 from tubemogul/index-merge-bug2
Null check in IncrementalIndexAdapter.getDimValueLookup()
2016-01-20 17:07:15 -08:00
Nishant 59ea186af7 fix reference counting for segments 2016-01-20 17:24:21 +05:30
Michael Schiff 50ceec78a2 null check in IncrementalIndexAdapter.getDimValueLookup() 2016-01-19 23:19:28 -08:00
jon-wei bc1e9b27c8 Consolidate IndexMergerTest and IndexMergerV9Test 2016-01-19 16:28:35 -08:00
jon-wei 747343e621 Preserve dimension order across indexes during ingestion 2016-01-19 13:34:11 -08:00
Fangjin Yang 0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
Himanshu Gupta a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Himanshu Gupta 52eb0f04a7 adding a new method getMergingFactory(..) to AggregatorFactory 2016-01-18 22:03:46 -06:00
Himanshu Gupta 77fc86c015 making AggregatorFactory abstract class 2016-01-18 22:03:46 -06:00
Himanshu Gupta 164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
zhxiaog 3459a202ce fixed #1873, add ability to express CONCAT as an extractionFn 2016-01-18 15:03:17 -08:00
Keuntae Park 238dd3be3c support cascade execution of extraction filters in extraction dimension spec 2016-01-18 11:10:19 +09:00
Fangjin Yang f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9
build v9 directly
2016-01-16 13:35:46 -06:00
Kurt Young 82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Kurt Young 1f2168fae5 add IndexMergerV9
add unit tests for IndexMergerV9 and fix some bugs

add more unit tests and fix bugs

handle null values and add more tests

minor changes & use LoggingProgressIndicator in IndexGeneratorReducer

make some static class public from IndexMerger

minor changes and add some comments

changes for comments
2016-01-16 11:25:28 +08:00
Kurt Young bb50d2a2b2 add some streaming writers 2016-01-16 11:25:26 +08:00
Fangjin Yang e0932ba1c2 Merge pull request #2267 from himanshug/fix_topn_multi_val_filter
Remap id's returned in XXXFilteredDimensionSpec.getRow() as per reduced cardinality
2016-01-14 17:06:54 -08:00
Fangjin Yang 7704699b40 Merge pull request #2265 from navis/strlen-dimension-ignored
Strlen sort spec ignores dimension
2016-01-14 17:06:33 -08:00
Himanshu Gupta ae6a111444 fix XXXFilteredDimensionSpec to remap the dictionary encodings as per new cardinality 2016-01-13 22:25:02 -06:00
binlijin a3140b2548 fix topN filtering on multi-valued dimension bug 2016-01-13 22:25:02 -06:00
navis.ryu ea9fabdf2f Strlen sort spec ignores dimension 2016-01-14 11:05:44 +09:00
Fangjin Yang 4c014c1574 Merge pull request #2228 from metamx/incremental-index-mem2
Improve heap usage for IncrementalIndex
2016-01-13 14:48:03 -08:00
navis.ryu 18479bb757 time-descending result of timeseries queries 2016-01-13 12:23:01 +09:00
Fangjin Yang d7ad93debc Merge pull request #2221 from binlijin/topN_minTopNThreshold
Allow change minTopNThreshold per topN query
2016-01-12 16:22:20 -08:00
Nishant 4863e2ca4f cache metric selectors instead of creating new ones for every metric in each row
clear selectors on close.

Add comments about thread safety.
2016-01-13 00:45:23 +05:30
Nishant dfe6abb721 Merge pull request #2250 from himanshug/agg_test_helper_fix
remove redundant registering of json modules in AggregationTestHelper
2016-01-12 11:42:00 +05:30
navis.ryu 976ebc45c0 Simplify information in IncrementalIndex 2016-01-12 10:18:11 +09:00
Himanshu Gupta b973604bf8 remove redundant registering of json modules in AggregationTestHelper 2016-01-11 19:03:22 -06:00
Xavier Léauté 46a7f2660d fix casing to be consistent with other classes 2016-01-08 10:19:06 -08:00
Fangjin Yang d0b10c29d7 Merge pull request #2197 from metamx/clearIncIndexClose
Make OnHeapIncrementalIndex clean maps on close()
2016-01-07 15:43:47 -08:00
Gian Merlino 4ecd901a1a Merge pull request #2219 from himanshug/identity_extraction_fn_singleton
make IdentityExtractionFn singleton
2016-01-07 10:08:28 -08:00
Fangjin Yang aaea95ed1b Merge pull request #2207 from himanshug/theta_sketch_select_query
fix bug for thetaSketch metric not working with select queries
2016-01-07 09:46:09 -08:00
binlijin 010c6e959c add test 2016-01-07 18:01:46 +08:00
binlijin a6bfcc5bfd Allow change minTopNThreshold per topN query 2016-01-07 14:51:00 +08:00
Fangjin Yang 4cc81d3eff Merge pull request #2096 from b-slim/add_use_case_unapply
Add use case unapply
2016-01-06 21:58:12 -08:00
Himanshu Gupta 217079d0c7 make IdentityExtractionFn singleton 2016-01-06 22:29:07 -06:00
Himanshu 902f51433d Merge pull request #2125 from mangeshpardeshiyahoo/master
Add extraction function support for Dimension Selector
2016-01-06 14:22:26 -06:00
Mangesh Pardeshi 75ee952197 Add extraction function support for dimension Selector 2016-01-06 13:47:07 -06:00
Slim Bouguerra 032d3bf6e6 Optimization of extraction filter by reversing the lookup 2016-01-06 11:16:11 -06:00
Himanshu Gupta 3f048f0b15 adding support to execute Select queries in AggregationTestHelper so that Select query based UTs can be written for complex aggregator implementations 2016-01-05 21:54:55 -06:00
Charles Allen 91fc32749b Make OnHeapIncrementalIndex clean maps on close() 2016-01-04 11:18:16 -08:00
Himanshu Gupta b47d807738 Add support for filtering at DimensionSpec level so that multivalued dimensions can be filtered correctly
also adding UTs for multi-valued dimensions
2015-12-30 17:59:47 -06:00
Himanshu Gupta fa5c3bb014 adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values 2015-12-30 15:06:24 -06:00
Nishant b68265399c Merge pull request #2168 from druid-io/remove-indexmaker
Remove IndexMaker
2015-12-30 12:24:29 +05:30
Fangjin Yang e14ad74088 Merge pull request #1936 from b-slim/between_range_with_predicat
adding Upper/Lower Bound Filter
2015-12-29 10:11:22 -08:00
fjy faf421726b remove IndexMaker 2015-12-28 14:19:02 -08:00
Gian Merlino 83f4130b5f SegmentMetadataQuery merging fixes.
- Fix merging when the INTERVALS analysisType is disabled, and add a test.
- Remove transformFn from CombiningSequence, use MappingSequence instead. transformFn did
  not work for "accumulate" anyway, which made the tests wrong (the intervals should have
  been condensed, but were not).
- Add analysisTypes to the Druids segmentMetadataQuery builder to make testing simpler.
2015-12-22 07:57:10 -08:00
Robin dded4441d3 for completeness, add unit test for groupby/having with unrecognized type 2015-12-21 12:06:56 -06:00
Himanshu Gupta e1631967e3 adding comments to explain merge failure in segmentMetadata query 2015-12-19 11:39:24 -06:00
Himanshu Gupta 7ecad1be24 Fix and UT for testing segment analysis merge 2015-12-19 00:24:02 -06:00
Fangjin Yang 7019d3c421 Merge pull request #2107 from jon-wei/fix_smq
More efficient SegmentMetadataQuery
2015-12-18 16:40:47 -08:00
Fangjin Yang 14229ba0f2 Merge pull request #1922 from metamx/jsonIgnoresFinalFields
Change DefaultObjectMapper to NOT overwrite final fields unless explicitly asked to
2015-12-18 15:38:32 -08:00
Fangjin Yang 71f554bf80 Merge pull request #2101 from himanshug/fix_extraction_dim_filter_cache_key
add extractionFn bytes to cache key in ExtractionDimFilter
2015-12-18 12:05:43 -08:00
Fangjin Yang 9e6874cc7e Merge pull request #2084 from binlijin/master
minor optimize IndexMerger's MMappedIndexRowIterable
2015-12-18 11:42:55 -08:00
Bingkun cc21a5fac7 Merge pull request #1999 from himanshug/remove_min_max_aggs
remove min/max aggregator factory
2015-12-18 13:38:52 -06:00
jon-wei 356b07c6c3 More efficient SegmentMetadataQuery 2015-12-17 12:46:23 -08:00
Jonathan Wei f8cf84f466 Merge pull request #1995 from himanshug/num_rows_seg_metadata_query
add numRows to segment metadata query response
2015-12-17 12:23:46 -08:00
Himanshu Gupta 82ea348003 add extractionFn bytes to cache key in ExtractionDimFilter 2015-12-16 14:00:38 -06:00
Himanshu 628643d80e Merge pull request #2091 from rasahner/noDefaultForGroupbyHaving
take away default for groupBy/having
2015-12-16 01:07:40 -06:00
sahner 3441cf3110 take away default for groupBy/having 2015-12-15 10:32:45 -06:00
Fangjin Yang e7f06cf61c Merge pull request #2075 from jon-wei/regex_extract
Configurable value replacement on match failure for RegexExtractionFn
2015-12-14 19:10:50 -08:00
jon-wei c88f75df7c Configurable value replacement on match failure for RegexExtractionFn 2015-12-14 17:57:41 -08:00
binlijin 362bea1090 minor optimize IndexMerger's MMappedIndexRowIterable 2015-12-11 15:04:46 +08:00
Xavier Léauté d531e69d1a Merge pull request #2079 from binlijin/master
reduce bytearray copy to minimal optimize VSizeIndexedWriter
2015-12-10 21:30:09 -08:00
Slim Bouguerra 77afdf25e3 adding Bound Filter 2015-12-10 08:47:21 -06:00
Slim Bouguerra ee1a39801a adding bulk lookup and reverse lookup 2015-12-10 08:29:41 -06:00
binlijin 0eafbd55b2 reduce bytearray copy to minimal optimize VSizeIndexedWriter 2015-12-10 16:34:39 +08:00
Fangjin Yang f4ba13a1ac Merge pull request #2029 from b-slim/add_reverse_fn
Adding reverse lookup function to LookupExtractor.
2015-12-09 12:50:13 -08:00
Xavier Léauté 9015a68c03 Merge pull request #2002 from navis/DRUID-2001
fixed #2001 GenericIndexed.fromIterable compares all values even when it's not sorted
2015-12-09 08:56:49 -08:00
Slim Bouguerra 85f339b687 introduction and implem of reverse lookup function unApply. 2015-12-09 10:02:57 -06:00
Nishant 6c23d8edb4 Merge pull request #2043 from mangeshpardeshiyahoo/master
Add dimension selector support for groupby/having filters
2015-12-08 12:08:53 +05:30
Mangesh Pardeshi d7ce120929 Add dimension selector support for groupby/having quries 2015-12-08 01:51:11 +00:00
Himanshu Gupta 431469e9c1 remove min/max aggregator factory which are replaced by double[min/max] aggregator factories 2015-12-05 22:36:49 -06:00
Himanshu Gupta 62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
Gian Merlino d21a640695 Merge pull request #2034 from b-slim/fix_cache_key
Fix getCacheKey for DimFilters
2015-12-04 09:13:06 -08:00
Slim Bouguerra fb4ff3cf54 fix getCacheKey 2015-12-04 08:07:08 -06:00
Charles Allen 9d02f47201 Update IncrementalIndexTest copyright notice 2015-12-03 18:03:08 -08:00
Charles Allen be8c6fafb0 Merge pull request #2017 from tubemogul/issue/63
fixes issue #63
2015-12-03 18:01:11 -08:00
Gian Merlino 045df54404 Merge pull request #1961 from metamx/druidMetricsVersion
Add the druid artifact version to metrics when emitted
2015-12-03 17:34:57 -08:00
Michael Schiff b6cc2428e1 fixes issue #63 2015-12-03 17:30:47 -08:00
Himanshu 0eab8417cb Merge pull request #2008 from codingwhatever/regex-search-query
Regex search query
2015-12-03 09:57:34 -06:00
Sam Groth 596b7ebd9a Adding RegexSearchQuerySpec 2015-12-03 09:16:02 -06:00