0eab8417cb
Merge pull request #2008 from codingwhatever/regex-search-query
...
Regex search query
2015-12-03 09:57:34 -06:00
596b7ebd9a
Adding RegexSearchQuerySpec
2015-12-03 09:16:02 -06:00
d02be6194d
Merge pull request #1967 from metamx/realtime-metrics-improvements
...
Add datasource and taskId to metrics emitted by peons
2015-12-02 23:48:13 -06:00
00c6027777
Merge pull request #1986 from metamx/substring
...
fixes #1874 adding a substring extraction function, tests, and documentation
2015-12-02 23:45:47 -06:00
68ef5f437a
fixes #1874 adding a substring extraction function, tests, and documentation
2015-12-01 23:50:32 -08:00
87357a0534
fixed #2001 GenericIndexed.fromIterable compares all values even when it's not sorted
2015-12-02 15:11:14 +09:00
1eb8211346
Add datasource and taskId to metrics emitted by peons
...
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.
fix spelling
review comment
review comment
2015-12-01 23:20:59 +05:30
cd2cff24ff
Fix serde for FragmentSearchQuerySpec and add some tests.
2015-11-30 17:34:35 -08:00
c73418c181
fixed #2003 ColumnSelectorBitmapIndexSelector throws NPE for dimension not supporting bitmap
2015-11-24 10:45:36 +09:00
7a89b2e1a6
add numRows to segment metadata query response
2015-11-20 01:25:02 -06:00
d93640bfcb
Merge pull request #1974 from jon-wei/dim_order_merge
...
Allow IndexMerger to use non-lexicographic dim order when merging indexes
2015-11-18 19:51:34 -06:00
e3e6159336
Merge pull request #1985 from metamx/FixLookupCacheKey
...
Change LookupExtractionFn cache key to be unique
2015-11-18 10:13:55 -08:00
7abe999418
Change LookupExtractionFn cache key to be unique
2015-11-17 18:02:40 -08:00
4afc62be29
Allow IndexMerger to use non-lexicographic dim order when merging indexes
2015-11-17 13:02:31 -08:00
d7eb2f717e
enable query caching on intermediate realtime persists
2015-11-17 10:58:00 -08:00
57f213d536
Better toString for groupBy, segmentMetadata queries.
2015-11-16 12:54:59 -08:00
cdceaf2d26
Fix IncrementalIndexAdapter getRows() Iterable
2015-11-12 13:10:42 -08:00
af34e9c8cb
Add the druid artifact version to metrics when emitted
2015-11-12 12:11:27 -08:00
286b8f8c6f
optimize index merge
2015-11-12 11:08:54 +08:00
fa6142e217
cleanup and remove unused imports
2015-11-11 12:25:21 -08:00
fd0935ecb9
fix spatial dimension transformer to work with hadoop
2015-11-10 19:16:51 -07:00
c511273efd
adding in filter
2015-11-06 16:23:24 -06:00
929b981710
Change DefaultObjectMapper to NOT overwrite final fields unless explicitly asked to
2015-11-05 18:10:13 -08:00
8f231fd3e3
cleanup druid codebase
2015-11-04 13:59:53 -08:00
8defe29270
Merge pull request #1901 from guobingkun/fix_typo_and_rename
...
Fix metadata typo and rename default extension directory
2015-11-03 14:02:11 -08:00
962f65cc76
fix metadata typo and rename default extension directory
2015-11-03 14:50:42 -06:00
cec09a9967
Merge pull request #1804 from himanshug/objectify_index_creators
...
static to non-static conversion for methods in Index[Merger/Maker/IO]
2015-11-03 11:25:32 -08:00
8b67417ac8
make methods in Index[Merger,Maker,IO] non-static so that they can have
...
appropriate ObjectMapper injected instead of creating one statically
2015-11-02 23:24:26 -06:00
e03fc2032f
changed equals/hashCode implementation
2015-11-02 17:21:35 +09:00
69c86716d6
addressed comments
2015-11-02 14:23:13 +09:00
032c3e986d
Make 'search' filter have a case sensitive option( #1878 )
2015-10-30 16:38:54 +09:00
25a0eb7ed5
Merge pull request #1799 from dclim/nested-groupby-aggregator-fix
...
Support multiple outer aggregators of same type and provide more help…
2015-10-29 18:01:31 -07:00
59872bd0cd
Merge pull request #1809 from metamx/fifoPriorityExecutorService
...
Make PrioritizedExecutorService optionally FIFO
2015-10-27 15:19:32 -07:00
060402a216
Merge pull request #1855 from himanshug/fix_having_specs
...
fix [GreaterThan,LessThan,Equals] HavingSpecs
2015-10-27 14:46:04 -07:00
ecdafa87c5
Make PrioritizedExecutorService optionally FIFO
2015-10-27 14:16:22 -07:00
a71c7270b9
making [GreaterThan,LessThan,Equals] HavingSpecs more robust by carefully using long vs float for comparison
2015-10-27 13:15:13 -05:00
5a082b2f5e
Merge pull request #1824 from metamx/UniformGranularitySpecHashEquals
...
Add hashCode and equals to UniformGranularitySpec
2015-10-26 09:34:01 -07:00
5f23703216
Merge pull request #1638 from guobingkun/remove_maven_client_code
...
Remove Maven client at runtime + Provide a way to load Druid extensions through local file system
2015-10-26 09:30:05 -07:00
7cecc55045
Add segment merge time as a metric
...
Add merge and persist cpu time
Fix typo
review comment
move cpu time measuring to VMUtils
review comments.
2015-10-22 12:28:03 +05:30
4914925d65
New extension loading mechanism
...
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
e4ac78e43d
bump next snapshot to 0.9.0
2015-10-20 13:46:13 -07:00
46ecdfa757
add comment explaining logic
2015-10-15 16:04:06 -06:00
4c2c7a2c37
update version to 0.8.3
2015-10-14 21:40:55 -07:00
f432b8e3f9
Add hashCode and equals to UniformGranularitySpec
...
* Also add hashCode != 0 to AllGranularity and NoneGranularity
2015-10-13 16:42:21 -07:00
c9d6994040
Merge pull request #1821 from himanshug/storage_adapter_update
...
cache max data timestamp in QueryableIndexStorageAdapter
2015-10-13 10:52:43 -07:00
490de1f98a
support multiple non-consecutive intervals in outer query of nested group-by
2015-10-13 10:16:06 -05:00
fbba30eb60
cache max data timestamp in QueryableIndexStorageAdapter so that TimestampCheckingOffset
...
does not have to get it per cursor.
2015-10-12 15:34:22 -05:00
8ed5d2c06a
Add hashCode and equals to stock lookups
2015-10-12 10:29:39 -07:00
2737fd83f5
in the IndexSizeExceededException put maxRowCount to confirm if it is correctly picked up from configuration
2015-10-06 15:23:14 -05:00
8654732ef6
make IndexSizeExceededException constructor take formatString and arguments than just fixed String
...
like ISE, IAE etc
2015-10-06 13:44:22 -05:00
f4e0a76820
Support multiple outer aggregators of same type and provide more helpful exception when the same inner aggregator is referenced by multiple types of outer aggregators
2015-10-01 15:15:12 -06:00
774765dc40
GroupByQueryRunnerTest for hyperUnique finalizing post aggregators
2015-10-01 00:09:29 -04:00
e3bb93e8c7
Revert "Merge pull request #1781 from dclim/nested-groupby-multiple-same-aggregator-fix-v2"
...
This reverts commit dae488b7c0397be4b897
2015-10-01 00:05:59 -04:00
8e20a1e1f3
Use DoubleSumAggregatorFactory instead of CountAggregatorFactory, add test for non-integers
2015-09-30 17:11:39 -06:00
70ae5ca922
Fix failure in nested groupBy with multiple aggregators with same fieldName
...
Version 2 - Throws an exception if an outer query references an
aggregator that doesn't exist in the inner query, and then uses the
inner query aggregator names to form the columns for the intermediate
incremental index.
Also deleted all the getRequiredColumns() methods which are no longer
being used.
We do something wacky by adding an aggregator factory for the post
aggregators when building the intermediate incremental index, otherwise
queries on post aggregate results fail because the data isn't in the
incremental index.
Closes #1419
2015-09-30 15:43:11 -06:00
8199ecf1a4
Merge pull request #1782 from jon-wei/smq_cachekey
...
Add analysisTypes to SegmentMetadataQuery cache key
2015-09-29 15:51:35 -07:00
41ff271339
Add analysisTypes to SegmentMetadataQuery cache key
2015-09-29 14:33:35 -07:00
2d847ad654
Merge pull request #1730 from metamx/union-queries-fix
...
fix #1727 - Union bySegment queries fix
2015-09-29 12:23:25 -07:00
573aa96bd6
fix #1727 - Union bySegment queries fix
...
Fixes #1727 .
revert to doing merging for results for union queries on broker.
revert unrelated changes
Add test for union query runner
Add test
remove unused imports
fix imports
fix renamed file
fix test
update docs.
2015-09-29 23:32:36 +05:30
62d4ced4dd
Separate ListColumnIncluderator cache key parts with nul bytes
2015-09-29 13:59:58 -04:00
e6a6284ebd
Allow SegmentMetadataQuery to skip cardinality and size calculations
2015-09-22 13:51:55 -07:00
aaa8a88464
Merge pull request #1739 from jon-wei/segment_realtime
...
Allow SegmentAnalyzer to read columns from StorageAdapter, allow SegmentMetadataQuery to query IncrementalIndexSegments on realtime node
2015-09-17 18:36:53 -07:00
df4c2bab10
Soften concurrency requirements on IncrementalIndexTest
2015-09-17 15:51:07 -07:00
367c50d4ba
Allow SegmentAnalyzer to read columns from StorageAdapter, allow SegmentMetadataQuery to query IncrementalIndexSegments on realtime node
2015-09-16 18:39:31 -07:00
6e1eb3b7fe
Add better concurrency testing to IncrementalIndexTest
2015-09-16 14:04:20 -07:00
9705c5139b
Merge pull request #1732 from jon-wei/segmentmeta
...
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-16 12:36:25 -07:00
8b071a7230
Merge pull request #1710 from metamx/incrementalIndexConcurrentTestLatching
...
Add some basic latching to concurrency testing in IncrementalIndexTest
2015-09-15 13:55:52 -07:00
193fb4fdfc
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-14 19:41:42 -07:00
bd605a097e
Merge pull request #1731 from metamx/regex-extraction-npe
...
fix NPE with regex extraction function
2015-09-14 15:55:05 -07:00
08a527d01a
fix NPE with regex extraction function
2015-09-14 14:45:30 -07:00
e569f4b6a7
Add dimension extraction functionality to SearchQuery
...
* Add IdentityExtractionFn
2015-09-14 11:36:15 -07:00
5ff92664f8
Merge pull request #1696 from metamx/cpuTimeReporting
...
Add CPU time to metrics for segment scanning.
2015-09-14 10:53:55 -05:00
34ef81572d
Merge pull request #1700 from himanshug/update_agg_test_helper
...
update indexing in the helper to use multiple persists and merge
2015-09-14 06:56:29 -07:00
8d3cdd8572
Don't check for sortedness if we already know GenericIndexedWriter isn't sorted
2015-09-11 16:32:09 -07:00
d6849805ea
Add some basic latching to concurrency testing in IncrementalIndexTest
2015-09-10 10:06:51 -07:00
5da58e48e0
use Rule based TemporaryFolder for cleanup of temp directory/files
2015-09-09 11:10:33 -05:00
44911039c5
update indexing in the helper to use multiple persists and final merge to
...
catch further issues in aggregator implementations
2015-09-09 11:10:33 -05:00
fcf5cae81d
Add CPU time to metrics for segment scanning.
2015-09-08 13:34:19 -07:00
4f61b42f40
Merge pull request #1578 from b-slim/fix_extraction_filter_2
...
Fix UT and documentation to the extraction filter
2015-09-01 10:46:20 -07:00
04ff6cd355
Merge pull request #1685 from gianm/close-loudly
...
Close output streams and channels loudly when creating segments.
2015-08-28 23:32:22 -05:00
940e1aa3eb
Replace funky imports with standard ones.
...
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
7d6fa2ba50
Close output streams and channels loudly when creating segments.
2015-08-28 17:14:03 -07:00
2e0dd1d792
adding UTs and addressing review comments to
...
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
2015-08-27 20:50:46 -05:00
2237a8cf0f
kafka 8 simple consumer firehose
2015-08-27 20:50:46 -05:00
c1388a1685
Merge pull request #1632 from Hailei/fix-subquery-innerquery-demension
...
Inner Query should build on sub query
2015-08-27 10:25:38 -07:00
2a866f49df
Downgrade Jackson to 2.4.6.
2015-08-26 18:25:55 -07:00
24aa762c79
Add test for #1632
2015-08-25 20:50:30 -07:00
51f6a9a2c9
update jackson to 2.6.1
2015-08-25 16:07:01 -07:00
c57c07f28a
add ability for client code to provide InputStream of input data in addition to File
...
It would be needed when input data file does not reside in the same jar
but you could still use getResourceAsStream() to read the data inside a file
2015-08-20 00:54:58 -05:00
3b2e41e42a
update for next release
2015-08-18 17:16:46 -07:00
7549f02578
support the case filter value is null
2015-08-17 15:09:37 -05:00
234a958817
Inner Query should build on sub query
2015-08-17 18:18:26 +08:00
db19d2d547
Revert "Update to guice 4.0"
2015-08-14 09:26:07 -07:00
be89105621
Merge pull request #1602 from metamx/more-code-cleanup
...
Some perf Improvements in Broker
2015-08-11 13:51:49 -07:00
fbdb841928
Merge pull request #1603 from metamx/optimize-lexicographic-topN
...
Optimizations for LexicographicTopNs
2015-08-11 13:35:34 -07:00
b8d8a8da9e
Optimisations for LexicographicTopNs
...
initial review for perf optimizations for lexicographic TopNs
fix compilation
create map with proper size
review comment
review comment
review comments
2015-08-12 00:37:48 +05:30
7e61216287
Update to guice 4.0
...
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
2015-08-10 13:57:18 -07:00
f0bc362981
clean code if is not needed anymore
2015-08-07 12:38:41 -05:00
64d638a386
optimize makeMatcher
2015-08-06 17:04:36 -05:00
1a46c4c71c
avoid creating mergeSeqence when not required
2015-08-06 14:25:13 +05:30
83de5a4716
addressing reviewers comments
2015-08-03 09:03:28 -05:00
dda0790a60
Fix extractionFilter by implementing make matcher
...
Fix getBitmapIndex to consider the case were dim is null
Unit Test for exractionFn with empty result and null_column
UT for TopN queries with Extraction filter
refactor in Extractiuon fileter makematcher for realtime segment and clean code in b/processing/src/test/java/io/druid/query/groupby/GroupByQueryRunnerTest.java
fix to make sure that empty string are converted to null
2015-08-03 09:02:17 -05:00
d11d9b6c45
dont waste memory in storing all lines from input
...
CharSource.readLines() reads all lines from input into a in-memory list
Since we need an iterator here, so this wastage can be easily prevented
2015-07-20 21:59:38 -05:00
0481c8ca26
Merge pull request #1406 from zhaown/fix-breaking-while-exceeding-max-intermediate-rows
...
Fix breaking while exceeding max intermediate rows.
2015-07-20 13:41:22 -07:00
f7a92db332
generic byte[] serde for InputRow
2015-07-20 12:01:53 -05:00
0439e8ec23
adding serde methods for intermediate aggregation object to ComplexMetricSerde
...
This provides the alternative to using ComplexMetricSerde.getObjectStrategy()
and using the serde methods from ObjectStrategy as that usage pattern is deprecated.
2015-07-20 12:01:53 -05:00
524b05f073
Fix breaking while exceeding max intermediate rows.
2015-07-19 10:41:53 +08:00
e21195f987
Merge pull request #1469 from guobingkun/table_config
...
Inconsistent property names for "druid.metadata.storage.tables.xxx"
2015-07-17 07:43:19 -07:00
19af3bc9bc
Merge pull request #1535 from metamx/alphanum-docs-tests
...
Update alphanumeric sort docs + more tests / examples
2015-07-16 22:09:41 -05:00
2c464ad936
correct reference in docs + more tests / examples
2015-07-16 19:50:05 -07:00
9616c10b1d
remove import static
2015-07-16 17:46:21 -07:00
c1308203b8
Merge pull request #1532 from metamx/fixTopNDimExtractionDoubleApply
...
Fix TopN dimension extractions being applied twice
2015-07-16 13:39:02 -07:00
3a0793aaf9
Merge pull request #1533 from metamx/extraCheckGroupByDimExtraction
...
Add more unit tests for group by
2015-07-15 21:09:00 -07:00
7d0b77c261
Add more unit tests for group by
2015-07-15 20:15:21 -07:00
a15a2c4047
fix histogram aggregator cache key
2015-07-15 17:33:36 -07:00
9092c665b7
Fix TopN dimension extractions being applied twice
2015-07-15 16:58:15 -07:00
456ad9ffba
Merge pull request #1529 from metamx/update-versions
...
inrement version
2015-07-15 13:25:31 -07:00
4cfb00bc8a
inrement version
2015-07-15 13:09:05 -07:00
5eadd395e2
Move lots of executor service creation to Execs
2015-07-14 15:38:49 -07:00
184b12bee8
fix groupBy caching to work with renamed aggregators
...
Issue - while storing results in cache we store the event map which
contains aggregator names mapped to values. Now when someone fire same
query after renaming aggs, the cache key will be same but the event
will contain metric values mapped to older names which leads to wrong
results.
Fix - modify cache to not store raw event but the actual list of values
only.
review comments + fix dimension renaming
review comment
2015-07-09 11:48:26 +05:30
9789417612
ModuleList is already part of Initialization
2015-07-01 11:37:40 -07:00
2c463ae435
Merge pull request #1489 from metamx/moveTestPackages
...
Move some test packages
2015-07-01 11:18:09 -07:00
5e19a615f1
Add coments to DimExtractionTopNAlgorithm
2015-07-01 10:32:45 -07:00
7a2a8a3d6e
Move extraction tests to more reasonable package
2015-07-01 10:30:50 -07:00
4a0ae7d8d5
Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code
2015-06-29 10:12:30 -05:00
28fa1642b9
add node time metrics to DirectDruidClient
2015-06-26 17:57:44 -07:00
36b4453789
Merge pull request #1455 from druid-io/fix-protobuf
...
Fix protobuf impl and docs
2015-06-22 23:15:40 -07:00
f9cdb0ad61
test for #1120
...
Make the changes described in #1120 to add test for the issue described
there.
2015-06-21 23:34:21 +05:30
9c74993559
fix protobuf impl and docs
2015-06-20 21:59:38 -07:00
0a5bb909a2
[maven-release-plugin] prepare for next development iteration
2015-06-18 17:35:19 -07:00
59c6b2b279
[maven-release-plugin] prepare release druid-0.8.0-rc1
2015-06-18 17:35:14 -07:00
6230ac90ae
Use IndexMerger for conversion
2015-06-10 11:34:58 -07:00
395ba79f8b
Merge pull request #1403 from metamx/mergerMakerTests
...
Improvements around resource handling in IndexMerger / IndexIO / QueryableIndex
2015-06-04 15:59:10 -07:00
ed8eb5c991
Improvements around resource handling in IndexMerger / IndexIO / QueryableIndex
...
* Fix resource leak in `io.druid.segment.IndexIO.DefaultIndexIOHandler#validateTwoSegments(java.io.File, java.io.File)`
* Un-deprecate `close()` in `QueryableIndex` and make it inherit `Closeable`
* Fix resource leaks in various unit tests
* Add `CloserRule` for closing out resources
2015-06-04 14:18:27 -07:00
50ad0e6474
Merge pull request #1412 from pjain1/alphaNumericTopN_NPE_fix
...
NPE fix for TopN query with alphaNumericTopN metric spec
2015-06-04 09:49:31 -05:00
a7b09e857c
NPE fix for alphaNumericTopN when pervious stop is not specified
2015-06-04 09:30:31 -05:00
35e2fde18e
Merge pull request #1386 from himanshug/aggregation_testing1
...
General class for testing any Aggregation Implementation
2015-06-03 23:43:36 -07:00
92d7316ed8
Merge pull request #1414 from metamx/timeout2TIMEOUT
...
Replace "timeout" with QueryContextKeys.TIMEOUT
2015-06-02 17:11:09 -07:00
1c4d42bc15
Replace "timeout" with QueryContextKeys.TIMEOUT
2015-06-02 14:49:21 -07:00
f48db09e35
Add optimizations for ExtractionFn by enabling MANY_TO_ONE vs ONE_TO_ONE codepaths
...
* Also adds LookupExtractionFn and MapLookupExtractor which takes in an explicit mapping of renames
* Add injective to javascript extraction fn
2015-06-02 12:22:56 -07:00
215c1ab01e
UTs for hyperUnique aggregation
2015-06-01 12:52:40 -05:00
160d5fe6b7
a general class for testing any [complex] aggregation implementation
2015-06-01 12:52:40 -05:00
55292bba13
Add more IndexMergerTests
2015-05-28 18:18:20 -07:00
1ebe622c7d
Add checkin GroupByQuery for null DimensionSpec in dimension list
2015-05-28 14:55:34 -07:00
f9c624c7db
Merge pull request #1361 from mrijke/groupby-limithavingorder-unittest
...
GroupBy Query with Having/Limit/Orderingspec inconsistencies (UnitTest)
2015-05-27 14:49:18 -07:00
1a3f04f0ed
Merge pull request #1354 from metamx/multi-valued-dimension-compression
...
Enabling compression for multiValued dimension
2015-05-26 23:43:53 -07:00
fd64c24e43
Fix roaring extraction filter on empty values
2015-05-26 13:54:18 -07:00
81415282aa
Enabling compression for multiValued dimension
...
Add test and refactoring
Add benchmark tests
2015-05-27 00:09:14 +05:30
e97d22a10a
Fix Extraction Filter cast problems for empty results
2015-05-22 15:20:11 -07:00
e1399b7ce4
Add unit test to show breaking Dimension Extraction Filter
2015-05-22 15:02:11 -07:00
75c092ccb1
Merge pull request #1375 from metamx/MetricManipulatorFnInstances
...
Modify MetricManipulatorFns to use instanced classes
2015-05-22 15:56:47 -04:00
042653ebcb
Modify MetricManipulatorFns to use instanced classes
2015-05-22 12:38:38 -07:00
723df735e9
force eagerness of processing of SegmentMetadata queries on the processing executor by converting the Sequence into List
2015-05-22 13:46:26 -05:00
5852b64852
adding UT for SegmentMetadata bySegment query which catches following regression caused by commit 55ebf0cfdf
...
it fails when we issue the SegmentMetadataQuery by setting {"bySegment" : true} in context with exception -
java.lang.ClassCastException: io.druid.query.Result cannot be cast to io.druid.query.metadata.metadata.SegmentAnalysis
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$4.compare(SegmentMetadataQueryQueryToolChest.java:222) ~[druid-processing-0.7.3-SNAPSHOT.jar:0.7.3-SNAPSHOT]
at com.google.common.collect.NullsFirstOrdering.compare(NullsFirstOrdering.java:44) ~[guava-16.0.1.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:46) ~[java-util-0.27.0.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:42) ~[java-util-0.27.0.jar:?]
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649) ~[?:1.7.0_80]
2015-05-22 13:45:54 -05:00
da0cc32bc8
Revert commit 55ebf0cfdf
...
which caused following regression
it fails when we issue the SegmentMetadataQuery by setting {"bySegment" : true} in context with exception -
java.lang.ClassCastException: io.druid.query.Result cannot be cast to io.druid.query.metadata.metadata.SegmentAnalysis
at io.druid.query.metadata.SegmentMetadataQueryQueryToolChest$4.compare(SegmentMetadataQueryQueryToolChest.java:222) ~[druid-processing-0.7.3-SNAPSHOT.jar:0.7.3-SNAPSHOT]
at com.google.common.collect.NullsFirstOrdering.compare(NullsFirstOrdering.java:44) ~[guava-16.0.1.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:46) ~[java-util-0.27.0.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:42) ~[java-util-0.27.0.jar:?]
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649) ~[?:1.7.0_80]
2015-05-22 13:39:34 -05:00
82da479464
Fix for GroupBy with Having+Limit+Orderspec
...
* Inverted function arguments to compose postProcFn for GroupBy queries
with havingspec + limitspec.
* Replaced query.getLimitSpec() with null in GroupByQueryToolChest's
mergeGroupByResults
* Added unittest to verify functionality
2015-05-19 18:35:48 +02:00
2fd3e9e8e5
return size = 0 in ColumnAnalysis if its unknown
...
that is if complex agg did not implement inputSizeFn() so
that segment metadata query shows atleast some information.
also instead of COMPLEX, return type of data stored.
2015-05-15 20:11:56 -05:00
3c3db7229c
Merge pull request #1355 from himanshug/long_max_min_aggregators
...
Long max/min aggregators
2015-05-13 12:08:11 -07:00
cebb550796
additional UTs for [DoubleMax/DoubleMin] aggregation
2015-05-13 09:25:41 -05:00
d0ec945129
adding aliases doubleMax and doubleMin for max and min respectively
...
renamed all [Max/Min]*.java to [DoubleMax/DoubleMin]*.java and created [Max/Min]AggregatorFactory.java which can be removed when we dont need the min/max aggregator type backward compatibility
2015-05-13 09:25:41 -05:00
2de38f7d29
UTs for long[Max/Min] aggregation
2015-05-13 09:25:22 -05:00
00436f93e2
long max/min aggregators implementation
2015-05-13 09:25:22 -05:00
7a6acf5c1b
update pom to 0.8
2015-05-11 19:41:58 -06:00
33265d63e1
Merge pull request #1262 from metamx/fix-null-dimension
...
fix handling of dimension having only null values
2015-05-06 13:51:26 -07:00
34be1e96fa
fix NPE
...
review comments
Add test
fix test for java8
2015-05-05 23:11:13 +05:30
8f8400e24e
fix handling of dimension having only null values
...
fixes #1211
fix value matcher
more improvements
more fixes for partial null column
fix handling of dimension having only null values
fixes #1211
fix value matcher
more improvements
more fixes for partial null column
review comment
IndexMaker speedups
* About 15% speedup
Conflicts:
processing/src/main/java/io/druid/segment/IndexMaker.java
fix handling of dimension having only null values
fixes #1211
fix value matcher
more improvements
more fixes for partial null column
fix handling of dimension having only null values
fixes #1211
fix value matcher
more improvements
more fixes for partial null column
review comment
review comments
review comment
fix failing tests
review comment
fix compilation
2015-05-04 22:07:45 +05:30
50158357ff
fixes #1330
...
fixes #1330 ,
Avoid creating Period instance as creating a Period from Long.MAX_VALUE
throws arithmetic exception.
After this query metric will emit duration in seconds instead of
minutes.
2015-05-04 20:34:28 +05:30
721505c017
Merge pull request #1208 from druid-io/rework-metrics
...
Schemaless metrics + additional metrics for things we care about
2015-04-27 15:04:54 -07:00
963e5765bf
Schemaless metrics + additional metrics for things we care about
2015-04-27 13:39:40 -07:00
27016c0289
Fix IndexIO segment validator to account for timestamp mismatches.
2015-04-27 12:42:16 -07:00
633fdb029e
Add option to ConvertSegmentTask to skip validation
...
* Validation is enabled by default
2015-04-27 08:37:55 -07:00
303727e6a9
IndexMaker speedups
...
* About 15% speedup
Conflicts:
processing/src/main/java/io/druid/segment/IndexMaker.java
2015-04-23 13:19:21 -07:00
f2300430d1
Cleanup some code in index creation.
...
* Add some unit tests
* Add io.druid.segment.IndexMerger.reprocess for quick re-indexing of data
* Add dim-value validation to validation checker (instead of ONLY index #)
* General code refactoring to make things a little easier to read
2015-04-23 12:41:42 -07:00
7939f43681
Merge pull request #1296 from druid-io/limit-test
...
Add test for order by metric and limit across multiple days
2015-04-22 11:28:06 -07:00
97d87a06d0
Add another test for limit across multiple days
2015-04-22 11:27:37 -07:00
28f69d6bd3
Merge pull request #1299 from metamx/improve-filter-datasource-metadata
...
Improve filtering of segments for dataSourceMetadataQuery
2015-04-22 11:07:35 -07:00
a0a28de551
fix serde issue when pulling timestamps from cache
2015-04-22 11:03:26 -07:00
2b4406671e
Merge pull request #1301 from druid-io/fix-type
...
fix count agg factory type
2015-04-21 09:24:20 -07:00
7805357ab1
fix count agg factory type
2015-04-21 09:23:04 -07:00
bb8c0cb50b
Improve filtering of segments for dataSourceMetadataQuery
...
dataSourceMetadataQuery only needs to be executed on latest segments at
present, modify filterSegments and add test.
2015-04-21 09:31:13 +05:30
f73f14ab91
Merge pull request #1297 from metamx/versionConverterTaskUpdates
...
Update VersionConverterTask for IndexSpec and allowing Forced updates
2015-04-20 16:44:35 -07:00
7479ac9012
Update VersionConverterTask for IndexSepc and allowing Forced updates
2015-04-20 16:17:06 -07:00
d260515a43
update druid-api version
2015-04-17 14:58:35 -07:00
cf155e4eba
Fix an issue that after broker forwards GroupByQuery to historical, havingSpec is still applied
...
on postAggregations which are removed in the forwarded query.
Add a unit test to replicate the issue.
Add a query that can replicate this issue into integration test.
2015-04-17 13:00:41 -05:00
f0a19349bf
fix up some comments for contributed test
2015-04-16 15:07:09 -07:00
90b17a5259
Merge pull request #1285 from venkateshk/limitspec-tests
...
Unit test to surface bug with limit-spec order by over specific query intervals
2015-04-16 13:52:58 -07:00
1d153674b6
remove overzealous check for backwards compatibility
2015-04-15 22:11:55 -07:00
ea5572d001
Merge pull request #1271 from metamx/strictErrorChecking
...
Add stricter checking for potential coding errors
2015-04-15 15:21:41 -07:00
abdeaa0746
Add stricter checking for potential coding errors
...
Can use via `mvn clean compile test-compile -P strict'
2015-04-15 14:52:25 -07:00
a2ba5b6183
Unit test to surface bug with limit-spec order by over specific query intervals
2015-04-15 06:31:22 -07:00
3a3046ccf3
add support for dimension compression
...
- compression for single-value dimensions using CompressedVSizeIntsIndexedSupplier
- makes dimension compression configurable via IndexSpec
- IndexSpec also enables configuring bitmap and metric compression
2015-04-14 10:44:18 -07:00
bafc5114b4
add toString, equals, and hashCode to BitmapSerdeFactory
2015-04-14 10:44:18 -07:00
d20128b89b
add compressed variable-size ints column type
2015-04-14 10:44:18 -07:00
ce928d9636
add compressed ints column type
2015-04-14 10:44:17 -07:00
5c23679238
add WritableSupplier and IndexedMultivalue
2015-04-14 10:44:17 -07:00
1abb9cce7c
make IndexedInts closeable + add fill method
2015-04-14 10:44:17 -07:00
ed0d49933e
fix memory leak in CompressedXXXIndexedSupplierTest
2015-04-14 10:44:16 -07:00
6790e6cf0f
add fromList to CompressedLongsIndexedSupplier
2015-04-14 10:44:16 -07:00
7517f0d0f0
Add some javadoc to the two Query processing interfaces to help aid in implementations of new Queries.
...
Also, remove some comments that did not have enough context to actually make sense to anyone but the original author (at least, I hope they make sense to the author, I definitely don't know what was being said).
2015-04-09 18:11:42 -07:00
208e307915
Merge pull request #1251 from metamx/uriSegmentLoaders
...
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
2015-03-30 17:43:51 -07:00
aea7f9d192
[maven-release-plugin] prepare for next development iteration
2015-03-30 16:35:24 -07:00
060d7aef03
[maven-release-plugin] prepare release druid-0.7.1
2015-03-30 16:35:20 -07:00
1c6cbea89c
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
...
This reverts commit f904bc7858
2015-03-30 13:40:04 -07:00
f904bc7858
Revert "Overhaul of SegmentPullers to add consistency and retries"
2015-03-30 13:15:50 -07:00
6d407e8677
Add URI handling to SegmentPullers
...
* Requires https://github.com/druid-io/druid-api/pull/37
* Requires https://github.com/metamx/java-util/pull/22
* Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl
* General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can
2015-03-30 12:33:23 -07:00
e5653f0752
Merge pull request #1190 from vigiglobe/master
...
Fix NPE when partionNumber 0 does not exist.
2015-03-26 13:25:39 -07:00
389ea4c32f
Merge pull request #1245 from b-slim/fix_injector_plus_ut
...
Bug fix @DruidSecondaryModule plus unit test
2015-03-26 10:04:44 -07:00
a9c47de571
Merge pull request #1243 from metamx/fix-union-timeline-lookup
...
fixes TimeboundaryQuery and DataSourceMetadata queries returning wrong values for union queries
2015-03-26 10:02:56 -07:00
1e6be7796e
bug fix @DruidSecondaryModule plus unit test
2015-03-26 10:44:52 -05:00
638bf9d4e9
return sorted List of TimeLineObjectHolder
2015-03-26 11:51:09 +05:30
942c17a2aa
Remove timeline chunk count assumptions.
...
* Replace with generic iterables
2015-03-24 22:40:49 +01:00
9983216871
use https maven repo URL to download dependencies
2015-03-20 14:09:07 -04:00
b389cfe404
[maven-release-plugin] prepare for next development iteration
2015-03-19 12:38:17 -07:00
60e7d543cc
[maven-release-plugin] prepare release druid-0.7.1-rc1
2015-03-19 12:38:13 -07:00
39e60b3405
fix race in groupByParallelQueryRunner
...
add UT and use a queue for better concurrency
2015-03-17 20:57:05 +05:30
127b6fd857
Merge pull request #1172 from himanshug/segment_metadata_eager
...
force eager the processing of segment metadata query on the processing executor
2015-03-12 10:19:48 -07:00
0a5a3fe2dc
fix file missing from rebase
2015-03-11 17:30:11 -07:00
e01ed16030
serde tests + equals/hashCode fixes for extraction functions
2015-03-11 16:48:28 -07:00
d3f5bddc5c
Add ability to apply extraction functions to the time dimension
...
- Moves DimExtractionFn under a more generic ExtractionFn interface to
support extracting dimension values other than strings
- pushes down extractionFn to the storage adapter from query engine
- 'dimExtractionFn' parameter has been deprecated in favor of 'extractionFn'
- adds a TimeFormatExtractionFn, allowing to project the '__time' dimension
- JavascriptDimExtractionFn renamed to JavascriptExtractionFn, adding
support for any dimension value types that map directly to Javascript
- update documentation for time column extraction and related changes
2015-03-11 16:45:42 -07:00
55ebf0cfdf
force eager the processing of segment metadata query on the processing threadpool by using ChainedExecutionQueryRunner in SegmentMetadataQueryRunnerFactory.mergeRunners(..)
2015-03-11 12:58:58 -05:00
217e674063
Handling aggregators and post aggregators with duplicate names
...
* add test for same-name groupBy hyperUniques post-agg
* add test for same-name post-agg in groupby with approx histogram
* Fixes https://github.com/druid-io/druid/issues/1045
* Throws an error if post aggs and aggs do not have unique names
* Add more groupBy tests for Having filters
2015-03-10 17:10:43 -07:00
0b467624ec
Merge pull request #694 from druid-io/arithmetic-op-strategies
...
normal division & configurable ordering for ArithmeticPostAggregator
2015-03-10 13:48:27 -07:00
2abdce1dc0
Merge pull request #1180 from metamx/logging-groupBy-NPE
...
add null check early to catch root cause for groupBy NPE while running bySegment query
2015-03-09 09:16:33 -07:00
6e935cca0a
add null check early to catch root cause
2015-03-09 21:10:28 +05:30
0d47c0c36d
normal division and configurable ordering for ArithmeticPostAggregator
...
Fixes #510
2015-03-04 12:44:24 -08:00
d685e2ab04
Merge pull request #1165 from friedhardware/fix-NPerror-select
...
Added null check for the pagingSpec on a Select Query.
2015-03-02 14:17:06 -08:00
e8605c63a9
Merge pull request #1150 from himanshug/broker-parallel-chunk-process
...
interval chunk query runner now processes individual chunk in a threadpool
2015-03-02 13:50:23 -08:00
29039fd541
interval chunk query runner now processes individual chunk in a thread pool and prints metrics query/time per chunk
2015-03-02 15:45:09 -06:00
e6130e0fdc
Added null check for the pagingSpec on a Select Query.
2015-03-02 12:41:59 -08:00
005f4da2c0
Merge pull request #1143 from metamx/update-rhino-1.7rc5
...
Update Rhino to 1.7RC5
2015-02-25 12:50:23 -08:00
b167dcf82c
[maven-release-plugin] prepare for next development iteration
2015-02-23 14:28:06 -08:00
e81ac2ba43
[maven-release-plugin] prepare release druid-0.7.0
2015-02-23 14:27:58 -08:00
562de6c621
Update docs and examples for log4j2 usage.
...
- Put configs early in classpath in examples so log4j2.xml will get picked up properly
- Add an example log4j2.xml file.
- Update Logging doc.
2015-02-19 11:40:56 -07:00
c4d721fffd
update Rhino to 1.7RC5
2015-02-19 09:48:18 -08:00
78df7f6165
Move Druid release artifacts to Sonatype
...
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
2015-02-13 14:26:31 -08:00
d29740ed9f
[maven-release-plugin] prepare for next development iteration
2015-02-12 16:16:00 -08:00
211fd15b7e
[maven-release-plugin] prepare release druid-0.7.0-rc3
2015-02-12 16:15:56 -08:00
90bc62eb5c
Merge pull request #1108 from metamx/improve-groupby-perf
...
Improve groupby by removing conversion to case insensitive row
2015-02-12 11:45:20 -08:00
15cf432b74
remove conversion to case insensitive row
...
this is not required after death to casing in 0.7
2015-02-11 19:40:36 +05:30
c5e99bf6ec
Merge pull request #1105 from metamx/fixEmptyExtractionFilter
...
Fix empty results on ExtractionFilter.
2015-02-10 14:25:58 -08:00
b9cb311a52
Fix empty results on ExtractionFilter.
...
* Now returns empty results rather than erroring out
* Added unit tests for multiples case
2015-02-10 14:04:38 -08:00
708759e1e0
Update http-client to 1.0.0
2015-02-10 13:36:47 -08:00
a7dcaffb53
fix `__time` column selector for incremental index
...
- also adds tests for selecting the time column
2015-02-06 12:06:05 -08:00
42e902b6e3
Merge pull request #1090 from metamx/alphanum-attribution
...
update code attribution
2015-02-04 15:51:34 -08:00
0fbc6071c9
update code attribution
2015-02-04 15:28:44 -08:00
25cf15824b
Merge pull request #1085 from gianm/dsmrv-fix
...
DataSourceMetadataResultValue fixes and JodaUtils adjustments.
2015-02-03 17:51:33 -08:00
085ad8d345
Fix DataSourceMetadataResultValue serde.
2015-02-03 17:39:42 -08:00
1f12c5b2f1
[maven-release-plugin] prepare for next development iteration
2015-02-03 12:06:49 -08:00
e82d431be7
[maven-release-plugin] prepare release druid-0.7.0-rc2
2015-02-03 12:06:41 -08:00
4eff269536
Merge pull request #1079 from druid-io/cleanup-deps
...
Remove non friendly dependencies from Druid
2015-02-03 11:56:41 -08:00
3e5d338c8e
Remove non friendly dependencies from Druid
2015-02-03 11:36:08 -08:00
71b4c5fa86
Merge pull request #1076 from metamx/remove-threadlocals
...
remove thread-locals in GenericIndexed in favor of wrapped objects
2015-02-02 20:02:33 -08:00
cb2e300eba
remove thread-locals in GenericIndexed in favor of wrapped objects to reduce GC pressure
2015-02-02 15:59:30 -08:00
42eba986ce
Towards consistent null handling
...
This commit also includes
1) the addition of a context parameter on timeseries queries that allows it to ignore empty buckets instead of generating results for them
2) A cleanup of an unused method on an interface
2015-02-02 12:53:07 -08:00
92e616de11
Merge pull request #1077 from metamx/remove-unused-imports
...
remove unused imports
2015-02-02 10:45:27 -08:00
ba932bb1f2
remove unused imports
2015-02-02 21:53:39 +05:30
d05032b98a
towards a community led druid
2015-01-31 20:57:36 -08:00
f24a89a22a
fix NPE for topN over missing hyperUniques column
2015-01-27 16:12:41 -08:00
226dd91a31
Add a hash map for storing groupBy partition index
...
* Improves groupBy performance by approx 15%
2015-01-26 08:42:02 -08:00
1f94de22c6
[maven-release-plugin] prepare for next development iteration
2015-01-20 14:23:55 -08:00
17476edc31
[maven-release-plugin] prepare release druid-0.7.0-rc1
2015-01-20 14:23:51 -08:00
3d27747f7e
Upgrade to log4j2
...
Default behavior is as before.
Added documentation for how to enable synchronous logging for select chatty classes:
* io.druid.client.ServerInventoryView
* io.druid.client.BatchServerInventoryView
* io.druid.curator.inventory.CuratorInventoryManager
* com.metamx.http.client.pool.ChannelResourceFactory
2015-01-20 12:35:18 -08:00
91a79dbf95
Merge pull request #1031 from metamx/ingestmetadata-query
...
DataSourceMetadata query
2015-01-19 21:55:35 -08:00
7bb038756c
Account for very slow writer threads in IncrementalIndexTest
2015-01-17 13:02:59 -08:00
b4041c13e5
Merge pull request #1029 from metamx/fixChainedExecutionQueryRunnerTest
...
Address spurious test failures
2015-01-16 13:08:32 -08:00
3b3aad78cb
Merge pull request #1027 from metamx/concurrentOnHeapIncrementalIndexFix
...
Fix concurrency issues in OnheapIncrementalIndex
2015-01-16 12:54:42 -08:00
197af967ef
Fix concurrency issues in OnheapIncrementalIndex
...
* Was encountering weird errors when fast writes were coming in while queries were happening.
* Added unit tests which tend to cause concurrency query problems
2015-01-16 12:01:46 -08:00
ebafa2a786
Fix spurious test failures in ChainedExecutionQueryRunnerTest
2015-01-15 16:49:16 -08:00
5bfcc43377
Merge pull request #1008 from metamx/stringConversionJavaUtilUpdate
...
Update all String conversions to and from byte[] to use the java-util StringUtils functions
2015-01-15 13:50:27 -08:00
c7452b75f6
Merge branch 'master' into ingestmetadata-query
2015-01-15 18:00:31 +05:30
d5f4182de4
global test timeouts + fix test race condition
2015-01-07 23:36:57 -08:00
852e863425
Merge pull request #981 from druid-io/strictModuleTyping
...
Use Module instead of generic Object in Guice related items
2015-01-05 12:43:20 -08:00
b1b5c9099e
Update all String conversions to and from byte[] to use the java-util StringUtils functions
...
* Speedup of GroupBy with javaScript filters by ~10%
* Requires https://github.com/metamx/java-util/pull/15
2015-01-05 11:22:32 -08:00
3fc6cf918d
add test for large chunks
2015-01-02 14:31:22 -08:00
f2f9cbeca8
throw error rather than returning garbage results
2015-01-02 14:29:21 -08:00
071943a367
fix LZF compression with buffers exceeding LZF chunk size
2015-01-02 11:39:50 -08:00
f2439899e7
fix bitmap factory serde
2014-12-23 15:07:32 -08:00
27a3169312
increase test timeouts
2014-12-19 17:09:43 -08:00
971afab36f
Lengthen CompressionStrategyTest::testKnownSizeConcurrency() to have 2m timeout on its test to account for shared Jenkins build lag
2014-12-19 12:53:20 -08:00
7c8d4a7433
Use Module instead of generic Object in Guice related items
2014-12-19 10:54:06 -08:00
be507b8cb4
Merge pull request #943 from mrijke/partialdimextractfn-nullpointer
...
Fix NullPointerException in PartialDimExtractionFn
2014-12-16 12:29:27 -07:00
80e4b68ee7
review comments
2014-12-16 21:16:48 +05:30
b3fe91bb50
Merge pull request #830 from metamx/union-merge-on-historical
...
Union merge on historical
2014-12-15 13:36:47 -07:00
3cb7999eb9
i hate hadoop dependencies
2014-12-15 09:52:46 -08:00
a0d3579a92
add docs + fix tests
2014-12-11 17:58:01 +05:30
7ad03087c0
Merge branch 'master' into ingestmetadata-query
2014-12-11 16:54:38 +05:30
32b4f55b8a
review comments refactoring
2014-12-11 16:33:14 +05:30
3763357f6e
Ingest metadata query implementation
2014-12-10 19:44:00 +05:30
d6d3ec6846
Merge pull request #948 from metamx/ingestion-docs
...
Redocumenting ingestion
2014-12-09 15:30:03 -07:00
9596c11f42
address cr
2014-12-09 14:19:18 -08:00
1a1b0e6f23
merge from master and review comments
2014-12-09 13:16:45 +05:30
1392e2731f
Merge pull request #936 from metamx/cachingRunnerImprovements
...
General Caching Query Runners cleanup (40% query time reduction for HLL)
2014-12-08 14:07:52 -08:00
7b65f0635d
General Caching Query Runners cleanup
...
* Add type strictness to CachingClusteredClient.
* Add background caching to CachingClusteredClient. Gives between 0% and 5% query speed increase.
* Add @BackgroundCaching annotation for injected ExecutorService items
* Add `numBackgroundThreads' configuration options to CacheConfig (default 0 aka same thread legacy behavior)
* Add unit tests for CacheConfig
* Add an abstract caching query runner class, currently it doesn't do anything exceppt simply make the two caching queries distinct.
* Add caching to CachingQueryRunner. Gives up to a WHOPPING 40% reduction in query time on HLL queries
* Updated docs with more info on cache settings.
2014-12-08 13:29:32 -08:00
90670a9c7e
Fix NullPointerException in PartialDimExtractionFn by explicity checking for dimValue == null, attempt 2
2014-12-08 22:26:35 +01:00
bd9bbf396c
Fix NullPointerException in PartialDimExtractionFn by explicity checking for dimValue == null
2014-12-08 20:11:58 +01:00
ad23e49777
use fixed-size mapdb cache to avoid heap growing uncontrollably
2014-12-05 15:34:50 -08:00
7cd45a6e1f
IncrementalIndex throws exception if limit exceeded
...
- For now uses a hardcoded ratio of aggregator to timeanddim buffer sizes
- canAppendRow is a workaround for realtime index since the
Firehose currently does not have a way of rolling back the last event in
case of error
- canAppendRow needs a fudge factor; there is a race between checking
if we can add a row and actually adding a row, because of the way MapDB
reports its size.
2014-12-04 14:38:16 -08:00
c7dbe6116c
write byte data as is in smile
2014-12-04 10:57:56 -08:00
c21a82a697
upgrade LZ4 to operate directly on ByteBuffers
2014-12-04 10:57:56 -08:00
0c521e0a77
update joda-time and fix min/max instant
2014-12-04 10:57:56 -08:00
269a51964e
fix size calculation
2014-12-04 17:22:24 +05:30
4dc0fdba8a
consider mapped size in limit calculation & review comments
2014-12-03 23:47:30 +05:30
529e7e0272
Merge pull request #927 from metamx/speedup-smile-bytes
...
Improve Smile serde performance by writing binary data as is
2014-12-03 10:02:08 -08:00
0f5d5840da
Merge pull request #924 from metamx/update-joda-time
...
Update Joda-Time and fix min/max instant overflow
2014-12-03 09:15:39 -08:00
da8bd7836b
Introduce buffer size
2014-12-03 16:28:22 +05:30
5fece517fa
write byte data as is in smile
2014-12-03 00:01:01 -08:00
18f50097a9
upgrade LZ4 to operate directly on ByteBuffers
2014-12-02 23:53:56 -08:00
bc173d14fc
a whole bunch of cleanup and fixes
2014-12-02 17:32:05 -08:00
a79389a9e5
update joda-time and fix min/max instant
2014-12-02 17:27:22 -08:00
b65933ffb8
make tests parameterised
2014-12-02 23:55:29 +05:30
6dc69c2f30
code cleanups & formatting
2014-12-02 22:44:33 +05:30
eac776f1a7
tests passing with on heap incremental index
2014-12-02 22:29:28 +05:30
4eee7e69b9
fix cardinality aggregator caching
2014-11-26 15:00:37 -08:00
5bc1be5ba0
Merge pull request #850 from metamx/druid-0.7.x-compressionstrategy
...
Compression strategy changes
2014-11-25 12:58:39 -08:00
c6043afa32
Removed empty function from CompressionStrategyTest
2014-11-25 12:57:06 -08:00
6943db5251
Changed branching logic for LZFCompressor to return null only on error, and avoid checking in most circumstances
2014-11-25 12:53:11 -08:00
9f945c2216
Removed lz4Fast from CompressedObjectStrategy for compression since it is not currently used
2014-11-24 16:11:03 -08:00
70e3108282
Multiple speed improvements revolving around topN with HLL
...
Change serializer / deserializer for HyperLogLog
* Changed DirectDruidClient's InputStream handling. Is now ~10% faster for data heavy queries, and has lower variance in execution speed.
* Changed HLL Collector's toByteStream() method to be better optimized for small values. Is notably faster for small result quantities which fall into the sparse HLL bucket codepath.
* No change for dense HLL which just uses a direct bytestream of the underlying byte data.
TopNNumericResultBuilder semi-aggressive loop unrolling for metricVals
Benchmark for HLL for sparse packing (small HLL bucket population):
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[0]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 216, GC.time: 0.42, time.total: 15.96, time.warmup: 0.22, time.bench: 15.74
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[1]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 217, GC.time: 0.45, time.total: 13.87, time.warmup: 0.02, time.bench: 13.85
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[2]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 55, GC.time: 0.16, time.total: 4.13, time.warmup: 0.00, time.bench: 4.12
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[3]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 55, GC.time: 0.16, time.total: 4.30, time.warmup: 0.00, time.bench: 4.30
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[4]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 8, GC.time: 0.03, time.total: 1.10, time.warmup: 0.00, time.bench: 1.09
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[5]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 8, GC.time: 0.03, time.total: 0.72, time.warmup: 0.00, time.bench: 0.72
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[6]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 1, GC.time: 0.00, time.total: 0.60, time.warmup: 0.00, time.bench: 0.60
HyperLogLogSerdeBenchmarkTest.benchmarkToByteBuffer[7]: [measured 100000 out of 100100 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.01, time.total: 0.26, time.warmup: 0.00, time.bench: 0.25
Updates to HyperLogLogCollector toByteBuffer() based on code review
Removed changes from DirectDruidClient from this branch and put it in another branch.
Changed HyperLogLogCollector to have protected getters and setters
Remove unused ByteOrder from HyperLogLogCollector
Copyright header on HyperLogLogSerdeBenchmarkTest
Now with less ass!
Reformat in TopNNumericResultsBuilder. No code change
Removed unused import in HyperLogLogCollector
Replace AppendableByteArrayInputStream in DirectDruidClient
* Replace with SequenceInputStream fueled by an enumeration of ChannelBufferInputStream which directly wrap the response context ChannelBuffer
Modify TopNQueryQueryToolChest to use Arrays instead of Lists
Modify TopNQueryQueryToolChest to use Arrays instead of Lists
Revert accidental changes to DirectDruidClient
They should be in another merge request:
https://github.com/metamx/druid/pull/893
Fixes from code review
* Extracting names from AggregatorFactory classes now done with TopNQueryQueryToolChest.extractFactoryName
* Renamed variable in TopNNumericResultBuilder
2014-11-24 16:02:00 -08:00
13cae41f6c
Merge branch 'master' into refactor-examples
2014-11-24 11:00:26 -08:00
9ced097abd
Merge pull request #895 from metamx/fix-interval-retry
...
A set of fixes to retry the query for missing segments in the timeline
2014-11-24 10:23:02 -08:00
c88aff6205
remove unneeded constant
2014-11-24 10:20:02 -08:00
9da66291e1
change naming to common config
2014-11-21 15:36:42 -08:00
8f80d9e189
Update CompressedObjectStrategy to try-with-resources but print log error if error while closing
2014-11-21 11:35:11 -08:00
fc9a54ea48
Fix CompressedObjectStrategy LZFCompressor to ignore error on close of ResourceHolder
2014-11-21 10:49:43 -08:00
f8ce68565b
Modified CompressedObjectStrategy to use 0xFF for Uncompressed
2014-11-21 10:33:53 -08:00
aa49e56ed6
Merge remote-tracking branch 'origin/master' into druid-0.7.x-compressionstrategy
2014-11-21 10:29:40 -08:00
ef62bccdec
ignore benchmark
2014-11-20 16:52:19 -08:00
e3260aa177
Filtered Aggregator fixes + enhancements
...
- fix NPE on IncrementIndex
- refactor code to support AND, OR filter
- tests for AND & OR filter
- handling for missing column / null values
2014-11-20 15:17:18 -08:00
47f5c1bd0a
fix retry interval is stupid
2014-11-20 12:50:56 -08:00
3d9d989a9f
A set of fixes to retry the query for missing intervals in the timeline
2014-11-20 12:04:37 -08:00
0ab34f86da
Revert "fix filtered Aggregator"
...
This reverts commit 6fd37ce023
2014-11-20 10:17:01 +05:30
6fd37ce023
fix filtered Aggregator
...
fix filtered Aggregator
remove unused name parameter for filtered aggregator
add tests
2014-11-20 09:29:26 +05:30
a49e673122
put back another missing test
2014-11-19 16:55:20 -08:00
14668846aa
add back some tests
2014-11-19 14:35:26 -08:00
fdeab0c6af
make Druid case sensitive
2014-11-19 14:27:31 -08:00
590d31799e
Merge pull request #876 from metamx/remove-backwards-compatible
...
Remove backwards compatible
2014-11-19 14:33:14 -07:00
18f44beee9
CompressedObjectStrategy improvements
...
* Added more unit tests
* Now properly uses safe / fast decompressor for LZ4
* Now chooses fastest lz4 instance instead of only looking at Java implmentations
* Encapsulate ResourceHolder in try-with-resources to make sure they close correctly
2014-11-19 11:10:59 -08:00
ccc757dc64
Merge remote-tracking branch 'origin/master' into druid-0.7.x-compressionstrategy
2014-11-19 09:39:35 -08:00
1bbc8fcbe5
Allow Smile to fall back to text
...
* Modify SmileFactory to set the delegate to text option.
* This option only occurs when a Reader type object is passed in to the deserialization stuff
* This is needed by the X-Druid-Response-Context header return value, which is JSON
2014-11-18 15:16:14 -08:00
42517f5d37
Merge pull request #884 from metamx/optimize-topN-pruning
...
optimise pruning of aggs
2014-11-18 14:19:30 -08:00
a96eaeb036
Merge pull request #882 from metamx/now_with_OPEN_SOURCE
...
Added src jar build to maven poms and re-formatted to conform to style guidelines.
2014-11-18 13:00:04 -08:00
6023d602e6
optimise pruning of aggs
...
optimise pruning of aggregators for topN
2014-11-19 00:17:25 +05:30
dc66e1708e
Added src jar build to maven poms and re-formatted to conform to style guidelines.
2014-11-18 09:05:30 -08:00
d914afe1cd
make defaultVersion configurable for non-jar testing
2014-11-17 13:54:32 -08:00
0c2d06475d
merge from master
2014-11-17 19:19:18 +05:30
cbffe3c648
merge from master and resolve conflicts
2014-11-17 18:07:08 +05:30
d5c4282766
fix broken ut
2014-11-14 13:45:42 -08:00
e1e171ad20
Merge pull request #865 from metamx/fix-retry-qr
...
Fix a couple of bugs with retry query runner after testing it locally
2014-11-14 13:33:51 -08:00
df1ad95075
remove useless config
2014-11-14 13:32:19 -08:00
d641d41f9e
address another cr
2014-11-14 13:29:59 -08:00
7736c3fc27
address cr
2014-11-14 13:28:32 -08:00
6ee8029462
Merge pull request #866 from metamx/mutableBitmapBenchmark
...
Add benchmarking for bitmaps
2014-11-14 14:16:21 -07:00
a4fc64ca3f
Merge pull request #856 from metamx/druid-845
...
Fix query by segment
2014-11-14 13:10:54 -08:00
4b7ab23289
Remove getIntervalString from BySegmentResultValue
2014-11-14 13:03:48 -08:00
bbc079b880
fix retry to actually return correct sequences
2014-11-14 12:10:04 -08:00
648759e9f6
Add deserialization benchmark to BitmapCreationBenchmark
2014-11-13 13:43:14 -08:00
483b2c7be0
Add copyright notice to BitmapCreationBenchmark
2014-11-13 12:55:02 -08:00
228fb0cf40
Add benchmarking for bitmaps
...
Here are the results on my laptop:
BitmapCreationBenchmark.testRandomAddition[0]: [measured 10 out of 20 rounds, threads: 1 (sequential)]
round: 0.49 [+- 0.07], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 16, GC.time: 0.01, time.total: 9.91, time.warmup: 5.06, time.bench: 4.86
BitmapCreationBenchmark.testLinearAdditionDescending[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.01 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 74, GC.time: 0.03, time.total: 5.82, time.warmup: 0.06, time.bench: 5.76
BitmapCreationBenchmark.testToImmutableByteArray[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 1.80, time.warmup: 0.02, time.bench: 1.78
BitmapCreationBenchmark.testRandomAddition[1]: [measured 10 out of 20 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 0.12, time.warmup: 0.08, time.bench: 0.04
BitmapCreationBenchmark.testLinearAdditionDescending[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 10, GC.time: 0.01, time.total: 4.26, time.warmup: 0.04, time.bench: 4.22
BitmapCreationBenchmark.testToImmutableByteArray[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.01 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 27, GC.time: 0.01, time.total: 5.11, time.warmup: 0.05, time.bench: 5.06
BitmapCreationBenchmark.testLinearAddition[0]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 3.48, time.warmup: 0.04, time.bench: 3.45
BitmapCreationBenchmark.testLinearAddition[1]: [measured 1000 out of 1010 rounds, threads: 1 (sequential)]
round: 0.00 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 9, GC.time: 0.00, time.total: 2.95, time.warmup: 0.03, time.bench: 2.92
2014-11-13 12:47:23,995 INFO [main] io.druid.segment.data.BitmapCreationBenchmark - Entry [0] is io.druid.segment.data.ConciseBitmapSerdeFactory
2014-11-13 12:47:23,995 INFO [main] io.druid.segment.data.BitmapCreationBenchmark - Entry [1] is io.druid.segment.data.RoaringBitmapSerdeFactory
2014-11-13 12:48:04 -08:00
6cdd6a6af7
change default settings for retry
2014-11-13 12:43:30 -08:00
2b0ed30344
add test
2014-11-13 12:38:35 -08:00
0d6816a037
Fix a couple of bugs with retry query runner after testing it locally
2014-11-13 11:53:29 -08:00
4a3c0fc5c4
Merge pull request #848 from metamx/druid-0.7.x-fastTopN-rebase
...
TopN performance improvements
2014-11-13 11:56:20 -07:00
34c3464bc9
Added more explanatory comments in PooledTopNAlgorithm
2014-11-13 10:47:56 -08:00
9a9238a801
Merge remote-tracking branch 'origin/master' into druid-845
2014-11-13 10:04:56 -08:00
19a37f773f
test for groupBy order
2014-11-12 22:48:16 -08:00
31fed7d329
Fix query by segment
...
* Changed topN queries to use joda Interval instead of string values
* topN by segment now implements BySegmentResultValue<Result<TopNResultValue>> instead of BySegmentResultValue<TopNResultValue>
* Added a unit test which failed uner the prior implementation.
2014-11-12 21:20:59 -08:00
4ac1aaf90e
Merge remote-tracking branch 'origin/master' into druid-0.7.x
...
Conflicts:
processing/src/main/java/io/druid/segment/QueryableIndexStorageAdapter.java
2014-11-12 14:08:51 -08:00
4b4f1c7d38
Merge pull request #846 from vikramakrishnan/cacheKeyFix
...
Include origin when creating the cacheKey for period Grans
2014-11-12 11:15:21 -07:00
7ef19009db
Merge pull request #834 from metamx/optimize-timestampchecking
...
skip timestamp checking if not required, remove duplicate code
2014-11-12 09:45:34 -07:00
8151d14786
Include origin when creating the cacheKey for period Grans
2014-11-12 21:36:46 +05:30
581e6830d5
Merge pull request #840 from metamx/powers-of-2-buffers
...
make buffer size a power of 2 and optimize buffer lookup
2014-11-11 19:54:00 -08:00
60e98c35d7
workaround for annotations requiring class literals
2014-11-11 18:48:49 -08:00
b580269f6d
Distinguish between default and legacy bitmaps
2014-11-11 18:35:45 -08:00
bc5c56e441
fix default impl
2014-11-11 18:00:46 -08:00
63ca2375a3
remove dead code and cleanup some defaults
2014-11-11 17:57:24 -08:00
5629307548
address a few more comments on roaring
2014-11-11 17:50:03 -08:00
a89b539b4f
Merge pull request #823 from metamx/roaring
...
Configurable bitmap indexes: roaring and concise
2014-11-11 17:26:38 -08:00
007e57f876
make buffer size a power of 2 and optimize
2014-11-11 16:24:37 -08:00
1cc162727b
address code review
2014-11-11 14:05:37 -08:00
3f5449d40a
loop unrolling provides no benefit for timeseries
2014-11-11 10:58:37 -08:00
e817db8b6c
unroll timeseries aggregations + naming
2014-11-11 10:09:37 -08:00
e6b7b03b5b
fix conversion
2014-11-10 17:13:28 -08:00
eb457c280e
revert change
2014-11-10 17:00:53 -08:00
336c73bdc2
cleanup dead code
2014-11-10 16:53:13 -08:00
df886fac1b
fix
2014-11-10 16:49:27 -08:00
d68bc3bdea
cleanup unused imports
2014-11-10 16:15:28 -08:00
92e71be864
Change TopNQueryRunnerBenchmark to use a ByteBuffer as per OffheapBufferPool
2014-11-10 15:40:54 -08:00
a093f3728a
Reformat on TopNNumericResultBuilder
2014-11-10 15:26:09 -08:00
2b0f4534bf
Modify formatting in TopNQueryRunnerBenchmark
2014-11-10 15:17:26 -08:00
fc78f215c4
Aggressive dimValue unrolling in PooledTopNAlgorithm
2014-11-10 15:14:45 -08:00
6188315293
Merge branch 'master' into druid-0.7.x
...
Conflicts:
processing/src/test/java/io/druid/query/search/SearchQueryRunnerTest.java
2014-11-10 14:52:10 -08:00
49e878cf1a
unroll multi-value dimensions
2014-11-10 14:21:56 -08:00
df9be030db
remove more legacy code
2014-11-10 14:09:00 -08:00
dbb6401fbe
Aggressive unrolling in PooledTopNAlgorithm
2014-11-10 14:04:37 -08:00
72c355e8ae
remove crazy handler code
2014-11-10 13:58:23 -08:00
fc34858e95
fix things for real-time ingestion
2014-11-10 13:48:21 -08:00
c5cc826998
make things actually work with roaring
2014-11-10 13:42:26 -08:00
358b2add17
make things actually work with roaring
2014-11-10 13:42:06 -08:00
9ac8589143
Merge remote-tracking branch 'origin/druid-0.7.x-fastTopN-rebase' into druid-0.7.x-fastTopN-rebase
2014-11-10 12:55:08 -08:00
70468400bf
stupid mistake
2014-11-10 12:54:48 -08:00
Himanshu
Sam Groth
Clint Wylie
navis.ryu
Nishant
Gian Merlino
Xavier Léauté
Charles Allen
jon-wei
Xavier Léauté
binlijin
dclim
Slim Bouguerra
fjy
Bingkun Guo
cheddar
lvjq
zhanghailei
zhaown
Parag Jain
Maarten Rijke
Neo
vkavuluri
Eric Tschetter
msprunck
Prajwal Tuladhar
Himanshu Gupta
Joshua Schumacher
James Estes
Maarten Rijke
Vikram Ramakrishnan