Commit Graph

2056 Commits

Author SHA1 Message Date
Justin Borromeo cd489a0208 Fixed failing test due to null resultFormat 2019-02-20 00:16:48 -08:00
Justin Borromeo 7baeade832 Changes based on Gian's comments 2019-02-19 17:52:06 -08:00
Justin Borromeo 35150fe1a6 Small changes 2019-02-15 15:57:53 -08:00
Justin Borromeo 4e69276d57 Removed unused import to satisfy PMD check 2019-02-11 12:09:54 -08:00
Justin Borromeo ecb0f483a9 improved doc 2019-02-11 10:37:11 -08:00
Justin Borromeo f0eddee665 Added more javadoc 2019-02-11 10:18:45 -08:00
Justin Borromeo 5f92dd7325 Unused import 2019-02-11 10:05:58 -08:00
Justin Borromeo 93e1636287 Added javadoc on ScanResultValueTimestampComparator 2019-02-11 10:03:14 -08:00
Justin Borromeo 134041c479 Renamed sort function 2019-02-08 13:13:54 -08:00
Justin Borromeo b2c8c77ad4 Fixing tests WIP 2019-02-06 17:39:48 -08:00
Justin Borromeo 85e72a614e Set to spaces over tabs 2019-02-06 15:42:02 -08:00
Justin Borromeo 7e872a8ebc Created an error message for when someone tries to time order a result
set > threshold limit
2019-02-06 15:36:24 -08:00
Justin Borromeo 305876a434 nit 2019-02-06 15:02:02 -08:00
Justin Borromeo 8212a21caf Improved conciseness 2019-02-06 14:40:35 -08:00
Justin Borromeo 10b5e0ca93 . 2019-02-06 13:42:12 -08:00
Justin Borromeo dfe4aa9681 Fixed codestyle and forbidden API errors 2019-02-06 13:41:18 -08:00
Justin Borromeo 4f51024b31 Wrote more tests for scan result value sort 2019-02-06 12:08:12 -08:00
Justin Borromeo 8b7d5f5081 Wrote tests for heapsort scan result values and fixed bug where iterator
wasn't returning elements in correct order
2019-02-06 11:55:09 -08:00
Justin Borromeo d1a1793f36 nit 2019-02-05 12:40:26 -08:00
Justin Borromeo 7deb06f6df Merge branch '6088-Create-Scan-Benchmark' into 6088-Time-Ordering-On-Scans-V2 2019-02-05 10:53:38 -08:00
Justin Borromeo b7d3a4900a Merge branch 'master' into 6088-Time-Ordering-On-Scans-V2 2019-02-05 10:23:32 -08:00
Jonathan Wei 8bc5eaa908
Set version to 0.14.0-incubating-SNAPSHOT (#7003) 2019-02-04 19:36:20 -08:00
Justin Borromeo 01b25ed112 Added time ordering to the scan benchmark 2019-02-04 14:36:18 -08:00
Justin Borromeo 12e51a2721 Added TimestampComparator tests 2019-02-04 12:02:13 -08:00
Justin Borromeo ad731a362b Change benchmark 2019-02-04 10:55:56 -08:00
Justin Borromeo 989bd2d50e Merge branch '6088-Create-Scan-Benchmark' into 6088-Time-Ordering-On-Scans-V2 2019-02-04 10:46:38 -08:00
Roman Leventov 0e926e8652 Prohibit assigning concurrent maps into Map-typed variables and fields and fix a race condition in CoordinatorRuleManager (#6898)
* Prohibit assigning concurrent maps into Map-types variables and fields; Fix a race condition in CoordinatorRuleManager; improve logic in DirectDruidClient and ResourcePool

* Enforce that if compute(), computeIfAbsent(), computeIfPresent() or merge() is called on a ConcurrentHashMap, it's stored in a ConcurrentHashMap-typed variable, not ConcurrentMap; add comments explaining get()-before-computeIfAbsent() optimization; refactor Counters; fix a race condition in Intialization.java

* Remove unnecessary comment

* Checkstyle

* Fix getFromExtensions()

* Add a reference to the comment about guarded computeIfAbsent() optimization; IdentityHashMap optimization

* Fix UriCacheGeneratorTest

* Workaround issue with MaterializedViewQueryQueryToolChest

* Strengthen Appenderator's contract regarding concurrency
2019-02-04 09:18:12 -08:00
Justin Borromeo 7b58471394 Licensing stuff 2019-02-02 03:48:18 -08:00
Justin Borromeo 79e8319383 Move ScanResultValue timestamp comparator to a separate class for testing 2019-02-01 18:22:58 -08:00
Justin Borromeo 7a6080f636 Stuff for time-ordered scan query 2019-02-01 18:00:58 -08:00
Justin Borromeo dba6e492a0 Merge branch 'master' into 6088-Create-Scan-Benchmark 2019-02-01 14:13:39 -08:00
Justin Borromeo 10e57d5f9e Moved Scan Builder to Druids class and started on Scan Benchmark setup 2019-02-01 14:04:13 -08:00
Roman Leventov f7df5fedcc Add several missing inspectRuntimeShape() calls (#6893)
* Add several missing inspectRuntimeShape() calls

* Add lgK to runtime shapes
2019-01-31 20:04:26 -08:00
Furkan KAMACI 30ec608038 Fix mixed up segment ids at SelectBinaryFnTest.java (#6946) 2019-01-30 20:04:16 -08:00
Clint Wylie de810286cd fix bug with expression virtual column selectors backed by a single long column (#6957)
* fix issue with SingleLongInputCachingExpressionColumnValueSelector when sql compatible null handling enabled

* add test with doubles to show same behavior for floats/doubles that lack the optimization of longs

* simplify

* fix import
2019-01-30 10:13:07 -05:00
Clint Wylie a6d81c0d16 Adds bloom filter aggregator to 'druid-bloom-filters' extension (#6397)
* blooming aggs

* partially address review

* fix docs

* minor test refactor after rebase

* use copied bloomkfilter

* add ByteBuffer methods to BloomKFilter to allow agg to use in place, simplify some things, more tests

* add methods to BloomKFilter to get number of set bits, use in comparator, fixes

* more docs

* fix

* fix style

* simplify bloomfilter bytebuffer merge, change methods to allow passing buffer offsets

* oof, more fixes

* more sane docs example

* fix it

* do the right thing in the right place

* formatting

* fix

* avoid conflict

* typo fixes, faster comparator, docs for comparator behavior

* unused imports

* use buffer comparator instead of deserializing

* striped readwrite lock for buffer agg, null handling comparator, other review changes

* style fixes

* style

* remove sync for now

* oops

* consistency

* inspect runtime shape of selector instead of selector plus, static comparator, add inner exception on serde exception

* CardinalityBufferAggregator inspect selectors instead of selectorPluses

* fix style

* refactor away from using ColumnSelectorPlus and ColumnSelectorStrategyFactory to instead use specialized aggregators for each supported column type, other review comments

* adjustment

* fix teamcity error?

* rename nil aggs to empty, change empty agg constructor signature, add comments

* use stringutils base64 stuff to be chill with master

* add aggregate combiner, comment
2019-01-29 20:05:17 +07:00
Benedict Jin 72a571fbf7 For performance reasons, use `java.util.Base64` instead of Base64 in Apache Commons Codec and Guava (#6913)
* * Add few methods about base64 into StringUtils

* Use `java.util.Base64` instead of others

* Add org.apache.commons.codec.binary.Base64 & com.google.common.io.BaseEncoding into druid-forbidden-apis

* Rename encodeBase64String & decodeBase64String

* Update druid-forbidden-apis
2019-01-25 17:32:29 -08:00
Himanshu Pandey e1033bb412 Issue#6892- Replaced Math.random() with ThreadLocalRandom.current().nextDouble() (#6914)
*  Replacing Math.random() with ThreadLocalRandom.current().nextDouble()

* Added java.lang.Math#random() in forbidden-apis.txt

* Minor change in the message - druid-forbidden-apis.txt
2019-01-25 19:49:20 +08:00
Clint Wylie 66f64cd8bd fix long/float/double dimension filtering for columns with nulls (#6906)
* fix long,float, double dimension filtering when sql compatible null handling is enabled and the column has null values

* revert unintended change

* fix tests
2019-01-23 22:36:52 -08:00
Roman Leventov 8eae26fd4e Introduce SegmentId class (#6370)
* Introduce SegmentId class

* tmp

* Fix SelectQueryRunnerTest

* Fix indentation

* Fixes

* Remove Comparators.inverse() tests

* Refinements

* Fix tests

* Fix more tests

* Remove duplicate DataSegmentTest, fixes #6064

* SegmentDescriptor doc

* Fix SQLMetadataStorageUpdaterJobHandler

* Fix DataSegment deserialization for ignoring id

* Add comments

* More comments

* Address more comments

* Fix compilation

* Restore segment2 in SystemSchemaTest according to a comment

* Fix style

* fix testServerSegmentsTable

* Fix compilation

* Add comments about why SegmentId and SegmentIdWithShardSpec are separate classes

* Fix SystemSchemaTest

* Fix style

* Compare SegmentDescriptor with SegmentId in Javadoc and comments rather than with DataSegment

* Remove a link, see https://youtrack.jetbrains.com/issue/IDEA-205164

* Fix compilation
2019-01-21 11:11:10 -08:00
Jihoon Son cc06e7e2df Fix fallback to cursor-based plan in UseIndexesStrategy (#6875)
* Fix fallback to cursor-based plan in UseIndexesStrategy

* fix build

* add a comment
2019-01-18 10:41:01 +08:00
Jonathan Wei 68f744ec0a
Fixed buckets histogram aggregator (#6638)
* Fixed buckets histogram aggregator

* PR comments

* More PR comments

* Checkstyle

* TeamCity

* More TeamCity

* PR comment

* PR comment

* Fix doc formatting
2019-01-17 14:51:16 -08:00
Dayue Gao 5b8a221713 Add SQL id, request logs, and metrics (#6302)
* use SqlLifecyle to manage sql execution, add sqlId

* add sql request logger

* fix UT

* rename sqlId to sqlQueryId, sql/time to sqlQuery/time, etc

* add docs and more sql request logger impls

* add UT for http and jdbc

* fix forbidden use of com.google.common.base.Charsets

* fix UT in QuantileSqlAggregatorTest, supressed unused warning of getSqlQueryId

* do not use default method in QueryMetrics interface

* capitalize 'sql' everywhere in the non-property parts of the docs

* use RequestLogger interface to log sql query

* minor bugfixes and add switching request logger

* add filePattern configs for FileRequestLogger

* address review comments, adjust sql request log format

* fix inspection error

* try SuppressWarnings("RedundantThrows") to fix inspection error on ComposingRequestLoggerProvider
2019-01-15 23:12:59 -08:00
Charles Allen 5d2947cd52 Use Guava Compatible immediate executor service (#6815)
* Use multi-guava version friendly direct executor implementation

* Don't use a singleton

* Fix strict compliation complaints

* Copy Guava's DirectExecutor

* Fix javadoc

* Imports are the devil
2019-01-11 10:42:19 -08:00
Richard Startin 99097617a1 use RoaringBitmapWriter for RoaringBitmap construction (#6764) 2019-01-08 17:18:41 -08:00
dongyifeng def823124c add version comparator for StringComparator (#6745)
* add version comparator for StringComparator

* add more test case and docs
2019-01-08 17:17:03 -08:00
Clint Wylie 9505074530 fix log typo (#6755)
* fix log typo, add DataSegmentUtils.getIdentifiersString util method

* fix indecisive oops
2018-12-18 15:10:25 -08:00
Clint Wylie 486c6f3cf9 emit logs that are only useful for debugging at debug level (#6741)
* make logs that are only useful for debugging be at debug level so log volume is much more chill

* info level messages for total merge buffer allocated/free

* more chill compaction logs
2018-12-17 14:20:28 +08:00
Atul Mohan 86e3ae5b48 Add fail message (#6720) 2018-12-11 08:05:50 -08:00
Gian Merlino b7709e1245 FileUtils: Sync directory entry too on writeAtomically. (#6677)
* FileUtils: Sync directory entry too on writeAtomically.

See the fsync(2) man page for why this is important:
https://linux.die.net/man/2/fsync

This also plumbs CompressionUtils's "zip" function through
writeAtomically, so the code for handling atomic local filesystem
writes is all done in the same place.

* Remove unused import.

* Avoid FileOutputStream.

* Allow non-atomic writes to overwrite.

* Add some comments. And no need to flush an unbuffered stream.
2018-12-08 17:12:59 +01:00