Commit Graph

203 Commits

Author SHA1 Message Date
Justin Borromeo 69b24bd851 Merge branch 'master' into 6088-Time-Ordering-On-Scans-N-Way-Merge 2019-02-22 18:13:26 -08:00
Justin Borromeo 06a5218917 Wrote docs 2019-02-22 16:59:57 -08:00
Justin Borromeo c9142e721c Merge branch 'master' into 6088-Time-Ordering-On-Scans-V2 2019-02-20 10:12:50 -08:00
Justin Borromeo 871b9d2f4c [Benchmarking] Call blackhole#consume() on collections instead of iterating through each element (#7002)
* Replaced iteration with blackhole#consume(the collection)

* Added javadoc on Sequence#toList()
2019-02-20 08:48:06 -08:00
Justin Borromeo 7baeade832 Changes based on Gian's comments 2019-02-19 17:52:06 -08:00
Justin Borromeo 35150fe1a6 Small changes 2019-02-15 15:57:53 -08:00
Jonathan Wei fafbc4a80e
Set version to 0.15.0-incubating-SNAPSHOT (#7014) 2019-02-07 14:02:52 -08:00
Justin Borromeo 2e3577cd3d Fixed benchmark queries 2019-02-07 13:01:25 -08:00
Justin Borromeo d3b335af42 added all query types to scan benchmark 2019-02-07 11:08:07 -08:00
Justin Borromeo ab00eade9f Kicking travis with change to benchmark param 2019-02-07 09:42:48 -08:00
Justin Borromeo b2c8c77ad4 Fixing tests WIP 2019-02-06 17:39:48 -08:00
Justin Borromeo 6723243ed2 Create Scan Benchmark (#6986)
* Moved Scan Builder to Druids class and started on Scan Benchmark setup

* Need to form queries

* It runs.

* Remove todos

* Change number of benchmark iterations

* Changed benchmark params

* More param changes

* Made Jon's changes and removed TODOs

* Broke some long lines into two lines

* Decrease segment size for less memory usage

* Committing a param change to kick teamcity
2019-02-06 14:45:01 -08:00
Justin Borromeo 148939e88b Merge branch '6088-Create-Scan-Benchmark' into 6088-Time-Ordering-On-Scans-V2 2019-02-06 13:26:17 -08:00
Justin Borromeo 5edbe2ae12 Merge github.com:apache/incubator-druid into 6088-Create-Scan-Benchmark 2019-02-06 13:18:55 -08:00
Justin Borromeo 60b7684db7 Committing a param change to kick teamcity 2019-02-06 13:02:13 -08:00
Justin Borromeo b6d4df3864 Decrease segment size for less memory usage 2019-02-05 16:45:20 -08:00
Justin Borromeo 7deb06f6df Merge branch '6088-Create-Scan-Benchmark' into 6088-Time-Ordering-On-Scans-V2 2019-02-05 10:53:38 -08:00
Justin Borromeo 86c5eee13b Broke some long lines into two lines 2019-02-05 10:31:27 -08:00
Justin Borromeo b7d3a4900a Merge branch 'master' into 6088-Time-Ordering-On-Scans-V2 2019-02-05 10:23:32 -08:00
Justin Borromeo 737a83321d Made Jon's changes and removed TODOs 2019-02-05 10:15:32 -08:00
Jonathan Wei 8bc5eaa908
Set version to 0.14.0-incubating-SNAPSHOT (#7003) 2019-02-04 19:36:20 -08:00
Justin Borromeo 796083f2bb Benchmark param change 2019-02-04 15:37:42 -08:00
Justin Borromeo 20c36644db More param changes 2019-02-04 15:36:35 -08:00
Justin Borromeo 9e6e71616b Changed benchmark params 2019-02-04 15:31:21 -08:00
Justin Borromeo 01b25ed112 Added time ordering to the scan benchmark 2019-02-04 14:36:18 -08:00
Justin Borromeo 432acaf085 Change number of benchmark iterations 2019-02-04 12:03:14 -08:00
Justin Borromeo 12e51a2721 Added TimestampComparator tests 2019-02-04 12:02:13 -08:00
Justin Borromeo e66339cd76 Remove todos 2019-02-04 10:56:41 -08:00
Justin Borromeo ad731a362b Change benchmark 2019-02-04 10:55:56 -08:00
Roman Leventov 0e926e8652 Prohibit assigning concurrent maps into Map-typed variables and fields and fix a race condition in CoordinatorRuleManager (#6898)
* Prohibit assigning concurrent maps into Map-types variables and fields; Fix a race condition in CoordinatorRuleManager; improve logic in DirectDruidClient and ResourcePool

* Enforce that if compute(), computeIfAbsent(), computeIfPresent() or merge() is called on a ConcurrentHashMap, it's stored in a ConcurrentHashMap-typed variable, not ConcurrentMap; add comments explaining get()-before-computeIfAbsent() optimization; refactor Counters; fix a race condition in Intialization.java

* Remove unnecessary comment

* Checkstyle

* Fix getFromExtensions()

* Add a reference to the comment about guarded computeIfAbsent() optimization; IdentityHashMap optimization

* Fix UriCacheGeneratorTest

* Workaround issue with MaterializedViewQueryQueryToolChest

* Strengthen Appenderator's contract regarding concurrency
2019-02-04 09:18:12 -08:00
Surekha 7baa33049c Introduce published segment cache in broker (#6901)
* Add published segment cache in broker

* Change the DataSegment interner so it's not based on DataSEgment's equals only and size is preserved if set

* Added a trueEquals to DataSegment class

* Use separate interner for realtime and historical segments

* Remove trueEquals as it's not used anymore, change log message

* PR comments

* PR comments

* Fix tests

* PR comments

* Few more modification to

* change the coordinator api
* removeall segments at once from MetadataSegmentView in order to serve a more consistent view of published segments
* Change the poll behaviour to avoid multiple poll execution at same time

* minor changes

* PR comments

* PR comments

* Make the segment cache in broker off by default

* Added a config to PlannerConfig
* Moved MetadataSegmentView to sql module

* Add doc for new planner config

* Update documentation

* PR comments

* some more changes

* PR comments

* fix test

* remove unintentional change, whether to synchronize on lifecycleLock is still in discussion in PR

* minor changes

* some changes to initialization

* use pollPeriodInMS

* Add boolean cachePopulated to check if first poll succeeds

* Remove poll from start()

* take the log message out of condition in stop()
2019-02-02 22:27:13 -08:00
Justin Borromeo 26930f8d20 It runs. 2019-02-01 16:38:49 -08:00
Justin Borromeo dd4ec1ac9c Need to form queries 2019-02-01 15:12:17 -08:00
Justin Borromeo dba6e492a0 Merge branch 'master' into 6088-Create-Scan-Benchmark 2019-02-01 14:13:39 -08:00
Justin Borromeo 10e57d5f9e Moved Scan Builder to Druids class and started on Scan Benchmark setup 2019-02-01 14:04:13 -08:00
Roman Leventov f7df5fedcc Add several missing inspectRuntimeShape() calls (#6893)
* Add several missing inspectRuntimeShape() calls

* Add lgK to runtime shapes
2019-01-31 20:04:26 -08:00
Roman Leventov 8eae26fd4e Introduce SegmentId class (#6370)
* Introduce SegmentId class

* tmp

* Fix SelectQueryRunnerTest

* Fix indentation

* Fixes

* Remove Comparators.inverse() tests

* Refinements

* Fix tests

* Fix more tests

* Remove duplicate DataSegmentTest, fixes #6064

* SegmentDescriptor doc

* Fix SQLMetadataStorageUpdaterJobHandler

* Fix DataSegment deserialization for ignoring id

* Add comments

* More comments

* Address more comments

* Fix compilation

* Restore segment2 in SystemSchemaTest according to a comment

* Fix style

* fix testServerSegmentsTable

* Fix compilation

* Add comments about why SegmentId and SegmentIdWithShardSpec are separate classes

* Fix SystemSchemaTest

* Fix style

* Compare SegmentDescriptor with SegmentId in Javadoc and comments rather than with DataSegment

* Remove a link, see https://youtrack.jetbrains.com/issue/IDEA-205164

* Fix compilation
2019-01-21 11:11:10 -08:00
Jonathan Wei 68f744ec0a
Fixed buckets histogram aggregator (#6638)
* Fixed buckets histogram aggregator

* PR comments

* More PR comments

* Checkstyle

* TeamCity

* More TeamCity

* PR comment

* PR comment

* Fix doc formatting
2019-01-17 14:51:16 -08:00
Dayue Gao 5b8a221713 Add SQL id, request logs, and metrics (#6302)
* use SqlLifecyle to manage sql execution, add sqlId

* add sql request logger

* fix UT

* rename sqlId to sqlQueryId, sql/time to sqlQuery/time, etc

* add docs and more sql request logger impls

* add UT for http and jdbc

* fix forbidden use of com.google.common.base.Charsets

* fix UT in QuantileSqlAggregatorTest, supressed unused warning of getSqlQueryId

* do not use default method in QueryMetrics interface

* capitalize 'sql' everywhere in the non-property parts of the docs

* use RequestLogger interface to log sql query

* minor bugfixes and add switching request logger

* add filePattern configs for FileRequestLogger

* address review comments, adjust sql request log format

* fix inspection error

* try SuppressWarnings("RedundantThrows") to fix inspection error on ComposingRequestLoggerProvider
2019-01-15 23:12:59 -08:00
Jihoon Son c35a39d70b
Add support maxRowsPerSegment for auto compaction (#6780)
* Add support maxRowsPerSegment for auto compaction

* fix build

* fix build

* fix teamcity

* add test

* fix test

* address comment
2019-01-10 09:50:14 -08:00
Jihoon Son fa7cb906e4 Fix auto compaction to consider intervals of running tasks (#6767)
* Fix auto compaction to consider intervals of running tasks

* adjust initial collection size
2018-12-27 18:03:44 -08:00
Shimi Kiviti 5fae522f12 replace Files.map() with FileUtils.map() (#6692) 2018-11-30 07:40:41 -08:00
Roman Leventov 887c645675
Find duplicate lines with checkstyle; enable some duplicate inspections in IntelliJ (#6558)
Not putting this to 0.13 milestone because the found bugs are not critical (one is a harmless DI config duplicate, and another is in a benchmark.

Change in `DumpSegment` is just an indentation change.
2018-11-26 16:55:42 +01:00
Roman Leventov 87b96fb1fd
Add checkstyle rules about imports and empty lines between members (#6543)
* Add checkstyle rules about imports and empty lines between members

* Add suppressions

* Update Eclipse import order

* Add empty line

* Fix StatsDEmitter
2018-11-20 12:42:15 +01:00
Roman Leventov 8f3fe9cd02 Prohibit String.replace() and String.replaceAll(), fix and prohibit some toString()-related redundancies (#6607)
* Prohibit String.replace() and String.replaceAll(), fix and prohibit some toString()-related redundancies

* Fix bug

* Replace checkstyle regexp with IntelliJ inspection
2018-11-15 13:21:34 -08:00
Gian Merlino 52f6bdc1eb
Optimization for expressions that hit a single long column. (#6599)
* Optimization for expressions that hit a single long column.

There was previously a single-long-input optimization that applied only
to the time column. These have been combined together. Also adds
type-specific value caching to ExprEval, which allowed simplifying
the SingleLongInputCachingExpressionColumnValueSelector code.

* Add more benchmarks.

* Don't use LRU cache for __time.

* Simplify a bit.

* Let the cache grow.
2018-11-13 09:36:32 -08:00
Roman Leventov 54351a5c75 Fix various bugs; Enable more IntelliJ inspections and update error-prone (#6490)
* Fix various bugs; Enable more IntelliJ inspections and update error-prone

* Fix NPE

* Fix inspections

* Remove unused imports
2018-11-06 14:38:08 -08:00
QiuMM 676f5e6d7f Prohibit some guava collection APIs and use JDK collection APIs directly (#6511)
* Prohibit some guava collection APIs and use JDK APIs directly

* reset files that changed by accident

* sort codestyle/druid-forbidden-apis.txt alphabetically
2018-10-29 13:02:43 +01:00
Roman Leventov 84ac18dc1b
Catch some incorrect method parameter or call argument formatting patterns with checkstyle (#6461)
* Catch some incorrect method parameter or call argument formatting patterns with checkstyle

* Fix DiscoveryModule

* Inline parameters_and_arguments.txt

* Fix a bug in PolyBind

* Fix formatting
2018-10-23 07:17:38 -03:00
Diego Costa d5ccd582b5 Reducing the usage of Jmh Invocation level on setup (#6459)
* Reducing the usage of invocation level setup

* Removing LoadStatusBenchmark as it has no use in the codebase
2018-10-14 20:37:11 -07:00