Commit Graph

10596 Commits

Author SHA1 Message Date
Torsten Hain 5356d8821b
fix typo in docker/druid.sh (#10433)
DRUID_NEWSIZE should not set MaxNewSize.
2020-09-28 14:38:21 -07:00
Clint Wylie b95bf444b2
add docs for kinesis lag metrics (#10435) 2020-09-28 13:13:53 -07:00
Clint Wylie 64df71f25f
more timeout handling in JsonParserIterator (#10426) 2020-09-28 13:12:42 -07:00
Clint Wylie 3d700a5e31
vectorize remaining math expressions (#10429)
* vectorize remaining math expressions

* fixes

* remove cannotVectorize() where no longer true

* disable vectorized groupby for numeric columns with nulls

* fixes
2020-09-26 23:30:14 -07:00
Chi Cao Minh cbd9ac8592
Web console autocompaction E2E test (#10425)
Add an E2E test for the common case web console workflow of setting up
autocompaction that changes the partitions from dynamic to hashed.

Also fix an issue with the async test setup to properly wait for the web
console to be ready.
2020-09-25 18:28:25 -07:00
Jihoon Son 0cc9eb4903
Store hash partition function in dataSegment and allow segment pruning only when hash partition function is provided (#10288)
* Store hash partition function in dataSegment and allow segment pruning only when hash partition function is provided

* query context

* fix tests; add more test

* javadoc

* docs and more tests

* remove default and hadoop tests

* consistent name and fix javadoc

* spelling and field name

* default function for partitionsSpec

* other comments

* address comments

* fix tests and spelling

* test

* doc
2020-09-24 16:32:56 -07:00
Jonathan Wei cb30b1fe23
Automatically determine numShards for parallel ingestion hash partitioning (#10419)
* Automatically determine numShards for parallel ingestion hash partitioning

* Fix inspection, tests, coverage

* Docs and some PR comments

* Adjust locking

* Use HllSketch instead of HyperLogLogCollector

* Fix tests

* Address some PR comments

* Fix granularity bug

* Small doc fix
2020-09-24 13:47:53 -07:00
Vadim Ogievetsky 89160c2f9b
better query view initial state (#10431) 2020-09-24 09:49:58 -07:00
Clint Wylie dad69481f0
add light weight version of /druid/coordinator/v1/lookups/nodeStatus (#10422)
* add light weight version /druid/coordinator/v1/lookups/nodeStatus

* review stuffs
2020-09-24 14:36:53 +08:00
Maytas Monsereenusorn 72f1b55f56
Add last_compaction_state to sys.segments table (#10413)
* Add is_compacted to sys.segments table

* change is_compacted to last_compaction_state

* fix tests

* fix tests

* address comments
2020-09-23 15:29:36 -07:00
Clint Wylie 19c4b16640
vectorized expressions and expression virtual columns (#10401)
* vectorized expression virtual columns

* cleanup

* fixes

* preserve float if explicitly specified

* oops

* null handling fixes, more tests

* what is an expression planner?

* better names

* remove unused method, add pi

* move vector processor builders into static methods

* reduce boilerplate

* oops

* more naming adjustments

* changes

* nullable

* missing hex

* more
2020-09-23 13:56:38 -07:00
Vadim Ogievetsky a60d034d01
Web console: compaction dialog update (#10417)
* compaction dialog update

* fix test snapshot

* Update web-console/src/dialogs/compaction-dialog/compaction-dialog.tsx

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>

* Update web-console/src/dialogs/compaction-dialog/compaction-dialog.tsx

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>

* feedback changes

Co-authored-by: Chi Cao Minh <chi.caominh@imply.io>
2020-09-23 09:34:19 -07:00
Gian Merlino 1af2eace41
Include Sequence-building time in CPU time metric. (#10377)
* Include Sequence-building time in CPU time metric.

Meaningful work can be done while building Sequences, and we should
count this work. On the Broker, this includes subquery processing
work done by the mergeResults call of the GroupByQueryQueryToolChest.

* Add test.
2020-09-23 14:33:55 +08:00
Vadim Ogievetsky ac0a45471e
Web console: add sort to tiers list (#10416)
* add sort to tiers list

* update snapshot
2020-09-22 19:00:55 -07:00
Vadim Ogievetsky 7cc0a7be68
Web console: clean up styling imports (#10410)
* fix styling for importing

* fix quotes
2020-09-21 17:30:25 -07:00
Tarun 49a09302f3
Issue fix for CSV loading with header and skip header not parsing well. (#10398) 2020-09-21 15:14:22 -07:00
Vadim Ogievetsky 6c5c86d800
Web console: fix lookup edit dialog, allow column renaming (#10406)
* column rename

* update licenses file

* remove empty file

* update license file

* move comment
2020-09-20 14:10:05 -07:00
sthetland ae247b6e63
Document change in results of groupBy queries with subtotalsSpec (#10405)
* subtotalsSpec results with null values

Document the format change in results of a groupBy query with a subtotalsSpec. This update applies to 0.18 and later.

* Review catches
2020-09-19 10:51:23 -07:00
Maytas Monsereenusorn e78d7862a8
Auto-compaction snapshot status API (#10371)
* Auto-compaction snapshot API

* Auto-compaction snapshot API

* Auto-compaction snapshot API

* Auto-compaction snapshot API

* Auto-compaction snapshot API

* Auto-compaction snapshot API

* Auto-compaction snapshot API

* fix when not all compacted segments are iterated

* add unit tests

* add unit tests

* add unit tests

* add unit tests

* add unit tests

* add unit tests

* add some tests to make code cov happy

* address comments

* address comments

* address comments

* address comments

* make code coverage happy

* address comments

* address comments

* address comments

* address comments
2020-09-18 16:37:58 -07:00
Igor Dvorzhak d0ee2e3a48
Upgrade ORC to 1.5.10 version (#10291) 2020-09-18 13:38:45 -07:00
Dylan Wylie f3eb0cfb3b
Avoid large limits causing int overflow in buffer size checks (#10356)
* Avoid large limits causing int overflow in buffer size checks

* fix lgtm overflow warning

Co-authored-by: Dylan <dwylie@spotx.tv>
2020-09-18 13:08:49 -07:00
belugabehr 74368d95af
Remove JODA Time Dependency from Avro Extensions (#10010) 2020-09-18 12:41:42 -07:00
Mainak Ghosh d9beda7f24
Adding the missing sqlQueryContext api (#10368)
* Adding the missing sqlQueryContext api

* Adding a serialization test for DefaultRequestLogEvent

* Fixing the unit test failure
2020-09-18 00:46:31 -07:00
Mainak Ghosh 14072d3ab0
Adding more dimensions to the audit log entry (#10373)
* Adding more dimensions to the audit log entry

* Making adding payload in audit metric optional

* Changing the name of the parameter to includePayloadAsDimensionInMetric. Adding a unit test

* Fixing the intellij code introspection issues
2020-09-17 18:36:28 -07:00
Suneet Saldanha 0b4c897fbe
Vectorized variance aggregators (#10390)
* wip vectorize

* close but not quite

* faster

* unit tests

* fix complex types for variance
2020-09-17 15:05:40 -07:00
Arvin.Z 1b05d6e542
recreate the balancer executor only when needed (#10280)
* recreate the balancer executor only when needed

* fix UT error

* shutdown the balancer executor in stopBeingLeader and stop

* remove commented code

* remove comments
2020-09-16 14:25:57 -05:00
Atul Mohan 94226f1b3d
Disable sending server version in response headers (#9832)
* Toggle sending of server version

* Remove config

Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-09-15 22:48:00 -07:00
Atul Mohan b6ad790dc7
Support combining inputsource for parallel ingestion (#10387)
* Add combining inputsource

* Fix documentation

Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-09-15 16:25:35 -07:00
Jihoon Son 8657b23ab2
Integration tests and docs for auto compaction with different partitioning (#10354)
* Working

* add test

* doc

* fix test

* split other integration test

* exclude other-index from other tests

* doc anchor fix

* adjust task slots and number of merge tasks

* spell check

* reduce maxNumConcurrentSubTasks to 1

* maxNumConcurrentSubtasks for range partitinoing

* reduce memory for historical

* change group name
2020-09-15 11:28:09 -07:00
Vadim Ogievetsky e465f05717
Web console: Improve number alignment in tables (#10389)
* Improve tables

* removed unused state interfaces

* better copy

* one more functional component

* updated e2e tests

* extract braced text correctly
2020-09-14 19:53:38 -07:00
Chi Cao Minh 5751d0edc1
Skip coverage check for tag builds (#10397)
The code coverage diff calculation assumes the TRAVIS_BRANCH environment
variable is the name of a branch; however, for tag builds it is the name
of the tag so the diff calculation fails. Since builds triggered by tags
do not have a code diff, the coverage check should be skipped to avoid
the error and to save some CI resources.
2020-09-14 19:46:33 -07:00
Suneet Saldanha f71ba6f2c2
Vectorized ANY aggregators (#10338)
* WIP vectorized ANY aggregators

* tests

* fix aggs

* cleanup

* code review + tests

* docs

* use NilVectorSelector when needed

* fix spellcheck

* dont instantiate vectors

* cleanup
2020-09-14 19:44:58 -07:00
Clint Wylie e012d5c41b
allow vectorized query engines to utilize vectorized virtual columns (#10388)
* allow vectorized query engines to utilize vectorized virtual column implementations

* javadoc, refactor, checkstyle

* intellij inspection and more javadoc

* better

* review stuffs

* fix incorrect refactor, thanks tests

* minor adjustments
2020-09-14 19:29:35 -07:00
Clint Wylie 184b202411
add computed Expr output types (#10370)
* push down ValueType to ExprType conversion, tidy up

* determine expr output type for given input types

* revert unintended name change

* add nullable

* tidy up

* fixup

* more better

* fix signatures

* naming things is hard

* fix inspection

* javadoc

* make default implementation of Expr.getOutputType that returns null

* rename method

* more test

* add output for contains expr macro, split operation and function auto conversion
2020-09-14 18:18:56 -07:00
Clint Wylie 084b23deed
benchmark for indexed table experiments (#10327)
* benchmark for indexed table experiments

* fix style

* teardown outside of measurement
2020-09-14 15:14:38 -07:00
Abhishek Agarwal f5e2645bbb
Support SearchQueryDimFilter in sql via new methods (#10350)
* Support SearchQueryDimFilter in sql via new methods

* Contains is a reserved word

* revert unnecessary change

* Fix toDruidExpression method

* rename methods

* java docs

* Add native functions

* revert change in dockerfile

* remove changes from dockerfile

* More tests

* travis fix

* Handle null values better
2020-09-14 09:57:54 -07:00
Cheng Pan 3d4b48e0aa
TransformSpecTest should extends InitializedNullHandlingTest (#10392) 2020-09-14 08:22:24 -07:00
Vadim Ogievetsky 3c8eacb2d4
Web console: improve query manager (convert to React hook) (#10360)
* Better query running

* update licenses

* update tests

* updated tests v2

* fade in cancel

* add exemplary tests

* update mkcomp

* fix inconsistent state update

* remove lastParsedQuery

* work if not a valid literal

* remove unused params

* fix licenses

* better state update

* get error message

* isEmpty tidy

* add tests around error message highlighting

* pull live query selector into a component

* add LiveQueryModeSelector tests

* update snapshots
2020-09-11 19:42:50 -07:00
Curt Buechter e3735602f2
Fix typo (#10385) 2020-09-11 16:31:36 -07:00
Jihoon Son 8f14ac814e
More structured way to handle parse exceptions (#10336)
* More structured way to handle parse exceptions

* checkstyle; add more tests

* forbidden api; test

* address comment; new test

* address review comments

* javadoc for parseException; remove redundant parseException in streaming ingestion

* fix tests

* unnecessary catch

* unused imports

* appenderator test

* unused import
2020-09-11 16:31:10 -07:00
Cheng Pan 8aea8cf1c6
Unit tests fail due to missing extend InitializedNullHandlingTest (#10382)
* CsvInputFormatTest should extend InitializedNullHandlingTest

* FirehoseFactoryToInputSourceAdaptorTest should extends InitializedNullHandlingTest
2020-09-11 16:23:46 -07:00
Lucas Capistrant 690e070c43
Fix doc for name of dynamic config to pause coordination (#10345) 2020-09-11 08:40:06 -05:00
Abhishek Agarwal a5c46dc84b
Add vectorization for druid-histogram extension (#10304)
* First draft

* Remove redundant code from FixedBucketsHistogramAggregator classes

* Add test cases for new classes

* Fix tests in sql compatible mode

* Typo fix

* Fix comment

* Add spelling

* Vectorize only for supported types

* Rename internal aggregator files

* Fix tests
2020-09-09 13:56:33 -07:00
Joy Kent e5f0da30ae
Fix stringFirst/stringLast rollup during ingestion (#10332)
* Add IndexMergerRollupTest

This changelist adds a test to merge indexes with StringFirst/StringLast aggregator.

* Fix StringFirstAggregateCombiner/StringLastAggregateCombiner

The segment-level type for stringFirst/stringLast is SerializablePairLongString,
not String. This changelist fixes it.

* Fix EarliestLatestAnySqlAggregator to handle COMPLEX type

This changelist allows EarliestLatestAnySqlAggregator to accept COMPLEX
type as an operand. For its return type, we set it to VARCHAR, since
COMPLEX column is only generated by stringFirst/stringLast during ingestion
rollup.

* Return value with smaller timestamp in StringFirstAggregatorFactory.combine function

* Add integration tests for stringFirst/stringLast during ingestion

* Use one EarliestLatestReturnTypeInference instance

Co-authored-by: Joy Kent <joy@automonic.ai>
2020-09-08 17:36:04 -07:00
Jihoon Son d32d1e7004
Fix result-level caching (#10341)
* create baseSequence early

* unit test

* add comment and a new test
2020-09-08 11:04:00 -07:00
Chi Cao Minh 176b715624
Ignore CVEs from htrace and ambari transitive deps (#10353)
* Ignore CVEs from htrace and ambari transitive deps

htrace CVEs are suppressed for now as addressing them requires updating
the hadoop version.

ambari CVEs are suppressed for now since ambari is updated to the latest
version and is no longer actively maintained.

* Fix compilation issue from ambari upgrade

* Add missing test coverage
2020-09-04 15:22:26 -07:00
Suneet Saldanha 91a153820e
fix NPE in StringGroupByColumnSelectorStrategy#bufferComparator (#10325)
* fix NPE in StringGroupByColumnSelectorStrategy#bufferComparator

* Add tests

* javadocs
2020-09-04 13:23:40 -07:00
Gian Merlino d7fcff3aba
StringFirstAggregatorFactory: Fix incorrect "combine" method. (#10351)
* StringFirstAggregatorFactory: Fix incorrect "combine" method.

There was a test, but it was wrong.

* Fix superclass.
2020-09-03 20:03:26 -07:00
LightGHLi a3bb6ee4a6
Add missing comma between JSON members in data-formats.md (#10343) 2020-09-03 20:03:06 -07:00
Suneet Saldanha a5cd5f1e84
Fix VARIANCE aggregator comparator (#10340)
* Fix VARIANCE aggregator comparator

The comparator for the variance aggregator used to compare values using the
count. This is now fixed to compare values using the variance. If the variance
is equal, the count and sum are used as tie breakers.

* fix tests + sql compatible mode

* code review

* more tests

* fix last test
2020-09-03 17:38:37 -07:00