Commit Graph

10504 Commits

Author SHA1 Message Date
Gian Merlino b6aaf59e8c
Add "offset" parameter to GroupBy query. (#10235)
* Add "offset" parameter to GroupBy query.

It works by doing the query as normal and then throwing away the first
"offset" number of rows on the broker.

* Stabilize GroupBy sorts.

* Fix inspections.

* Fix suppression.

* Fixups.

* Move TopNSequence to druid-core.

* Addl comments.

* NumberedElement equals verification.

* Changes from review.
2020-08-05 15:39:58 -07:00
Vadim Ogievetsky 9a29496b6c
Web console: Improve retention rules dialog in all sorts of ways (#10226)
* improve ret rules

* tidy up tests
2020-08-04 16:43:31 -07:00
keefe roedersheimer 4bb198eb4c
change search filter to includes (#10141) 2020-08-04 16:37:07 -07:00
Abhishek Radhakrishnan 34a4113752
Add vectorization support for the longMin aggregator. (#10211)
* Fix minor formatting in docs.

* Add Nullhandling initialization for test to run from IDE.

* Vectorize longMin aggregator.

- A new vectorized class for the vectorized long min aggregator.
- Changes to AggregatorFactory to support vectorize functionality.
- Few changes to schema evolution test to add LongMinAggregatorFactory.

* Add longSum to the supported vectorized aggregator implementations.

* Add MIN() long min to calcite query test that can vectorize.

* Add simple long aggregations test.

* Fixup formatting per checkstyle guide.

* fixup and add more tests for long min aggregator.

* Override test for groupBy since timestamps are handled differently.

* Null compatibility check in test.

* Review comment: Add a test case to LongMinAggregationTest.
2020-08-01 15:32:09 -07:00
Clint Wylie cfb7a893e7
fill out missing test coverage for druid-datasketches postaggs (#9730)
* fill out missing test coverage for druid-datasketches postaggs

* fixup

* fixup merge

* oops

* oops again
2020-07-31 10:08:07 -07:00
frank chen 646fa84d04
Support unit on byte-related properties (#10203)
* support unit suffix on byte-related properties

* add doc

* change default value of byte-related properites in example files

* fix coding style

* fix doc

* fix CI

* suppress spelling errors

* improve code according to comments

* rename Bytes to HumanReadableBytes

* add getBytesInInt to get value safely

* improve doc

* fix problem reported by CI

* fix problem reported by CI

* resolve code review comments

* improve error message

* improve code & doc according to comments

* fix CI problem

* improve doc

* suppress spelling check errors
2020-07-31 09:58:48 +08:00
Jian Wang 271f90f205
Add segment pruning for hash based shard spec (#9810)
* Add segment pruning for hash based partitioning

* Update doc

* Add additional test

* Address comments

* Fix unit test failure

Co-authored-by: Jian Wang <jwang@pinterest.com>
2020-07-30 18:44:26 -07:00
Maytas Monsereenusorn 574b062f1f
Cluster wide default query context setting (#10208)
* Cluster wide default query context setting

* Cluster wide default query context setting

* Cluster wide default query context setting

* add docs

* fix docs

* update props

* fix checkstyle

* fix checkstyle

* fix checkstyle

* update docs

* address comments

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix NPE
2020-07-29 15:19:18 -07:00
Jihoon Son 63c1746fe4
Fix timeseries query constructor when postAggregator has an expression reading timestamp result column (#10198)
* Fix timeseries query constructor when postAggregator has an expression reading timestamp result column

* fix npe

* Fix postAgg referencing timestampResultField and add a test for it

* fix test

* doc

* revert doc
2020-07-27 10:54:44 -07:00
Suneet Saldanha 2f28be3f2a
Suppress CVE-2020-7692 (#10214)
Druid is not a native app, so this CVE should not apply.
2020-07-27 10:52:44 -07:00
Clint Wylie 79dffefbf8
add explicit example for jdbc query context on connection properties (#10182)
* add explicit example for jdbc query context on connection properties

* make comment clearer

* Update sql.md

* Update sql.md
2020-07-24 13:43:04 -07:00
Vadim Ogievetsky 6d8799f2df
Update QueryView to use latest DruidQueryToolkit (#10201)
* Update to latest DruidQueryToolkit

* add THEN keyword

* do not crash on invalid JSON
2020-07-23 22:45:01 -07:00
Richard Startin e363b1cd20
Update RoaringBitmap to 0.9.0 (#9987) 2020-07-23 19:29:25 -07:00
Jihoon Son 6fdce36e41
Add integration tests for query retry on missing segments (#10171)
* Add integration tests for query retry on missing segments

* add missing dependencies; fix travis conf

* address comments

* Integration tests extension

* remove unused dependency

* remove druid_main

* fix java agent port
2020-07-22 22:30:35 -07:00
Jihoon Son 26d099f39b
Fix sys.servers table to not throw NPE and handle brokers/indexers/peons properly for broadcast segments (#10183)
* Fix sys.servers table to not throw NPE and handle brokers/indexers/peons properly for broadcast segments

* fix tests and add missing tests

* revert null handling fix

* unused import

* move out util methods from DiscoveryDruidNode
2020-07-21 17:52:51 -07:00
mans2singh d4bd6e5207
ingestion and tutorial doc update (#10202) 2020-07-21 17:52:23 -07:00
Clint Wylie 266243ac75
include staged maven artifacts in example vote thread (#10200) 2020-07-21 16:30:28 -07:00
Maytas Monsereenusorn dd7a32ad48
Fix ITSqlInputSourceTest (#10194)
* Fix ITSqlInputSourceTest.java

* Fix ITSqlInputSourceTest.java

* Fix ITSqlInputSourceTest.java

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2020-07-21 09:52:13 -07:00
Jihoon Son 41982116f4
Report missing segments when there is no segment for the query datasource in historicals (#10199)
* Report missing segments when there is no segment for the query
datasource in historicals

* test

* missing part for test

* another test
2020-07-20 21:02:52 -07:00
Clint Wylie 0deefd6408
update release process guide to include web-console versions (#10176) 2020-07-20 13:59:06 -07:00
Maytas Monsereenusorn 0cabc53bd5
Add integration tests for Appends (#10186)
* append test

* add append IT

* fix checkstyle

* fix checkstyle

* Remove parallel

* fix checkstyle

* fix

* fix

* address comments

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2020-07-20 13:43:13 -07:00
Suneet Saldanha eeb9012743
Change color of Run button for native queries (#10170)
* Change color of Run button for native queries

When a user tries to run a native query, change the color of the button to
Druid's secondary color to indicate that the user is not running a SQL query.

Before this change, the web-console would indicate this by changing the text
of the button from Run (SQL queries) to Rune (native queries). Rune could be
confusing to users as this appears to be a typo.

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* code review
2020-07-15 15:46:02 -07:00
Maytas Monsereenusorn 454eed06a8
JettyTest.testNumConnectionsMetricHttp is rarely flaky (#10169) 2020-07-14 10:36:46 -07:00
Suneet Saldanha e6c9142129
Add validation for authenticator and authorizer name (#10106)
* Add validation for authorizer name

* fix deps

* add javadocs

* Do not use resource filters

* Fix BasicAuthenticatorResource as well

* Add integration tests

* fix test

* fix
2020-07-13 21:15:54 -07:00
Nishant Bangarwa 971d8a353b
Add groupBy limitSpec to queryCache key (#10093)
* Add groupBy limitSpec to queryCache key

* Only add limitSpec to cache key if pushdown is set to true

* review comment
2020-07-13 19:15:09 -07:00
Nishant Bangarwa 2b48de074a
Add additional properties for Kafka AdminClient and consumer from test config file (#10137)
* Add kafka test configs from file for AdminClient and consumer

* review comment
2020-07-13 18:16:05 +05:30
Joseph Glanville f3023c6058
Fix formatting in druid-pac4j documentation (#10174)
Superfluous column broke table formatting.
2020-07-12 18:51:42 -07:00
Clint Wylie 0a8fd62f22
split web-console e2e-tests from unit tests (#10173)
* split web-console e2e-test from unit test

* fix stuff

* smaller change

* oops
2020-07-12 18:51:08 -07:00
Clint Wylie ebea73d081
fix website build (#10172)
* fix mvn website build to use mvn supplied nodejs, fix broken redirects, move block from custom.css to custom.scss so will be correctly generated

* sidebar

* fix lol
2020-07-11 22:01:43 -07:00
Suneet Saldanha 58f2e51161
Do not echo back username on auth failure (#10097)
* Do not echo back username on auth failure

* use bad username

* Remove username from exception messages

* fix tests

* fix the tests

* hopefully this time

* this time the tests work

* fixed this time

* fix

* upgrade to Jetty 9.4.30

* Unknown users echo back Unauthorized

* fix
2020-07-10 12:19:10 -07:00
Nishant Bangarwa b7f4ce70cc
Update ambari-metrics-common to version 2.6.1.0.0 (#10165)
* Switch to apache version of ambari-metrics-common

* Add test

* Fix intellij inspection

* Fix intellij inspection
2020-07-10 12:18:40 -07:00
Maytas Monsereenusorn 54a8fb827d
Fix flaky tests in DruidCoordinatorTest (#10157)
* Fix flaky tests in DruidCoordinatorTest

* Imporve fail msg

* Fix flaky tests in DruidCoordinatorTest
2020-07-08 20:03:52 -07:00
Antoine Huret 88d20a61a6
renamed authenticationChain to authenticatorChain (#10143) 2020-07-08 19:58:21 -07:00
Gian Merlino 9587fc0b84
Fix documentation for Kinesis fetchThreads. (#10156)
* Fix documentation for Kinesis fetchThreads

The default was changed in #9819, but the documentation wasn't updated.

* Add 'procs' to spelling.
2020-07-08 19:47:09 -07:00
Jihoon Son 53a2550571
Follow-up for RetryQueryRunner fix (#10144)
* address comments; use guice instead of query context

* typo

* QueryResource tests

* address comments

* catch queryException

* fix spell check
2020-07-08 13:28:11 -07:00
Maytas Monsereenusorn 4e8570b71b
Add integration tests for all InputFormat (#10088)
* Add integration tests for Avro OCF InputFormat

* Add integration tests for Avro OCF InputFormat

* add tests

* fix bug

* fix bug

* fix failing tests

* add comments

* address comments

* address comments

* address comments

* fix test data

* reduce resource needed for IT

* remove bug fix

* fix checkstyle

* add bug fix
2020-07-08 12:50:29 -07:00
Maytas Monsereenusorn 859ff6e9c0
Reduce memory footprint of integration test by not starting unneeded containers (#10150)
* Reduce memory footprint of integration test

* fix README

* fix README

* fix error in script

* fix security IT
2020-07-08 09:46:18 -07:00
Franklyn Dsouza 1b9aacb1cd
Fix avg sql aggregator (#10135)
* new average aggregator

* method to create count aggregator factory

* test everything

* update other usages

* fix style

* fix more tests

* fix datasketches tests
2020-07-08 08:38:56 -07:00
Jihoon Son c776e412e0
Update dictionary for spell check (#10152) 2020-07-07 23:12:39 -07:00
Gian Merlino 11c0da8097
Add availability and consistency docs. (#10149)
* Add availability and consistency docs.

Describes transactional ingestion and atomic replacement. Also, this patch
deletes some bad advice from the javadocs for SegmentTransactionalInsertAction.

* Fix missing word.
2020-07-07 15:22:52 -07:00
Egor Riashin d54a5e009f
ui: fix missing columns during Transform step (#10086)
Co-authored-by: egor-ryashin <egor.ryashin@metamarkets.com>
2020-07-07 15:16:52 -07:00
Gian Merlino eeaf609fc0
Update Jetty to 9.4.30.v20200611. (#10098)
* Update Jetty to 9.4.30.v20200611.

This is the latest version currently available in the 9.4.x line.

* Various adjustments.

* Class name fixes.

* Remove unused HttpClientModule code.

* Add coverage suppressions.

* Another coverage suppression.

* Fix wildcards.
2020-07-07 14:24:02 -07:00
Parag Jain 98ac7dfeff
mask secrets in MM task command log (#10128)
* mask secrets in MM task command log

* unit test for masked iterator

* checkstyle fix
2020-07-07 10:25:15 -07:00
Clint Wylie 010fe047e1
AbstractOptimizableDimFilter should be public (#10142) 2020-07-06 15:19:32 -07:00
Clint Wylie c86e7ce30b
bump version to 0.20.0-SNAPSHOT (#10124) 2020-07-06 15:08:32 -07:00
Gian Merlino ddda2a4f18
VersionedIntervalTimeline: Fix thread-unsafe call to "lookup". (#10130) 2020-07-05 09:32:18 -07:00
Fullstop000 bcf41922ce
Remove unsupported task types in doc (#10111) 2020-07-04 18:13:53 -07:00
Jihoon Son 2b93dc6019
Fix CachingClusteredClient when querying specific segments (#10125)
* Fix CachingClusteredClient when querying specific segments

* delete useless test

* roll back timeout
2020-07-02 17:50:50 -07:00
Jonathan Wei ed981ef88e
Add DimFilter.toOptimizedFilter(), ensure that join filter pre-analysis operates on optimized filters (#10056)
* Ensure that join filter pre-analysis operates on optimized filters, add DimFilter.toOptimizedFilter

* Remove aggressive equality check that was used for testing

* Use Suppliers.memoize

* Checkstyle
2020-07-01 22:26:17 -07:00
Atul Mohan 367eaedbb4
Clarify change in behavior for druid.server.maxSize (#10105)
* Clarify maxSize docs

* Add info about maxSize

Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-07-01 22:22:18 -07:00