Commit Graph

10674 Commits

Author SHA1 Message Date
Clint Wylie b36dab0fe6
fix connectionId issue with JDBC prepared statement queries and router (#10272)
* fix router jdbc prepared statement connectionId issue

* column metadata too

* style

* remove tls

* try tls again

* add keystore stuffs

* use keyManager password

* add unit test

* simplify
2020-08-19 00:18:06 -07:00
Jihoon Son 9a81740281
Don't log the entire task spec (#10278)
* Don't log the entire task spec

* fix lgtm

* fix serde

* address comments and add tests

* fix tests

* remove unnecessary codes
2020-08-18 11:03:13 -07:00
Suneet Saldanha 0891b1f833
Add note about aggregations on floats (#10285)
* Add note about aggreations on floats

Floating point math is known to be unstable. Due to the way aggregators work
across segments it's possible for the same query operating on the same data to
produce slightly different results.

The same problem exists with any aggregators that are not commutative since
the merge order across segments is not guaranteed.

* Also talk about doubles

* Apply suggestions from code review
2020-08-17 13:29:57 -07:00
Vatsal Bajpai ee40d00be1
typo fix from hear to here (#10292)
Should be `There are no other changes that need to be made here`
2020-08-17 07:54:21 -07:00
Xavier Léauté 225490474d
Update Kafka dependencies to 2.6.0 (#10286)
* update Kafka dependencies to Kafka 2.6.0
* switch to Scala 2.13 build of Kafka
* update integration tests
* update Kafka tutorial
2020-08-15 07:56:40 -07:00
Himanshu 12ae84165e
remove DruidLeaderClient.goAsync(..) that does not follow redirect. Replace its usage by DruidLeaderClient.go(..) with InputStreamFullResponseHandler (#9717)
* remove DruidLeaderClient.goAsync(..) that does not follow redirect.
Replace its usage by DruidLeaadereClient.go(..) with
InputStreamFullResponseHandler

* remove ByteArrayResponseHolder dependency from JsonParserIterator

* add UT to cover lines in InputStreamFullResponseHandler

* refactor SystemSchema to reduce branches

* further reduce branches

* Revert "add UT to cover lines in InputStreamFullResponseHandler"

This reverts commit 330aba3dd9.

* UTs for InputStreamFullResponseHandler

* remove unused imports
2020-08-14 10:51:18 -07:00
Gian Merlino 6cca7242de
Add "offset" parameter to the Scan query. (#10233)
* Add "offset" parameter to the Scan query.

It works by doing the query as normal and then throwing away the first
"offset" number of rows on the broker.

* Fix constructor call.

* Fix up JSONs.

* Fix call to ScanQuery.

* Doc update.

* Fix javadocs.

* Spotbugs, LGTM suppressions.

* Javadocs.

* Fix suppression.

* Stabilize Scan query result order, add tests.

* Update LGTM comment.

* Fixup.

* Test different batch sizes too.

* Nicer tests.

* Fix comment.
2020-08-13 14:56:24 -07:00
Clint Wylie e053348f74
add hasNulls to ColumnCapabilities, ColumnAnalysis (#10219)
* add isNullable to ColumnCapabilities, ColumnAnalysis

* better builder

* fix segment metadata queries in integration tests

* adjustments

* cleanup

* fix spotbugs

* treat unknown as true in segmentmetadata

* rename to hasNulls, add docs

* fixup

* test the dim indexer selector isNull fix for numeric columns

* fixes

* oof
2020-08-13 14:55:32 -07:00
Jihoon Son a61263b4a9
Allow forceLimitPushDown in SQL (#10253)
* Allow forceLimitPushDown in SQL

* fix test

* fix test

* review comments

* fix test
2020-08-13 13:30:41 -07:00
Vadim Ogievetsky 748a83cb78
Web console: fix json input (#10271)
* fix json input

* tidy up

* add error extraction test
2020-08-13 12:20:58 -07:00
Gian Merlino 89860b7d6a
Fix javadoc mistake in DefaultLimitSpec. (#10269)
Javadoc for getLimit should say it's a limit, not an offset.
2020-08-13 12:17:26 -07:00
Gian Merlino d36a0f61da
Clarify documentation on dimensions, dimensionExclusions. (#10265)
In particular: exclusions are ignored if dimensions are set.
2020-08-12 08:06:53 -07:00
Gian Merlino e273264332
Fix two id-over-maxId errors in StringDimensionIndexer. (#10245)
1) lookupId could return IDs beyond maxId if called with a recently added value.
2) getRow could return an ID for null beyond maxId, if null was recently
   encountered in a dimension that initially didn't appear at all. (In this case,
   the dictionary ID for null can be > 0).

Also add a comment explaining how this stuff is supposed to work.
2020-08-11 20:32:10 -07:00
Suneet Saldanha 6baea0b4d5
Fix broken sampler for re-indexing (#10196)
* Fix broken sampler for re-indexer

When re-indexing a Druid datasource, the web-console would generate an
invalid inputFormat since the type is not specified.

* code review
2020-08-11 19:22:26 -07:00
Clint Wylie c72f96a4ba
fix bug with expressions on sparse string realtime columns without explicit null valued rows (#10248)
* fix bug with realtime expressions on sparse string columns

* fix test

* add comment back

* push capabilities for dimensions to dimension indexers since they know things

* style

* style

* fixes

* getting a bit carried away

* missed one

* fix it

* benchmark build fix

* review stuffs

* javadoc and comments

* add comment

* more strict check

* fix missed usaged of impl instead of interface
2020-08-11 11:07:17 -07:00
Jihoon Son 35284e5166
Make stale bot less aggressive (#10261) 2020-08-10 20:59:02 -07:00
Abhishek Radhakrishnan dc16abae34
Vectorization support for long, double, float min & max aggregators. (#10260)
* LongMaxVectorAggregator support and test case.

* DoubleMinVectorAggregator and test cases.

* DoubleMaxVectorAggregator and unit test.

* FloatMinVectorAggregator and FloatMaxVectorAggregator.

* Documentation update to include the other vector aggregators.

* Bug fix.

* checkstyle formatting fixes.

* CalciteQueryTest cases update.

* Separate test classes for FloatMaxAggregation and FloatMniAggregation.

* remove the cannotVectorize for float max/min aggregator in test.

* Tests in GroupByQueryRunner, GroupByTimeseriesQueryRunner and TimeseriesQueryRunner.
2020-08-10 15:18:55 -07:00
Atul Mohan 06539bc828
Set default server.maxsize to the sum of segment cache (#10255)
* Default server.maxsize

* Remove maxsize refs from config

Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-08-10 09:21:22 -07:00
Vadim Ogievetsky 1fae0e15c9
improve JSON paste (#10256) 2020-08-07 17:01:06 -07:00
Gian Merlino 170031744e
Combine InDimFilter, InFilter. (#10119)
* Combine InDimFilter, InFilter.

There are two motivations:

1. Ensure that when HashJoinSegmentStorageAdapter compares its Filter
   to the original one, and it is an "in" type, the comparison is by
   reference and does not need to check deep equality. This is useful
   when the "in" filter is very large.
2. Simplify things. (There isn't a great reason for the DimFilter and
   Filter logic to be separate, and combining them reduces some
   duplication.)

* Fix test.
2020-08-06 18:34:21 -07:00
Gian Merlino b6aaf59e8c
Add "offset" parameter to GroupBy query. (#10235)
* Add "offset" parameter to GroupBy query.

It works by doing the query as normal and then throwing away the first
"offset" number of rows on the broker.

* Stabilize GroupBy sorts.

* Fix inspections.

* Fix suppression.

* Fixups.

* Move TopNSequence to druid-core.

* Addl comments.

* NumberedElement equals verification.

* Changes from review.
2020-08-05 15:39:58 -07:00
Vadim Ogievetsky 9a29496b6c
Web console: Improve retention rules dialog in all sorts of ways (#10226)
* improve ret rules

* tidy up tests
2020-08-04 16:43:31 -07:00
keefe roedersheimer 4bb198eb4c
change search filter to includes (#10141) 2020-08-04 16:37:07 -07:00
Abhishek Radhakrishnan 34a4113752
Add vectorization support for the longMin aggregator. (#10211)
* Fix minor formatting in docs.

* Add Nullhandling initialization for test to run from IDE.

* Vectorize longMin aggregator.

- A new vectorized class for the vectorized long min aggregator.
- Changes to AggregatorFactory to support vectorize functionality.
- Few changes to schema evolution test to add LongMinAggregatorFactory.

* Add longSum to the supported vectorized aggregator implementations.

* Add MIN() long min to calcite query test that can vectorize.

* Add simple long aggregations test.

* Fixup formatting per checkstyle guide.

* fixup and add more tests for long min aggregator.

* Override test for groupBy since timestamps are handled differently.

* Null compatibility check in test.

* Review comment: Add a test case to LongMinAggregationTest.
2020-08-01 15:32:09 -07:00
Clint Wylie cfb7a893e7
fill out missing test coverage for druid-datasketches postaggs (#9730)
* fill out missing test coverage for druid-datasketches postaggs

* fixup

* fixup merge

* oops

* oops again
2020-07-31 10:08:07 -07:00
frank chen 646fa84d04
Support unit on byte-related properties (#10203)
* support unit suffix on byte-related properties

* add doc

* change default value of byte-related properites in example files

* fix coding style

* fix doc

* fix CI

* suppress spelling errors

* improve code according to comments

* rename Bytes to HumanReadableBytes

* add getBytesInInt to get value safely

* improve doc

* fix problem reported by CI

* fix problem reported by CI

* resolve code review comments

* improve error message

* improve code & doc according to comments

* fix CI problem

* improve doc

* suppress spelling check errors
2020-07-31 09:58:48 +08:00
Jian Wang 271f90f205
Add segment pruning for hash based shard spec (#9810)
* Add segment pruning for hash based partitioning

* Update doc

* Add additional test

* Address comments

* Fix unit test failure

Co-authored-by: Jian Wang <jwang@pinterest.com>
2020-07-30 18:44:26 -07:00
Maytas Monsereenusorn 574b062f1f
Cluster wide default query context setting (#10208)
* Cluster wide default query context setting

* Cluster wide default query context setting

* Cluster wide default query context setting

* add docs

* fix docs

* update props

* fix checkstyle

* fix checkstyle

* fix checkstyle

* update docs

* address comments

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix checkstyle

* fix NPE
2020-07-29 15:19:18 -07:00
Jihoon Son 63c1746fe4
Fix timeseries query constructor when postAggregator has an expression reading timestamp result column (#10198)
* Fix timeseries query constructor when postAggregator has an expression reading timestamp result column

* fix npe

* Fix postAgg referencing timestampResultField and add a test for it

* fix test

* doc

* revert doc
2020-07-27 10:54:44 -07:00
Suneet Saldanha 2f28be3f2a
Suppress CVE-2020-7692 (#10214)
Druid is not a native app, so this CVE should not apply.
2020-07-27 10:52:44 -07:00
Clint Wylie 79dffefbf8
add explicit example for jdbc query context on connection properties (#10182)
* add explicit example for jdbc query context on connection properties

* make comment clearer

* Update sql.md

* Update sql.md
2020-07-24 13:43:04 -07:00
Vadim Ogievetsky 6d8799f2df
Update QueryView to use latest DruidQueryToolkit (#10201)
* Update to latest DruidQueryToolkit

* add THEN keyword

* do not crash on invalid JSON
2020-07-23 22:45:01 -07:00
Richard Startin e363b1cd20
Update RoaringBitmap to 0.9.0 (#9987) 2020-07-23 19:29:25 -07:00
Jihoon Son 6fdce36e41
Add integration tests for query retry on missing segments (#10171)
* Add integration tests for query retry on missing segments

* add missing dependencies; fix travis conf

* address comments

* Integration tests extension

* remove unused dependency

* remove druid_main

* fix java agent port
2020-07-22 22:30:35 -07:00
Jihoon Son 26d099f39b
Fix sys.servers table to not throw NPE and handle brokers/indexers/peons properly for broadcast segments (#10183)
* Fix sys.servers table to not throw NPE and handle brokers/indexers/peons properly for broadcast segments

* fix tests and add missing tests

* revert null handling fix

* unused import

* move out util methods from DiscoveryDruidNode
2020-07-21 17:52:51 -07:00
mans2singh d4bd6e5207
ingestion and tutorial doc update (#10202) 2020-07-21 17:52:23 -07:00
Clint Wylie 266243ac75
include staged maven artifacts in example vote thread (#10200) 2020-07-21 16:30:28 -07:00
Maytas Monsereenusorn dd7a32ad48
Fix ITSqlInputSourceTest (#10194)
* Fix ITSqlInputSourceTest.java

* Fix ITSqlInputSourceTest.java

* Fix ITSqlInputSourceTest.java

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2020-07-21 09:52:13 -07:00
Jihoon Son 41982116f4
Report missing segments when there is no segment for the query datasource in historicals (#10199)
* Report missing segments when there is no segment for the query
datasource in historicals

* test

* missing part for test

* another test
2020-07-20 21:02:52 -07:00
Clint Wylie 0deefd6408
update release process guide to include web-console versions (#10176) 2020-07-20 13:59:06 -07:00
Maytas Monsereenusorn 0cabc53bd5
Add integration tests for Appends (#10186)
* append test

* add append IT

* fix checkstyle

* fix checkstyle

* Remove parallel

* fix checkstyle

* fix

* fix

* address comments

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix
2020-07-20 13:43:13 -07:00
Suneet Saldanha eeb9012743
Change color of Run button for native queries (#10170)
* Change color of Run button for native queries

When a user tries to run a native query, change the color of the button to
Druid's secondary color to indicate that the user is not running a SQL query.

Before this change, the web-console would indicate this by changing the text
of the button from Run (SQL queries) to Rune (native queries). Rune could be
confusing to users as this appears to be a typo.

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* Update web-console/src/views/query-view/run-button/run-button.scss

* code review
2020-07-15 15:46:02 -07:00
Maytas Monsereenusorn 454eed06a8
JettyTest.testNumConnectionsMetricHttp is rarely flaky (#10169) 2020-07-14 10:36:46 -07:00
Suneet Saldanha e6c9142129
Add validation for authenticator and authorizer name (#10106)
* Add validation for authorizer name

* fix deps

* add javadocs

* Do not use resource filters

* Fix BasicAuthenticatorResource as well

* Add integration tests

* fix test

* fix
2020-07-13 21:15:54 -07:00
Nishant Bangarwa 971d8a353b
Add groupBy limitSpec to queryCache key (#10093)
* Add groupBy limitSpec to queryCache key

* Only add limitSpec to cache key if pushdown is set to true

* review comment
2020-07-13 19:15:09 -07:00
Nishant Bangarwa 2b48de074a
Add additional properties for Kafka AdminClient and consumer from test config file (#10137)
* Add kafka test configs from file for AdminClient and consumer

* review comment
2020-07-13 18:16:05 +05:30
Joseph Glanville f3023c6058
Fix formatting in druid-pac4j documentation (#10174)
Superfluous column broke table formatting.
2020-07-12 18:51:42 -07:00
Clint Wylie 0a8fd62f22
split web-console e2e-tests from unit tests (#10173)
* split web-console e2e-test from unit test

* fix stuff

* smaller change

* oops
2020-07-12 18:51:08 -07:00
Clint Wylie ebea73d081
fix website build (#10172)
* fix mvn website build to use mvn supplied nodejs, fix broken redirects, move block from custom.css to custom.scss so will be correctly generated

* sidebar

* fix lol
2020-07-11 22:01:43 -07:00
Suneet Saldanha 58f2e51161
Do not echo back username on auth failure (#10097)
* Do not echo back username on auth failure

* use bad username

* Remove username from exception messages

* fix tests

* fix the tests

* hopefully this time

* this time the tests work

* fixed this time

* fix

* upgrade to Jetty 9.4.30

* Unknown users echo back Unauthorized

* fix
2020-07-10 12:19:10 -07:00