Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,387 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

11387 Commits

Author SHA1 Message Date
Maytas Monsereenusorn b36242b404
Fix bug in Variance Buffer Aggregator resulting in intermittent NaN when druid.generic.useDefaultValueForNull=false (#11617) 
* Fix bug in Variance Aggregator resulting in intermittent NaN when druid.generic.useDefaultValueForNull=false

* fix checkstyle

* address comments
 2021-08-20 09:13:51 -07:00
Clint Wylie ec334a641b
MySQL extension with MariaDB connector docs (#11608) 
* add docs for mariadb support via mysql extensions

* add logging so you know what druid knows

* homogenize

* spelling

* missed a couple
 2021-08-19 01:52:26 -07:00
hqx871 38ebaee0fd
VirtualColumnRegistry reuse virtual column should take account of value type (#11546) 
Co-authored-by: huangqixiang.871 <huangqixiang.871@bytedance.com>
 2021-08-19 01:46:27 -07:00
Maytas Monsereenusorn ce4dd48bb8
Support custom coordinator duties (#11601) 
* impl

* fix checkstyle

* fix checkstyle

* fix checkstyle

* add test

* add test

* add test

* add integration tests

* add integration tests

* add more docs

* address comments

* address comments

* address comments

* add test

* fix checkstyle

* fix test
 2021-08-19 11:54:11 +07:00
Charles Smith 91cd573472
fixes web console introduction and addresses linking issues (#11609) 
* fixes web console introduction and addresses  linking issues

* fix merge conflict
 2021-08-18 08:37:05 -07:00
Arvin.Z 504e54402b
update default compression format for bitmap (#11610) 
Co-authored-by: azheng <azheng@adobe.com>
 2021-08-18 14:54:27 +05:30
Karan Kumar d1bad92880
Made the instructions of adding extra resources as part of extensions simpler (#11577) 2021-08-17 17:33:55 +05:30
imply-jhan 332e68edb5
improve the metric definition (#11602) 2021-08-17 12:31:42 +07:00
Yi Yuan bf863343f8
delete some code (#11552) 
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
 2021-08-16 10:40:40 -07:00
Gian Merlino 4e5f9cdacf
Add pushes to DataSketches in SQL docs. (#11578) 
* Add pushes to DataSketches in SQL docs.

These notices were already in the native docs, but they were missing
from the SQL docs.

* Grammar fix.
 2021-08-16 10:38:56 -07:00
Jihoon Son 177264c649
resultFormat name in camel case (#11585) 
* resultFormat name in camel case

* test for letter case
 2021-08-14 18:30:21 +08:00
dependabot[bot] 776ddf76f4
Bump parquet.version from 1.11.1 to 1.12.0 (#11346) 
* Bump parquet.version from 1.11.1 to 1.12.0

Bumps `parquet.version` from 1.11.1 to 1.12.0.

Updates `parquet-column` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

Updates `parquet-avro` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

Updates `parquet-hadoop` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

---
updated-dependencies:
- dependency-name: org.apache.parquet:parquet-column
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-avro
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-hadoop
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update license

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
 2021-08-13 19:17:57 -07:00
Peter Marshall 8aaefb91e3
Docs - MiddleManager Affinity "strong" definition (#11480) 
* Affinity "strong" definition

Reworded "strong" to emphasise meaning and consequences - OTBO https://the-asf.slack.com/archives/CJ8D1JTB8/p1609558156092800

* Spelling corrections

* Update docs/configuration/index.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
 2021-08-13 19:17:16 -07:00
sthetland 95c5bc3a6d
Clarify when changes to credentialIterations take effect (#11590) 
This change updates doc to clarify when and how a change to druid.auth.authenticator.basic.credentialIterations takes effect: changes apply only to new users or existing users upon changing their password via the credentials API, which may not be the expectation.
 2021-08-13 17:02:07 -07:00
Parag Jain c7b46671b3
option to use deep storage for storing shuffle data (#11507) 
Fixes #11297.
Description

Description and design in the proposal #11297
Key changed/added classes in this PR

    *DataSegmentPusher
    *ShuffleClient
    *PartitionStat
    *PartitionLocation
    *IntermediaryDataManager
 2021-08-13 16:40:25 -04:00
frank chen e40be0ae28
Add SQL functions to format numbers into human readable format (#10635) 
* add binary_byte_format/decimal_byte_format/decimal_format

* clean code

* fix doc

* fix review comments

* add spelling check rules

* remove extra param

* improve type handling and null handling

* remove extra zeros

* fix tests and add space between unit suffix and number as most size-format functions do

* fix tests

* add examples

* change function names according to review comments

* fix merge

Signed-off-by: frank chen <frank.chen021@outlook.com>

* no need to configure NullHandling explicitly for tests

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix tests in SQL-Compatible mode

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Resolve review comments

* Update SQL test case to check null handling

* Fix intellij inspections

* Add more examples

* Fix example
 2021-08-13 10:27:49 -07:00
Charles Smith 6524d838d7
Docs refactor of ingestion. Carries #11541 (#11576) 
* Docs refactor of ingestion. Carries #11541

* Update docs/misc/math-expr.md

* add Apache license

* fix header, add topics to sidebar

* Update docs/ingestion/partitioning.md

* pick up changes to  and  md from c7fdf1d, #11479

Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
 2021-08-13 08:42:03 -07:00
Kashif Faraz aaf0aaad8f
Enable routing of SQL queries at Router (#11566) 
This PR adds a new property druid.router.sql.enable which allows the
Router to handle SQL queries when set to true.

This change does not affect Avatica JDBC requests and they are still routed
by hashing the Connection ID.

To allow parsing of the request object as a SqlQuery (contained in module druid-sql),
some classes have been moved from druid-server to druid-services with
the same package name.
 2021-08-13 18:44:39 +05:30
Harini Rajendran ccd362d228
Fix FileIteratingFirehoseTest to extend NullHandlingTest (#11581) 2021-08-12 08:26:04 -07:00
Rohan Garg 9da7199142
Lock hadoop dependencies to 2.8.5 (#11583) 2021-08-12 15:16:47 +05:30
Gian Merlino faebefecae
Docs: add pointers from api-reference to sql docs. (#11548) 2021-08-11 09:00:33 -07:00
Jonathan Wei 2a6421d0d9
Suppress CVEs for jdom2, kafka-clients, libthrift, solr-solrj (#11572) 2021-08-11 15:46:57 +05:30
Suneet Saldanha 640f63094a
fix little typo (#11573) 
* fix little typo

* Update docs/misc/math-expr.md

Co-authored-by: Charles Smith <techdocsmith@gmail.com>

Co-authored-by: Charles Smith <techdocsmith@gmail.com>
 2021-08-10 21:43:01 -07:00
Clint Wylie 9af7ba9d2a
STRING_AGG SQL aggregator function (#11241) 
* add string_agg

* oops

* style and fix test

* spelling

* fixup

* review stuffs
 2021-08-10 13:47:09 -07:00
Rohan Garg 2004a94675
Cleanup test dependencies in hdfs-storage extension (#11563) 
* Cleanup test dependencies in hdfs-storage extension

* Fix working directory in LocalFileSystem in indexing-hadoop test
 2021-08-10 07:52:32 -07:00
benkrug bef6f43e3d
Update math-expr.md (#11254) 
* Update math-expr.md
 2021-08-09 17:46:05 -07:00
frank chen bf5d829b71
Add more guidelines on the use of aliyun-oss-extensions (#11420) 
* Add more description

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Update prefixes usage and Add troubleshooting section

* Add endpoint configuration recommendation

* Fix link

* resolve review comments
 2021-08-09 17:27:35 -07:00
Charles Smith 941c5ffb05
clarify JVM tmp dir requires execute on files (#11542) 
* clarify JVM tmp dir requires execute on files

* code SysMonitor for spellcheck
 2021-08-09 17:25:10 -07:00
Clint Wylie f2ac6cd96e
fix parse exception handling for stream parsers (#11556) 
* fix parse exception handling

* fix style and inspections
 2021-08-09 12:40:44 -07:00
Maytas Monsereenusorn 06bae29979
Fix ingestion task failure when no input split to process (#11553) 
* fix ingestion task failure when no input split to process

* add IT

* fix IT
 2021-08-09 23:11:08 +07:00
Paul Rogers 3e7cba738f
Minor edits to architecture page to improve flow (#11465) 
* Minor edits to architecture page to improve flow

* Fixed spelling issue
 2021-08-09 07:48:29 -07:00
Abhishek Agarwal 2eff0902aa
suppress kafka-clients CVE (#11562) 
The CVE details are here - https://nvd.nist.gov/vuln/detail/CVE-2021-26291. I am marking it suppressed since we are only using kafka-clients jar in druid. We use maven-artifact jar ourselves but it is only used for comparing versions
 2021-08-09 19:02:25 +05:30
Jihoon Son e9d964d504
Improve concurrency between DruidSchema and BrokerServerView (#11457) 
* Improve concurrency between DruidSchema and BrokerServerView

* unused imports and workaround for error prone faiure

* count only known segments

* add comments
 2021-08-06 14:07:13 -07:00
Kashif Faraz 39a3db7943
Add unit test for config `druid.broker.segment.watchedTiers` (#11555) 2021-08-07 00:12:40 +05:30
Yi Yuan 59c8430d29
change document (#11545) 
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
 2021-08-06 07:57:12 -07:00
Vadim Ogievetsky 257bc5c62f
Web console: fix count aggregation input in the data loader (#11485) 
* add typeIs

* fix unused field in count metric

* better types

* typos

* work with readonly types

* factor out apply cancel buttons

* form editor

* selection type

* unsaved changes

* form editor spec

* tidy up sampler

* more menu controls

* update e2e test
 2021-08-05 12:30:30 -07:00
Peter Marshall 60e3955adb
Docs - clarify datasource API sources (#11489) 
* Update api-reference.md

Added note OTBO Druid slack

* Update api-reference.md

Changed to an alternative explanation

* Update api-reference.md

Oops fixed.

* Update docs/operations/api-reference.md

Co-authored-by: Suneet Saldanha <suneet@apache.org>

* Update docs/operations/api-reference.md

Co-authored-by: Suneet Saldanha <suneet@apache.org>

Co-authored-by: Suneet Saldanha <suneet@apache.org>
 2021-08-05 11:29:33 -07:00
Suneet Saldanha e423e99997
Update default maxSegmentsInNodeLoadingQueue (#11540) 
* Update default maxSegmentsInNodeLoadingQueue

Update the default maxSegmentsInNodeLoadingQueue from 0 (unbounded) to 100.

An unbounded maxSegmentsInNodeLoadingQueue can cause cluster instability.
Since this is the default druid operators need to run into this instability
and then look through the docs to see that the recommended value for a large
cluster is 1000. This change makes it so the default will prevent clusters
from falling over as they grow over time.

* update tests

* codestyle
 2021-08-05 11:26:58 -07:00
Maytas Monsereenusorn 3257913737
Improve query error logging (#11519) 
* Improve query error logging

* add docs

* address comments

* address comments
 2021-08-05 22:51:09 +07:00
Maytas Monsereenusorn 4470ca6a92
Fix hostname validation not skipping with `druid.client.https.validateHostnames=false` in java 8u275 and later (#11538) 
* fix skip hostname validation in java 8u275 and later

* add unit test

* fix checkstyle
 2021-08-05 15:42:55 +07:00
Yi Yuan 23d7d71ea5
Add Environment Variable DynamicConfigProvider (#11377) 
* add_environment_variable_DynamicConfigProvider

* fix code

* code fixed

* code fixed

* add document

* fix doc

* fix doc

* add more unit test

* fix style

* fix document

* bug fixed

* fix unit test

* fix comment

* fix test

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
 2021-08-04 20:26:58 -07:00
wx930910 578625b771
Replace TestInputRowHandler with mocking object (#11529) 
* Replace TestInputRowHandler with mocking object

* Change EasyMock object to Mockito object. Make test logic concise

* correct code format
 2021-08-04 16:45:22 -07:00
Suneet Saldanha 361bfdcaa5
Better logging for lookups (#11539) 
* Better logging for lookups

The default pollPeriod of 0 means that lookups are loaded once only at startup

Add a warning message to warn operators about this. I suspect that most
operators using jdbc or uri would expect eventual consistency with the source
of the lookups if using jdbc or uri. So make this a warning to make it easier
to debug if an operator notices a data inconsistency issue.

* oops
 2021-08-04 16:44:54 -07:00
dependabot[bot] 590583699f
Bump maven-enforcer-plugin from 3.0.0-M3 to 3.0.0 (#11543) 
Bumps [maven-enforcer-plugin](https://github.com/apache/maven-enforcer) from 3.0.0-M3 to 3.0.0.
- [Release notes](https://github.com/apache/maven-enforcer/releases)
- [Commits](https://github.com/apache/maven-enforcer/compare/enforcer-3.0.0-M3...enforcer-3.0.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-enforcer-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-04 08:43:49 -07:00
Rohan Garg 1a562f444c
Cleanup hadoop dependencies in indexing modules (#11516) 
* Remove hadoop-yarn-common dependency

(cherry picked from commit d767c8f3d204d9d27d8122d55680c3c9f1cfe473)

* Remove hdfs dependency from druid core
 2021-08-03 17:56:54 -07:00
dependabot[bot] eceacf74c0
Bump java-dogstatsd-client from 2.6.1 to 2.13.0 (#11533) 
Bumps [java-dogstatsd-client](https://github.com/DataDog/java-dogstatsd-client) from 2.6.1 to 2.13.0.
- [Release notes](https://github.com/DataDog/java-dogstatsd-client/releases)
- [Changelog](https://github.com/DataDog/java-dogstatsd-client/blob/master/CHANGELOG.md)
- [Commits](https://github.com/DataDog/java-dogstatsd-client/compare/java-dogstatsd-client-2.6.1...v2.13.0)

---
updated-dependencies:
- dependency-name: com.datadoghq:java-dogstatsd-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-03 17:53:45 -07:00
dependabot[bot] 511bc964ff
Bump docker-java-transport-netty from 3.2.8 to 3.2.11 (#11532) 
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java) from 3.2.8 to 3.2.11.
- [Release notes](https://github.com/docker-java/docker-java/releases)
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.8...3.2.11)

---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-03 17:53:22 -07:00
Yi Yuan aa7cb50f24
Add DynamicConfigProvider for Schema Registry (#11362) 
* add_DynamicConfigProvider_for_schema_registry

* bug fixed

* add document

* fix document

* fix spot bug

* fix document

* inject ObjectMapper

* add DynamicConfigProviderUtils

* add UT

* bug fixed

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
 2021-08-03 13:24:52 -07:00
frank chen 55a01a030a
Clarify that Broker caching for groupBy v2 queries does not work (#11370) 
* Add a note

* Update docs/configuration/index.md

Co-authored-by: sthetland <steve.hetland@imply.io>

* clarify that both of non-result level cache and result level cache are not supported

Co-authored-by: sthetland <steve.hetland@imply.io>
 2021-08-03 10:01:15 -07:00
Yi Yuan f1e52ab356
add doc (#11531) 
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
 2021-08-03 12:20:29 +08:00