78b4be467e
Add sleep function for testing ( #11626 )
...
* Add sleep function for testing
* sql function
* javadoc
2021-08-24 14:30:31 +07:00
59e560e24d
fix for numShards description ( #11611 )
...
Co-authored-by: devin-kim <devin.kim@kakaocorp.com>
2021-08-23 14:05:03 -07:00
66964a261b
fixes syntax for TRIM ( #11619 )
...
* fixes syntax for TRIM
* trim erroneous quote
* fix typo
2021-08-23 11:44:19 -07:00
b36242b404
Fix bug in Variance Buffer Aggregator resulting in intermittent NaN when druid.generic.useDefaultValueForNull=false ( #11617 )
...
* Fix bug in Variance Aggregator resulting in intermittent NaN when druid.generic.useDefaultValueForNull=false
* fix checkstyle
* address comments
2021-08-20 09:13:51 -07:00
ec334a641b
MySQL extension with MariaDB connector docs ( #11608 )
...
* add docs for mariadb support via mysql extensions
* add logging so you know what druid knows
* homogenize
* spelling
* missed a couple
2021-08-19 01:52:26 -07:00
38ebaee0fd
VirtualColumnRegistry reuse virtual column should take account of value type ( #11546 )
...
Co-authored-by: huangqixiang.871 <huangqixiang.871@bytedance.com>
2021-08-19 01:46:27 -07:00
ce4dd48bb8
Support custom coordinator duties ( #11601 )
...
* impl
* fix checkstyle
* fix checkstyle
* fix checkstyle
* add test
* add test
* add test
* add integration tests
* add integration tests
* add more docs
* address comments
* address comments
* address comments
* add test
* fix checkstyle
* fix test
2021-08-19 11:54:11 +07:00
91cd573472
fixes web console introduction and addresses linking issues ( #11609 )
...
* fixes web console introduction and addresses linking issues
* fix merge conflict
2021-08-18 08:37:05 -07:00
504e54402b
update default compression format for bitmap ( #11610 )
...
Co-authored-by: azheng <azheng@adobe.com>
2021-08-18 14:54:27 +05:30
d1bad92880
Made the instructions of adding extra resources as part of extensions simpler ( #11577 )
2021-08-17 17:33:55 +05:30
332e68edb5
improve the metric definition ( #11602 )
2021-08-17 12:31:42 +07:00
bf863343f8
delete some code ( #11552 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-16 10:40:40 -07:00
4e5f9cdacf
Add pushes to DataSketches in SQL docs. ( #11578 )
...
* Add pushes to DataSketches in SQL docs.
These notices were already in the native docs, but they were missing
from the SQL docs.
* Grammar fix.
2021-08-16 10:38:56 -07:00
177264c649
resultFormat name in camel case ( #11585 )
...
* resultFormat name in camel case
* test for letter case
2021-08-14 18:30:21 +08:00
776ddf76f4
Bump parquet.version from 1.11.1 to 1.12.0 ( #11346 )
...
* Bump parquet.version from 1.11.1 to 1.12.0
Bumps `parquet.version` from 1.11.1 to 1.12.0.
Updates `parquet-column` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases )
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md )
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0 )
Updates `parquet-avro` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases )
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md )
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0 )
Updates `parquet-hadoop` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases )
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md )
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0 )
---
updated-dependencies:
- dependency-name: org.apache.parquet:parquet-column
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-avro
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-hadoop
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update license
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-13 19:17:57 -07:00
8aaefb91e3
Docs - MiddleManager Affinity "strong" definition ( #11480 )
...
* Affinity "strong" definition
Reworded "strong" to emphasise meaning and consequences - OTBO https://the-asf.slack.com/archives/CJ8D1JTB8/p1609558156092800
* Spelling corrections
* Update docs/configuration/index.md
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
* Update docs/configuration/index.md
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2021-08-13 19:17:16 -07:00
95c5bc3a6d
Clarify when changes to credentialIterations take effect ( #11590 )
...
This change updates doc to clarify when and how a change to druid.auth.authenticator.basic.credentialIterations takes effect: changes apply only to new users or existing users upon changing their password via the credentials API, which may not be the expectation.
2021-08-13 17:02:07 -07:00
c7b46671b3
option to use deep storage for storing shuffle data ( #11507 )
...
Fixes #11297 .
Description
Description and design in the proposal #11297
Key changed/added classes in this PR
*DataSegmentPusher
*ShuffleClient
*PartitionStat
*PartitionLocation
*IntermediaryDataManager
2021-08-13 16:40:25 -04:00
e40be0ae28
Add SQL functions to format numbers into human readable format ( #10635 )
...
* add binary_byte_format/decimal_byte_format/decimal_format
* clean code
* fix doc
* fix review comments
* add spelling check rules
* remove extra param
* improve type handling and null handling
* remove extra zeros
* fix tests and add space between unit suffix and number as most size-format functions do
* fix tests
* add examples
* change function names according to review comments
* fix merge
Signed-off-by: frank chen <frank.chen021@outlook.com>
* no need to configure NullHandling explicitly for tests
Signed-off-by: frank chen <frank.chen021@outlook.com>
* fix tests in SQL-Compatible mode
Signed-off-by: frank chen <frank.chen021@outlook.com>
* Resolve review comments
* Update SQL test case to check null handling
* Fix intellij inspections
* Add more examples
* Fix example
2021-08-13 10:27:49 -07:00
6524d838d7
Docs refactor of ingestion. Carries #11541 ( #11576 )
...
* Docs refactor of ingestion. Carries #11541
* Update docs/misc/math-expr.md
* add Apache license
* fix header, add topics to sidebar
* Update docs/ingestion/partitioning.md
* pick up changes to and md from c7fdf1d#11479
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
2021-08-13 08:42:03 -07:00
aaf0aaad8f
Enable routing of SQL queries at Router ( #11566 )
...
This PR adds a new property druid.router.sql.enable which allows the
Router to handle SQL queries when set to true.
This change does not affect Avatica JDBC requests and they are still routed
by hashing the Connection ID.
To allow parsing of the request object as a SqlQuery (contained in module druid-sql),
some classes have been moved from druid-server to druid-services with
the same package name.
2021-08-13 18:44:39 +05:30
ccd362d228
Fix FileIteratingFirehoseTest to extend NullHandlingTest ( #11581 )
2021-08-12 08:26:04 -07:00
9da7199142
Lock hadoop dependencies to 2.8.5 ( #11583 )
2021-08-12 15:16:47 +05:30
faebefecae
Docs: add pointers from api-reference to sql docs. ( #11548 )
2021-08-11 09:00:33 -07:00
2a6421d0d9
Suppress CVEs for jdom2, kafka-clients, libthrift, solr-solrj ( #11572 )
2021-08-11 15:46:57 +05:30
640f63094a
fix little typo ( #11573 )
...
* fix little typo
* Update docs/misc/math-expr.md
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2021-08-10 21:43:01 -07:00
9af7ba9d2a
STRING_AGG SQL aggregator function ( #11241 )
...
* add string_agg
* oops
* style and fix test
* spelling
* fixup
* review stuffs
2021-08-10 13:47:09 -07:00
2004a94675
Cleanup test dependencies in hdfs-storage extension ( #11563 )
...
* Cleanup test dependencies in hdfs-storage extension
* Fix working directory in LocalFileSystem in indexing-hadoop test
2021-08-10 07:52:32 -07:00
bef6f43e3d
Update math-expr.md ( #11254 )
...
* Update math-expr.md
2021-08-09 17:46:05 -07:00
bf5d829b71
Add more guidelines on the use of aliyun-oss-extensions ( #11420 )
...
* Add more description
Signed-off-by: frank chen <frank.chen021@outlook.com>
* Update prefixes usage and Add troubleshooting section
* Add endpoint configuration recommendation
* Fix link
* resolve review comments
2021-08-09 17:27:35 -07:00
941c5ffb05
clarify JVM tmp dir requires execute on files ( #11542 )
...
* clarify JVM tmp dir requires execute on files
* code SysMonitor for spellcheck
2021-08-09 17:25:10 -07:00
f2ac6cd96e
fix parse exception handling for stream parsers ( #11556 )
...
* fix parse exception handling
* fix style and inspections
2021-08-09 12:40:44 -07:00
06bae29979
Fix ingestion task failure when no input split to process ( #11553 )
...
* fix ingestion task failure when no input split to process
* add IT
* fix IT
2021-08-09 23:11:08 +07:00
3e7cba738f
Minor edits to architecture page to improve flow ( #11465 )
...
* Minor edits to architecture page to improve flow
* Fixed spelling issue
2021-08-09 07:48:29 -07:00
2eff0902aa
suppress kafka-clients CVE ( #11562 )
...
The CVE details are here - https://nvd.nist.gov/vuln/detail/CVE-2021-26291 . I am marking it suppressed since we are only using kafka-clients jar in druid. We use maven-artifact jar ourselves but it is only used for comparing versions
2021-08-09 19:02:25 +05:30
e9d964d504
Improve concurrency between DruidSchema and BrokerServerView ( #11457 )
...
* Improve concurrency between DruidSchema and BrokerServerView
* unused imports and workaround for error prone faiure
* count only known segments
* add comments
2021-08-06 14:07:13 -07:00
39a3db7943
Add unit test for config `druid.broker.segment.watchedTiers` ( #11555 )
2021-08-07 00:12:40 +05:30
59c8430d29
change document ( #11545 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-06 07:57:12 -07:00
257bc5c62f
Web console: fix count aggregation input in the data loader ( #11485 )
...
* add typeIs
* fix unused field in count metric
* better types
* typos
* work with readonly types
* factor out apply cancel buttons
* form editor
* selection type
* unsaved changes
* form editor spec
* tidy up sampler
* more menu controls
* update e2e test
2021-08-05 12:30:30 -07:00
60e3955adb
Docs - clarify datasource API sources ( #11489 )
...
* Update api-reference.md
Added note OTBO Druid slack
* Update api-reference.md
Changed to an alternative explanation
* Update api-reference.md
Oops fixed.
* Update docs/operations/api-reference.md
Co-authored-by: Suneet Saldanha <suneet@apache.org>
* Update docs/operations/api-reference.md
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-05 11:29:33 -07:00
e423e99997
Update default maxSegmentsInNodeLoadingQueue ( #11540 )
...
* Update default maxSegmentsInNodeLoadingQueue
Update the default maxSegmentsInNodeLoadingQueue from 0 (unbounded) to 100.
An unbounded maxSegmentsInNodeLoadingQueue can cause cluster instability.
Since this is the default druid operators need to run into this instability
and then look through the docs to see that the recommended value for a large
cluster is 1000. This change makes it so the default will prevent clusters
from falling over as they grow over time.
* update tests
* codestyle
2021-08-05 11:26:58 -07:00
3257913737
Improve query error logging ( #11519 )
...
* Improve query error logging
* add docs
* address comments
* address comments
2021-08-05 22:51:09 +07:00
4470ca6a92
Fix hostname validation not skipping with `druid.client.https.validateHostnames=false` in java 8u275 and later ( #11538 )
...
* fix skip hostname validation in java 8u275 and later
* add unit test
* fix checkstyle
2021-08-05 15:42:55 +07:00
23d7d71ea5
Add Environment Variable DynamicConfigProvider ( #11377 )
...
* add_environment_variable_DynamicConfigProvider
* fix code
* code fixed
* code fixed
* add document
* fix doc
* fix doc
* add more unit test
* fix style
* fix document
* bug fixed
* fix unit test
* fix comment
* fix test
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-04 20:26:58 -07:00
578625b771
Replace TestInputRowHandler with mocking object ( #11529 )
...
* Replace TestInputRowHandler with mocking object
* Change EasyMock object to Mockito object. Make test logic concise
* correct code format
2021-08-04 16:45:22 -07:00
361bfdcaa5
Better logging for lookups ( #11539 )
...
* Better logging for lookups
The default pollPeriod of 0 means that lookups are loaded once only at startup
Add a warning message to warn operators about this. I suspect that most
operators using jdbc or uri would expect eventual consistency with the source
of the lookups if using jdbc or uri. So make this a warning to make it easier
to debug if an operator notices a data inconsistency issue.
* oops
2021-08-04 16:44:54 -07:00
590583699f
Bump maven-enforcer-plugin from 3.0.0-M3 to 3.0.0 ( #11543 )
...
Bumps [maven-enforcer-plugin](https://github.com/apache/maven-enforcer ) from 3.0.0-M3 to 3.0.0.
- [Release notes](https://github.com/apache/maven-enforcer/releases )
- [Commits](https://github.com/apache/maven-enforcer/compare/enforcer-3.0.0-M3...enforcer-3.0.0 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-enforcer-plugin
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 08:43:49 -07:00
1a562f444c
Cleanup hadoop dependencies in indexing modules ( #11516 )
...
* Remove hadoop-yarn-common dependency
(cherry picked from commit d767c8f3d204d9d27d8122d55680c3c9f1cfe473)
* Remove hdfs dependency from druid core
2021-08-03 17:56:54 -07:00
eceacf74c0
Bump java-dogstatsd-client from 2.6.1 to 2.13.0 ( #11533 )
...
Bumps [java-dogstatsd-client](https://github.com/DataDog/java-dogstatsd-client ) from 2.6.1 to 2.13.0.
- [Release notes](https://github.com/DataDog/java-dogstatsd-client/releases )
- [Changelog](https://github.com/DataDog/java-dogstatsd-client/blob/master/CHANGELOG.md )
- [Commits](https://github.com/DataDog/java-dogstatsd-client/compare/java-dogstatsd-client-2.6.1...v2.13.0 )
---
updated-dependencies:
- dependency-name: com.datadoghq:java-dogstatsd-client
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 17:53:45 -07:00
511bc964ff
Bump docker-java-transport-netty from 3.2.8 to 3.2.11 ( #11532 )
...
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java ) from 3.2.8 to 3.2.11.
- [Release notes](https://github.com/docker-java/docker-java/releases )
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.8...3.2.11 )
---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 17:53:22 -07:00
Jihoon Son
Daegi Kim
Charles Smith
Maytas Monsereenusorn
Clint Wylie
hqx871
Arvin.Z
Karan Kumar
imply-jhan
Yi Yuan
Gian Merlino
dependabot[bot]
Peter Marshall
sthetland
Parag Jain
frank chen
Kashif Faraz
Harini Rajendran
Rohan Garg
Jonathan Wei
Suneet Saldanha
benkrug
Charles Smith
Paul Rogers
Abhishek Agarwal
Vadim Ogievetsky
wx930910