2a6421d0d9
Suppress CVEs for jdom2, kafka-clients, libthrift, solr-solrj ( #11572 )
2021-08-11 15:46:57 +05:30
640f63094a
fix little typo ( #11573 )
...
* fix little typo
* Update docs/misc/math-expr.md
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2021-08-10 21:43:01 -07:00
9af7ba9d2a
STRING_AGG SQL aggregator function ( #11241 )
...
* add string_agg
* oops
* style and fix test
* spelling
* fixup
* review stuffs
2021-08-10 13:47:09 -07:00
2004a94675
Cleanup test dependencies in hdfs-storage extension ( #11563 )
...
* Cleanup test dependencies in hdfs-storage extension
* Fix working directory in LocalFileSystem in indexing-hadoop test
2021-08-10 07:52:32 -07:00
bef6f43e3d
Update math-expr.md ( #11254 )
...
* Update math-expr.md
2021-08-09 17:46:05 -07:00
bf5d829b71
Add more guidelines on the use of aliyun-oss-extensions ( #11420 )
...
* Add more description
Signed-off-by: frank chen <frank.chen021@outlook.com>
* Update prefixes usage and Add troubleshooting section
* Add endpoint configuration recommendation
* Fix link
* resolve review comments
2021-08-09 17:27:35 -07:00
941c5ffb05
clarify JVM tmp dir requires execute on files ( #11542 )
...
* clarify JVM tmp dir requires execute on files
* code SysMonitor for spellcheck
2021-08-09 17:25:10 -07:00
f2ac6cd96e
fix parse exception handling for stream parsers ( #11556 )
...
* fix parse exception handling
* fix style and inspections
2021-08-09 12:40:44 -07:00
06bae29979
Fix ingestion task failure when no input split to process ( #11553 )
...
* fix ingestion task failure when no input split to process
* add IT
* fix IT
2021-08-09 23:11:08 +07:00
3e7cba738f
Minor edits to architecture page to improve flow ( #11465 )
...
* Minor edits to architecture page to improve flow
* Fixed spelling issue
2021-08-09 07:48:29 -07:00
2eff0902aa
suppress kafka-clients CVE ( #11562 )
...
The CVE details are here - https://nvd.nist.gov/vuln/detail/CVE-2021-26291 . I am marking it suppressed since we are only using kafka-clients jar in druid. We use maven-artifact jar ourselves but it is only used for comparing versions
2021-08-09 19:02:25 +05:30
e9d964d504
Improve concurrency between DruidSchema and BrokerServerView ( #11457 )
...
* Improve concurrency between DruidSchema and BrokerServerView
* unused imports and workaround for error prone faiure
* count only known segments
* add comments
2021-08-06 14:07:13 -07:00
39a3db7943
Add unit test for config `druid.broker.segment.watchedTiers` ( #11555 )
2021-08-07 00:12:40 +05:30
59c8430d29
change document ( #11545 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-06 07:57:12 -07:00
257bc5c62f
Web console: fix count aggregation input in the data loader ( #11485 )
...
* add typeIs
* fix unused field in count metric
* better types
* typos
* work with readonly types
* factor out apply cancel buttons
* form editor
* selection type
* unsaved changes
* form editor spec
* tidy up sampler
* more menu controls
* update e2e test
2021-08-05 12:30:30 -07:00
60e3955adb
Docs - clarify datasource API sources ( #11489 )
...
* Update api-reference.md
Added note OTBO Druid slack
* Update api-reference.md
Changed to an alternative explanation
* Update api-reference.md
Oops fixed.
* Update docs/operations/api-reference.md
Co-authored-by: Suneet Saldanha <suneet@apache.org>
* Update docs/operations/api-reference.md
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-05 11:29:33 -07:00
e423e99997
Update default maxSegmentsInNodeLoadingQueue ( #11540 )
...
* Update default maxSegmentsInNodeLoadingQueue
Update the default maxSegmentsInNodeLoadingQueue from 0 (unbounded) to 100.
An unbounded maxSegmentsInNodeLoadingQueue can cause cluster instability.
Since this is the default druid operators need to run into this instability
and then look through the docs to see that the recommended value for a large
cluster is 1000. This change makes it so the default will prevent clusters
from falling over as they grow over time.
* update tests
* codestyle
2021-08-05 11:26:58 -07:00
3257913737
Improve query error logging ( #11519 )
...
* Improve query error logging
* add docs
* address comments
* address comments
2021-08-05 22:51:09 +07:00
4470ca6a92
Fix hostname validation not skipping with `druid.client.https.validateHostnames=false` in java 8u275 and later ( #11538 )
...
* fix skip hostname validation in java 8u275 and later
* add unit test
* fix checkstyle
2021-08-05 15:42:55 +07:00
23d7d71ea5
Add Environment Variable DynamicConfigProvider ( #11377 )
...
* add_environment_variable_DynamicConfigProvider
* fix code
* code fixed
* code fixed
* add document
* fix doc
* fix doc
* add more unit test
* fix style
* fix document
* bug fixed
* fix unit test
* fix comment
* fix test
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-04 20:26:58 -07:00
578625b771
Replace TestInputRowHandler with mocking object ( #11529 )
...
* Replace TestInputRowHandler with mocking object
* Change EasyMock object to Mockito object. Make test logic concise
* correct code format
2021-08-04 16:45:22 -07:00
361bfdcaa5
Better logging for lookups ( #11539 )
...
* Better logging for lookups
The default pollPeriod of 0 means that lookups are loaded once only at startup
Add a warning message to warn operators about this. I suspect that most
operators using jdbc or uri would expect eventual consistency with the source
of the lookups if using jdbc or uri. So make this a warning to make it easier
to debug if an operator notices a data inconsistency issue.
* oops
2021-08-04 16:44:54 -07:00
590583699f
Bump maven-enforcer-plugin from 3.0.0-M3 to 3.0.0 ( #11543 )
...
Bumps [maven-enforcer-plugin](https://github.com/apache/maven-enforcer ) from 3.0.0-M3 to 3.0.0.
- [Release notes](https://github.com/apache/maven-enforcer/releases )
- [Commits](https://github.com/apache/maven-enforcer/compare/enforcer-3.0.0-M3...enforcer-3.0.0 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-enforcer-plugin
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 08:43:49 -07:00
1a562f444c
Cleanup hadoop dependencies in indexing modules ( #11516 )
...
* Remove hadoop-yarn-common dependency
(cherry picked from commit d767c8f3d204d9d27d8122d55680c3c9f1cfe473)
* Remove hdfs dependency from druid core
2021-08-03 17:56:54 -07:00
eceacf74c0
Bump java-dogstatsd-client from 2.6.1 to 2.13.0 ( #11533 )
...
Bumps [java-dogstatsd-client](https://github.com/DataDog/java-dogstatsd-client ) from 2.6.1 to 2.13.0.
- [Release notes](https://github.com/DataDog/java-dogstatsd-client/releases )
- [Changelog](https://github.com/DataDog/java-dogstatsd-client/blob/master/CHANGELOG.md )
- [Commits](https://github.com/DataDog/java-dogstatsd-client/compare/java-dogstatsd-client-2.6.1...v2.13.0 )
---
updated-dependencies:
- dependency-name: com.datadoghq:java-dogstatsd-client
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 17:53:45 -07:00
511bc964ff
Bump docker-java-transport-netty from 3.2.8 to 3.2.11 ( #11532 )
...
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java ) from 3.2.8 to 3.2.11.
- [Release notes](https://github.com/docker-java/docker-java/releases )
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.8...3.2.11 )
---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-03 17:53:22 -07:00
aa7cb50f24
Add DynamicConfigProvider for Schema Registry ( #11362 )
...
* add_DynamicConfigProvider_for_schema_registry
* bug fixed
* add document
* fix document
* fix spot bug
* fix document
* inject ObjectMapper
* add DynamicConfigProviderUtils
* add UT
* bug fixed
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-03 13:24:52 -07:00
55a01a030a
Clarify that Broker caching for groupBy v2 queries does not work ( #11370 )
...
* Add a note
* Update docs/configuration/index.md
Co-authored-by: sthetland <steve.hetland@imply.io>
* clarify that both of non-result level cache and result level cache are not supported
Co-authored-by: sthetland <steve.hetland@imply.io>
2021-08-03 10:01:15 -07:00
f1e52ab356
add doc ( #11531 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-08-03 12:20:29 +08:00
a2da407b70
Add error msg to parallel task's TaskStatus ( #11486 )
...
* Add error msg to parallel task's TaskStatus
* Consolidate failure block
* Add failure test
* Make it fail
* Add fail while stopped
* Simplify hash task test using a runner that fails after so many runs (parameter)
* Remove unthrown exception
* Use runner names to identify phase
* Added range partition kill test & fixed a timing bug with the custom runner
* Forbidden api
* Style
* Unit test code cleanup
* Added message to invalid state exception and improved readability of the phase error messages for the parallel task failure unit tests
2021-08-02 12:11:28 -07:00
cf674c833c
Bump maven-resources-plugin from 3.1.0 to 3.2.0 ( #11525 )
...
Bumps [maven-resources-plugin](https://github.com/apache/maven-resources-plugin ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/apache/maven-resources-plugin/releases )
- [Commits](https://github.com/apache/maven-resources-plugin/compare/maven-resources-plugin-3.1.0...maven-resources-plugin-3.2.0 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-02 09:38:34 -07:00
2e850b5655
Bump aws.sdk.version from 1.11.884 to 1.12.37 ( #11367 )
...
* Bump aws.sdk.version from 1.11.884 to 1.12.8
Bumps `aws.sdk.version` from 1.11.884 to 1.12.8.
Updates `aws-java-sdk-core` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
Updates `aws-java-sdk-ec2` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
Updates `aws-java-sdk-s3` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
Updates `aws-java-sdk-sts` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
Updates `aws-java-sdk-kinesis` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
Updates `aws-java-sdk-rds` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases )
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8 )
---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-core
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-ec2
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-s3
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-sts
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-kinesis
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-rds
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Fix license and bump to latest aws
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-01 00:12:57 -07:00
dbed4424b5
Upgrade ORC to 1.6.9 ( #11518 )
2021-07-31 23:33:03 -07:00
8ba7f6a48c
Fix incorrect result of exact topN on an inner join with limit ( #11517 )
2021-07-31 15:55:49 -07:00
98312d54cf
Fix CI for master ( #11522 )
2021-07-30 15:41:21 -07:00
949484728f
docs fix for doubleMean description ( #11513 )
...
* fix for doubleMean description
* include quantile aggregator description from Suneet
* update hyperlink to quantiles aggregator
2021-07-30 12:39:44 -07:00
05a7da792f
compaction/status API retains status for datasources that no longer existed causing in-memory used to grow unbounded ( #11510 )
...
* fix compaction status api
* fix checkstyle
* address comment
2021-07-30 22:19:24 +07:00
995d99d9e4
add ingest/notices/queueSize metric to give visibility into supervisor notices queue size ( #11417 )
2021-07-30 07:59:26 -07:00
b83742179a
Reduce method invocation of reservoir sampling ( #11257 )
...
* reduce method invocation of reservoir sampling
* add a dynamic parameter and add benchmark
* rebase
2021-07-30 22:09:50 +08:00
59e61e127a
Add a section for updating the release branch before making a tag in the release process ( #11449 )
2021-07-29 17:05:33 -07:00
4bca7f014e
update error-prone to 2.8.0 with fix for crashing check ( #11494 )
...
* error-prone 2.8.0 fixes https://github.com/google/error-prone/issues/2396
* fix for a few ignored return values
* fix unknown args in sub-modules
2021-07-29 09:13:46 -07:00
9b250c54aa
Allow kill task to mark segments as unused ( #11501 )
...
* Allow kill task to mark segments as unused
* Add IndexerSQLMetadataStorageCoordinator test
* Update docs/ingestion/data-management.md
Co-authored-by: Jihoon Son <jihoonson@apache.org>
* Add warning to kill task doc
Co-authored-by: Jihoon Son <jihoonson@apache.org>
2021-07-29 10:48:43 -05:00
280c08045f
Update awesome-code-style ( #11503 )
2021-07-28 09:25:18 -07:00
0de1837ff7
Docs - partitioning note re: skew / dim concatenation + nav update ( #11488 )
...
* Update native-batch.md
Knowledge from https://the-asf.slack.com/archives/CJ8D1JTB8/p1595434977062400
* Update native-batch.md
* Fixed broken link + some grammar
2021-07-27 09:17:01 -07:00
8a4e27f51d
Select broker based on query context parameter `brokerService` ( #11495 )
...
This change allows the selection of a specific broker service (or broker tier) by the Router.
The newly added ManualTieredBrokerSelectorStrategy works as follows:
Check for the parameter brokerService in the query context. If this is a valid broker service, use it.
Check if the field defaultManualBrokerService has been set in the strategy. If this is a valid broker service, use it.
Move on to the next strategy
2021-07-27 20:56:05 +05:30
60fdf7a734
Rollup measurement query amended ( #11479 )
...
By user request from https://groups.google.com/g/druid-user/c/bFkOtE-1eQg - gives the measure as a floating point instead of an integer.
2021-07-27 06:29:29 -07:00
676efb1b3f
Fix integration test credential resource path handling ( #11487 )
...
This PR fixes an issue with the integration test copy_resources.sh script.
The "install druid jars" portion was removing the $SHARED_DIR/docker directory, which wipes out the $SHARED_DIR/docker/extensions and $SHARED_DIR/docker/credentials directories created just before, which leads to issues later in the script when copying resources to the $SHARED_DIR/docker/credentials/ dir.
2021-07-27 12:32:34 +05:30
c068906fca
Make intermediate store for shuffle tasks an extension point ( #11492 )
...
* add interface
* add docs
* fix errors
* fix injection
* fix injection
* update javadoc
2021-07-27 11:29:43 +07:00
3f456fe305
Address CVE-2021-35515 CVE-2021-36090 ( #11496 )
...
* Address CVE-2021-35515 CVE-2021-36090
Bump commons-compress to deal with new CVEs
* fix licenses
2021-07-26 14:54:32 -07:00
973e5bf7d0
Docs - HLL lgK tip and slight layout change ( #11482 )
...
* HLL lgK and a tip
Knowledge transfer from https://the-asf.slack.com/archives/CJ8D1JTB8/p1600699967024200 . Attempted to make a connection between the SQL HLL function and the HLL underneath without getting too complicated. Also added a note about using K over 16 being pretty much pointless.
* Corrected spelling
* Create datasketches-hll.md
Put roll-up back to rollup
* Update docs/development/extensions-core/datasketches-hll.md
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
2021-07-26 12:28:53 -07:00
Jonathan Wei
Suneet Saldanha
Clint Wylie
Rohan Garg
benkrug
frank chen
Charles Smith
Maytas Monsereenusorn
Paul Rogers
Abhishek Agarwal
Jihoon Son
Kashif Faraz
Yi Yuan
Vadim Ogievetsky
Peter Marshall
wx930910
dependabot[bot]
Agustin Gonzalez
Dongjoon Hyun
Victoria Lim
Harini Rajendran
Yuanli Han
Xavier Léauté
John Gozde