Commit Graph

11058 Commits

Author SHA1 Message Date
Xavier Léauté a1c20d7457
update jackson dependencies to use bom (#11353)
Switching to the bom dependency declaration simplifies managing jackson
dependencies. It also removes the need to override individual library
versions for CVE fixes, since the bom takes care of that internally.

This change aligns our jackson dependency versions on 2.10.5(.x):
- updates jackson libraries from 2.10.2 to 2.10.5
- jackson-databind remains at 2.10.5.1 as defined in the bom

Release notes: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10
2021-06-16 18:37:30 -07:00
Xavier Léauté 712f2a5d00
upgrade error-prone to 2.7.1 and support checks with Java 11+ (#11363)
* upgrade error-prone to 2.7.1 and support checks with Java 11+

- upgrade error-prone to 2.7.1
- support running error-prone with Java 11 and above using -Xplugin
  instead of custom compiler
- add compiler arguments to ignore warnings/errors in Java 15/16
- introduce strictCompile property to enable strict profiles since we
  now need multiple strict profiles for Java 8
- properly exclude all generated source files from error-prone
- fix druid-processing overriding annotation processors from parent pom
- fix druid-core disabling most non-default checks
- align plugin and annotation errorprone versions
- fix / suppress additional issues found by error-prone:
  * fix bug in SeekableStreamSupervisor initializing ArrayList size with
    the taskGroupdId
  * fix missing @Override annotations
- remove outdated compiler plugin in benchmarks
- remove deleted ParameterPackage error-prone rule
- re-enable checks on benchmark module as well

* fix IntelliJ inspections

* disable LongFloatConversion due to bug in error-prone with JDK 8

* add comment about InsecureCrypto
2021-06-16 12:55:34 -07:00
Clint Wylie bfbd7ec432
fix a bugs related to SQL type inference return type nullability (#11327)
* fix a bunch of type inference nullability bugs

* fixes

* style

* fix test

* fix concat
2021-06-15 12:26:59 -07:00
Clint Wylie 920aa414ca
enrich expression cache key information to support expressions which depend on external state (#11358)
* enrich expression cache key information to support expressions which depend on external state such as lookups

* cache rules everything around me

* low carb

* rename
2021-06-14 17:26:43 -07:00
Xavier Léauté 6edc017fad
update jacoco for Java 15/16 support (#11361) 2021-06-12 13:57:41 +08:00
Clint Wylie 50327b8f63
ignore bySegment query context for SQL queries (#11352)
* ignore bySegment query context for SQL queries

* revert unintended change
2021-06-11 13:49:03 -07:00
Charles Smith a1ed3a407d
clarify bySegment is native only (#11331) 2021-06-11 13:48:17 -07:00
John Gozde ac9b1f81b4
Web console: Remove support for IE11 and other older browsers (#11357)
* Use common browserlist and update to drop IE11

* Change TypeScript target to ES2016

* Update browserslist for "supports es6" support

* Show a warning if accessed from an unsupported browser

* Inline browser-update styles; detect SyntaxErrors too

* Better wording

* Upgrade to the latest Blueprint

* Refactor RunButton to be FC, use useHotkeys

* Remove dead license

* Update snapshots

* Address feedback

* Wording

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>
2021-06-10 19:05:40 -07:00
Clint Wylie 6b272c857f
adjust topn heap algorithm to only use known cardinality path when dictionary is unique (#11186)
* adjust topn heap algorithm to only use known cardinality path when dictionary is unique

* better check and add comment

* adjust comment more
2021-06-10 18:32:22 -05:00
Jihoon Son 51f983101e
Fix wrong encoding in PredicateFilteredDimensionSelector.getRow (#11339) 2021-06-10 09:14:34 -07:00
dependabot[bot] 167044f715
Bump fastutil from 8.2.3 to 8.5.4 (#11347)
* Bump fastutil from 8.2.3 to 8.5.4

Bumps [fastutil](https://github.com/vigna/fastutil) from 8.2.3 to 8.5.4.
- [Release notes](https://github.com/vigna/fastutil/releases)
- [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES)
- [Commits](https://github.com/vigna/fastutil/compare/8.2.3...8.5.4)

---
updated-dependencies:
- dependency-name: it.unimi.dsi:fastutil
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml
* update maven dependency list for -core and -extra libraries to pass maven dependency checks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-10 07:43:18 -07:00
Xavier Léauté db5ee83ca6
pin ZooKeeper dependencies to 3.5.x (#11354) 2021-06-10 07:41:53 -07:00
Yi Yuan 8de0d36c52
Allow query through router when load moving average extension (#11276)
* init commit

* change NoopQuerySegmentWalker name

* change doc

* move NoopQuerySegmentWalker and add document

* fix doc

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-06-10 18:46:53 +08:00
Egor Riashin 9047fa3d9c
S3 ingestion can assume role (#10995)
* feature s3 assume role

* feature s3 assume role

* feature s3 assume role

* feature s3 assume role

* feature s3 assume role

* feature s3 assume role

* tests fix

* spelling fix

* sts fix

Co-authored-by: egor-ryashin <egor.ryashin@rilldata.com>
2021-06-09 16:02:35 +05:30
Oops 651810f9e5
remove DEPRECATION part (#11326) 2021-06-09 15:52:43 +08:00
dependabot[bot] 1a27ed0b86
Bump junit from 4.12 to 4.13.2 (#11344)
Bumps [junit](https://github.com/junit-team/junit4) from 4.12 to 4.13.2.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.12.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.12...r4.13.2)

---
updated-dependencies:
- dependency-name: junit:junit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 18:18:39 -07:00
Xavier Léauté 8f4662d05a
restrict travis builds to main and release branches (#11349) 2021-06-08 16:34:17 -07:00
dependabot[bot] be10a236d5
Bump commons-io from 2.6 to 2.9.0 (#11338)
* Bump commons-io from 2.6 to 2.9.0

Bumps commons-io from 2.6 to 2.9.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml to reflect version bumps
* fix tests relying on specific log messages

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-08 10:02:59 -07:00
Yi Yuan 145cf9e5c3
fix document about input format (#11342)
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-06-08 23:44:54 +08:00
dependabot[bot] c82ede9afc
Bump maven-antrun-plugin from 1.8 to 3.0.0 (#11336)
Bumps [maven-antrun-plugin](https://github.com/apache/maven-antrun-plugin) from 1.8 to 3.0.0.
- [Release notes](https://github.com/apache/maven-antrun-plugin/releases)
- [Commits](https://github.com/apache/maven-antrun-plugin/compare/maven-antrun-plugin-1.8...maven-antrun-plugin-3.0.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-antrun-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 08:28:11 -07:00
frank chen 2ee7e31e5b
Fix syntax error (#11332) 2021-06-07 22:35:02 -07:00
dependabot[bot] 1e8b5360b3
Bump docker-java-transport-netty from 3.2.0 to 3.2.8 (#11337)
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java) from 3.2.0 to 3.2.8.
- [Release notes](https://github.com/docker-java/docker-java/releases)
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.0...3.2.8)

---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-07 18:58:38 -07:00
Xavier Léauté dbec8da7ea
enable dependabot for dependency updates (#11079)
* enable dependabot for dependency updates
* exclude dependabot config from license checks
2021-06-07 13:56:32 -07:00
Abhishek Agarwal 7ffe402b87
Fix priority parsing in HiLoQueryLaningStrategy (#11302) 2021-06-07 15:27:27 +05:30
Abhishek Agarwal 44d629319d
handle timestamps of complex types when parsing protobuf messages (#11293)
* handle timestamps correctly when parsing protobuf

* Add timestamp handling to ProtobufReader

* disable checkstyle for generated sourcecode

* Fix test

* try this

* refactor tests
2021-06-07 15:19:39 +05:30
Vadim Ogievetsky 0c5d1c9725
Web console: add more query fixing auto suggestions (#11203)
* add more query fixing auto suggestions

* update query gen

* update toolkit

* update licenses

* fix funky quotes

* funky => fancy

* revert engine change

* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
frank chen d5139c9543
Fix permission problems in docker (#11299)
* Create /opt/data to fix permission problem

* eliminate symlink to avoid compatibility problem on AWS Fargate

* Add a workaround section

* Update instruction for named volume

* Use named volume in docker-compose

* Revert some doc change

* Resolve review comments
2021-06-01 17:33:27 -07:00
zachjsh 27f1b6cbf3
Fix Index hadoop failing with index.zip is not a valid DFS filename (#11316)
* * Fix bug

* * simplify class loading

* * fix example configs for integration tests

* Small classloader cleanup

Co-authored-by: jon-wei <jon.wei@imply.io>
2021-06-01 19:14:50 -04:00
Hellis a24817d20a
Migrate helm chart from helm v2 to v3 (#11290)
* move all files from helm to helm/druid

* migrate druid helm chart from helm v2 to v3

* add license headers

* bump version to 0.3.0
2021-05-28 13:46:02 +08:00
frank chen e664bfd433
Improve doc of movingAverage (#11262)
* Make doc more directive

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Add limitation

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Suppress spelling check error
2021-05-28 13:10:55 +08:00
frank chen 04fefb0ca3
Fix ClassCastException (#11266)
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-05-27 21:25:51 -07:00
Vadim Ogievetsky 31c811d894
Web console: Fix maxRowsPerSegment validation in hashed compaction spec (#11308)
* allow defining of maxRowsPerSegment for now

* use common util

* update snapshots

* fix test

* fix e2e test
2021-05-27 16:36:42 -07:00
Maytas Monsereenusorn e5633d7842
Fix bug: 502 bad gateway thrown when we edit/delete any auto compaction config created 0.21.0 or before (#11311)
* fix bug

* add test

* fix IT

* fix checkstyle

* address comments
2021-05-27 16:34:32 -07:00
frank chen 60843bd11f
Add configuration suggestion to `druid.indexer.storage.type` (#11304) 2021-05-27 06:44:47 -07:00
Clint Wylie 7a2bcb3ca1
chill, travis (#11283)
* maybe make ci more selective in what it do

* maybe this

* skip self and travis config

* can it be in before_script? lets find out

* nope

* fixes

* oops

* only skip stuff for PR builds, add some tests

* revert unintended change

* more comments, more tests, more better
2021-05-27 05:40:40 -07:00
Rohan Garg 6c7177714f
Add test for join on __time column (#11289) 2021-05-26 22:20:39 -07:00
Xavier Léauté b517c3339b
remove ZooKeeper 3.4 support + pass tests with Java 15 (#11073)
With this change, Druid will only support ZooKeeper 3.5.x and later.

In order to support Java 15 we need to switch to ZK 3.5.x client libraries and drop support for ZK 3.4.x
(see #10780 for the detailed reasons) 

* remove ZooKeeper 3.4.x compatibility
* exclude additional ZK 3.5.x netty dependencies to ensure we use our version
* keep ZooKeeper version used for integration tests in sync with client library version
* remove the need to specify ZK version at runtime for docker
* add support to run integration tests with JDK 15
* build and run unit tests with Java 15 in travis
2021-05-25 12:49:49 -07:00
fhan 82380b67e0
Improve IT job 79 ITNestedQueryPushDownTest integration test (#11268)
* improve occasional failure caused by resource competition

* adjust more configs in tiny-cluters.yaml

Co-authored-by: yfhanfei <yfhanfei@ZBMac-C02DW5SMMD6P.local>
2021-05-24 10:12:34 +08:00
Clint Wylie 2bfcee5824
Fix issue with empty array converting to string expression instead of string array (#11270) 2021-05-22 09:31:28 +08:00
Agustin Gonzalez 4ba5738ffb
Add an issues section to deal with common issues when building druid (#11271) 2021-05-21 09:04:51 -07:00
Charles Smith 403dcf5cfb
fixes some typos, edits for style (#11258) 2021-05-21 08:58:39 -07:00
Charles Smith fcb4eaa3d4
add docs for high-churn datasource cleanup (#11245)
* add docs for high-churn datasource cleanup

* fix most comments except for task log

* address  comments

* update strategy recommendation

* address addtional comments

* fix

* address comments

* address comments from @sthetland
2021-05-20 09:48:42 -07:00
Maria Sitkovets 259207753d
Fix is null selector returning incorrect value for Long data type (#11170)
* Fix is null selector returning incorrect value for Long data type

* Fix style errors

* Refactor getObject method to also cache null column values

* Make lastInput variable nullable

* Refactor unit test

* Use new boolean lastInputIsNull instead of Long for lastInput to avoid boxing

* Refactor to remove Long for input variable

* Make a separate null caching variable

* Cleaner null caching implementation
2021-05-19 20:47:02 -07:00
Agustin Gonzalez 383daa4029
Catch exception inside ITRetryUtil to fix one of the causes for flaky integration tests (#11265)
* Do not stop retrying when an exception is encountered. Save & propagate last exception if retry count is exceeded.

* Add one more log message to help with debugging

* Limit schema registry heap to attempt to control OOMs
2021-05-19 13:56:02 -07:00
Prashanth Sagar 4c3077390f
Update check-licenses.py (#11269) 2021-05-19 13:52:00 -07:00
Clint Wylie 933350d106
integration test runner xmx (#11273)
* integration test runner xmx

* smaller
2021-05-19 12:59:50 -07:00
Yi Yuan 3be8e29269
Add integration test for protobuf (#11126)
* add file test

* test

* for test

* bug fixed

* test

* test

* test

* bug fixed

* delete auto scaler

* add input format

* add extensions

* bug fixed

* bug fixed

* bug fixed

* revert

* add schema registry test

* bug fixed

* bug fixed

* delete desc

* delete change

* add desc

* bug fixed

* test inputformat

* bug fixed

* bug fixed

* bug fixed

* bug fixed

* delete io exception

* change builder not static

* change pom

* bug fixed

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-05-17 15:45:07 -07:00
Clint Wylie 6d08a7051e
fix bug with aggregator expressions on realtime index with string columns always producing 0 values (#11185)
* fix bug with aggregator expressions on realtime index with string columns always producing 0 values

* more test

* rework some stuff

* javadocs
2021-05-17 11:59:13 -07:00
Clint Wylie aa62073faa
fix sql planner bug with inner offset causing loop (#11259)
* fix sql planner bug with inner offset causing loop

* move check up
2021-05-15 14:26:41 -07:00
Vadim Ogievetsky d85e8b708b
Web console: fix SegmentTimeline query (#11204)
* fix SegmentTimeline query

* test query
2021-05-14 14:48:23 -07:00