db5ee83ca6
pin ZooKeeper dependencies to 3.5.x ( #11354 )
2021-06-10 07:41:53 -07:00
8de0d36c52
Allow query through router when load moving average extension ( #11276 )
...
* init commit
* change NoopQuerySegmentWalker name
* change doc
* move NoopQuerySegmentWalker and add document
* fix doc
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-06-10 18:46:53 +08:00
9047fa3d9c
S3 ingestion can assume role ( #10995 )
...
* feature s3 assume role
* feature s3 assume role
* feature s3 assume role
* feature s3 assume role
* feature s3 assume role
* feature s3 assume role
* tests fix
* spelling fix
* sts fix
Co-authored-by: egor-ryashin <egor.ryashin@rilldata.com>
2021-06-09 16:02:35 +05:30
651810f9e5
remove DEPRECATION part ( #11326 )
2021-06-09 15:52:43 +08:00
1a27ed0b86
Bump junit from 4.12 to 4.13.2 ( #11344 )
...
Bumps [junit](https://github.com/junit-team/junit4 ) from 4.12 to 4.13.2.
- [Release notes](https://github.com/junit-team/junit4/releases )
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.12.md )
- [Commits](https://github.com/junit-team/junit4/compare/r4.12...r4.13.2 )
---
updated-dependencies:
- dependency-name: junit:junit
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 18:18:39 -07:00
8f4662d05a
restrict travis builds to main and release branches ( #11349 )
2021-06-08 16:34:17 -07:00
be10a236d5
Bump commons-io from 2.6 to 2.9.0 ( #11338 )
...
* Bump commons-io from 2.6 to 2.9.0
Bumps commons-io from 2.6 to 2.9.0.
---
updated-dependencies:
- dependency-name: commons-io:commons-io
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml to reflect version bumps
* fix tests relying on specific log messages
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-08 10:02:59 -07:00
145cf9e5c3
fix document about input format ( #11342 )
...
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-06-08 23:44:54 +08:00
c82ede9afc
Bump maven-antrun-plugin from 1.8 to 3.0.0 ( #11336 )
...
Bumps [maven-antrun-plugin](https://github.com/apache/maven-antrun-plugin ) from 1.8 to 3.0.0.
- [Release notes](https://github.com/apache/maven-antrun-plugin/releases )
- [Commits](https://github.com/apache/maven-antrun-plugin/compare/maven-antrun-plugin-1.8...maven-antrun-plugin-3.0.0 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-antrun-plugin
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 08:28:11 -07:00
2ee7e31e5b
Fix syntax error ( #11332 )
2021-06-07 22:35:02 -07:00
1e8b5360b3
Bump docker-java-transport-netty from 3.2.0 to 3.2.8 ( #11337 )
...
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java ) from 3.2.0 to 3.2.8.
- [Release notes](https://github.com/docker-java/docker-java/releases )
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.0...3.2.8 )
---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-07 18:58:38 -07:00
dbec8da7ea
enable dependabot for dependency updates ( #11079 )
...
* enable dependabot for dependency updates
* exclude dependabot config from license checks
2021-06-07 13:56:32 -07:00
7ffe402b87
Fix priority parsing in HiLoQueryLaningStrategy ( #11302 )
2021-06-07 15:27:27 +05:30
44d629319d
handle timestamps of complex types when parsing protobuf messages ( #11293 )
...
* handle timestamps correctly when parsing protobuf
* Add timestamp handling to ProtobufReader
* disable checkstyle for generated sourcecode
* Fix test
* try this
* refactor tests
2021-06-07 15:19:39 +05:30
0c5d1c9725
Web console: add more query fixing auto suggestions ( #11203 )
...
* add more query fixing auto suggestions
* update query gen
* update toolkit
* update licenses
* fix funky quotes
* funky => fancy
* revert engine change
* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
d5139c9543
Fix permission problems in docker ( #11299 )
...
* Create /opt/data to fix permission problem
* eliminate symlink to avoid compatibility problem on AWS Fargate
* Add a workaround section
* Update instruction for named volume
* Use named volume in docker-compose
* Revert some doc change
* Resolve review comments
2021-06-01 17:33:27 -07:00
27f1b6cbf3
Fix Index hadoop failing with index.zip is not a valid DFS filename ( #11316 )
...
* * Fix bug
* * simplify class loading
* * fix example configs for integration tests
* Small classloader cleanup
Co-authored-by: jon-wei <jon.wei@imply.io>
2021-06-01 19:14:50 -04:00
a24817d20a
Migrate helm chart from helm v2 to v3 ( #11290 )
...
* move all files from helm to helm/druid
* migrate druid helm chart from helm v2 to v3
* add license headers
* bump version to 0.3.0
2021-05-28 13:46:02 +08:00
e664bfd433
Improve doc of movingAverage ( #11262 )
...
* Make doc more directive
Signed-off-by: frank chen <frank.chen021@outlook.com>
* Add limitation
Signed-off-by: frank chen <frank.chen021@outlook.com>
* Suppress spelling check error
2021-05-28 13:10:55 +08:00
04fefb0ca3
Fix ClassCastException ( #11266 )
...
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-05-27 21:25:51 -07:00
31c811d894
Web console: Fix maxRowsPerSegment validation in hashed compaction spec ( #11308 )
...
* allow defining of maxRowsPerSegment for now
* use common util
* update snapshots
* fix test
* fix e2e test
2021-05-27 16:36:42 -07:00
e5633d7842
Fix bug: 502 bad gateway thrown when we edit/delete any auto compaction config created 0.21.0 or before ( #11311 )
...
* fix bug
* add test
* fix IT
* fix checkstyle
* address comments
2021-05-27 16:34:32 -07:00
60843bd11f
Add configuration suggestion to `druid.indexer.storage.type` ( #11304 )
2021-05-27 06:44:47 -07:00
7a2bcb3ca1
chill, travis ( #11283 )
...
* maybe make ci more selective in what it do
* maybe this
* skip self and travis config
* can it be in before_script? lets find out
* nope
* fixes
* oops
* only skip stuff for PR builds, add some tests
* revert unintended change
* more comments, more tests, more better
2021-05-27 05:40:40 -07:00
6c7177714f
Add test for join on __time column ( #11289 )
2021-05-26 22:20:39 -07:00
b517c3339b
remove ZooKeeper 3.4 support + pass tests with Java 15 ( #11073 )
...
With this change, Druid will only support ZooKeeper 3.5.x and later.
In order to support Java 15 we need to switch to ZK 3.5.x client libraries and drop support for ZK 3.4.x
(see #10780 for the detailed reasons)
* remove ZooKeeper 3.4.x compatibility
* exclude additional ZK 3.5.x netty dependencies to ensure we use our version
* keep ZooKeeper version used for integration tests in sync with client library version
* remove the need to specify ZK version at runtime for docker
* add support to run integration tests with JDK 15
* build and run unit tests with Java 15 in travis
2021-05-25 12:49:49 -07:00
82380b67e0
Improve IT job 79 ITNestedQueryPushDownTest integration test ( #11268 )
...
* improve occasional failure caused by resource competition
* adjust more configs in tiny-cluters.yaml
Co-authored-by: yfhanfei <yfhanfei@ZBMac-C02DW5SMMD6P.local>
2021-05-24 10:12:34 +08:00
2bfcee5824
Fix issue with empty array converting to string expression instead of string array ( #11270 )
2021-05-22 09:31:28 +08:00
4ba5738ffb
Add an issues section to deal with common issues when building druid ( #11271 )
2021-05-21 09:04:51 -07:00
403dcf5cfb
fixes some typos, edits for style ( #11258 )
2021-05-21 08:58:39 -07:00
fcb4eaa3d4
add docs for high-churn datasource cleanup ( #11245 )
...
* add docs for high-churn datasource cleanup
* fix most comments except for task log
* address comments
* update strategy recommendation
* address addtional comments
* fix
* address comments
* address comments from @sthetland
2021-05-20 09:48:42 -07:00
259207753d
Fix is null selector returning incorrect value for Long data type ( #11170 )
...
* Fix is null selector returning incorrect value for Long data type
* Fix style errors
* Refactor getObject method to also cache null column values
* Make lastInput variable nullable
* Refactor unit test
* Use new boolean lastInputIsNull instead of Long for lastInput to avoid boxing
* Refactor to remove Long for input variable
* Make a separate null caching variable
* Cleaner null caching implementation
2021-05-19 20:47:02 -07:00
383daa4029
Catch exception inside ITRetryUtil to fix one of the causes for flaky integration tests ( #11265 )
...
* Do not stop retrying when an exception is encountered. Save & propagate last exception if retry count is exceeded.
* Add one more log message to help with debugging
* Limit schema registry heap to attempt to control OOMs
2021-05-19 13:56:02 -07:00
4c3077390f
Update check-licenses.py ( #11269 )
2021-05-19 13:52:00 -07:00
933350d106
integration test runner xmx ( #11273 )
...
* integration test runner xmx
* smaller
2021-05-19 12:59:50 -07:00
3be8e29269
Add integration test for protobuf ( #11126 )
...
* add file test
* test
* for test
* bug fixed
* test
* test
* test
* bug fixed
* delete auto scaler
* add input format
* add extensions
* bug fixed
* bug fixed
* bug fixed
* revert
* add schema registry test
* bug fixed
* bug fixed
* delete desc
* delete change
* add desc
* bug fixed
* test inputformat
* bug fixed
* bug fixed
* bug fixed
* bug fixed
* delete io exception
* change builder not static
* change pom
* bug fixed
Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-05-17 15:45:07 -07:00
6d08a7051e
fix bug with aggregator expressions on realtime index with string columns always producing 0 values ( #11185 )
...
* fix bug with aggregator expressions on realtime index with string columns always producing 0 values
* more test
* rework some stuff
* javadocs
2021-05-17 11:59:13 -07:00
aa62073faa
fix sql planner bug with inner offset causing loop ( #11259 )
...
* fix sql planner bug with inner offset causing loop
* move check up
2021-05-15 14:26:41 -07:00
d85e8b708b
Web console: fix SegmentTimeline query ( #11204 )
...
* fix SegmentTimeline query
* test query
2021-05-14 14:48:23 -07:00
3649c608d2
array handling improvements ( #11233 )
...
* fix jdbc array handling, split handling for some array and multi value operator, split and add more tests
* formatting
2021-05-13 18:50:32 -07:00
4100c5edc0
Fix taskQueue to honor ( #11243 )
...
useLineageBasedSegmentAllocation in default taskContext
2021-05-14 09:30:50 +08:00
4a3c834ecf
i dig the optimism, but need more time ( #11250 )
2021-05-13 11:16:10 -07:00
790262e5d0
add estimated byte size limit enforcement for heap based expression aggregator ( #11236 )
2021-05-12 01:21:50 -07:00
3455352241
Add feature to automatically remove compaction configurations for inactive datasources ( #11232 )
...
* add auto cleanup
* add auto cleanup
* add auto cleanup
* add tests
* add tests
* use retryutils
* use retryutils
* use retryutils
* address comments
2021-05-11 18:49:18 -07:00
d11be88c4b
Fix required field treatment ( #11228 )
2021-05-11 17:58:37 -07:00
8e5048e643
Avoid memory mapping hydrants after they are persisted & after they are merged for native batch ingestion ( #11123 )
...
* Avoid mapping hydrants in create segments phase for native ingestion
* Drop queriable indices after a given sink is fully merged
* Do not drop memory mappings for realtime ingestion
* Style fixes
* Renamed to match use case better
* Rollback memoization code and use the real time flag instead
* Null ptr fix in FireHydrant toString plus adjustments to memory pressure tracking calculations
* Style
* Log some count stats
* Make sure sinks size is obtained at the right time
* BatchAppenderator unit test
* Fix comment typos
* Renamed methods to make them more readable
* Move persisted metadata from FireHydrant class to AppenderatorImpl. Removed superfluous differences and fix comment typo. Removed custom comparator
* Missing dependency
* Make persisted hydrant metadata map concurrent and better reflect the fact that keys are Java references. Maintain persisted metadata when dropping/closing segments.
* Replaced concurrent variables with normal ones
* Added batchMemoryMappedIndex "fallback" flag with default "false". Set this to "true" make code fallback to previous code path.
* Style fix.
* Added note to new setting in doc, using Iterables.size (and removing a dependency), and fixing a typo in a comment.
* Forgot to commit this edited documentation message
2021-05-11 14:34:26 -07:00
4326e699bd
Add feature to automatically remove datasource metadata based on retention period ( #11227 )
...
* add auto clean up datasource metadata
* add test
* fix checkstyle
* add comments
* fix error
* address comments
* Address comments
* fix test
* fix test
* fix typo
* add comment
* fix test
* fix test
2021-05-11 01:22:33 -07:00
fae7ebf489
change errant 'none' configuration to 'manual': ( #11218 )
2021-05-10 22:04:18 -07:00
3a660bc6ee
Make sure updating coordinator config is protected against race condition ( #11144 )
...
* Make sure changing coordinator config is protected against concurrent updates
* Make sure updating coordinator config is protected against race condition
* add retry
* fix checkstyle
* add tests
* add tests
* add more tests
* add tests
* fix
* fix checkstyle
2021-05-10 13:58:08 -07:00
f6662b4893
fix count and average SQL aggregators on constant virtual columns ( #11208 )
...
* fix count and average SQL aggregators on constant virtual columns
* style
* even better, why are we tracking virtual columns in aggregations at all if we have a virtual column registry
* oops missed a few
* remove unused
* this will fix it
2021-05-10 13:41:48 -07:00
Xavier Léauté
Yi Yuan
Egor Riashin
Oops
dependabot[bot]
frank chen
Abhishek Agarwal
Vadim Ogievetsky
zachjsh
Hellis
Maytas Monsereenusorn
Clint Wylie
Rohan Garg
fhan
Agustin Gonzalez
Charles Smith
Maria Sitkovets
Prashanth Sagar
Jihoon Son