Commit Graph

11037 Commits

Author SHA1 Message Date
dependabot[bot] 1e8b5360b3
Bump docker-java-transport-netty from 3.2.0 to 3.2.8 (#11337)
Bumps [docker-java-transport-netty](https://github.com/docker-java/docker-java) from 3.2.0 to 3.2.8.
- [Release notes](https://github.com/docker-java/docker-java/releases)
- [Changelog](https://github.com/docker-java/docker-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/docker-java/docker-java/compare/3.2.0...3.2.8)

---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java-transport-netty
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-07 18:58:38 -07:00
Xavier Léauté dbec8da7ea
enable dependabot for dependency updates (#11079)
* enable dependabot for dependency updates
* exclude dependabot config from license checks
2021-06-07 13:56:32 -07:00
Abhishek Agarwal 7ffe402b87
Fix priority parsing in HiLoQueryLaningStrategy (#11302) 2021-06-07 15:27:27 +05:30
Abhishek Agarwal 44d629319d
handle timestamps of complex types when parsing protobuf messages (#11293)
* handle timestamps correctly when parsing protobuf

* Add timestamp handling to ProtobufReader

* disable checkstyle for generated sourcecode

* Fix test

* try this

* refactor tests
2021-06-07 15:19:39 +05:30
Vadim Ogievetsky 0c5d1c9725
Web console: add more query fixing auto suggestions (#11203)
* add more query fixing auto suggestions

* update query gen

* update toolkit

* update licenses

* fix funky quotes

* funky => fancy

* revert engine change

* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
frank chen d5139c9543
Fix permission problems in docker (#11299)
* Create /opt/data to fix permission problem

* eliminate symlink to avoid compatibility problem on AWS Fargate

* Add a workaround section

* Update instruction for named volume

* Use named volume in docker-compose

* Revert some doc change

* Resolve review comments
2021-06-01 17:33:27 -07:00
zachjsh 27f1b6cbf3
Fix Index hadoop failing with index.zip is not a valid DFS filename (#11316)
* * Fix bug

* * simplify class loading

* * fix example configs for integration tests

* Small classloader cleanup

Co-authored-by: jon-wei <jon.wei@imply.io>
2021-06-01 19:14:50 -04:00
Hellis a24817d20a
Migrate helm chart from helm v2 to v3 (#11290)
* move all files from helm to helm/druid

* migrate druid helm chart from helm v2 to v3

* add license headers

* bump version to 0.3.0
2021-05-28 13:46:02 +08:00
frank chen e664bfd433
Improve doc of movingAverage (#11262)
* Make doc more directive

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Add limitation

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Suppress spelling check error
2021-05-28 13:10:55 +08:00
frank chen 04fefb0ca3
Fix ClassCastException (#11266)
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-05-27 21:25:51 -07:00
Vadim Ogievetsky 31c811d894
Web console: Fix maxRowsPerSegment validation in hashed compaction spec (#11308)
* allow defining of maxRowsPerSegment for now

* use common util

* update snapshots

* fix test

* fix e2e test
2021-05-27 16:36:42 -07:00
Maytas Monsereenusorn e5633d7842
Fix bug: 502 bad gateway thrown when we edit/delete any auto compaction config created 0.21.0 or before (#11311)
* fix bug

* add test

* fix IT

* fix checkstyle

* address comments
2021-05-27 16:34:32 -07:00
frank chen 60843bd11f
Add configuration suggestion to `druid.indexer.storage.type` (#11304) 2021-05-27 06:44:47 -07:00
Clint Wylie 7a2bcb3ca1
chill, travis (#11283)
* maybe make ci more selective in what it do

* maybe this

* skip self and travis config

* can it be in before_script? lets find out

* nope

* fixes

* oops

* only skip stuff for PR builds, add some tests

* revert unintended change

* more comments, more tests, more better
2021-05-27 05:40:40 -07:00
Rohan Garg 6c7177714f
Add test for join on __time column (#11289) 2021-05-26 22:20:39 -07:00
Xavier Léauté b517c3339b
remove ZooKeeper 3.4 support + pass tests with Java 15 (#11073)
With this change, Druid will only support ZooKeeper 3.5.x and later.

In order to support Java 15 we need to switch to ZK 3.5.x client libraries and drop support for ZK 3.4.x
(see #10780 for the detailed reasons) 

* remove ZooKeeper 3.4.x compatibility
* exclude additional ZK 3.5.x netty dependencies to ensure we use our version
* keep ZooKeeper version used for integration tests in sync with client library version
* remove the need to specify ZK version at runtime for docker
* add support to run integration tests with JDK 15
* build and run unit tests with Java 15 in travis
2021-05-25 12:49:49 -07:00
fhan 82380b67e0
Improve IT job 79 ITNestedQueryPushDownTest integration test (#11268)
* improve occasional failure caused by resource competition

* adjust more configs in tiny-cluters.yaml

Co-authored-by: yfhanfei <yfhanfei@ZBMac-C02DW5SMMD6P.local>
2021-05-24 10:12:34 +08:00
Clint Wylie 2bfcee5824
Fix issue with empty array converting to string expression instead of string array (#11270) 2021-05-22 09:31:28 +08:00
Agustin Gonzalez 4ba5738ffb
Add an issues section to deal with common issues when building druid (#11271) 2021-05-21 09:04:51 -07:00
Charles Smith 403dcf5cfb
fixes some typos, edits for style (#11258) 2021-05-21 08:58:39 -07:00
Charles Smith fcb4eaa3d4
add docs for high-churn datasource cleanup (#11245)
* add docs for high-churn datasource cleanup

* fix most comments except for task log

* address  comments

* update strategy recommendation

* address addtional comments

* fix

* address comments

* address comments from @sthetland
2021-05-20 09:48:42 -07:00
Maria Sitkovets 259207753d
Fix is null selector returning incorrect value for Long data type (#11170)
* Fix is null selector returning incorrect value for Long data type

* Fix style errors

* Refactor getObject method to also cache null column values

* Make lastInput variable nullable

* Refactor unit test

* Use new boolean lastInputIsNull instead of Long for lastInput to avoid boxing

* Refactor to remove Long for input variable

* Make a separate null caching variable

* Cleaner null caching implementation
2021-05-19 20:47:02 -07:00
Agustin Gonzalez 383daa4029
Catch exception inside ITRetryUtil to fix one of the causes for flaky integration tests (#11265)
* Do not stop retrying when an exception is encountered. Save & propagate last exception if retry count is exceeded.

* Add one more log message to help with debugging

* Limit schema registry heap to attempt to control OOMs
2021-05-19 13:56:02 -07:00
Prashanth Sagar 4c3077390f
Update check-licenses.py (#11269) 2021-05-19 13:52:00 -07:00
Clint Wylie 933350d106
integration test runner xmx (#11273)
* integration test runner xmx

* smaller
2021-05-19 12:59:50 -07:00
Yi Yuan 3be8e29269
Add integration test for protobuf (#11126)
* add file test

* test

* for test

* bug fixed

* test

* test

* test

* bug fixed

* delete auto scaler

* add input format

* add extensions

* bug fixed

* bug fixed

* bug fixed

* revert

* add schema registry test

* bug fixed

* bug fixed

* delete desc

* delete change

* add desc

* bug fixed

* test inputformat

* bug fixed

* bug fixed

* bug fixed

* bug fixed

* delete io exception

* change builder not static

* change pom

* bug fixed

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-05-17 15:45:07 -07:00
Clint Wylie 6d08a7051e
fix bug with aggregator expressions on realtime index with string columns always producing 0 values (#11185)
* fix bug with aggregator expressions on realtime index with string columns always producing 0 values

* more test

* rework some stuff

* javadocs
2021-05-17 11:59:13 -07:00
Clint Wylie aa62073faa
fix sql planner bug with inner offset causing loop (#11259)
* fix sql planner bug with inner offset causing loop

* move check up
2021-05-15 14:26:41 -07:00
Vadim Ogievetsky d85e8b708b
Web console: fix SegmentTimeline query (#11204)
* fix SegmentTimeline query

* test query
2021-05-14 14:48:23 -07:00
Clint Wylie 3649c608d2
array handling improvements (#11233)
* fix jdbc array handling, split handling for some array and multi value operator, split and add more tests

* formatting
2021-05-13 18:50:32 -07:00
Jihoon Son 4100c5edc0
Fix taskQueue to honor (#11243)
useLineageBasedSegmentAllocation in default taskContext
2021-05-14 09:30:50 +08:00
Clint Wylie 4a3c834ecf
i dig the optimism, but need more time (#11250) 2021-05-13 11:16:10 -07:00
Clint Wylie 790262e5d0
add estimated byte size limit enforcement for heap based expression aggregator (#11236) 2021-05-12 01:21:50 -07:00
Maytas Monsereenusorn 3455352241
Add feature to automatically remove compaction configurations for inactive datasources (#11232)
* add auto cleanup

* add auto cleanup

* add auto cleanup

* add tests

* add tests

* use retryutils

* use retryutils

* use retryutils

* address comments
2021-05-11 18:49:18 -07:00
Vadim Ogievetsky d11be88c4b
Fix required field treatment (#11228) 2021-05-11 17:58:37 -07:00
Agustin Gonzalez 8e5048e643
Avoid memory mapping hydrants after they are persisted & after they are merged for native batch ingestion (#11123)
* Avoid mapping hydrants in create segments phase for native ingestion

* Drop queriable indices after a given sink is fully merged

* Do not drop memory mappings for realtime ingestion

* Style fixes

* Renamed to match use case better

* Rollback memoization code and use the real time flag instead

* Null ptr fix in FireHydrant toString plus adjustments to memory pressure tracking calculations

* Style

* Log some count stats

* Make sure sinks size is obtained at the right time

* BatchAppenderator unit test

* Fix comment typos

* Renamed methods to make them more readable

* Move persisted metadata from FireHydrant class to AppenderatorImpl. Removed superfluous differences and fix comment typo. Removed custom comparator

* Missing dependency

* Make persisted hydrant metadata map concurrent and better reflect the fact that keys are Java references. Maintain persisted metadata when dropping/closing segments.

* Replaced concurrent variables with normal ones

* Added   batchMemoryMappedIndex "fallback" flag with default "false". Set this to "true" make code fallback to previous code path.

* Style fix.

* Added note to new setting in doc, using Iterables.size (and removing a dependency), and fixing a typo in a comment.

* Forgot to commit this edited documentation message
2021-05-11 14:34:26 -07:00
Maytas Monsereenusorn 4326e699bd
Add feature to automatically remove datasource metadata based on retention period (#11227)
* add auto clean up datasource metadata

* add test

* fix checkstyle

* add comments

* fix error

* address comments

* Address comments

* fix test

* fix test

* fix typo

* add comment

* fix test

* fix test
2021-05-11 01:22:33 -07:00
Charles Smith fae7ebf489
change errant 'none' configuration to 'manual': (#11218) 2021-05-10 22:04:18 -07:00
Maytas Monsereenusorn 3a660bc6ee
Make sure updating coordinator config is protected against race condition (#11144)
* Make sure changing coordinator config is protected against concurrent updates

* Make sure updating coordinator config is protected against race condition

* add retry

* fix checkstyle

* add tests

* add tests

* add more tests

* add tests

* fix

* fix checkstyle
2021-05-10 13:58:08 -07:00
Clint Wylie f6662b4893
fix count and average SQL aggregators on constant virtual columns (#11208)
* fix count and average SQL aggregators on constant virtual columns

* style

* even better, why are we tracking virtual columns in aggregations at all if we have a virtual column registry

* oops missed a few

* remove unused

* this will fix it
2021-05-10 13:41:48 -07:00
Vadim Ogievetsky 15de29a2c4
Handle not having default rules better (#11214) 2021-05-10 15:13:15 -04:00
Clint Wylie 691d7a1d54
SQL timeseries no longer skip empty buckets with all granularity (#11188)
* SQL timeseries no longer skip empty buckets with all granularity

* add comment, fix tests

* the ol switcheroo

* revert unintended change

* docs and more tests

* style

* make checkstyle happy

* docs fixes and more tests

* add docs, tests for array_agg

* fixes

* oops

* doc stuffs

* fix compile, match doc style
2021-05-10 10:13:37 -07:00
frank chen fa113fb4a9
Fix default value (#11220) 2021-05-10 10:11:26 -07:00
Yuanli Han 14f1f2aa76
Fix a broken link in the development doc (#11226) 2021-05-10 16:14:06 +08:00
Yuanli Han 8647040f4d
Allow user to set group.id for Kafka ingestion task (#11147)
* allow user to set group.id for Kafka ingestion task

* fix test coverage by removing deprecated code and add doc

* fix typo

* Update docs/development/extensions-core/kafka-ingestion.md

Co-authored-by: frank chen <frankchen@apache.org>

Co-authored-by: frank chen <frankchen@apache.org>
2021-05-09 11:56:19 +08:00
Jihoon Son 2df42143ae
Fix idempotence of segment allocation and task report apis in native batch ingestion (#11189)
* Fix idempotence of segment allocation and task report apis in native
batch ingestion

* better error and javadoc

* checkstyle and dependency

* fix tests and add more tests

* task config instead of context; add doc

* unused import and dependency

* typo in doc

* fix unintended changes

* fix wrong import

* remove unnecessary error handling

* add task context back

* default task context

* fix test and doc

* address comments

* unused imports
2021-05-07 14:29:48 -07:00
Charles Smith cf2cde1d2d
add links to release notes, light refactor of landing page (#11051)
* add links to release notes, light refactor of landing page

* Update docs/design/index.md
2021-05-07 14:26:47 -07:00
Xavier Léauté 3b9dad4c9e
Consolidate the number of Dockerfiles (#11187)
* Consolidate the number of Dockerfiles

* add build-arguments to choose which Java base image to use at runtime
* default to building image with Java 11
* base k8s integration test image off of the default image: this ensures
  our docker image now gets tested as part of integration tests.

* upgrade maven help plugin to 3.2.0
2021-05-07 10:41:34 -07:00
Gian Merlino a1f850d707
Fix vectorized cardinality bug on certain string columns. (#11199)
* Fix vectorized cardinality bug on certain string columns.

Fixes a bug introduced in #11182, related to the fact that in some cases,
ColumnProcessors.makeVectorProcessor will call "makeObjectProcessor"
instead of "makeSingleValueDimensionProcessor" or
"makeMultiValueDimensionProcessor". CardinalityVectorProcessorFactory
improperly ignored calls to "makeObjectProcessor".

In addition to fixing the bug, I added this detail to the javadocs for
VectorColumnProcessorFactory, to prevent others from running into the
same thing in the future. They do not currently call out this case.

* Improve test coverage.

* Additional fixes.
2021-05-07 08:37:10 -07:00
John Bampton a8c00d8d9b
chore: fix case of GitHub (#10928) 2021-05-07 01:15:43 -07:00