Commit Graph

11001 Commits

Author SHA1 Message Date
Maytas Monsereenusorn 4326e699bd
Add feature to automatically remove datasource metadata based on retention period (#11227)
* add auto clean up datasource metadata

* add test

* fix checkstyle

* add comments

* fix error

* address comments

* Address comments

* fix test

* fix test

* fix typo

* add comment

* fix test

* fix test
2021-05-11 01:22:33 -07:00
Charles Smith fae7ebf489
change errant 'none' configuration to 'manual': (#11218) 2021-05-10 22:04:18 -07:00
Maytas Monsereenusorn 3a660bc6ee
Make sure updating coordinator config is protected against race condition (#11144)
* Make sure changing coordinator config is protected against concurrent updates

* Make sure updating coordinator config is protected against race condition

* add retry

* fix checkstyle

* add tests

* add tests

* add more tests

* add tests

* fix

* fix checkstyle
2021-05-10 13:58:08 -07:00
Clint Wylie f6662b4893
fix count and average SQL aggregators on constant virtual columns (#11208)
* fix count and average SQL aggregators on constant virtual columns

* style

* even better, why are we tracking virtual columns in aggregations at all if we have a virtual column registry

* oops missed a few

* remove unused

* this will fix it
2021-05-10 13:41:48 -07:00
Vadim Ogievetsky 15de29a2c4
Handle not having default rules better (#11214) 2021-05-10 15:13:15 -04:00
Clint Wylie 691d7a1d54
SQL timeseries no longer skip empty buckets with all granularity (#11188)
* SQL timeseries no longer skip empty buckets with all granularity

* add comment, fix tests

* the ol switcheroo

* revert unintended change

* docs and more tests

* style

* make checkstyle happy

* docs fixes and more tests

* add docs, tests for array_agg

* fixes

* oops

* doc stuffs

* fix compile, match doc style
2021-05-10 10:13:37 -07:00
frank chen fa113fb4a9
Fix default value (#11220) 2021-05-10 10:11:26 -07:00
Yuanli Han 14f1f2aa76
Fix a broken link in the development doc (#11226) 2021-05-10 16:14:06 +08:00
Yuanli Han 8647040f4d
Allow user to set group.id for Kafka ingestion task (#11147)
* allow user to set group.id for Kafka ingestion task

* fix test coverage by removing deprecated code and add doc

* fix typo

* Update docs/development/extensions-core/kafka-ingestion.md

Co-authored-by: frank chen <frankchen@apache.org>

Co-authored-by: frank chen <frankchen@apache.org>
2021-05-09 11:56:19 +08:00
Jihoon Son 2df42143ae
Fix idempotence of segment allocation and task report apis in native batch ingestion (#11189)
* Fix idempotence of segment allocation and task report apis in native
batch ingestion

* better error and javadoc

* checkstyle and dependency

* fix tests and add more tests

* task config instead of context; add doc

* unused import and dependency

* typo in doc

* fix unintended changes

* fix wrong import

* remove unnecessary error handling

* add task context back

* default task context

* fix test and doc

* address comments

* unused imports
2021-05-07 14:29:48 -07:00
Charles Smith cf2cde1d2d
add links to release notes, light refactor of landing page (#11051)
* add links to release notes, light refactor of landing page

* Update docs/design/index.md
2021-05-07 14:26:47 -07:00
Xavier Léauté 3b9dad4c9e
Consolidate the number of Dockerfiles (#11187)
* Consolidate the number of Dockerfiles

* add build-arguments to choose which Java base image to use at runtime
* default to building image with Java 11
* base k8s integration test image off of the default image: this ensures
  our docker image now gets tested as part of integration tests.

* upgrade maven help plugin to 3.2.0
2021-05-07 10:41:34 -07:00
Gian Merlino a1f850d707
Fix vectorized cardinality bug on certain string columns. (#11199)
* Fix vectorized cardinality bug on certain string columns.

Fixes a bug introduced in #11182, related to the fact that in some cases,
ColumnProcessors.makeVectorProcessor will call "makeObjectProcessor"
instead of "makeSingleValueDimensionProcessor" or
"makeMultiValueDimensionProcessor". CardinalityVectorProcessorFactory
improperly ignored calls to "makeObjectProcessor".

In addition to fixing the bug, I added this detail to the javadocs for
VectorColumnProcessorFactory, to prevent others from running into the
same thing in the future. They do not currently call out this case.

* Improve test coverage.

* Additional fixes.
2021-05-07 08:37:10 -07:00
John Bampton a8c00d8d9b
chore: fix case of GitHub (#10928) 2021-05-07 01:15:43 -07:00
benkrug 49c8307b72
Update datasource.md (#10864)
* Update datasource.md

Change "table" to "datasource" in join discussion: This means that all datasources
other than the leftmost "base" table must fit in memory.

According to docs on datasources, "datasource" is the more general term, and a table is a kind of datasource.  In the context here, then, "datasource" is applicable.

* left-hand table -> left-hand datasource

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>

Co-authored-by: sthetland <steve.hetland@imply.io>
Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
2021-05-07 01:14:45 -07:00
Lasse Krogh Mammen 9be2a5cdc2
Add documentation re alphabetical sorted of MV dimensions (#10695) 2021-05-07 01:12:32 -07:00
Maytas Monsereenusorn d73f72e508
Add feature to automatically remove supervisor based on retention period (#11200)
* add auto clean up

* add test

* add test

* fix test

* Address comments

* Address comments
2021-05-06 22:25:23 -07:00
Maytas Monsereenusorn 351059ca43
Suppressing false positive CVE-2020-7791 (#11215)
* suppressing false positive CVE-2020-7791

* add comments
2021-05-06 15:24:12 -07:00
Atul Mohan ac95f99d59
Upgrade jetty version (#11207) 2021-05-06 18:21:36 +08:00
imply-jbalik 4adb121234
Fix example of prefixes for Cloud Input Sources(eg. S3) (#11192)
Fixed a syntax error in "prefix" lines in docs/ingestion/native-batch.md

S3 requires a trailing slash for directory like structures, so this updates the examples to include the trailing slashes.
2021-05-05 21:19:31 -07:00
Yuanli Han 34169c8550
fix doc (#11202)
(cherry picked from commit ffb3c049726b5e461c6f7f8b6f4b75d2cb907dcc)
2021-05-05 06:17:07 -07:00
Lucas Capistrant bb3c810b36
Create dynamic config that can limit number of non-primary replicants loaded per coordination cycle (#11135)
* lay the groundwork for throttling replicant loads per RunRules execution

* Add dynamic coordinator config to control new replicant threshold.

* remove redundant line

* add some unit tests

* fix checkstyle error

* add documentation for new dynamic config

* improve docs and logs

* Alter how null is handled for new config. If null, manually set as default
2021-05-05 07:39:36 -05:00
Clint Wylie 6f0b4d90d8
clarify that appenderator persist byte sizes are estimations (#11198) 2021-05-05 17:44:04 +08:00
zachjsh 99f39c7202
Hadoop segment index file rename (#11194)
* Do stuff

* Do more stuff

* * Do more stuff

* * Do more stuff

* * working

* * cleanup

* * more cleanup

* * more cleanup

* * add license header

* * Add unit tests

* * add java docs

* * add more unit tests

* * Cleanup test

* * Move removing of workingPath to index task rather than in hadoop job.

* * Address review comments

* * remove unused import

* * Address review comments

* Do not overwrite segment descriptor for segment if it already exists.

* * add comments to FileSystemHelper class

* * fix local hadoop integration test

* * Fix failing test failures when running with java11

* Revert "Revert "Adjust HadoopIndexTask temp segment renaming to avoid potential race conditions (#11075)" (#11151)"

This reverts commit 49a9c3ffb7.

* * remove JobHelperPowerMockTest

* * remove FileSystemHelper class
2021-05-04 20:22:18 -04:00
Clint Wylie cd1ddcd36b
Web Console: allow pollPeriod to be defined in JDBC lookup config (#11193)
* allow pollPeriod to be defined in JDBC lookup config

* add test

* save the whales
2021-05-04 00:11:26 -07:00
frank chen 204901a602
Fix Smile encoding for HTTP response (#10980)
* fix Smile encoding bug

Signed-off-by: frank chen <frank.chen021@outlook.com>

* Add unit tests

* Add IT for smile encoding

* Fix cases

* Update javadoc

Co-authored-by: Jihoon Son <jihoonson@apache.org>

* resolve comments

Co-authored-by: Jihoon Son <jihoonson@apache.org>
2021-05-03 22:43:47 -07:00
Clint Wylie 554f1ffeee
ARRAY_AGG sql aggregator function (#11157)
* ARRAY_AGG sql aggregator function

* add javadoc

* spelling

* review stuff, return null instead of empty when nil input

* review stuff

* Update sql.md

* use type inference for finalize, refactor some things
2021-05-03 22:17:10 -07:00
imply-jbalik 6f7701e742
fixed array syntax (#11191) 2021-05-03 21:38:16 -07:00
Gian Merlino bef7cc911f
Vectorize the cardinality aggregator. (#11182)
* Vectorize the cardinality aggregator.

Does not include a byRow implementation, so if byRow is true then
the aggregator still goes through the non-vectorized path.

Testing strategy:

- New tests that exercise both styles of "aggregate" for supported types.
- Some existing tests have also become active (note the deleted
  "cannotVectorize" lines).

* Adjust whitespace.
2021-05-03 20:27:02 -07:00
sthetland ca1412d574
Reduce visibility of Tranquility documentation (#11134)
* reduce visibility of tranquility doc

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
2021-05-03 16:48:24 -07:00
Maytas Monsereenusorn 84aac4832d
Add feature to automatically remove rules based on retention period (#11164)
* Add feature to automatically remove rules based on retention period

* Add feature to automatically remove rules based on retention period

* address comments
2021-05-03 11:50:45 -07:00
Gian Merlino 809e001939
Vectorize the DataSketches quantiles aggregator. (#11183)
* Vectorize the DataSketches quantiles aggregator.

Also removes synchronization for the BufferAggregator and VectorAggregator
implementations, since it is not necessary (similar to #11115).

Extends DoublesSketchAggregatorTest and DoublesSketchSqlAggregatorTest
to run all test cases in vectorized mode.

* Style fix.
2021-05-02 16:14:21 -07:00
Gian Merlino 046069f35a
Add a way to retrieve UTF-8 bytes directly via DimensionDictionarySelector. (#11172)
* Add a way to retrieve UTF-8 bytes directly via DimensionDictionarySelector.

The idea is that certain operations (like count distinct on strings) will
be faster if they are able to run directly on UTF-8 bytes instead of on
Java Strings decoded by "lookupName".

* Add license header.

* Updates suggested by robots.
2021-04-30 10:56:11 -07:00
Gian Merlino 6d82c3cbf1
StringComparators: No need to convert to UTF-8 for lexicographic comparison. (#11171)
Lexicographic ordering of UTF-8 byte sequences and in-memory UTF-16
strings are equivalent. So, we can skip the (expensive) conversion and
get an equivalent result. Thank you, Unicode!
2021-04-30 10:54:20 -07:00
benkrug fdab95ea99
Update index.md (#11174)
tiny change for readability
2021-04-30 09:40:19 -07:00
frank chen aec65242fc
fix docker volume permissions (#11167)
Docker volume directory was accidentally removed due to reordering of statements.
This causes ownership and permissions on the volume directory to be reset, preventing startup.

fixes #11166
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-04-30 08:51:53 -07:00
Benedict Jin ed81548ff7
Add helm chart for Apache Druid (#11163)
* Add helm chart for Apache Druid

* Add license headers
2021-04-29 12:38:50 +08:00
Jihoon Son 941440afb6
Suggest using svn mv; clarify the process for doc updating (#11162) 2021-04-29 10:08:23 +08:00
Gian Merlino 7d808e357c
InDimFilter: Fix cache key computation to avoid collisions. (#11168)
The prior code did not include separation between values, and encoded
null ambiguously. This patch fixes both of those issues by encoding
strings as length + value instead of just value.

I think cache key computation was OK prior to #9800. Prior to that
patch, the cache key was computed using CacheKeyBuilder.appendStrings,
which encodes strings as UTF-8 and inserts a separator byte (0xff)
between them that cannot appear in a UTF-8 stream.
2021-04-28 17:28:29 -07:00
Gian Merlino ad028de538
InDimFilter: Fix NPE involving certain Set types. (#11169)
* InDimFilter: Fix NPE involving certain Set types.

Normally, InDimFilters that come from JSON have HashSets for "values".
However, programmatically-generated filters (like the ones from #11068)
may use other set types. Some set types, like TreeSets with natural
ordering, will throw NPE on "contains(null)", which causes the
InDimFilter's ValueMatcher to throw NPE if it encounters a null value.

This patch adds code to detect if the values set can support
contains(null), and if not, wrap that in a null-checking lambda.

Also included:

- Remove unneeded NullHandling.needsEmptyToNull method.
- Update IndexedTableJoinable to generate a TreeSet that does not
  require lambda-wrapping. (This particular TreeSet is how I noticed
  the bug in the first place.)

* Test fixes.

* Improve test coverage
2021-04-28 14:13:42 -07:00
Jeet Patel 7139c60868
Change the `id` for `kubernetes` doc link to work (#11176)
* Change the `id` for doc link to work

* Added `druid-kubernetes-extensions` to the list
2021-04-28 10:12:28 -07:00
Jihoon Son 8215cc3238
Unit test for DefaultOperandTypeChecker (#11152)
* Less strict operand type check and implicit casting

* fix ci

* Clean up unnecessary changes

* more cleanup

* unused import
2021-04-27 18:47:38 -07:00
Jihoon Son 261c1f271f
Keep traitSet of logicalValues (#11138) 2021-04-27 18:45:23 -07:00
Clint Wylie 57ddae782e
fix serde issues with time-min-max extension (#11146)
* fix serde issues with time-min-max extension

* fix pom dependencies
2021-04-27 10:33:13 -07:00
Vadim Ogievetsky 1b07d554a8
Misc QueryView UX improvments (#11158) 2021-04-27 10:26:01 -07:00
Xavier Léauté 0296f20551
upgrade Apache Kafka to 2.8.0 (#11139)
* upgrade to Apache Kafka 2.8.0 (release notes:
  https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html)
* pass Kafka version as a Docker argument in integration tests
  to keep in sync with maven version
* fix use of internal Kafka APIs in integration tests
2021-04-24 08:27:07 -07:00
Jeet Patel 31042cddf5
Fix `defaultMetricDimensions.json` path link (#11156) 2021-04-24 11:08:03 +08:00
Gian Merlino a47c0d2579
Clarify meaning of "root-level fields" in the documentation. (#11143) 2021-04-24 11:06:08 +08:00
Harini Rajendran 8a3be6bccc
Fix TimeSeriesUnionQueryRunnerTest by extending InitializedNullHandlingTest (#11154) 2021-04-23 08:56:03 -07:00
John Gozde 9745d9e1c3
Web console: Switch to ESLint (#11142)
* Initial eslint config

* I guess eslint sorts underscores differently

* Trim curlies (in jsx)

* Re-organize rules

* Use consistent quote props

* Restructure eslint rules as additions/overrides to recommended configs

* Fix the 'recommended' stuff

* Add prefer-readonly

* Add prefer-object-spread

* Prettify

* Add eslint-plugin-react-hooks

* Switch to eslint-plugin-simple-sort-order

So much better

* Add no-extraneous-dependencies

* ban-tslint-comment for funzies

* If we enabled no-shadow, we'd probably want this option

* Not prefer-for-of

* no-confusing-void-expression, no-confusing-non-null-assertion

* Add some no-unnecessary-* rules

* non-nullable-type-assertion-style!

* prefer-includes

* Reorganize

* prefer-things

* switch-exhaustiveness-check

* We don't need the jsdoc plugin, prettier has our backs

* Remove a useless rule

* Drop TSLint and (temporarily) awesome-code-style

* Removing Object.assign revealed a type issue

* Bring back awesome-code-style for sasslint config

* Disable react/jsx-no-target-blank

* Add prettify-check script

* Add license to eslint config

* Format readme

* Update README for eslint, IDE settings

* Add 'autofix' script

* Switch to @awesome-code-style
2021-04-22 19:33:03 -07:00