f46cc4faaf
Revert "fixed input source sampler buildReader exp"
...
This reverts commit e688db8
2020-12-07 18:34:59 +03:00
e688db8503
fixed input source sampler buildReader exp
2020-12-07 18:28:25 +03:00
b7641f644c
Two fixes related to encoding of % symbols. ( #10645 )
...
* Two fixes related to encoding of % symbols.
1) TaskResourceFilter: Don't double-decode task ids. request.getPathSegments()
returns already-decoded strings. Applying StringUtils.urlDecode on
top of that causes erroneous behavior with '%' characters.
2) Update various ThreadFactoryBuilder name formats to escape '%'
characters. This fixes situations where substrings starting with '%'
are erroneously treated as format specifiers.
ITs are updated to include a '%' in extra.datasource.name.suffix.
* Avoid String.replace.
* Work around surefire bug.
* Fix xml encoding.
* Another try at the proper encoding.
* Give up on the emojis.
* Less ambitious testing.
* Fix an additional problem.
* Adjust encodeForFormat to return null if the input is null.
2020-12-06 22:35:11 -08:00
17f39ab91e
Fix misspellings in druid-forbidden-apis. ( #10634 )
...
These caused certain APIs to not actually be properly forbidden.
Also removed two MoreExecutors entries for methods that don't exist in
our version of Guava.
2020-12-05 15:26:57 -08:00
7eb5f59a9a
Fix string byte calculation in StringDimensionIndexer ( #10623 )
...
* fix string byte calculation
* fix tests
* fix test
2020-12-04 00:51:48 -08:00
52d46cebc3
Move common configurations to TuningConfig ( #10478 )
...
* Move common methods that are used in HadoopTuningConfig and in AppenderatorConfig to TuningConfig
* Rename rowFlushBoundary in HadoopTuningConfig to maxRowsInMemory to match TuningConfig API
2020-12-03 18:13:32 -08:00
229b5f359f
Remove hard limitation that druid(after 0.15.0) only can consume Kafka version 0.11.x or better ( #10551 )
...
* remove build in kafka consumer config :
* modify druid docs of kafka indexing service
* yuezhang
* modify doc
* modify docs
* fix kafkaindexTaskTest.java
* revert uncessary change
* add more logs and modify docs
* revert jdk version
* modify docs
* modify-kafka-version v2
* modify docs
* modify docs
* modify docs
* modify docs
* modify docs
* done
* remove useless import
* change code and add UT
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-12-03 17:37:59 -08:00
ae6c43de71
Add an integration test for HTTP inputSource ( #10620 )
2020-12-03 15:51:56 -08:00
2cd017b7aa
Fix the config initialization on container restart ( #10458 )
2020-12-03 12:03:00 -08:00
813e18774e
make dimension column extensible with COMPLEX type ( #10277 )
...
* make dimension column extensible with COMPLEX type
* more changes
Change-Id: I9707dd644b8d71030b74a8c1d6fff0c0020d960d
* processing module changes for build fix
Change-Id: I146f95a41b79d20edb1721be13f0e9641f788e0e
* rename ColumnCapabilities.getTypeName() to getComplexTypeName()
* rename ColumnBuilder.setTypeName(..) -> ColumnBuilder.setComplexTypeName(..)
2020-12-03 08:58:17 -08:00
c94be8a945
Revert "Update google client libraries ( #10536 )" ( #10599 )
...
This reverts commit 4537016cad
2020-12-03 20:14:52 +05:30
7e9522870f
introduce DynamicConfigProvider interface and make kafka consumer props extensible ( #10309 )
...
* introduce DynamicConfigProvider interface and make kafka consumer props extensible
* fix intellij inspection error
* make DynamicConfigProvider generic
Change-Id: I2e3e89f8617b6fe7fc96859deca4011f609dc5a3
* deprecate PasswordProvider
2020-12-02 16:38:27 -08:00
f965464f36
Fix empty directory handling ( #10319 )
...
Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-12-02 10:37:08 -08:00
2e02eebd9d
Add context dimension to DefaultQueryMetrics ( #10578 )
...
* Add context dimension to DefaultQueryMetrics
* remove redundant addition of context dimension from DruidMetrics now that QueryMetrics adds it by default
* update SearchQueryMetrics to reflect the same pattern as other default dimensions in QueryMetrics
* add PublicApi annotation for context in QueryMetrics Interface
2020-12-01 18:34:03 -08:00
e7e07eab11
[Improve Doc] : Modify the disadvantages of the lazyLoadOnStart feature. ( #10608 )
...
* modify docs
* modify docs
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-12-01 18:33:22 -08:00
24f1e35b5d
fix desc of 'required' for granularity property ( #10616 )
2020-12-01 18:29:51 -08:00
5b06c7a3a9
Web console: improve how code is imported, use API instance ( #10597 )
...
* fix imports
* clean up imports
* update DQT to fix escaping
2020-12-01 13:16:14 -08:00
d47d6cf081
Add time-to-first-result benchmark for groupBy ( #10612 )
2020-12-01 10:32:37 -08:00
2560bf0a19
Add new coordinator metrics for coordinator duty runtimes ( #10603 )
...
* Add new coordinator metrics for duty runtimes
* fix spelling for a constant variable value
* add comment clarifying why the global runtime metric is emitted where it is
* Remove duty alias in lieu of using the class name for metrics
* fix docs
* CoordinatorStats tests + add duty stats to accumulate() logic
2020-11-29 14:47:35 -08:00
30bcb0fd74
DataSourcesSnapshotBenchmark to measure iterateAllUsedSegmentsInSnapshot perf ( #10604 )
2020-11-29 14:42:14 -08:00
7462b0b953
Allow missing intervals for Parallel task with hash/range partitioning ( #10592 )
...
* Allow missing intervals for Parallel task
* fix row filter
* fix tests
* fix log
2020-11-25 14:50:22 -08:00
d0c2ede50c
Added CronScheduler support as a proof to clock drift while emitting metrics ( #10448 )
...
Co-authored-by: Ayush Kulshrestha <ayush.kulshrestha@miqdigital.com>
2020-11-25 12:31:38 +01:00
fe693a4f01
Improve doc and exception message for invalid user configurations ( #10598 )
...
* improve doc and exception message
* add spelling check rules and remove unused import
* add a test to improve test coverage
2020-11-23 15:03:13 -08:00
31740b3b29
Fix : Druid throws java.util.concurrent.RejectedExecutionException when ingest task is stopping. ( #10555 )
...
* check exec status before return Signal
* add more log
* change log level to debug and add UT
* change log leverl to warn and merge master
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-11-23 14:52:03 -08:00
4537016cad
Update google client libraries ( #10536 )
...
modify license.yaml
Update google oauth client version
2020-11-20 15:23:30 -08:00
111b431c07
Introduce query/timeout/count metric ( #10567 )
...
* Add timeout metric
* Add tests
2020-11-20 15:17:26 -08:00
3cafd531de
fix issue causing incorrect config in Docker ( #10595 )
...
Previously, when the Docker entrypoint script generated the config
files, it would append the configuration without including a newline.
This could result in incorrect configuration. This has been fixed by
always appending a newline before any configuration.
Co-authored-by: Bryson Chen <brysonjackychen@gmail.com>
Co-authored-by: Bryson Chen <brysonjackychen@gmail.com>
2020-11-20 14:52:38 -08:00
2201ffa2f0
druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup ( #10591 )
...
* druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup
* address review comment
* remove unintentional change
2020-11-20 14:46:19 -08:00
ba915b7f56
Security overview documentation ( #10339 )
...
* initial file
* initial file
* security overview added
* ldap added
* spacing adjustments
* nits
* security graphics and doc review
* Update docs/operations/security-overview.md
Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
* Update docs/operations/security-user-auth.md
Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
* Update docs/operations/security-overview.md
Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
* Update docs/operations/security-overview.md
Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
* updates frm review
* review comments
* finish up review and light edits
* broken links
* spell check
Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
2020-11-19 15:24:58 -08:00
2f4d6da33f
Updates segment metadata query documentation ( #10589 )
...
* updates segment metadata query documentation to be clearer about cardinality estimation
* typo in documentation
2020-11-20 00:08:27 +05:30
abb03d38b0
double confirm before delete all the ununsed segmensts ( #10581 )
...
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-11-19 10:07:52 -06:00
d7d2c804ad
Add zero period support to TIMESTAMPADD ( #10550 )
...
* Allow zero period for TIMESTAMPADD
* update test cases
* add empty zone test case
* add unit test cases for TimestampShiftMacro
2020-11-18 18:26:53 -08:00
1272fb17e5
modify druid.historical.cache.maxEntrySize property in Unified format ( #10590 )
...
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2020-11-17 16:36:50 -06:00
9964dd4cb2
Web console: fix data loader schema table column ordering bug and other polish ( #10588 )
...
* remove unused fields
* keep tables live
* advanced
* fix schema view
* better indication
* tests pass
* Show more instead of show advanced
* fix tests
* extract dynamic configs
* update snapshots
* fix issues
* update snapshot
* reword without >
2020-11-17 13:25:03 -08:00
3447934a75
Ensure Krb auth before killing YARN apps in graceful shutdown ( #9785 )
2020-11-16 09:59:14 -08:00
9e6978609a
Add TravisCI job that builds and tests on ARM64 CPU architecture ( #10562 )
2020-11-16 21:08:43 +05:30
e83d5cb59e
Fix ingestion failure of pretty-formatted JSON message ( #10383 )
...
* support multi-line text
* add test cases
* split json text into lines case by case
* improve exception handle
* fix CI
* use IntermediateRowParsingReader as base of JsonReader
* update doc
* ignore the non-immutable field in test case
* add more test cases
* mark `lineSplittable` as final
* fix testcases
* fix doc
* add a test case for SqlReader
* return all raw columns when exception occurs
* fix CI
* fix test cases
* resolve review comments
* handle ParseException returned by index.add
* apply Iterables.getOnlyElement
* fix CI
* fix test cases
* improve code in more graceful way
* fix test cases
* fix test cases
* add a test case to check multiple json string in one text block
* fix inspection check
2020-11-13 13:59:23 -08:00
21e3c4b39c
Add missing docs for timeout exceptions ( #10554 )
...
* Add missing docs for timeout exceptions
* Add info on auth failures
2020-11-13 08:45:40 -06:00
c3cad461bc
Correct getRandomBalancerSegmentHolderTest ( #10569 )
2020-11-12 23:21:05 +05:30
6563599de4
modify access to protected SQLMetadataConnector methods to allow extensions to create SQL metadata tables using implementation specific constructs (payload type, serial type, etc) ( #10573 )
2020-11-12 23:20:01 +05:30
f1b5745eb6
Add helm shield ( #10526 )
2020-11-10 16:18:36 -06:00
20e7e53ede
Use FileUtils.createTempDir in IndexGeneratorJob. ( #10568 )
...
Simplifies code by using a method designed for this purpose.
2020-11-09 22:40:28 -08:00
cd231d8511
Run integration test queries once ( #10564 )
...
* Run integration test queries once
* missed a few
2020-11-09 17:34:27 -08:00
83667172ce
Add missing coordinator dynamic config to the web-console dialog for dynamic coordinator config ( #10545 )
...
* Add missing coordinator dynamic config to the web-console dialog for dynamic coordinator config
* update jest snapshots after making console change
2020-11-09 13:00:52 -06:00
6c8a77b7a9
Bump jetty to latest version ( #10563 )
...
This addresses CVE-2020-27216 which was flagged by the security vulnerability
job.
2020-11-09 08:51:36 -08:00
3436297354
Clarify how ORDER BY works with UNION ALL ( #10561 )
...
Hopefully a bit clearer.
2020-11-05 20:12:03 -08:00
6ccddedb7a
Improved exception handling in case of query timeouts ( #10464 )
...
* Separate timeout exceptions
* Add more tests
Co-authored-by: Atul Mohan <atulmohan@yahoo-inc.com>
2020-11-03 09:00:33 -06:00
d8e5a159e8
Update index.md ( #10549 )
...
Removing the extra `_` in the default for middlemanager category
2020-11-03 13:44:47 +05:30
9286153145
doc wrong description of configuration ( #10546 )
2020-11-02 17:57:16 -08:00
6b14bdb3a5
Add support for Blacklisting some domains for HTTPInputSource ( #10535 )
...
fix inspections
refactor class name
change name
add allowList as well
distinguish between empty and null list
Fix CI
2020-11-02 21:47:25 +05:30
egor-ryashin
Gian Merlino
Maytas Monsereenusorn
Liran Funaro
zhangyue19921010
Jihoon Son
Valdemar
Himanshu
Suneet Saldanha
Atul Mohan
Lucas Capistrant
frank chen
Vadim Ogievetsky
Ayush Kulshrestha
Nishant Bangarwa
David Palmer
sthetland
michaelschiff
Martin Grigorov
Clint Wylie
Benedict Jin
Mainak Ghosh
Husky Zeng