Commit Graph

12880 Commits

Author SHA1 Message Date
Vadim Ogievetsky 981662e9f4
Web console: add a nice UI for overlord dynamic configs and improve the docs (#13993)
* in progress

* better form

* doc updates

* doc changes

* add inline docs

* fix tests

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* Update docs/configuration/index.md

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* final fixes

* fix case

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Update docs/configuration/index.md

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* fix overflow

* fix spelling

---------

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2023-03-31 10:12:25 -07:00
Clint Wylie e3211e3be0
actually backwards compatible frontCoded string encoding strategy (#13996) 2023-03-31 02:24:12 -07:00
soullkk 51f3db2ce6
Fix peon errors when executing tasks in ipv6(#13972) (#13995) 2023-03-31 09:18:10 +05:30
Soumyava 1eeecf5fb2
Fixing regression issues on unnest (#13976)
* select sum(c) on an unnested column now does not return 'Type mismatch' error and works properly
* Making sure an inner join query works properly
* Having on unnested column with a group by now works correctly
* count(*) on an unnested query now works correctly
2023-03-31 09:06:43 +05:30
abhagraw eb31207402
Using MinIO to run S3DeepStorage ITs (#13997)
* Using MinIO to S3DeepStorage ITs

* Adding S3DeepStorageTest to github actions revised ITs
2023-03-30 12:15:53 -07:00
Kashif Faraz 47face9ca9
Handle null values in BrokerServerView.serverAddedSegment (#13980)
Due to race conditions, the BrokerServerView may sometimes try to add a segment to a server which has already been removed from the inventory. This results in an NPE and keeps the BrokerServerView from processing all change requests.
2023-03-30 16:19:05 +05:30
Nicholas Lippis 61a35262ec
Kubernetes task runner live reports (#13986)
Implement Live Reports for the KubernetesTaskRunner
2023-03-30 10:30:22 +05:30
George Shiqi Wu 44abe2b96f
Fix bug in k8s task runner in handling deleted jobs (#14001)
With the KubernetesTaskRunner, if a task is manually shutdown via the web console while running or the corresponding k8s job is manually deleted, the thread responsible for overseeing the task gets stuck in a loop because the fabric8 client sends one event to it that the job is null when the job is deleted, but this doesn't pass the condition.

This means that the thread is stuck waiting on a fabric8 event (the job being successful) that will never come up until maxTaskDuration (default 4 hours). If a user of the extension is trying to use a limited taskqueue maxSize, this can cause problems as the k8s executor pool is unable to pick up additional tasks (since threads are stuck waiting on the old tasks that have already been deleted).
2023-03-30 10:09:52 +05:30
zachjsh 3bb67721f7
Allow for Input source security in SQL layer (#13989)
This change introduces the concept of input source type security model, proposed in #13837.. With this change, this feature is only available at the SQL layer, but we will expand to native layer in a follow up PR.

To enable this feature, the user must set the following property to true:

druid.auth.enableInputSourceSecurity=true

The default value for this property is false, which will continue the existing functionality of having the usage all external sources being authorized against the hardcoded resource action

new ResourceAction(new Resource(ResourceType.EXTERNAL, ResourceType.EXTERNAL), Action.READ

When this config is enabled, the users will be required to be authorized for the following resource action

new ResourceAction(new Resource(ResourceType.EXTERNAL, {INPUT_SOURCE_TYPE}, Action.READ

where {INPUT_SOURCE_TYPE} is the type of the input source being used;, http, inline, s3, etc..

Documentation has not been added for the feature as it is not complete at the moment, as we still need to enable this for the native layer in a follow up pr.
2023-03-29 22:15:33 -04:00
Vadim Ogievetsky abb7133153
Web console: use EXTEND syntax (#13985)
* use EXTEND syntax

* update licenses

* update demo queries

* updated snapshots

* add join algorithm selector

* dismiss
2023-03-29 16:19:49 -07:00
Tejaswini Bandlamudi f715887172
Debug docker logs on ITs failure. (#13978) 2023-03-29 09:06:41 -07:00
Karan Kumar e4c5122a60
Fixing checkstyle (#14000) 2023-03-29 20:21:21 +05:30
Sandeep ccdf30e399
Bump Joda-Time version for current DateTimeZone data (#13999) 2023-03-29 20:15:49 +05:30
Karan Kumar 8dce3ca4d5
OOM fix for running MSQ jobs with `intermediateSuperSorterStorageMaxLocalBytes` set (#13974)
While using intermediateSuperSorterStorageMaxLocalBytes the super sorter was retaining references of the memory allocator.

The fix clears the current outputChannel when close() is called on the ComposingWritableFrameChannel.java
2023-03-29 18:00:00 +05:30
Tejaswini Bandlamudi 3c096c01a2
cache mvn dependencies across tests without building (#13962) 2023-03-29 16:27:36 +05:30
Nicholas Lippis 488f1d8363
Do not print error message if pod not found when getting task location (#13971)
Do not print error message if pod not found when getting task location
2023-03-29 13:27:06 +05:30
Clint Wylie 2219e68fa3
add backwards compat mode for frontCoded stringEncodingStrategy (#13988) 2023-03-28 14:44:44 -07:00
Paul Rogers 76fe26d4ba
Fix typos, add tests for http() function (#13954) 2023-03-28 14:41:06 -07:00
frankgrimes97 2f98675285
Tuple sketch SQL support (#13887)
This PR is a follow-up to #13819 so that the Tuple sketch functionality can be used in SQL for both ingestion using Multi-Stage Queries (MSQ) and also for analytic queries against Tuple sketch columns.
2023-03-28 18:47:12 +05:30
Karan Kumar c2fe6a4956
Reworking s3 connector with various improvements (#13960)
* Reworking s3 connector with
1. Adding retries
2. Adding max fetch size
3. Using s3Utils for most of the api's
4. Fixing bugs in DurableStorageCleaner
5. Moving to Iterator for listDir call
2023-03-28 17:05:16 +05:30
Rishabh Singh e8e8082573
Update OIDCConfig with scope information (#13973)
Allow users to provide custom scope through OIDC configuration
2023-03-28 14:50:00 +05:30
Clint Wylie d5b1b5bc8e
nested columns + arrays = array columns! (#13803)
array columns!
changes:
* add support for storing nested arrays of string, long, and double values as specialized nested columns instead of breaking them into separate element columns
* nested column type mimic behavior means that columns ingested with only root arrays of primitive values will be ARRAY typed columns
* neat test refactor stuff
* add v4 segment test
* add array element indexes
* add tests for unnest and array columns
* fix unnest column value selector cursor handling of null and empty arrays
2023-03-27 12:42:35 -07:00
Gian Merlino 062d72b67e
Add timeout to TaskStartTimeoutFault. (#13970)
* Add timeout to TaskStartTimeoutFault.

Makes the error message a bit more useful.

* Update docs.
2023-03-27 23:37:19 +05:30
Arnout Engelen daff7fe73b
Document how to report security issues (#13886)
Document how to report security issues on the security overview page, so we can link this page from the homepage. That should make all the other important security information easier to find as well.
2023-03-27 11:26:37 +05:30
kaijianding 13ffeb50ba
should retry when failed to pause realtime task (#11515) 2023-03-25 19:03:13 +05:30
Atul Mohan 19db32d6b4
Add JWT authenticator support for validating ID Tokens (#13242)
Expands the OIDC based auth in Druid by adding a JWT Authenticator that validates ID Tokens associated with a request. The existing pac4j authenticator works for authenticating web users while accessing the console, whereas this authenticator is for validating Druid API requests made by Direct clients. Services already supporting OIDC can attach their ID tokens to the Druid requests
under the Authorization request header.
2023-03-25 18:41:40 +05:30
Rishabh Singh 598eaad7e1
Fix HSTS for middle manager (#13975)
Fix HSTS for middle manager
2023-03-25 14:01:09 +05:30
Gian Merlino 549018d076 Revert "Update docs."
This reverts commit de27c7d3c1.
2023-03-24 17:16:12 -07:00
Gian Merlino de27c7d3c1 Update docs. 2023-03-24 17:15:27 -07:00
Nicholas Lippis 8a72544bd2
Hook up pod template adapter (#13966)
* Hook up PodTemplateTaskAdapter

* Make task adapter TYPE parameters final

* Rename adapters types

* Include specified adapter name in exception message

* Documentation for sidecarSupport deprecation

* Fix order

* Set TASK_ID as environment variable in PodTemplateTaskAdapter (#13969)

* Update docs/development/extensions-contrib/k8s-jobs.md

Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>

* Hook up PodTemplateTaskAdapter

* Make task adapter TYPE parameters final

* Rename adapters types

* Include specified adapter name in exception message

* Documentation for sidecarSupport deprecation

* Fix order

* fix spelling errors

---------

Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
2023-03-24 12:13:46 -06:00
Jill Osborne 976d39281f
Fix some broken links in docs (#13968) 2023-03-24 10:48:23 -07:00
Nicholas Lippis 36df2495e1
Set TASK_ID as environment variable in PodTemplateTaskAdapter (#13969) 2023-03-23 16:45:01 -06:00
Abhishek Agarwal 139a058ba7
Use sonatype maven central for plugin repositories (#13961)
* Change search order of maven repositories

* Update pom.xml
2023-03-23 15:35:47 +05:30
abhagraw c52d15d65d
Fixing security vulnerability check errors (#13956)
* Fixing security vulnerability check errors

* Updating javax.el to jakarta.el

* Adding cron job trigger on changes to suppressions file
2023-03-23 11:10:06 +05:30
Paul Rogers da42ee5bfa
Added TYPE(native) data type for external tables (#13958) 2023-03-22 21:43:29 -07:00
Soumyava 2ad133c06e
Unnest changes for moving the filter on right side of correlate to inside the unnest datasource (#13934)
* Refactoring and bug fixes on top of unnest. The filter now is passed inside the unnest cursors. Added tests for scenarios such as
1. filter on unnested column which involves a left filter rewrite
2. filter on unnested virtual column which pushes the filter to the right only and involves no rewrite
3. not filters
4. SQL functions applied on top of unnested column
5. null present in first row of the column to be unnested
2023-03-22 18:24:00 -07:00
Vadim Ogievetsky 8d125b7c7f
Web console: segment writing progress indication (#13929)
* add segment writing progress indication

* update with more metrics

* add push metric
2023-03-22 16:34:38 -07:00
Nicholas Lippis d81d13b9ba
Pod template task adapter (#13896)
* Pod template task adapter

* Use getBaseTaskDirPaths

* Remove unused task from getEnv

* Use Optional.ifPresent() instead of Optional.map()

* Pass absolute path

* Don't pass task to getEnv

* Assert the correct adapter is created

* Javadocs and Comments

* Add exception message to assertions
2023-03-22 14:20:24 -06:00
Clint Wylie 086eb26b74
fix join and unnest planning to ensure that duplicate join prefixes are not used (#13943)
* fix join and unnest planning to ensure that duplicate join prefixes are not used

* wont somebody please think of the children
2023-03-22 12:53:55 -07:00
Adarsh Sanjeev 7bab407495
Add segment generator counters to MSQ reports (#13909)
* Add segment generator counters to reports

* Remove unneeded annotation

* Fix checkstyle and coverage

* Add persist and merged as new metrics

* Address review comments

* Fix checkstyle

* Create metrics class to handle updating counters

* Address review comments

* Add rowsPushed as a new metrics
2023-03-22 09:17:26 -07:00
Clint Wylie f4392a3155
expression transform improvements and fixes (#13947)
changes:
* fixes inconsistent handling of byte[] values between ExprEval.bestEffortOf and ExprEval.ofType, which could cause byte[] values to end up as java toString values instead of base64 encoded strings in ingest time transforms
* improved ExpressionTransform binding to re-use ExprEval.bestEffortOf when evaluating a binding instead of throwing it away
* improved ExpressionTransform array handling, added RowFunction.evalDimension that returns List<String> to back Row.getDimension and remove the automatic coercing of array types that would typically happen to expression transforms unless using Row.getDimension
* added some tests for ExpressionTransform with array inputs
* improved ExpressionPostAggregator to use partial type information from decoration
* migrate some test uses of InputBindings.forMap to use other methods
2023-03-21 23:26:53 -07:00
Kashif Faraz b7752a909c
Enable round-robin segment assignment and batch segment allocation by default (#13942)
Changes:
- Set `useRoundRobinSegmentAssignment` in coordinator dynamic config to `true` by default.
- Set `batchSegmentAllocation` in `TaskLockConfig` (used in Overlord runtime properties) to `true` by default.
2023-03-22 08:20:01 +05:30
Victoria Lim ede9903ff4
pip install for Python Druid API (#13938)
Broken test appears unrelated to this PR

* make druidapi pip installable

* include druidapi in prerequisites

* add license to setup.py

* updates from Paul's review

* note about editable install

* Apply suggestions from code review

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>

* update install instructions

* found unrelated typos

* standardize install cmd with pip

---------

Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
2023-03-21 11:37:39 -07:00
Gian Merlino 1c7a03a47b
Lower default maxRowsInMemory for realtime ingestion. (#13939)
* Lower default maxRowsInMemory for realtime ingestion.

The thinking here is that for best ingestion throughput, we want
intermediate persists to be as big as possible without using up all
available memory. So, we rely mainly on maxBytesInMemory. The default
maxRowsInMemory (1 million) is really just a safety: in case we have
a large number of very small rows, we don't want to get overwhelmed
by per-row overheads.

However, maximum ingestion throughput isn't necessarily the primary
goal for realtime ingestion. Query performance is also important. And
because query performance is not as good on the in-memory dataset, it's
helpful to keep it from growing too large. 150k seems like a reasonable
balance here. It means that for a typical 5 million row segment, we
won't trigger more than 33 persists due to this limit, which is a
reasonable number of persists.

* Update tests.

* Update server/src/main/java/org/apache/druid/segment/indexing/RealtimeTuningConfig.java

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

* Fix test.

* Fix link.

---------

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>
2023-03-21 10:36:36 -07:00
Jill Osborne 4f95285406
Correct nested columns JSON example (#13953) 2023-03-21 09:17:26 -07:00
Atul Mohan 617c325c70
Make zk connection retries configurable (#13913)
* This makes the zookeeper connection retry count configurable. This is presently hardcoded to 29 tries which ends up taking a long time for the druid node to shutdown in case of ZK connectivity loss.
Having a shorter retry count helps k8s deployments to fail fast. In situations where the underlying k8s node loses network connectivity or is no longer able to talk to zookeeper, failing fast can trigger pod restarts which can then reassign the pod to a healthy k8s node.
Existing behavior is preserved, but users can override this property if needed.
2023-03-21 14:45:28 +05:30
Adarsh Sanjeev 143fdcfacf
Change test name so it triggers in CI (#13844)
As the name of the class did not end or start with "Test", CalciteSelectQueryMSQTest was not triggered in CI. This PR renames the test.
2023-03-20 15:55:52 +05:30
Tejaswini Bandlamudi 1c250a0bc0
Fix error in cron job ITs workflow (#13945) 2023-03-17 17:29:45 +05:30
John Gozde 38adac4369
Dart sass (#13937)
* Run npx saas-migrator division

* Switch to dart sass

* Upgrade blueprint

* Remove deprecated import syntax

* Prettify

* Snapshots
2023-03-16 12:44:24 -07:00
Karan Kumar bf13156b55
Regression bug fix where ever LimitFrameProcessor's were used. (#13941) 2023-03-16 09:18:18 -07:00