7e147ee905
Web console: Reset to specific offsets dialog ( #14863 )
...
* add dialog
* copy changes
2023-08-17 15:38:56 -07:00
59415ba9b2
Web console: expose new coordinator properties in the dialog ( #14791 )
...
* expose new coordinator properties in the dialog
* escape
2023-08-17 15:37:23 -07:00
37db5d9b81
Reset offsets supervisor API ( #14772 )
...
* Add supervisor /resetOffsets API.
- Add a new endpoint /druid/indexer/v1/supervisor/<supervisorId>/resetOffsets
which accepts DataSourceMetadata as a body parameter.
- Update logs, unit tests and docs.
* Add a new interface method for backwards compatibility.
* Rename
* Adjust tests and javadocs.
* Use CoreInjectorBuilder instead of deprecated makeInjectorWithModules
* UT fix
* Doc updates.
* remove extraneous debugging logs.
* Remove the boolean setting; only ResetHandle() and resetInternal()
* Relax constraints and add a new ResetOffsetsNotice; cleanup old logic.
* A separate ResetOffsetsNotice and some cleanup.
* Minor cleanup
* Add a check & test to verify that sequence numbers are only of type SeekableStreamEndSequenceNumbers
* Add unit tests for the no op implementations for test coverage
* CodeQL fix
* checkstyle from merge conflict
* Doc changes
* DOCUSAURUS code tabs fix. Thanks, Brian!
2023-08-17 14:13:10 -07:00
2cc3bd6383
Bump joda-time:joda-time from 2.12.4 to 2.12.5 ( #14855 )
...
* Bump joda-time:joda-time from 2.12.4 to 2.12.5
Bumps [joda-time:joda-time](https://github.com/JodaOrg/joda-time ) from 2.12.4 to 2.12.5.
- [Release notes](https://github.com/JodaOrg/joda-time/releases )
- [Changelog](https://github.com/JodaOrg/joda-time/blob/main/RELEASE-NOTES.txt )
- [Commits](https://github.com/JodaOrg/joda-time/compare/v2.12.4...v2.12.5 )
---
updated-dependencies:
- dependency-name: joda-time:joda-time
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-17 11:24:22 -07:00
2a7fbf2ab4
Bump org.apache.directory.api:api-util from 1.0.3 to 2.1.3 ( #14852 )
...
Bumps [org.apache.directory.api:api-util](https://github.com/apache/directory-ldap-api ) from 1.0.3 to 2.1.3.
- [Commits](https://github.com/apache/directory-ldap-api/compare/1.0.3...2.1.3 )
---
updated-dependencies:
- dependency-name: org.apache.directory.api:api-util
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:56:34 -07:00
fffb2e4fe7
Speed up SQLMetadataStorageActionHandlerTest ( #14856 )
...
Changes
- Reduce test time of `SQLMetadataStorageActionHandlerTest.testMigration`
- Slightly modify log messages to adhere to Druid style
2023-08-17 18:02:43 +05:30
b97cc45d81
Add clarification to the docs for multi-topic Kafka ingestion ( #14847 )
...
Follow-up to #14828 . Added some more clarification about how topicPattern is used.
2023-08-17 12:52:06 +05:30
dc2ae1e99c
Web console: improving the helper queries by allowing for running inline helper queries ( #14801 )
...
* remove helper queries
* fix tests
* take care of zero queries also
* switch to better place
2023-08-16 23:50:43 -07:00
5d4ac64178
Adapt maxSegmentsToMove based on cluster skew ( #14584 )
...
Changes:
- No change in behaviour if `smartSegmentLoading` is disabled
- If `smartSegmentLoading` is enabled
- Compute `balancerComputeThreads` based on `numUsedSegments`
- Compute `maxSegmentsToMove` based on `balancerComputeThreads`
- Compute `segmentsToMoveToFixSkew` based on usage skew
- Compute `segmentsToMove = Math.min(maxSegmentsToMove, segmentsToMoveToFixSkew)`
Limits:
- 1 <= `balancerComputeThreads` <= 8
- `maxSegmentsToMove` <= 20% of total segments
- `minSegmentsToMove` = 0.15% of total segments
2023-08-17 11:14:54 +05:30
cb27d0d2ed
Web console: enable Kafka multi-topic ingestion from the data loader ( #14833 )
...
* multi topic ux
* updated to match new api
2023-08-17 09:57:34 +05:30
6b4dda964d
Docusaurus2 upgrade for master ( #14411 )
...
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-08-16 19:01:21 -07:00
6b14dde50e
deprecate config-magic in favor of json configuration stuff ( #14695 )
...
* json config based processing and broker merge configs to deprecate config-magic
2023-08-16 18:23:57 -07:00
26d82fd342
fix filtering bug in filtering unnest cols and dim cols: Received a non-applicable rewrite ( #14587 )
2023-08-16 17:57:16 -07:00
f585f0a8ed
202306-docs-notebook topn ( #14478 )
...
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
2023-08-16 14:50:49 -07:00
2561477e87
Jupyter nested columns tutorial ( #14788 )
2023-08-16 14:45:37 -07:00
faf79470ae
Bump io.dropwizard.metrics:metrics-graphite from 3.1.2 to 4.2.19 ( #14842 )
...
* Bump io.dropwizard.metrics:metrics-graphite from 3.1.2 to 4.2.19
Bumps [io.dropwizard.metrics:metrics-graphite](https://github.com/dropwizard/metrics ) from 3.1.2 to 4.2.19.
- [Release notes](https://github.com/dropwizard/metrics/releases )
- [Commits](https://github.com/dropwizard/metrics/compare/v3.1.2...v4.2.19 )
---
updated-dependencies:
- dependency-name: io.dropwizard.metrics:metrics-graphite
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* align graphite-emitter dropwizard version with core
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 13:58:35 -07:00
3954685aae
Report more metrics to monitor K8s task runner ( #14771 )
...
* Report pod running metrics to monitor K8s task runner
* refine method definition
* fix checkstyle
* implement task metrics
* more comment
* address comments
* update doc for the new metrics reported
* fix checkstyle
* refine method definition
* minor refine
2023-08-16 14:03:53 -04:00
97c3773012
Bump commons-cli:commons-cli from 1.3.1 to 1.5.0 ( #14837 )
...
* Bump commons-cli:commons-cli from 1.3.1 to 1.5.0
Bumps commons-cli:commons-cli from 1.3.1 to 1.5.0.
---
updated-dependencies:
- dependency-name: commons-cli:commons-cli
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:37:56 -07:00
8be7751dbd
Bump org.tukaani:xz from 1.8 to 1.9 ( #14839 )
...
* Bump org.tukaani:xz from 1.8 to 1.9
Bumps org.tukaani:xz from 1.8 to 1.9.
---
updated-dependencies:
- dependency-name: org.tukaani:xz
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:37:29 -07:00
e55fe67535
Bump apache.curator.version from 5.4.0 to 5.5.0 ( #14843 )
...
* Bump apache.curator.version from 5.4.0 to 5.5.0
Bumps `apache.curator.version` from 5.4.0 to 5.5.0.
Updates `org.apache.curator:curator-client` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-framework` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-recipes` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-x-discovery` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-test` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
---
updated-dependencies:
- dependency-name: org.apache.curator:curator-client
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-framework
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-recipes
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-x-discovery
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:36:58 -07:00
7911a04064
Refactoring of multi-topic kafka ingestion docs ( #14828 )
...
In this PR, I have gotten rid of multiTopic parameter and instead added a topicPattern parameter. Kafka supervisor will pass topicPattern or topic as the stream name to the core ingestion engine. There is validation to ensure that only one of topic or topicPattern will be set. This new setting is easier to understand than overloading the topic field that earlier could be interpreted differently depending on the value of some other field.
2023-08-16 18:00:11 +05:30
d9221e46e4
Completely disable cachingCost balancer strategy ( #14798 )
...
`cachingCost` has been deprecated in #14484 and is not advised to be used in
production clusters as it may cause usage skew across historicals which the
coordinator is unable to rectify. This PR completely disables `cachingCost` strategy
as it has now been rendered redundant due to recent performance improvements
made to `cost` strategy.
Changes
- Disable `cachingCost` strategy
- Add `DisabledCachingCostBalancerStrategyFactory` for the time being so that we
can give a proper error message before falling back to `CostBalancerStrategy`. This
will be removed in subsequent releases.
- Retain `CachingCostBalancerStrategy` for testing/benchmarking purposes.
- Add javadocs to `DiskNormalizedCostBalancerStrategy`
2023-08-16 11:43:52 +05:30
9be0f64f50
Bump org.apache.commons:commons-compress from 1.21 to 1.23.0 ( #14820 )
...
* Bump org.apache.commons:commons-compress from 1.21 to 1.23.0
Bumps org.apache.commons:commons-compress from 1.21 to 1.23.0.
---
updated-dependencies:
- dependency-name: org.apache.commons:commons-compress
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 20:08:54 -04:00
0b2563fea3
Web console: adding format notice for CSV and TSV ( #14783 )
...
* adding format notice for CSV and TSV
* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx
Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx
Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
* Update web-console/src/druid-models/ingestion-spec/ingestion-spec.tsx
Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
* fix tests
---------
Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
2023-08-15 15:35:50 -07:00
8fa78594ea
Druid SQL API documentation refactor ( #14711 )
2023-08-15 13:45:25 -07:00
a38579ab3c
Retention rules API documentation refactor ( #14623 )
2023-08-15 13:44:44 -07:00
aeeeed3b35
Bump protobuf.version from 3.21.7 to 3.24.0 ( #14823 )
...
* Bump protobuf.version from 3.21.7 to 3.24.0
Bumps `protobuf.version` from 3.21.7 to 3.24.0.
Updates `com.google.protobuf:protobuf-java` from 3.21.7 to 3.24.0
- [Release notes](https://github.com/protocolbuffers/protobuf/releases )
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl )
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.21.7...v3.24.0 )
Updates `com.google.protobuf:protobuf-java-util` from 3.21.7 to 3.24.0
---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: com.google.protobuf:protobuf-java-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
* fix licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 12:15:55 -07:00
8abdaa239b
Bump dropwizard.metrics.version from 4.0.0 to 4.2.19 ( #14824 )
...
* Bump dropwizard.metrics.version from 4.0.0 to 4.2.19
Bumps `dropwizard.metrics.version` from 4.0.0 to 4.2.19.
Updates `io.dropwizard.metrics:metrics-core` from 4.0.0 to 4.2.19
- [Release notes](https://github.com/dropwizard/metrics/releases )
- [Commits](https://github.com/dropwizard/metrics/compare/v4.0.0...v4.2.19 )
Updates `io.dropwizard.metrics:metrics-jmx` from 4.0.0 to 4.2.19
- [Release notes](https://github.com/dropwizard/metrics/releases )
- [Commits](https://github.com/dropwizard/metrics/compare/v4.0.0...v4.2.19 )
---
updated-dependencies:
- dependency-name: io.dropwizard.metrics:metrics-core
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: io.dropwizard.metrics:metrics-jmx
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 12:15:29 -07:00
e33d2db235
202307-notebooks Template amends ( #14683 )
...
Co-authored-by: writer-jill <jill.osborne@imply.io>
2023-08-15 11:25:56 -07:00
2fdf5b195f
Bump org.assertj:assertj-core from 3.19.0 to 3.24.2 ( #14815 )
...
Bumps org.assertj:assertj-core from 3.19.0 to 3.24.2.
---
updated-dependencies:
- dependency-name: org.assertj:assertj-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-15 07:39:48 -07:00
f0a79fa0e4
Bump org.apache.maven.plugins:maven-source-plugin from 2.2.1 to 3.3.0 ( #14812 )
...
Bumps [org.apache.maven.plugins:maven-source-plugin](https://github.com/apache/maven-source-plugin ) from 2.2.1 to 3.3.0.
- [Commits](https://github.com/apache/maven-source-plugin/compare/maven-source-plugin-2.2.1...maven-source-plugin-3.3.0 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-source-plugin
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-15 07:39:19 -07:00
0967048dca
Bump org.scala-lang:scala-library from 2.13.9 to 2.13.11 ( #14826 )
...
Bumps [org.scala-lang:scala-library](https://github.com/scala/scala ) from 2.13.9 to 2.13.11.
- [Release notes](https://github.com/scala/scala/releases )
- [Commits](https://github.com/scala/scala/compare/v2.13.9...v2.13.11 )
---
updated-dependencies:
- dependency-name: org.scala-lang:scala-library
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-15 07:38:34 -07:00
76c7963979
Bump com.github.oshi:oshi-core from 6.4.2 to 6.4.4 ( #14814 )
...
* Bump com.github.oshi:oshi-core from 6.4.2 to 6.4.4
Bumps [com.github.oshi:oshi-core](https://github.com/oshi/oshi ) from 6.4.2 to 6.4.4.
- [Release notes](https://github.com/oshi/oshi/releases )
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md )
- [Commits](https://github.com/oshi/oshi/compare/oshi-parent-6.4.2...oshi-parent-6.4.4 )
---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
2023-08-15 16:14:19 +05:30
5f54ae7d27
Bump org.apache.maven.plugins:maven-surefire-plugin ( #14813 )
...
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire ) from 3.0.0-M7 to 3.1.2.
- [Release notes](https://github.com/apache/maven-surefire/releases )
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.0.0-M7...surefire-3.1.2 )
2023-08-14 23:07:53 -07:00
d5d483fdc9
Bump org.apache.rat:apache-rat-plugin from 0.12 to 0.15 ( #14817 )
2023-08-14 23:07:03 -07:00
50b3d96df5
increase dependabot PR limit for Java dependencies ( #14804 )
...
Many dependabot PRs are currently stuck due to API changes or
incompatibilities. Temporarily Increasing the limit so we can get
updates for other dependencies.
2023-08-14 19:51:59 -07:00
30b5dd4ca7
Add support to read from multiple kafka topics in same supervisor ( #14424 )
...
This PR adds support to read from multiple Kafka topics in the same supervisor. A multi-topic ingestion can be useful in scenarios where a cluster admin has no control over input streams. Different teams in an org may create different input topics that they can write the data to. However, the cluster admin wants all this data to be queryable in one data source.
2023-08-14 22:24:49 +05:30
e16096735b
Fix 404 when segment is used but not in the Coordinator snapshot ( #14762 )
...
* Fix 404 when used segment has not been updated in the Coordinator snapshot
* Add unit test
2023-08-14 13:20:43 +05:30
786e772d26
Remove config `druid.coordinator.compaction.skipLockedIntervals` ( #14807 )
...
The value of `druid.coordinator.compaction.skipLockedIntervals` should always be `true`.
2023-08-14 12:31:15 +05:30
0dc305f9e4
Upgrade hibernate validator version to fix CVE-2019-10219 ( #14757 )
2023-08-14 11:50:51 +05:30
e2d2afce46
Bump postgresql from 42.4.1 to 42.6.0 ( #13959 )
...
* Bump postgresql from 42.4.1 to 42.6.0
Bumps [postgresql](https://github.com/pgjdbc/pgjdbc ) from 42.4.1 to 42.6.0.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases )
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md )
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.4.1...REL42.6.0 )
---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-12 19:17:00 -04:00
afe22907a5
Calcite upgrade 1.35 ( #14510 )
...
* Update to Calcite 1.35.0
* Update from.ftl for Calcite 1.35.0.
* Fixed tests in Calcite upgrade by doing the following:
1. Added a new rule, CoreRules.PROJECT_FILTER_TRANSPOSE_WHOLE_PROJECT_EXPRESSIONS, to Base rules
2. Refactored the CorrelateUnnestRule
3. Updated CorrelateUnnestRel accordingly
4. Fixed a case with selector filters on the left where Calcite was eliding the virtual column
5. Additional test cases for fixes in 2,3,4
6. Update to StringListAggregator to fail a query if separators are not propagated appropriately
* Refactored for testcases to pass after the upgrade, introduced 2 new data sources for handling filters and select projects
* Added a literalSqlAggregator as the upgraded Calcite involved changes to subquery remove rule. This corrected plans for 2 queries with joins and subqueries by replacing an useless literal dimension with a post agg. Additionally a test with COUNT DISTINCT and FILTER which was failing with Calcite 1.21 is added here which passes with 1.35
* Updated to latest avatica and updated code as SqlUnknownTimeStamp is now used in Calcite which needs to be resolved to a timestamp literal
* Added a wrapper segment ref to use for unnest and filter segment reference
2023-08-11 12:47:16 -07:00
c8a11702db
Support broadcast segmetns ( #14789 )
2023-08-11 11:14:05 -07:00
ec28672d07
Web console: allow format picking for download ( #14794 )
...
* allow format picking for download
* better popover
* ux review tweaks
2023-08-11 09:43:29 -07:00
b0c78ff295
Web console: make retention dialog clearer ( #14793 )
...
* make retention dialog clearer
* tweak
* another tweak
* Update web-console/src/dialogs/retention-dialog/retention-dialog.tsx
Co-authored-by: Suneet Saldanha <suneet@apache.org>
* update snapshot for copy
---------
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2023-08-11 09:43:00 -07:00
a0234c4e13
Add sampling factor for DeterminePartitionsJob ( #13840 )
...
There are two type of DeterminePartitionsJob:
- When the input data is not assume grouped, there may be duplicate rows.
In this case, two MR jobs are launched. The first one do group job to remove duplicate rows.
And a second one to perform global sorting to find lower and upper bound for target segments.
- When the input data is assume grouped, we only need to launch the global sorting
MR job to find lower and upper bound for segments.
Sampling strategy:
- If the input data is assume grouped, sample by random at the mapper side of the global sort mr job.
- If the input data is not assume grouped, sample at the mapper of the group job. Use hash on time
and all dimensions and mod by sampling factor to sample, don't use random method because there
may be duplicate rows.
2023-08-11 10:42:25 +05:30
353f7bed7f
Adding data generation pod to jupyter notebooks deployment ( #14742 )
...
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2023-08-10 15:43:05 -07:00
82d82dfbd6
Add stats to KillUnusedSegments coordinator duty ( #14782 )
...
### Description
Added the following metrics, which are calculated from the `KillUnusedSegments` coordinatorDuty
`"killTask/availableSlot/count"`: calculates the number remaining task slots available for auto kill
`"killTask/maxSlot/count"`: calculates the maximum number of tasks available for auto kill
`"killTask/task/count"`: calculates the number of tasks submitted by auto kill.
#### Release note
NEW: metrics added for auto kill
`"killTask/availableSlot/count"`: calculates the number remaining task slots available for auto kill
`"killTask/maxSlot/count"`: calculates the maximum number of tasks available for auto kill
`"killTask/task/count"`: calculates the number of tasks submitted by auto kill.
2023-08-10 18:36:53 -04:00
23306c4d80
retry when killing s3 based segments ( #14776 )
...
### Description
s3 deleteObjects request sent when killing s3 based segments now being retried, if failure is retry-able.
2023-08-10 14:04:16 -04:00
37ed0f4a17
Bump jclouds.version from 1.9.1 to 2.0.3 ( #14746 )
...
* Updates `org.apache.jclouds:*` from 1.9.1 to 2.0.3
* Pin jclouds to 2.0.x since 2.1.x requires Guava 18+
* replace easymock with mockito
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 06:24:01 -07:00
Vadim Ogievetsky
Abhishek Radhakrishnan
dependabot[bot]
Kashif Faraz
Abhishek Agarwal
317brian
Clint Wylie
Pranav
Peter Marshall
Jill Osborne
YongGang
Nhi Pham
Xavier Léauté
AmatyaAvadhanula
Rishabh Singh
Soumyava
George Shiqi Wu
hqx871
Sergio Ferragut
zachjsh