druid/extensions-core
TSFenwick 9a9038c7ae
Speed up kill tasks by deleting segments in batch (#14131)
* allow for batched delete of segments instead of deleting segment data one by one

create new batchdelete method in datasegment killer that has default functionality
of iterating through all segments and calling delete on them. This will enable
a slow rollout of other deepstorage implementations to move to a batched delete
on their own time

* cleanup batchdelete segments

* batch delete with the omni data deleter

cleaned up code
just need to add tests and docs for this functionality

* update java doc to explain how it will try to use batch if function is overwritten

* rename killBatch to kill
add unit tests

* add omniDataSegmentKillerTest for deleting multiple segments at a time. fix checkstyle

* explain test peculiarity better

* clean up batch kill in s3.

* remove unused return value. cleanup comments and fix checkstyle

* default to batch delete. more specific java docs. list segments that couldn't be deleted
if there was a client error or server error

* simplify error handling

* add tests where an exception is thrown when killing multiple s3 segments

* add test for failing to delete two calls with the s3 client

* fix javadoc for kill(List<DataSegment> segments) clean up tests remove feature flag

* fix typo in javadocs

* fix test failure

* fix checkstyle and improve tests

* fix intellij inspections issues

* address comments, make delete multiple segments not assume same bucket

* fix test errors

* better grammar and punctuation. fix test. and better logging for exception

* remove unused code

* avoid extra arraylist instantiation

* fix broken test

* fix broken test

* fix tests to use assert.throws
2023-07-27 15:34:44 -07:00
..
avro-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
azure-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
datasketches add equality, null, and range filter (#14542) 2023-07-18 12:15:22 -07:00
druid-aws-rds-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
druid-basic-security Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
druid-bloom-filter add equality, null, and range filter (#14542) 2023-07-18 12:15:22 -07:00
druid-catalog Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
druid-kerberos Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
druid-pac4j Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
druid-ranger-security Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
ec2-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
google-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
hdfs-storage Clean up the core API required for Iceberg extension (#14614) 2023-07-21 13:01:33 +05:30
histogram add equality, null, and range filter (#14542) 2023-07-18 12:15:22 -07:00
kafka-extraction-namespace Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
kafka-indexing-service Add ingest/input/bytes metric and Kafka consumer metrics. (#14582) 2023-07-20 10:56:22 +08:00
kinesis-indexing-service Add ingest/input/bytes metric and Kafka consumer metrics. (#14582) 2023-07-20 10:56:22 +08:00
kubernetes-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
lookups-cached-global Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
lookups-cached-single Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
multi-stage-query Merge core CoordinatorClient with MSQ CoordinatorServiceClient. (#14652) 2023-07-27 13:23:37 -07:00
mysql-metadata-storage Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
orc-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
parquet-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
postgresql-metadata-storage Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
protobuf-extensions Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
s3-extensions Speed up kill tasks by deleting segments in batch (#14131) 2023-07-27 15:34:44 -07:00
simple-client-sslcontext Prepare master branch for next release, 28.0.0 (#14595) 2023-07-18 09:22:30 +05:30
stats add equality, null, and range filter (#14542) 2023-07-18 12:15:22 -07:00
testing-tools add equality, null, and range filter (#14542) 2023-07-18 12:15:22 -07:00