3d6ef6a90e
This is a preliminary of #46250 making the snapshot delete work by doing all the metadata updates first and then bulk deleting all of the now unreferenced blobs. Before this change, the metadata updates for each shard and subsequent deletion of the blobs that have become unreferenced due to the delete would happen sequentially shard-by-shard parallelising only over all the indices in the snapshot. This change makes it so the all the metadata updates happen in parallel on a shard level first. Once all of the updates of shard-level metadata have finished, all the now unreferenced blobs are deleted in bulk. This has two benefits (outside of making #46250 a smaller change): * We have a lower likelihood of failing to update shard level metadata because it happens with priority and a higher degree of parallelism * Deleting of unreferenced data in the shards should go much faster in many cases (rolling indices, large number of indices with many unchanged shards) as well because a number of small bulk deletions (just two blobs for `index-N` and `snap-` for each unchanged shard) are grouped into larger bulk deletes of `100-1000` blobs depending on Cloud provider (even though the final bulk deletes are happening sequentially this should be much faster in almost all cases as you'd parallelism of 50 (GCS) to 500 (S3) snapshot threads to achieve the same delete rates when deleting from unchanged shards). |
||
|---|---|---|
| .. | ||
| licenses | ||
| src | ||
| build.gradle | ||