OpenSearch/qa
Armin Braun d456f7870a
Deduplicate Index Metadata in BlobStore (#50278) (#59514)
This PR introduces two new fields in to `RepositoryData` (index-N) to track the blob name of `IndexMetaData` blobs and their content via setting generations and uuids. This is used to deduplicate the `IndexMetaData` blobs (`meta-{uuid}.dat` in the indices folders under `/indices` so that new metadata for an index is only written to the repository during a snapshot if that same metadata can't be found in another snapshot.
This saves one write per index in the common case of unchanged metadata thus saving cost and making snapshot finalization drastically faster if many indices are being snapshotted at the same time.

The implementation is mostly analogous to that for shard generations in #46250 and piggy backs on the BwC mechanism introduced in that PR (which means this PR needs adjustments if it doesn't go into `7.6`).

Relates to #45736 as it improves the efficiency of snapshotting unchanged indices
Relates to #49800 as it has the potential of loading the index metadata for multiple snapshots of the same index concurrently much more efficient speeding up future concurrent snapshot delete
2020-07-14 22:18:42 +02:00
..
ccs-unavailable-clusters Remove deprecated usage of testCompile configuration (#57921) (#58083) 2020-06-14 22:30:44 +02:00
die-with-dignity RestHandlers declare handled routes (#52123) 2020-02-09 22:48:32 -07:00
evil-tests Remove deprecated usage of testCompile configuration (#57921) (#58083) 2020-06-14 22:30:44 +02:00
full-cluster-restart Make data streams a basic licensed feature. (#59392) 2020-07-13 17:26:46 +02:00
logging-config ThreadPool and ThreadContext are not closeable (#43249) (#49273) 2019-11-19 13:15:16 -07:00
mixed-cluster 7.x - Create plugin for yamlTest task (#56841) (#59090) 2020-07-06 14:16:26 -05:00
multi-cluster-search Make data streams a basic licensed feature. (#59392) 2020-07-13 17:26:46 +02:00
no-bootstrap-tests Prevent unexpected native controller output hanging the process (#56685) 2020-05-13 22:57:00 +01:00
os Use consistent system jdk in 7.x packaging tests (#59257) 2020-07-08 15:23:59 -07:00
remote-clusters Scripting: Deprecate general cache settings (#55753) (#58283) 2020-06-18 11:54:23 -06:00
repository-multi-version Deduplicate Index Metadata in BlobStore (#50278) (#59514) 2020-07-14 22:18:42 +02:00
rolling-upgrade Allow warnings about 'global' template in upgrade tests (#59242) 2020-07-08 13:40:55 -06:00
smoke-test-client Replace compile configuration usage with api (7.x backport) (#58721) 2020-06-30 15:57:41 +02:00
smoke-test-http Separate coordinating and primary bytes in stats (#59487) 2020-07-14 12:37:06 -06:00
smoke-test-ingest-disabled 7.x - Create plugin for yamlTest task (#56841) (#59090) 2020-07-06 14:16:26 -05:00
smoke-test-ingest-with-all-dependencies 7.x - Create plugin for yamlTest task (#56841) (#59090) 2020-07-06 14:16:26 -05:00
smoke-test-multinode 7.x - Create plugin for yamlTest task (#56841) (#59090) 2020-07-06 14:16:26 -05:00
smoke-test-plugins 7.x - Create plugin for yamlTest task (#56841) (#59090) 2020-07-06 14:16:26 -05:00
translog-policy Omit translog bwc test before 6.3.0 with default distro (#57266) 2020-05-28 12:38:30 -07:00
unconfigured-node-name Apply 2-space indent to all gradle scripts (#49071) 2019-11-14 11:01:23 +00:00
verify-version-constants Move bwcVersions extension property to BuildParams (back port) (#56381) 2020-05-11 09:39:13 +02:00
wildfly Replace compile configuration usage with api (7.x backport) (#58721) 2020-06-30 15:57:41 +02:00
build.gradle Apply 2-space indent to all gradle scripts (#49071) 2019-11-14 11:01:23 +00:00