mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-05 20:48:22 +00:00
e907b7c11e
We can be better at checking `buffer_size` and `chunk_size` for S3 repositories. For example, we know that: * `buffer_size` should be more than `5mb` * `chunk_size` should be no more than `5tb` * `buffer_size` should be lower than `chunk_size` Otherwise, setting `buffer_size` is useless. For the record: `chunk_size` is a Snapshot setting whatever the implementation is. `buffer_size` is an S3 implementation setting. Let say that you are snapshotting a 500mb file. If you set `chunk_size` to `200mb`, then Snapshot service will call S3 repository to snapshot 3 files with the following sizes: * `200mb` * `200mb` * `100mb` If you set `buffer_size` to `100mb` (AWS maximum size recommendation), the first file of `200mb` will be uploaded on S3 using the multipart feature in 2 chunks and the workflow is basically the following: * create the multipart request and get back an `id` from AWS S3 platform * upload part1: `100mb` * upload part2: `100mb` * "commit" the full upload using the `id`. Closes #17244.