mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-03-09 14:34:43 +00:00
0b8b147a74
This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage. The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot. The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly. Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments in Azure Storage. The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the blobs can be performed. Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99