Add remote store backpressure and API documentation (#4215)

* Add remote store backpressure and API documentation

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Reworded experimental warning

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Rewording for clarity

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Revert back to index name

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add table intro sentence

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Implemented doc review comments

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Update _tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Add index pattern

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

---------

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>

_tuning-your-cluster/availability-and-recovery/index.md

@@ -1,9 +1,11 @@
 ---
 layout: default
-title: Availability and Recovery
+title: Availability and recovery
 nav_order: 20
 has_children: true
 has_toc: true
 ---
 
-The following OpenSearch features help ensure consistent uptime so that your cluster can complete and scale based on your use case, as well as creating snapshots. 
+# Availability and recovery
+
+The following OpenSearch features help ensure reliability and consistent uptime so that your cluster can complete and scale based on your use case.

_tuning-your-cluster/availability-and-recovery/remote-store/index.md

@@ -2,14 +2,16 @@
 layout: default
 title: Remote-backed storage
 nav_order: 40
-parent: Availability and Recovery
+has_children: true
+parent: Availability and recovery
 redirect_from: 
   - /opensearch/remote/
+  - /tuning-your-cluster/availability-and-recovery/remote/
 ---
 
 # Remote-backed storage
 
-Remote-backed storage is an experimental feature. Therefore, we do not recommend the use of remote-backed storage in a production environment. For updates on the progress of remote-backed storage, or if you want leave feedback that could help improve the feature, refer to the issue on [GitHub](https://github.com/opensearch-project/OpenSearch/issues/1968).
+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
 {: .warning}
 
 Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to remote storage. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information.

_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md

@@ -0,0 +1,48 @@
+---
+layout: default
+title: Remote segment backpressure
+nav_order: 10
+parent: Remote-backed storage
+grand_parent: Availability and recovery
+---
+
+# Remote segment backpressure
+
+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
+{: .warning}
+
+Remote segment backpressure is a shard-level rejection mechanism that dynamically rejects indexing requests when the remote segment store falls behind the local committed segments on the primary shard. With remote segment backpressure, you can prevent the lag between the remote store and the local primary store. The lag can be caused by slow or failed remote store interaction, remote store throttling, long garbage collection pauses, or high CPU utilization.
+
+## Thresholds
+
+Remote segment backpressure is activated if any of the following thresholds is breached:
+
+- **Consecutive failure**: The backpressure is activated if there are _N_ or more consecutive failures. The value of _N_ is configurable in `remote_store.segment.pressure.consecutive_failures.limit`.
+- **Bytes lag**:  The bytes lag is computed by adding the size of all files that are present in local committed segments but not in the remote store. The backpressure is activated if the bytes lag is greater than _K_ multiplied by the moving average of the size (in bytes) of files uploaded after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.bytes_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_bytes_moving_average_window_size`.  
+- **Time lag**: The time lag is computed by comparing the timestamps of the most recent local refresh and the most recent remote store segment upload. The backpressure is activated if the time lag is greater than _K_ multiplied by the moving average of the time taken to upload new segments and metadata file after each refresh. The variance factor _K_ is configurable in `remote_store.segment.pressure.time_lag.variance_factor`. The moving window size is configurable in `remote_store.segment.pressure.upload_time_moving_average_window_size`.  
+
+## Handling segment merges 
+
+At every segment merge, a corresponding refresh is initiated. Because this refresh has new merged segments, the bytes lag instantly spikes. To compensate for this spike, the bytes lag and time lag are evaluated only if the remote store is behind the local primary store by more than one refresh. However, backpressure induced by consecutive failures activates regardless of refresh lag (the number of refreshes by which the remote store is lagging behind the local store).
+
+## Remote segment backpressure settings
+
+Remote segment backpressure adds several settings to the standard OpenSearch cluster settings. The settings are dynamic, so you can change the default backpressure behavior without restarting your cluster. 
+
+The following table lists the settings used for activating backpressure. For threshold calculation, see [Thresholds](#thresholds).
+
+|Setting	|Data type	|Description	|
+|:---	|:---	|:---	|
+|`remote_store.segment.pressure.enabled`	|Boolean	| If `true`, enables remote segment backpressure. Default is `false`. |
+|`remote_store.segment.pressure.consecutive_failures.limit`	|Integer |The minimum consecutive failure count for activating remote segment backpressure. Default is `5`.	|
+|`remote_store.segment.pressure.bytes_lag.variance_factor`	|Float | The variance factor that is used together with the moving average to calculate the dynamic bytes lag threshold for activating remote segment backpressure. Default is `10`.	|
+|`remote_store.segment.pressure.upload_bytes_moving_average_window_size`	|Integer	|The moving average window size that is used to calculate the dynamic bytes lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/). Default is `20`.	|
+|`remote_store.segment.pressure.time_lag.variance_factor`	|Float 	|The variance factor that is used together with the moving average to calculate the dynamic time lag threshold for activating remote segment backpressure. Default is `10`.	|
+|`remote_store.segment.pressure.upload_time_moving_average_window_size`	|Integer	|The moving average window size that is used to calculate the dynamic time lag threshold for activating remote segment backpressure. The moving average is also exposed through the [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/).	Default is `20`.|
+
+The following table lists the settings used for statistics.
+
+|Setting	|Data type	|Description	|
+|:---	|:---	|:---	|
+|`remote_store.segment.pressure.upload_bytes_per_sec_moving_average_window_size`	|Integer |The moving average window size, which is exposed through [Remote Store Stats API]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/).	Default is `20`.|
+

_tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api.md

@@ -0,0 +1,207 @@
+---
+layout: default
+title: Remote Store Stats API 
+nav_order: 20
+parent: Remote-backed storage
+grand_parent: Availability and recovery
+---
+
+# Remote Store Stats API
+Introduced 2.8
+{: .label .label-purple }
+
+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
+{: .warning}
+
+Use the Remote Store Stats API to monitor shard-level remote store performance.
+
+## Path and HTTP methods
+
+```json
+GET _remotestore/stats/<index_name>
+GET _remotestore/stats/<index_name>/<shard_id>
+```
+
+## Path parameters
+
+The following table lists the available path parameters. All path parameters are optional.
+
+Parameter | Type | Description
+:--- | :--- | :---
+`index_name` | String | The index name or index pattern.
+`shard_id` | String | The shard ID.
+
+## Remote store stats for an index
+
+Use the following API to get remote store statistics for all shards of an index. 
+
+#### Example request
+
+```json
+GET _remotestore/stats/<index_name>
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+<details open markdown="block">
+  <summary>
+    Response
+  </summary>
+  {: .text-delta }
+
+```json
+{
+  "_shards": {
+    "total": 2,
+    "successful": 2,
+    "failed": 0
+  },
+  "stats": [
+    {
+      "shard_id": "[so][1]",
+      "refresh_time_lag_in_millis": 0,
+      "refresh_lag": 0,
+      "bytes_lag": 0,
+      "backpressure_rejection_count": 0,
+      "consecutive_failure_count": 0,
+      "total_remote_refresh": {
+        "started": 56,
+        "succeeded": 56,
+        "failed": 0
+      },
+      "total_uploads_in_bytes": {
+        "started": 1524670599,
+        "succeeded": 1524670599,
+        "failed": 0
+      },
+      "remote_refresh_size_in_bytes": {
+        "last_successful": 12711179,
+        "moving_avg": 30726409
+      },
+      "upload_latency_in_bytes_per_sec": {
+        "moving_avg": 25276841.3
+      },
+      "remote_refresh_latency_in_millis": {
+        "moving_avg": 964.7
+      }
+    },
+    {
+      "shard_id": "[so][0]",
+      "refresh_time_lag_in_millis": 5727,
+      "refresh_lag": 1,
+      "bytes_lag": 0,
+      "backpressure_rejection_count": 0,
+      "consecutive_failure_count": 0,
+      "total_remote_refresh": {
+        "started": 57,
+        "succeeded": 56,
+        "failed": 0
+      },
+      "total_uploads_in_bytes": {
+        "started": 1568138701,
+        "succeeded": 1568138701,
+        "failed": 0
+      },
+      "remote_refresh_size_in_bytes": {
+        "last_successful": 12705142,
+        "moving_avg": 32766119.75
+      },
+      "upload_latency_in_bytes_per_sec": {
+        "moving_avg": 25523682.95
+      },
+      "remote_refresh_latency_in_millis": {
+        "moving_avg": 990.55
+      }
+    }
+  ]
+}
+```
+</details>
+
+### Response fields
+
+The following table lists the available response fields. 
+
+|Field	|Description	|
+|:---	|:---	|
+|`refresh_time_lag_in_millis`	|The time (in milliseconds) the remote refresh is behind the local refresh.	|
+|`refresh_lag`	| The number of refreshes by which the remote store is lagging behind the local store.	|
+|`bytes_lag`	| The bytes lag between the remote and local store.	|
+|`backpressure_rejection_count`	| The total number of write rejections made because of remote store backpressure.	|
+|`consecutive_failure_count`	| The number of consecutive remote refresh failures since the last success.	|
+|`total_remote_refresh`	|The total number of remote refreshes.	|
+|`total_uploads_in_bytes`	| The total number of bytes in all uploads to the remote store. |
+|`remote_refresh_size_in_bytes.last_successful`	|The size of data uploaded in the last successful refresh.	|
+|`remote_refresh_size_in_bytes.moving_avg`	|The average size of data (in bytes) uploaded in the last _N_ refreshes. _N_ is defined in `remote_store.segment.pressure.upload_bytes_moving_average_window_size`. For details, see [Remote segment backpressure]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure/).	|
+|`upload_latency_in_bytes_per_sec.moving_avg`	|The average speed of remote store uploads (in bytes per second) for the last _N_ uploads. _N_ is defined in `remote_store.segment.pressure.upload_bytes_per_sec_moving_average_window_size`. For details, see [Remote segment backpressure]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure/).		|
+|`remote_refresh_latency_in_millis.moving_avg`	|The average time taken by a single remote refresh during the last _N_ remote refreshes. _N_ is defined in `remote_store.segment.pressure.upload_time_moving_average_window_size`. For details, see [Remote segment backpressure]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure/).	|
+
+## Remote store stats for a single shard
+
+Use the following API to get remote store statistics for a single shard.
+
+#### Example request
+
+```json
+GET _remotestore/stats/<index_name>/<shard_id>
+```
+{% include copy-curl.html %}
+
+#### Example response
+
+<details open markdown="block">
+  <summary>
+    Response
+  </summary>
+  {: .text-delta }
+
+```json
+{
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "failed": 0
+  },
+  "stats": [
+    {
+      "shard_id": "[so][0]",
+      "refresh_time_lag_in_millis": 5727,
+      "refresh_lag": 1,
+      "bytes_lag": 0,
+      "backpressure_rejection_count": 0,
+      "consecutive_failure_count": 0,
+      "total_remote_refresh": {
+        "started": 57,
+        "succeeded": 56,
+        "failed": 0
+      },
+      "total_uploads_in_bytes": {
+        "started": 1568138701,
+        "succeeded": 1568138701,
+        "failed": 0
+      },
+      "remote_refresh_size_in_bytes": {
+        "last_successful": 12705142,
+        "moving_avg": 32766119.75
+      },
+      "upload_latency_in_bytes_per_sec": {
+        "moving_avg": 25523682.95
+      },
+      "remote_refresh_latency_in_millis": {
+        "moving_avg": 990.55
+      }
+    }
+  ]
+}
+```
+</details>
+
+### Remote store stats for a local shard
+
+Provide the `local` query parameter set to `true` to only fetch the shards present on the node that is serving the request:
+
+```json
+GET _remotestore/stats/<index_name>?local=true
+```
+{% include copy-curl.html %}

_tuning-your-cluster/availability-and-recovery/search-backpressure.md

@@ -3,7 +3,7 @@ layout: default
 title: Search backpressure
 nav_order: 60
 has_children: false
-parent: Availability and Recovery
+parent: Availability and recovery
 redirect_from: 
   - /opensearch/search-backpressure/
 ---

_tuning-your-cluster/availability-and-recovery/segment-replication/backpressure.md

@@ -4,7 +4,7 @@ title: Segment replication backpressure
 nav_order: 75
 parent: Segment replication
 has_children: false
-grand_parent: Availability and Recovery
+grand_parent: Availability and recovery
 ---
 
 # Segment replication backpressure

_tuning-your-cluster/availability-and-recovery/segment-replication/index.md

@@ -3,7 +3,7 @@ layout: default
 title: Segment replication 
 nav_order: 70
 has_children: true
-parent: Availability and Recovery
+parent: Availability and recovery
 datatable: true
 redirect_from:
   - /opensearch/segment-replication/

_tuning-your-cluster/availability-and-recovery/shard-indexing-backpressure.md

@@ -3,7 +3,7 @@ layout: default
 title: Shard indexing backpressure
 nav_order: 62
 has_children: true
-parent: Availability and Recovery
+parent: Availability and recovery
 redirect_from: 
   - /opensearch/shard-indexing-backpressure/
 ---

_tuning-your-cluster/availability-and-recovery/shard-indexing-settings.md

@@ -3,7 +3,7 @@ layout: default
 title: Settings
 parent: Shard indexing backpressure
 nav_order: 50
-grand_parent: Availability and Recovery
+grand_parent: Availability and recovery
 redirect_from: 
   - /opensearch/shard-indexing-settings/
 ---

_tuning-your-cluster/availability-and-recovery/stats-api.md

@@ -3,7 +3,7 @@ layout: default
 title: Stats API
 parent: Shard indexing backpressure
 nav_order: 2
-grand_parent: Availability and Recovery
+grand_parent: Availability and recovery
 has_children: false
 redirect_from: 
   - /opensearch/stats-api/