diff --git a/_benchmark/quickstart.md b/_benchmark/quickstart.md index d4671da3..52415cb6 100644 --- a/_benchmark/quickstart.md +++ b/_benchmark/quickstart.md @@ -138,9 +138,16 @@ opensearch-benchmark execute-test --pipeline=benchmark-only --workload=percolato When the `execute_test` command runs, all tasks and operations in the `percolator` workload run sequentially. +### Validating the test + +After an OpenSearch Benchmark test runs, take the following steps to verify that it has run properly: + +- Note the number of documents in the OpenSearch or OpenSearch Dashboards index that you plan to run the benchmark against. +- In the results returned by OpenSearch Benchmark, compare the `workload.json` file for your specific workload and verify that the document count matches the number of documents. For example, based on the [percolator](https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/percolator/workload.json#L19) `workload.json` file, you should expect to see `2000000` documents in your cluster. + ### Understanding the results -Benchmark returns the following response once the benchmark completes: +OpenSearch Benchmark returns the following response once the benchmark completes: ```bash ------------------------------------------------------ @@ -248,6 +255,8 @@ Each task run by the `percolator` workload represents a specific OpenSearch API * **Service Time:** The amount of time, excluding wait time, taken for the request and the response to be sent and received by Benchmark. * **Error Rate:** The percentage of operations run during the task that were not successful or returned a 200 error code. +For more details about how the summary report is generated, see [Summary report]({{site.url}}{{site.baseurl}}/benchmark/reference/summary-report/). + ## Running OpenSearch Benchmark on your own cluster diff --git a/_benchmark/reference/commands/command-flags.md b/_benchmark/reference/commands/command-flags.md index 7ba72ebd..b857ce12 100644 --- a/_benchmark/reference/commands/command-flags.md +++ b/_benchmark/reference/commands/command-flags.md @@ -43,7 +43,7 @@ Defines which variables to inject into the workload. Variables injected must be ## test-procedure -Defines a test procedure to use. You can find a list of test procedures by using `opensearch-benchmark list test-procedures`. +Defines the test procedures to use with each workload. You can find a list of test procedures that the workload supports by specifying the workload in the `info` command, for example, `opensearch-benchmark info --workload=`. To look up information on a specific test procedure, use the command `opensearch-benchmark info --workload= --test-procedure=`. ## test-execution-id diff --git a/_benchmark/reference/commands/execute-test.md b/_benchmark/reference/commands/execute-test.md index e81c4e28..4b551988 100644 --- a/_benchmark/reference/commands/execute-test.md +++ b/_benchmark/reference/commands/execute-test.md @@ -151,7 +151,7 @@ The following options determine which workload is used to run the test: The following options define what test procedures the test uses and which operations are contained inside the procedure: - `--test-execution-id`: Defines a unique ID for this test run. -- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test-procedures`. +Defines the test procedures to use with each workload. You can find a list of test procedures that the workload supports by specifying the workload in the `info` command, for example, `opensearch-benchmark info --workload=`. To look up information on a specific test procedure, use the command `opensearch-benchmark info --workload= --test-procedure=`. - `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run. - `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run. - `--enable-assertions`: Enables assertion checks for tasks. Default is `false`. diff --git a/_benchmark/reference/summary-report.md b/_benchmark/reference/summary-report.md new file mode 100644 index 00000000..d5396cd3 --- /dev/null +++ b/_benchmark/reference/summary-report.md @@ -0,0 +1,260 @@ +--- +layout: default +title: Summary report +nav_order: 40 +parent: Metrics +--- + +# Summary report + +At the end of each run, OpenSearch Benchmark shows a summary report based on the metric keys defined in the workload. This page gives details on each line of the summary report and that line's associated metric key. + +## Cumulative indexing time of primary shards + +**Corresponding metrics key**: `indexing_total_time` + +The cumulative time used for indexing as reported by the Index Stats API. Note that this is not wall-clock time, for example, if M indexing threads ran for N minutes, report M * N minutes, not N minutes. + +## Cumulative indexing time across primary shards + +**Corresponding metrics key:** `indexing_total_time` (property: `per-shard`) + +The minimum, median, and maximum cumulative time used for indexing across primary shards as reported by the Index Stats API. + +## Cumulative indexing throttle time of primary shards + +**Corresponding metrics key:** `indexing_throttle_time` + +The cumulative time that the indexing has been throttled as reported by the Index Stats API. Note that this is not wall-clock time, for example, if M indexing threads ran for N minutes, report M * N minutes, not N minutes. + + +## Cumulative indexing throttle time across primary shards + +**Corresponding metrics key:** `indexing_throttle_time` (property: `per-shard`) + +The minimum, median, and maximum cumulative time used that indexing has been throttled across primary shards as reported by the Index Stats API. + + +## Cumulative merge time of primary shards + +**Corresponding metrics key:** `merges_total_time` + +The cumulative runtime of merges of primary shards, as reported by the index stats API. Note that this is not wall-clock time. + +## Cumulative merge count of primary shards + +**Corresponding metrics key:** `merges_total_count` + +The cumulative number of merges of primary shards, as reported by the Index Stats API under `_all/primaries`. + + +## Cumulative merge time across primary shards + +**Corresponding metrics key:** `merges_total_time` (property: `per-shard`) + +The minimum, median, and maximum cumulative time of merges across primary shards as reported by the Index Stats API. + + +## Cumulative refresh time of primary shards + +**Corresponding metrics key**: `refresh_total_time` + +The cumulative time used for index refresh of primary shards as reported by the Index Stats API. Note that this is not wall-clock time. + +## Cumulative refresh count of primary shards + +**Corresponding metrics key:** `refresh_total_count` + +The cumulative number of refreshes of primary shards as reported by the Index Stats API under `_all/primaries`. + +## Cumulative refresh time across primary shards + +**Corresponding metrics key:** `refresh_total_time` (property: `per-shard`) + +The minimum, median, and maximum cumulative time for index refresh across primary shards as reported by the Index Stats API. + +## Cumulative flush time of primary shards + +**Corresponding metrics key:** `flush_total_time` + +The cumulative time used for index flush of primary shards as reported by the Index Stats API. Note that this is not wall-clock time. + +## Cumulative flush count of primary shards + +**Corresponding metrics key**: `flush_total_count` + +The cumulative number of flushes of primary shards as reported by the Index Stats API under `_all/primaries`. + + +## Cumulative flush time across primary shards + +**Corresponding metrics key:** `flush_total_time` (property: `per-shard`) + +The minimum, median, and maximum time for index flush across primary shards as reported by the Index Stats API. + +## Cumulative merge throttle time of primary shards + +**Corresponding metrics key:** `merges_total_throttled_time` + +The cumulative time within merges that have been throttled as reported by the Index Stats API. Note that this is not wall-clock time. + +## Cumulative merge throttle time across primary shards + +Corresponding metrics key: `merges_total_throttled_time` (property: `per-shard`) + +The minimum, median, and maximum cumulative time that merges have been throttled across primary shards as reported by the Index Stats API. + +## ML processing time + +Corresponding metrics key: `ml_processing_time` + +The minimum, mean, median, and maximum time in milliseconds that a machine learning (ML) job has spent processing a single bucket. + + +## Total young gen GC time + +**Corresponding metrics key**: `node_total_young_gen_gc_time` + +The total runtime of the young generation (gen) garbage collector (GC) across the whole cluster as reported by the Node Stats API. + +## Total young gen GC count + +**Corresponding metrics key:** `node_total_young_gen_gc_count` + +The total number of young gen GCs across the whole cluster as reported by the Node Stats API. + + +## Total old gen GC time + +**Corresponding metrics key:** `node_total_old_gen_gc_time` + +The total runtime of the old gen GC across the whole cluster as reported by the Node Stats API. + +## Total old gen GC count + +**Corresponding metrics key:** `node_total_old_gen_gc_count` + +The total number of old gen GCs across the whole cluster as reported by the Node Stats API. + +## Total ZGC cycles GC time + +**Corresponding metrics key**: `node_total_zgc_cycles_gc_count` + +The total number of garbage collections performed by the Z garbage collector (ZGC) across the whole cluster as reported by the Node Stats API. + +## Total ZGC pauses GC time + +**Corresponding metrics key**: `node_total_zgc_pauses_gc_time` + +The total time spent in stop-the-world pauses by the ZGC across the whole cluster as reported by the Node Stats API. + + +## Total ZGC pauses GC count + +**Corresponding metrics key**: `node_total_zgc_pauses_gc_count` + +The total number of stop-the-world pauses performed by the ZGC across the whole cluster as reported by the Node Stats API. + + +## Store size + +**Corresponding metrics key**: `store_size_in_bytes` + +The index size in bytes (excluding the translog) as reported by the Index Stats API. + +## Translog size + +**Corresponding metrics key**: `translog_size_in_bytes` + +The translog size in bytes as reported by the Index Stats API. + +## Heap used for X + +**Corresponding metrics keys**: `segments_*_in_bytes` + +The number of bytes used for the corresponding item as reported by the Index Stats API. The item may be any of the following: + +- Doc values +- Terms +- Norms +- Points +- Stored fields + + +## Segments count + +**Corresponding metrics key**: `segments_count` + +The total number of segments as reported by the Index Stats API. + + +## Total ingest pipeline count + +**Corresponding metrics key**: `ingest_pipeline_cluster_count` + +The total number of documents ingested by all nodes within the cluster over the race duration. + +## Total ingest pipeline time + +**Corresponding metrics key**: `ingest_pipeline_cluster_time` + +The total time in milliseconds spent preprocessing ingest documents by all nodes within the cluster over the race duration. + + +## Total ingest pipeline failed + +**Corresponding metrics key**: `ingest_pipeline_cluster_failed` + +The total number of failed ingest operations by all nodes within the cluster over the race duration. + + +## Throughput + +**Corresponding metrics key**: `throughput` + +Reports the minimum, mean, median, and maximum throughput for each task. + +The number of operations that OpenSearch can perform within a certain time period per second. The report includes the minimum, mean, median, and maximum throughput for each task. + + +## Latency + +**Corresponding metrics key**: `latency` + +The time period between submission of a request and receiving the complete response. It includes the wait time the request spends waiting before it is processed by OpenSearch. OpenSearch reports several percentile numbers for each task. Which percentiles are shown depends on how many requests OpenSearch can capture during the latency period. + + +## Service time + +**Corresponding metrics key**: `service_time` + +The time period between sending a request and receiving the corresponding response. It does not include waiting time. While many load testing tools refer to this metric as _latency_, it is not the same. OpenSearch reports several percentile numbers for each task. Which percentiles are shown depends on how many requests OpenSearch can capture during the latency period. + + + +## Processing time + +Processing time is only reported if the setting `output.processingtime` is set to `true` in the OpenSearch Benchmark configuration file. +{: note.} + +**Corresponding metrics key**: `processing_time` + + +The time period between start of request processing and retrieval of the complete response. Unlike `service_time`, this metric includes OpenSearch’s client-side processing overhead. The larger the difference between `service_time` and `processing_time`, the higher the overhead in the client. Depending on your processing goals, this can point to a potential client-side bottleneck that requires investigation. + + +## Error rate + +Corresponding metrics key: `service_time`. Each `service_time` record has a `meta.success` flag. + +The ratio of erroneous responses relative to the total number of responses. Any exception thrown by the Python OpenSearch client is considered erroneous, for example, HTTP response codes 4xx, 5xx, or network errors (network unreachable). You can investigate the root cause by inspecting OpenSearch and OpenSearch Benchmark logs and rerunning the benchmark. + + +## Disk usage + +**Corresponding metrics keys**: `disk_usage_total` +**Metric metadata**: `index` and `field` + +The total number of bytes that a single field uses on disk. Recorded for each field returned by the Disk Usage API even if the total is `0`. + + diff --git a/_benchmark/reference/telemetry.md b/_benchmark/reference/telemetry.md new file mode 100644 index 00000000..dca4ee50 --- /dev/null +++ b/_benchmark/reference/telemetry.md @@ -0,0 +1,184 @@ +--- +layout: default +title: Telemetry devices +nav_order: 45 +has_children: true +--- + +# Telemetry devices + +Telemetry devices give you additional insights on benchmark results. To view a list of the available telemetry devices, use the command `opensearch-benchmark list telemetry`. + +All telemetry devices with a `--stats` can be used with clusters not provisioned by OpenSearch Benchmark. These devices are referred to as **Runtime level telemetry devices**. Alternatively, **Setup level telemetry devices** encompass devices that can only be used when OpenSearch Benchmark provisions a cluster. + +This page lists the telemetry devices supported by OpenSearch Benchmark. + +## jfr + +The `jfr` telemetry device enables the [Java Flight Recorder (JFR)](https://docs.oracle.com/javacomponents/jmc-5-5/jfr-runtime-guide/index.html) on the benchmark candidate. Up to Java Development Kit (JDK) 11, JFR ships only with Oracle JDK. OpenSearch Benchmark assumes that Oracle JDK is used for benchmarking. If you run benchmarks on JDK 11 or later, [JFR](https://jdk.java.net/jmc/) is also available on OpenJDK. + +To enable `jfr`, invoke **Workload** with the command `opensearch-benchmark workload --workload=pmc --telemetry jfr`. Then `jfr` will write a flight recording file that can be opened in Java Mission Control. OpenSearch Benchmark prints the location of the flight recording file on the command line. + +The `jfr` devices support the following parameters: + + +- `recording-template`: The name of a custom flight recording template. It is your responsibility to correctly install these recording templates on each target machine. If none is specified, the default recording JFR template is used. +- `jfr-delay`: The length of time to wait before starting to record. Optional. +- `jfr-duration`: The length of time to record. Optional. + +## jit + +The `jit` telemetry device enables JIT compiler logs for the benchmark candidate. If the HotSpot disassembler library is available, the logs will contain the disassembled JIT compiler output, which can be used for low-level analysis. + +## gc + +The `gc` telemetry device enables garbage collector (GC) logs for the benchmark candidate. You can use tools such as GCViewer to analyze the GC logs. + +If the runtime JDK is Java 9 or higher, you can specify the `gc-log-config` parameter. The GC logging configuration consists of a list of tags and levels, such as the default value `gc*=info,safepoint=info,age*=trace`. Run `java -Xlog:help` to view a list of available levels and tags. + +## headdump + +The `heapdump` telemetry device captures a heap dump after a benchmark has finished and right before the node is shut down. + +## node-stats + +The `node-stats` telemetry device regularly calls the cluster [Node Stats API]({{site.url}}{{site.baseurl}}/api-reference/nodes-apis/nodes-stats/) and records metrics from the following stats and their associated keys: + +- Index stats: `indices` +- Thread pool stats: `thread_pool` +- JVM buffer pool stats: `jvm.buffer_pools` +- JVM gc stats: `jvm.gc` +- OS mem stats: `os.mem` +- OS cgroup stats: `os.cgroup` +- JVM mem stats: `jvm.mem` +- Circuit breaker stats: `breakers` +- Network-related stats: `transport` +- Process CPU stats: `process.cpu` + +The `node-stats` device supports the following parameters: + +- `node-stats-sample-interval`: A positive number greater than zero denoting the sampling interval in seconds. Default is `1`. +- `node-stats-include-indices`: A Boolean indicating whether index stats should be included. Default is `false`. +- `node-stats-include-indices-metrics`: A comma-separated string specifying the index stats metrics to include. This is useful, for example, to restrict the collected index stats metrics. Specifying this parameter implicitly enables collection of index stats, so you don’t also need to specify `node-stats-include-indices: true.` For example, `--telemetry-params="node-stats-include-indices-metrics:'docs'"` will collect the docs metrics from the index stats. If you want to use multiple fields, pass a JSON file to `telemetry-params`. Default is `docs,store,indexing,search,merges,query_cache,fielddata,segments,translog,request_cache`. +- `node-stats-include-thread-pools`: A Boolean indicating whether thread pool stats should be included. Default is `true`. +- `node-stats-include-buffer-pools`: A Boolean indicating whether buffer pool stats should be included. Default is `true`. +- `node-stats-include-breakers`: A Boolean indicating whether circuit breaker stats should be included. Default is `true`. +- `node-stats-include-gc`: A Boolean indicating whether JVM GC stats should be included. Default is `true`. +- `node-stats-include-mem`: A Boolean indicating whether both JVM heap and OS mem stats should be included. Default is `true`. +- `node-stats-include-cgroup`: A Boolean to include operating system cgroup stats. Memory stats are omitted since OpenSearch outputs them as string values. Use the `os_mem_*` fields instead. Default is `true`. +- `node-stats-include-network`: A Boolean indicating whether network-related stats should be included. Default is `true`. +- `node-stats-include-process`: A Boolean indicating whether process CPU stats should be included. Default is `true`. +- `node-stats-include-indexing-pressure`: A Boolean indicating whether indexing presser stats should be included. Default is `true`. + +## recovery-stats + +The `recovery-stats` telemetry device regularly calls the [CAT Recovery API]({{site.url}}{{site.baseurl}}/api-reference/cat/cat-recovery/) and records one metrics document per shard. + +This telemetry device supports the following parameters: + +- `searchable-snapshots-stats-indices` A string with the index pattern, or list of index patterns, that searchable snapshots stats should additionally be collected from. If unset, only cluster-level stats will be collected. Default is `None`. +- `searchable-snapshots-stats-sample-interval`: A positive number greater than zero denoting the sampling interval in seconds. Default is `1`. + +## shard-stats + +The `shard-stats` telemetry device regularly calls the cluster [Node Stats API]({{site.url}}{{site.baseurl}}/api-reference/nodes-apis/nodes-stats/) using a `level=shard` cluster parameter and records one metrics document per shard. + +This device supports the `shard-stats-sample-interval` parameter, which defines the sampling interval in seconds. Default is `60`. + +## data-stream-stats + +The `data-stream-stats` telemetry device regularly calls the [Data Stream Stats API]({{site.url}}{{site.baseurl}}/im-plugin/data-streams/#step-2-create-a-data-stream) and records one metrics document for cluster-level stats (`_all`) and one metrics document per data stream. + +The following is an example of recorded documents given two data streams in the cluster: + +```json +{ + "data_streams" : [ + { + "name" : "logs-nginx", + "timestamp_field" : { + "name" : "request_time" + }, + "indices" : [ + { + "index_name" : ".ds-logs-nginx-000001", + "index_uuid" : "-VhmuhrQQ6ipYCmBhn6vLw" + } + ], + "generation" : 1, + "status" : "GREEN", + "template" : "logs-template-nginx" + } + ] +}, +{ + "name": "data-stream-stats", + "data_stream": "my-data-stream-1", + "backing_indices": 1, + "store_size_bytes": 439137, + "maximum_timestamp": 1579936446448 +}, +{ + "name": "data-stream-stats", + "data_stream": "my-data-stream-2", + "backing_indices": 1, + "store_size_bytes": 439199, + "maximum_timestamp": 1579936446448 +} +``` + +This telemetry device supports the `data-stream-stats-sample-interval` parameter, which defines the sampling interval in seconds. Default is `10`. + +## ingest-pipeline-stats + +The `ingest-pipeline-stats` telemetry device makes a call at the beginning and end of the benchmark to the Node Stats API and records the deltas in the form of the following documents: + +- Three results documents for each cluster: `ingest_pipeline_cluster_count`, `ingest_pipeline_cluster_time`, `ingest_pipeline_cluster_failed` +- One metrics document for each node’s respective stats: `ingest_pipeline_node_count`, `ingest_pipeline_node_time`, `ingest_pipeline_node_failed` +- One metrics document for each pipeline’s respective stats: `ingest_pipeline_pipeline_count`, `ingest_pipeline_pipeline_time`, `ingest_pipeline_pipeline_failed` +- One metrics document for each pipeline processor’s respective stats: `ingest_pipeline_processor_count`, `ingest_pipeline_processor_time`, `ingest_pipeline_processor_failed` + + +The following example shows each document record given a single cluster, single node, and single pipeline: + +```json +{ + "name": "ingest_pipeline_cluster_count", + "value": 1001, + "meta": { + "cluster_name": "docker-cluster" + } +}, +{ + "name": "ingest_pipeline_node_count", + "value": 1001, + "meta": { + "cluster_name": "docker-cluster", + "node_name": "node-001" + } +}, +{ + "name": "ingest_pipeline_pipeline_count", + "value": 1001, + "meta": { + "cluster_name": "docker-cluster", + "node_name": "node-001", + "ingest_pipeline": "test-pipeline-1" + } +}, +{ + "name": "ingest_pipeline_processor_count", + "value": 1001, + "meta": { + "cluster_name": "docker-cluster", + "node_name": "node-001", + "ingest_pipeline": "test-pipeline-1", + "processor_name": "uppercase_1", + "type": "uppercase" + } +} +``` + + + + diff --git a/_benchmark/user-guide/creating-custom-workloads.md b/_benchmark/user-guide/creating-osb-workloads.md similarity index 95% rename from _benchmark/user-guide/creating-custom-workloads.md rename to _benchmark/user-guide/creating-osb-workloads.md index 17c2a69c..76c57390 100644 --- a/_benchmark/user-guide/creating-custom-workloads.md +++ b/_benchmark/user-guide/creating-osb-workloads.md @@ -1,14 +1,16 @@ --- layout: default -title: Creating custom workloads +title: Creating OpenSearch Benchmark workloads nav_order: 10 parent: User guide -redirect_from: /benchmark/creating-custom-workloads/ +redirect_from: + - /benchmark/creating-custom-workloads/ + - /benchmark/user-guide/creating-custom-workloads --- # Creating custom workloads -OpenSearch Benchmark includes a set of [workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) that you can use to benchmark data from your cluster. Additionally, if you want to create a workload that is tailored to your own data, you can create a custom workload using one of the following options: +OpenSearch Benchmark (OSB) includes a set of [workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) that you can use to benchmark data from your cluster. Additionally, if you want to create a workload that is tailored to your own data, you can create a custom workload using one of the following options: - [Creating custom workloads](#creating-custom-workloads) - [Creating a workload from an existing cluster](#creating-a-workload-from-an-existing-cluster) @@ -28,14 +30,14 @@ If you already have an OpenSearch cluster with indexed data, use the following s ### Prerequisites -Before creating a custom workload, make sure you have the following prerequisites: +Before creating a custom OSB workload, make sure you have the following prerequisites in place: - An OpenSearch cluster with an index that contains 1000 or more documents. If your cluster's index does not contain at least 1000 documents, the workload can still run tests, however, you cannot run workloads using `--test-mode`. - You must have the correct permissions to access your OpenSearch cluster. For more information about cluster permissions, see [Permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/). ### Customizing the workload -To begin creating a custom workload, use the `opensearch-benchmark create-workload` command. +To begin creating a custom OSB workload, use the `opensearch-benchmark create-workload` command. ``` opensearch-benchmark create-workload \ @@ -77,7 +79,7 @@ Extracting documents for index [movies]... 2000/2000 docs [10 ------------------------------- ``` -As part of workload creation, OpenSearch Benchmark generates the following files. You can access them in the directory specified by the `--output-path` option. +As part of workload creation, OSB generates the following files. You can access them in the directory specified by the `--output-path` option. - `workload.json`: Contains general workload specifications. - `.json`: Contains mappings and settings for the extracted indexes. diff --git a/_benchmark/user-guide/telemetry.md b/_benchmark/user-guide/telemetry.md new file mode 100644 index 00000000..7cc7f6b7 --- /dev/null +++ b/_benchmark/user-guide/telemetry.md @@ -0,0 +1,8 @@ +--- +layout: default +title: Enabling telemetry devices +nav_order: 15 +parent: User guide +--- + +Telemetry results will not appear in the summary report. To visualize telemetry results, ingest the data into OpenSearch and visualize the data in OpenSearch Dashboards. \ No newline at end of file