Add Benchmark command line reference (#4173)
* Add Benchmark command reference Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Add initial Benchmark command line reference Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Address technical feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * More technical feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * More technical feedback Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Fix list description Signed-off-by: Naarcha-AWS <naarcha@amazon.com> * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Naarcha-AWS <naarcha@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>
This commit is contained in:
Naarcha-AWS
GitHub
parent
2e92a215d6
commit
377f4f09ea
|
|
@ -0,0 +1,132 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: compare
|
||||||
|
nav_order: 55
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
# compare
|
||||||
|
|
||||||
|
The `compare` command helps you analyze the difference between two benchmark tests. This can help you analyze the performance impact of changes made from a previous test based on a specific Git revision.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
You can compare two different workload tests using their `TestExecution IDs`. To find a list of tests run from a specific workload, use `opensearch-benchmark list test_executions`. You should receive an output similar to the following:
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
____ _____ __ ____ __ __
|
||||||
|
/ __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__
|
||||||
|
/ / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
|
||||||
|
/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,<
|
||||||
|
\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_|
|
||||||
|
/_/
|
||||||
|
Recent test-executions:
|
||||||
|
|
||||||
|
Recent test_executions:
|
||||||
|
|
||||||
|
TestExecution ID TestExecution Timestamp Workload Workload Parameters TestProcedure ProvisionConfigInstance User Tags workload Revision Provision Config Revision
|
||||||
|
------------------------------------ ------------------------- ---------- --------------------- ------------------- ------------------------- ----------- ------------------- ---------------------------
|
||||||
|
729291a0-ee87-44e5-9b75-cc6d50c89702 20230524T181718Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
f91c33d0-ec93-48e1-975e-37476a5c9fe5 20230524T170134Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
d942b7f9-6506-451d-9dcf-ef502ab3e574 20230524T144827Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
a33845cc-c2e5-4488-a2db-b0670741ff9b 20230523T213145Z geonames append-no-conflicts
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, use `compare` to call a `--baseline` test and a `--contender` test for comparison.
|
||||||
|
|
||||||
|
```
|
||||||
|
opensearch-benchmark compare --baseline=417ed42-6671-9i79-11a1-e367636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7
|
||||||
|
```
|
||||||
|
|
||||||
|
You should receive the following response comparing the final benchmark metrics for both tests:
|
||||||
|
|
||||||
|
```
|
||||||
|
____ _____ __ ____ __ __
|
||||||
|
/ __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__
|
||||||
|
/ / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
|
||||||
|
/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,<
|
||||||
|
\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_|
|
||||||
|
/_/
|
||||||
|
|
||||||
|
Comparing baseline
|
||||||
|
TestExecution ID: 729291a0-ee87-44e5-9b75-cc6d50c89702
|
||||||
|
TestExecution timestamp: 2023-05-24 18:17:18
|
||||||
|
|
||||||
|
with contender
|
||||||
|
TestExecution ID: a33845cc-c2e5-4488-a2db-b0670741ff9b
|
||||||
|
TestExecution timestamp: 2023-05-23 21:31:45
|
||||||
|
|
||||||
|
|
||||||
|
------------------------------------------------------
|
||||||
|
_______ __ _____
|
||||||
|
/ ____(_)___ ____ _/ / / ___/_________ ________
|
||||||
|
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
|
||||||
|
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
|
||||||
|
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
|
||||||
|
------------------------------------------------------
|
||||||
|
Metric Baseline Contender Diff
|
||||||
|
-------------------------------------------------------- ---------- ----------- -----------------
|
||||||
|
Min Indexing Throughput [docs/s] 19501 19118 -383.00000
|
||||||
|
Median Indexing Throughput [docs/s] 20232 19927.5 -304.45833
|
||||||
|
Max Indexing Throughput [docs/s] 21172 20849 -323.00000
|
||||||
|
Total indexing time [min] 55.7989 56.335 +0.53603
|
||||||
|
Total merge time [min] 12.9766 13.3115 +0.33495
|
||||||
|
Total refresh time [min] 5.20067 5.20097 +0.00030
|
||||||
|
Total flush time [min] 0.0648667 0.0681833 +0.00332
|
||||||
|
Total merge throttle time [min] 0.796417 0.879267 +0.08285
|
||||||
|
Query latency term (50.0 percentile) [ms] 2.10049 2.15421 +0.05372
|
||||||
|
Query latency term (90.0 percentile) [ms] 2.77537 2.84168 +0.06630
|
||||||
|
Query latency term (100.0 percentile) [ms] 4.52081 5.15368 +0.63287
|
||||||
|
Query latency country_agg (50.0 percentile) [ms] 112.049 110.385 -1.66392
|
||||||
|
Query latency country_agg (90.0 percentile) [ms] 128.426 124.005 -4.42138
|
||||||
|
Query latency country_agg (100.0 percentile) [ms] 155.989 133.797 -22.19185
|
||||||
|
Query latency scroll (50.0 percentile) [ms] 16.1226 14.4974 -1.62519
|
||||||
|
Query latency scroll (90.0 percentile) [ms] 17.2383 15.4079 -1.83043
|
||||||
|
Query latency scroll (100.0 percentile) [ms] 18.8419 18.4241 -0.41784
|
||||||
|
Query latency country_agg_cached (50.0 percentile) [ms] 1.70223 1.64502 -0.05721
|
||||||
|
Query latency country_agg_cached (90.0 percentile) [ms] 2.34819 2.04318 -0.30500
|
||||||
|
Query latency country_agg_cached (100.0 percentile) [ms] 3.42547 2.86814 -0.55732
|
||||||
|
Query latency default (50.0 percentile) [ms] 5.89058 5.83409 -0.05648
|
||||||
|
Query latency default (90.0 percentile) [ms] 6.71282 6.64662 -0.06620
|
||||||
|
Query latency default (100.0 percentile) [ms] 7.65307 7.3701 -0.28297
|
||||||
|
Query latency phrase (50.0 percentile) [ms] 1.82687 1.83193 +0.00506
|
||||||
|
Query latency phrase (90.0 percentile) [ms] 2.63714 2.46286 -0.17428
|
||||||
|
Query latency phrase (100.0 percentile) [ms] 5.39892 4.22367 -1.17525
|
||||||
|
Median CPU usage (index) [%] 668.025 679.15 +11.12499
|
||||||
|
Median CPU usage (stats) [%] 143.75 162.4 +18.64999
|
||||||
|
Median CPU usage (search) [%] 223.1 229.2 +6.10000
|
||||||
|
Total Young Gen GC time [s] 39.447 40.456 +1.00900
|
||||||
|
Total Young Gen GC count 10 11 +1.00000
|
||||||
|
Total Old Gen GC time [s] 7.108 7.703 +0.59500
|
||||||
|
Total Old Gen GC count 10 11 +1.00000
|
||||||
|
Index size [GB] 3.25475 3.25098 -0.00377
|
||||||
|
Total written [GB] 17.8434 18.3143 +0.47083
|
||||||
|
Heap used for segments [MB] 21.7504 21.5901 -0.16037
|
||||||
|
Heap used for doc values [MB] 0.16436 0.13905 -0.02531
|
||||||
|
Heap used for terms [MB] 20.0293 19.9159 -0.11345
|
||||||
|
Heap used for norms [MB] 0.105469 0.0935669 -0.01190
|
||||||
|
Heap used for points [MB] 0.773487 0.772155 -0.00133
|
||||||
|
Heap used for points [MB] 0.677795 0.669426 -0.00837
|
||||||
|
Segment count 136 121 -15.00000
|
||||||
|
Indices Stats(90.0 percentile) [ms] 3.16053 3.21023 +0.04969
|
||||||
|
Indices Stats(99.0 percentile) [ms] 5.29526 3.94132 -1.35393
|
||||||
|
Indices Stats(100.0 percentile) [ms] 5.64971 7.02374 +1.37403
|
||||||
|
Nodes Stats(90.0 percentile) [ms] 3.19611 3.15251 -0.04360
|
||||||
|
Nodes Stats(99.0 percentile) [ms] 4.44111 4.87003 +0.42892
|
||||||
|
Nodes Stats(100.0 percentile) [ms] 5.22527 5.66977 +0.44450
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
You can use the following options to customize the results of your test comparison:
|
||||||
|
|
||||||
|
- `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution.
|
||||||
|
- `--contender`: The TestExecution ID for the contender being compared to the baseline.
|
||||||
|
- `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`.
|
||||||
|
- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`.
|
||||||
|
- `--results-file`: When provided a file path, writes the compare results to the file indicated in the path.
|
||||||
|
- `--show-in-results`: Determines whether or not to include the comparison in the results file.
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: download
|
||||||
|
nav_order: 60
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
# download
|
||||||
|
|
||||||
|
Use the `download` command to select which OpenSearch distribution version to download.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
The following example downloads OpenSearch version 2.7.0:
|
||||||
|
|
||||||
|
```
|
||||||
|
opensearch-benchmark download --distribution-version=2.7.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Benchmark then returns the location of the OpenSearch artifact:
|
||||||
|
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"opensearch": "/Users/.benchmark/benchmarks/distributions/opensearch-2.7.0.tar.gz"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
Use the following options to customize how OpenSearch Benchmark downloads OpenSearch:
|
||||||
|
|
||||||
|
- `--provision-config-repository`: Defines the repository from which OpenSearch Benchmark loads `provision-configs` and `provision-config-instances`.
|
||||||
|
- `--provision-config-revision`: Defines a specific Git revision in the `provision-config` that OpenSearch Benchmark should use.
|
||||||
|
- `--provision-config-path`: Defines the path to the `--provision-config-instance` and any OpenSearch plugin configurations to use.
|
||||||
|
- `--distribution-version`: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see [Version history](https://opensearch.org/docs/version-history/).
|
||||||
|
- `--distribution-repository`: Defines the repository from where the OpenSearch distribution should be downloaded. Default is `release`.
|
||||||
|
- `--provision-config-instance`: Defines the `--provision-config-instance` to use. You can see possible configuration instances using the command `opensearch-benchmark list provision-config-instances`.
|
||||||
|
- `--provision-config-instance-params`: A comma-separated list of key-value pairs injected verbatim as variables for the `provision-config-instance`.
|
||||||
|
- `--target-os`: The target operating system (OS) for which the OpenSearch artifact should be downloaded. Default is the current OS.
|
||||||
|
- `--target-arch`: The name of the CPU architecture for which an artifact should be downloaded.
|
||||||
|
|
@ -0,0 +1,178 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: execute-test
|
||||||
|
nav_order: 65
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
# execute-test
|
||||||
|
|
||||||
|
Whether you're using the included [OpenSearch Benchmark workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) or a [custom workload]({{site.url}}{{site.baseurl}}/benchmark/creating-a-custom-workload/), use the `execute-test` command to gather data about the performance of your OpenSearch cluster according the selected workload.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
The following example executes a test using the `geonames` workload in test mode:
|
||||||
|
|
||||||
|
```
|
||||||
|
opensearch-benchmark execute-test --workload=geonames --test-mode
|
||||||
|
```
|
||||||
|
|
||||||
|
After the test runs, OpenSearch Benchmark responds with a summary of the benchmark metrics:
|
||||||
|
|
||||||
|
```
|
||||||
|
------------------------------------------------------
|
||||||
|
_______ __ _____
|
||||||
|
/ ____(_)___ ____ _/ / / ___/_________ ________
|
||||||
|
/ /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
|
||||||
|
/ __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
|
||||||
|
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
|
||||||
|
------------------------------------------------------
|
||||||
|
|
||||||
|
| Metric | Task | Value | Unit |
|
||||||
|
|-------------------------------:|---------------------:|----------:|-------:|
|
||||||
|
| Total indexing time | | 28.0997 | min |
|
||||||
|
| Total merge time | | 6.84378 | min |
|
||||||
|
| Total refresh time | | 3.06045 | min |
|
||||||
|
| Total flush time | | 0.106517 | min |
|
||||||
|
| Total merge throttle time | | 1.28193 | min |
|
||||||
|
| Median CPU usage | | 471.6 | % |
|
||||||
|
| Total Young Gen GC | | 16.237 | s |
|
||||||
|
| Total Old Gen GC | | 1.796 | s |
|
||||||
|
| Index size | | 2.60124 | GB |
|
||||||
|
| Total written | | 11.8144 | GB |
|
||||||
|
| Heap used for segments | | 14.7326 | MB |
|
||||||
|
| Heap used for doc values | | 0.115917 | MB |
|
||||||
|
| Heap used for terms | | 13.3203 | MB |
|
||||||
|
| Heap used for norms | | 0.0734253 | MB |
|
||||||
|
| Heap used for points | | 0.5793 | MB |
|
||||||
|
| Heap used for stored fields | | 0.643608 | MB |
|
||||||
|
| Segment count | | 97 | |
|
||||||
|
| Min Throughput | index-append | 31925.2 | docs/s |
|
||||||
|
| Median Throughput | index-append | 39137.5 | docs/s |
|
||||||
|
| Max Throughput | index-append | 39633.6 | docs/s |
|
||||||
|
| 50.0th percentile latency | index-append | 872.513 | ms |
|
||||||
|
| 90.0th percentile latency | index-append | 1457.13 | ms |
|
||||||
|
| 99.0th percentile latency | index-append | 1874.89 | ms |
|
||||||
|
| 100th percentile latency | index-append | 2711.71 | ms |
|
||||||
|
| 50.0th percentile service time | index-append | 872.513 | ms |
|
||||||
|
| 90.0th percentile service time | index-append | 1457.13 | ms |
|
||||||
|
| 99.0th percentile service time | index-append | 1874.89 | ms |
|
||||||
|
| 100th percentile service time | index-append | 2711.71 | ms |
|
||||||
|
| ... | ... | ... | ... |
|
||||||
|
| ... | ... | ... | ... |
|
||||||
|
| Min Throughput | painless_dynamic | 2.53292 | ops/s |
|
||||||
|
| Median Throughput | painless_dynamic | 2.53813 | ops/s |
|
||||||
|
| Max Throughput | painless_dynamic | 2.54401 | ops/s |
|
||||||
|
| 50.0th percentile latency | painless_dynamic | 172208 | ms |
|
||||||
|
| 90.0th percentile latency | painless_dynamic | 310401 | ms |
|
||||||
|
| 99.0th percentile latency | painless_dynamic | 341341 | ms |
|
||||||
|
| 99.9th percentile latency | painless_dynamic | 344404 | ms |
|
||||||
|
| 100th percentile latency | painless_dynamic | 344754 | ms |
|
||||||
|
| 50.0th percentile service time | painless_dynamic | 393.02 | ms |
|
||||||
|
| 90.0th percentile service time | painless_dynamic | 407.579 | ms |
|
||||||
|
| 99.0th percentile service time | painless_dynamic | 430.806 | ms |
|
||||||
|
| 99.9th percentile service time | painless_dynamic | 457.352 | ms |
|
||||||
|
| 100th percentile service time | painless_dynamic | 459.474 | ms |
|
||||||
|
|
||||||
|
----------------------------------
|
||||||
|
[INFO] SUCCESS (took 2634 seconds)
|
||||||
|
----------------------------------
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
Use the following options to customize the `execute-test` command for your use case. Options in this section are categorized by their use case.
|
||||||
|
|
||||||
|
## General settings
|
||||||
|
|
||||||
|
The following options shape how each test runs and how results are output.
|
||||||
|
|
||||||
|
- `--test-mode`: Runs the given workload in test mode, which is useful when checking a workload for errors.
|
||||||
|
- `--user-tag`: Defines user-specific key-value pairs to be used in metric record as meta information, for example, `intention:baseline-ticket-12345`.
|
||||||
|
- `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`.
|
||||||
|
- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`.
|
||||||
|
- `--results-file`: When provided a file path, writes the compare results to the file indicated in the path.
|
||||||
|
- `--show-in-results`: Determines whether or not to include the comparison in the results file.
|
||||||
|
|
||||||
|
|
||||||
|
### Distributions
|
||||||
|
|
||||||
|
The following options set which version OpenSearch and OpenSearch plugins the benchmark test uses.
|
||||||
|
|
||||||
|
- `--distribution-version`: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see [Version history](https://opensearch.org/docs/version-history/).
|
||||||
|
- `--distribution-repository`: Defines the repository from where the OpenSearch distribution should be downloaded. Default is `release`.
|
||||||
|
- `--revision`: Defines the current source code revision to use for running a benchmark test. Default is `current`.
|
||||||
|
- `current`: Uses the source tree's current revision based on your OpenSearch distribution.
|
||||||
|
- `latest`: Fetches the latest revision from the main branch of the source tree.
|
||||||
|
- You can also use a time stamp or commit ID from the source tree. When using a timestamp, specify `@ts` where "ts" is a valid ISO 8601 timestamp, for example `@2013-07-27T10:37:00Z`.
|
||||||
|
- `--opensearch-plugins`: Defines which [OpenSearch plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) to install. By default, no plugins are installed.
|
||||||
|
- `--plugin-params:` Defines a comma-separated list of key:value pairs that are injected verbatim into all plugins as variables.
|
||||||
|
- `--runtime-jdk`: The major version of JDK to use.
|
||||||
|
- `--client-options`: Defines a comma-separated list of clients to use. All options are passed to the OpenSearch Python client. Default is `timeout:60`.
|
||||||
|
|
||||||
|
### Cluster
|
||||||
|
|
||||||
|
The following option relates to the target cluster of the benchmark.
|
||||||
|
|
||||||
|
- `--target-hosts`: Defines a comma-separated list of host-port pairs which should be targeted if using the pipeline `benchmark-only`. Default is `localhost:9200`.
|
||||||
|
|
||||||
|
|
||||||
|
### Distributed workload generation
|
||||||
|
|
||||||
|
The following options help those who want to use multiple hosts to generate load to the benchmark cluster.
|
||||||
|
|
||||||
|
- `--load-worker-coordinator-hosts`: Defines a comma-separated list of hosts that coordinate loads. Default is `localhost`.
|
||||||
|
- `--enable-worker-coordinator-profiling`: Enables an analysis of the performance of OpenSearch Benchmark's worker coordinator. Default is `false`.
|
||||||
|
|
||||||
|
### Provisioning
|
||||||
|
|
||||||
|
The following options help customize how OpenSearch Benchmark provisions OpenSearch and workloads.
|
||||||
|
|
||||||
|
- `--provision-config-repository`: Defines the repository from which OpenSearch Benchmark loads `provision-configs` and `provision-config-instances`.
|
||||||
|
- `--provision-config-path`: Defines the path to the `--provision-config-instance` and any OpenSearch plugin configurations to use.
|
||||||
|
- `--provision-config-revision`: Defines a specific Git revision in the `provision-config` that OpenSearch Benchmark should use.
|
||||||
|
- `--provision-config-instance`: Defines the `--provision-config-instance` to use. You can see possible configuration instances using the command `opensearch-benchmark list provision-config-instances`.
|
||||||
|
- `--provision-config-instance-params`: A comma-separated list of key-value pairs injected verbatim as variables for the `provision-config-instance`.
|
||||||
|
|
||||||
|
|
||||||
|
### Workload
|
||||||
|
|
||||||
|
The following options determine which workload is used to run the test.
|
||||||
|
|
||||||
|
- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads.
|
||||||
|
- `--workload-path`: Defines the path to a downloaded or custom workload.
|
||||||
|
- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use.
|
||||||
|
- `--workload`: Defines the workload to use based on the workload's name. You can find a list of preloaded workloads using `opensearch-benchmark list workloads`.
|
||||||
|
|
||||||
|
### Test procedures
|
||||||
|
|
||||||
|
The following options define what test procedures the test uses and which operations are contained inside the procedure.
|
||||||
|
|
||||||
|
- `--test-execution-id`: Defines a unique ID for this test run.
|
||||||
|
- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test-procedures`.
|
||||||
|
- `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
|
||||||
|
- `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run.
|
||||||
|
- `--enable-assertions`: Enables assertion checks for tasks. Default is `false`.
|
||||||
|
|
||||||
|
### Pipelines
|
||||||
|
|
||||||
|
The `--pipeline` option selects a pipeline to run. You can find a list of pipelines supported by OpenSearch Benchmark by running `opensearch-benchmark list pipelines`.
|
||||||
|
|
||||||
|
|
||||||
|
### Telemetry
|
||||||
|
|
||||||
|
The following options enable telemetry devices on OpenSearch Benchmark.
|
||||||
|
|
||||||
|
- `--telemetry`: Enables the provided telemetry devices when the devices are provided using a comma-separated list. You can find a list of possible telemetry devices by using `opensearch-benchmark list telemetry`.
|
||||||
|
- `--telemetry-params`: Defines a comma-separated list of key-value pairs that are injected verbatim into the telemetry devices as parameters.
|
||||||
|
|
||||||
|
|
||||||
|
### Errors
|
||||||
|
|
||||||
|
The following options set how OpenSearch Benchmark handles errors when running tests.
|
||||||
|
|
||||||
|
- `--on-error`: Controls how OpenSearch Benchmark responds to errors. Default is `continue`.
|
||||||
|
- `continue`: Continues to run the test despite the error.
|
||||||
|
- `abort`: Aborts the test when an error occurs.
|
||||||
|
- `--preserve-install`: Keeps the Benchmark candidate and its index. Default is `false`.
|
||||||
|
- `--kill-running-processes`: When set to `true`, stops any OpenSearch Benchmark processes currently running and allows OpenSearch Benchmark to continue to run. Default is `false`.
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: generate
|
||||||
|
nav_order: 70
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
The `generate` command generates visualization based on benchmark results.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
The following example generates a time-series chart, which outputs into the `.benchmark` directory.
|
||||||
|
|
||||||
|
```
|
||||||
|
opensearch-benchmark generate --chart-type="time-series"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
The following options customize the visualization produced by the `generate` command:
|
||||||
|
|
||||||
|
- `--chart-spec-path`: Sets the path to the JSON files containing chart specifications that can be used to generate charts.
|
||||||
|
- `--chart-type`: Generates the indicated chart type, either `time-series` or `bar`. Default is `time-series`.
|
||||||
|
- `--output-path`: The path and name where the chart outputs. Default is `stdout`.
|
||||||
|
|
@ -0,0 +1,26 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Command reference
|
||||||
|
nav_order: 50
|
||||||
|
has_children: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# OpenSearch Benchmark command reference
|
||||||
|
|
||||||
|
This section provides a list of commands supported by OpenSearch Benchmark, including commonly used commands such as `execute-test` and `list`.
|
||||||
|
|
||||||
|
- [compare]({{site.url}}{{site.baseurl}}/benchmark/commands/compare/)
|
||||||
|
- [download]({{site.url}}{{site.baseurl}}/benchmark/commands/download/)
|
||||||
|
- [execute-test]({{site.url}}{{site.baseurl}}/benchmark/commands/execute-test/)
|
||||||
|
- [generate]({{site.url}}{{site.baseurl}}/benchmark/commands/generate/)
|
||||||
|
- [info]({{site.url}}{{site.baseurl}}/benchmark/commands/info/)
|
||||||
|
- [list]({{site.url}}{{site.baseurl}}/benchmark/commands/list/)
|
||||||
|
|
||||||
|
## List of common options
|
||||||
|
|
||||||
|
All OpenSearch Benchmark commands support the following options:
|
||||||
|
|
||||||
|
- `--h` or `--help`: Provides options and other useful information about each command.
|
||||||
|
- `--quiet`: Hides as much of the results output as possible. Default is `false`.
|
||||||
|
- `--offline`: Indicates whether OpenSearch Benchmark has a connection to the Internet. Default is `false`.
|
||||||
|
|
||||||
|
|
@ -0,0 +1,158 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: info
|
||||||
|
nav_order: 75
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
# info
|
||||||
|
|
||||||
|
The `info` command prints details about a OpenSearch Benchmark component.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
The following example returns information about a workload named `nyc_taxis`.
|
||||||
|
|
||||||
|
```
|
||||||
|
opensearch-benchmark info --workload=nyc_taxis
|
||||||
|
```
|
||||||
|
|
||||||
|
OpenSearch Benchmark returns information about the workload, as shown in the following example response.
|
||||||
|
|
||||||
|
```
|
||||||
|
____ _____ __ ____ __ __
|
||||||
|
/ __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__
|
||||||
|
/ / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
|
||||||
|
/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,<
|
||||||
|
\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_|
|
||||||
|
/_/
|
||||||
|
|
||||||
|
Showing details for workload [nyc_taxis]:
|
||||||
|
|
||||||
|
* Description: Taxi rides in New York in 2015
|
||||||
|
* Documents: 165,346,692
|
||||||
|
* Compressed Size: 4.5 GB
|
||||||
|
* Uncompressed Size: 74.3 GB
|
||||||
|
|
||||||
|
===================================
|
||||||
|
TestProcedure [searchable-snapshot]
|
||||||
|
===================================
|
||||||
|
|
||||||
|
Measuring performance for Searchable Snapshot feature. Based on the default test procedure 'append-no-conflicts'.
|
||||||
|
|
||||||
|
Schedule:
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. delete-index
|
||||||
|
2. create-index
|
||||||
|
3. check-cluster-health
|
||||||
|
4. index (8 clients)
|
||||||
|
5. refresh-after-index
|
||||||
|
6. force-merge
|
||||||
|
7. refresh-after-force-merge
|
||||||
|
8. wait-until-merges-finish
|
||||||
|
9. create-snapshot-repository
|
||||||
|
10. delete-snapshot
|
||||||
|
11. create-snapshot
|
||||||
|
12. wait-for-snapshot-creation
|
||||||
|
13. delete-local-index
|
||||||
|
14. restore-snapshot
|
||||||
|
15. default
|
||||||
|
16. range
|
||||||
|
17. distance_amount_agg
|
||||||
|
18. autohisto_agg
|
||||||
|
19. date_histogram_agg
|
||||||
|
|
||||||
|
====================================================
|
||||||
|
TestProcedure [append-no-conflicts] (run by default)
|
||||||
|
====================================================
|
||||||
|
|
||||||
|
Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only. After that a couple of queries are run.
|
||||||
|
|
||||||
|
Schedule:
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. delete-index
|
||||||
|
2. create-index
|
||||||
|
3. check-cluster-health
|
||||||
|
4. index (8 clients)
|
||||||
|
5. refresh-after-index
|
||||||
|
6. force-merge
|
||||||
|
7. refresh-after-force-merge
|
||||||
|
8. wait-until-merges-finish
|
||||||
|
9. default
|
||||||
|
10. range
|
||||||
|
11. distance_amount_agg
|
||||||
|
12. autohisto_agg
|
||||||
|
13. date_histogram_agg
|
||||||
|
|
||||||
|
==============================================
|
||||||
|
TestProcedure [append-no-conflicts-index-only]
|
||||||
|
==============================================
|
||||||
|
|
||||||
|
Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only.
|
||||||
|
|
||||||
|
Schedule:
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. delete-index
|
||||||
|
2. create-index
|
||||||
|
3. check-cluster-health
|
||||||
|
4. index (8 clients)
|
||||||
|
5. refresh-after-index
|
||||||
|
6. force-merge
|
||||||
|
7. refresh-after-force-merge
|
||||||
|
8. wait-until-merges-finish
|
||||||
|
|
||||||
|
=====================================================
|
||||||
|
TestProcedure [append-sorted-no-conflicts-index-only]
|
||||||
|
=====================================================
|
||||||
|
|
||||||
|
Indexes the whole document corpus in an index sorted by pickup_datetime field in descending order (most recent first) and using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only.
|
||||||
|
|
||||||
|
Schedule:
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. delete-index
|
||||||
|
2. create-index
|
||||||
|
3. check-cluster-health
|
||||||
|
4. index (8 clients)
|
||||||
|
5. refresh-after-index
|
||||||
|
6. force-merge
|
||||||
|
7. refresh-after-force-merge
|
||||||
|
8. wait-until-merges-finish
|
||||||
|
|
||||||
|
======================
|
||||||
|
TestProcedure [update]
|
||||||
|
======================
|
||||||
|
|
||||||
|
Schedule:
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. delete-index
|
||||||
|
2. create-index
|
||||||
|
3. check-cluster-health
|
||||||
|
4. update (8 clients)
|
||||||
|
5. refresh-after-index
|
||||||
|
6. force-merge
|
||||||
|
7. refresh-after-force-merge
|
||||||
|
8. wait-until-merges-finish
|
||||||
|
|
||||||
|
|
||||||
|
-------------------------------
|
||||||
|
[INFO] SUCCESS (took 2 seconds)
|
||||||
|
-------------------------------
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
You can use the following options with the `info` command:
|
||||||
|
|
||||||
|
|
||||||
|
- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads.
|
||||||
|
- `--workload-path`: Defines the path to a downloaded or custom workload.
|
||||||
|
- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use.
|
||||||
|
- `--workload`: Defines the workload to use based on the workload's name. You can find a list of preloaded workloads using `opensearch-benchmark list workloads`.
|
||||||
|
- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test_procedures`.
|
||||||
|
- `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
|
||||||
|
- `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run.
|
||||||
|
|
@ -0,0 +1,70 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: list
|
||||||
|
nav_order: 80
|
||||||
|
parent: Command reference
|
||||||
|
---
|
||||||
|
|
||||||
|
# list
|
||||||
|
|
||||||
|
The `list` command lists the following elements used by OpenSearch Benchmark:
|
||||||
|
|
||||||
|
- `telemetry`: Telemetry devices
|
||||||
|
- `workloads`: Workloads
|
||||||
|
- `pipelines`: Pipelines
|
||||||
|
- `test_executions`: Single run of a workload
|
||||||
|
- `provision_config_instances`: Provisioned configuration instances
|
||||||
|
- `opensearch-plugins`: OpenSearch plugins
|
||||||
|
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
The following example lists any workload test runs and detailed information about each test.
|
||||||
|
|
||||||
|
```
|
||||||
|
`opensearch-benchmark list test_executions
|
||||||
|
```
|
||||||
|
|
||||||
|
OpenSearch Benchmark returns information about each test.
|
||||||
|
|
||||||
|
```
|
||||||
|
benchmark list test_executions
|
||||||
|
|
||||||
|
____ _____ __ ____ __ __
|
||||||
|
/ __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__
|
||||||
|
/ / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
|
||||||
|
/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,<
|
||||||
|
\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_|
|
||||||
|
/_/
|
||||||
|
|
||||||
|
|
||||||
|
Recent test_executions:
|
||||||
|
|
||||||
|
TestExecution ID TestExecution Timestamp Workload Workload Parameters TestProcedure ProvisionConfigInstance User Tags workload Revision Provision Config Revision
|
||||||
|
------------------------------------ ------------------------- ---------- --------------------- ------------------- ------------------------- ----------- ------------------- ---------------------------
|
||||||
|
729291a0-ee87-44e5-9b75-cc6d50c89702 20230524T181718Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
f91c33d0-ec93-48e1-975e-37476a5c9fe5 20230524T170134Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
d942b7f9-6506-451d-9dcf-ef502ab3e574 20230524T144827Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
a33845cc-c2e5-4488-a2db-b0670741ff9b 20230523T213145Z geonames append-no-conflicts 4gheap 30260cf
|
||||||
|
ba643ed3-0db5-452e-a680-2b0dc0350cf2 20230522T224450Z geonames append-no-conflicts external 30260cf
|
||||||
|
8d366ec5-3322-4e09-b041-a4b02e870033 20230519T201514Z geonames append-no-conflicts external 30260cf
|
||||||
|
4574c13e-8742-41af-a4fa-79480629ecf0 20230519T195617Z geonames append-no-conflicts external 30260cf
|
||||||
|
3e240d18-fc87-4c49-9712-863196efcef4 20230519T195412Z geonames append-no-conflicts external 30260cf
|
||||||
|
90f066ae-3d83-41e9-bbeb-17cb0480d578 20230519T194448Z geonames append-no-conflicts external 30260cf
|
||||||
|
78602e07-0ff8-4f00-9a0e-746fb64e4129 20230519T193258Z geonames append-no-conflicts external 30260cf
|
||||||
|
|
||||||
|
-------------------------------
|
||||||
|
[INFO] SUCCESS (took 0 seconds)
|
||||||
|
-------------------------------
|
||||||
|
```
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
You can use the following options with the `test` command:
|
||||||
|
|
||||||
|
- `--limit`: Limits the number of search results for recent test executions. Default is `10`.
|
||||||
|
- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads.
|
||||||
|
- `--workload-path`: Defines the path to a downloaded or custom workload.
|
||||||
|
- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use.
|
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue