From 377f4f09ea4a6c188079b3677512cc0949ce0850 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 25 May 2023 17:13:06 -0500 Subject: [PATCH] Add Benchmark command line reference (#4173) * Add Benchmark command reference Signed-off-by: Naarcha-AWS * Add initial Benchmark command line reference Signed-off-by: Naarcha-AWS * Address technical feedback Signed-off-by: Naarcha-AWS * More technical feedback Signed-off-by: Naarcha-AWS * More technical feedback Signed-off-by: Naarcha-AWS * Fix list description Signed-off-by: Naarcha-AWS * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Naarcha-AWS Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> --- _benchmark/commands/compare.md | 132 +++++++++++++++++++++ _benchmark/commands/download.md | 40 +++++++ _benchmark/commands/execute-test.md | 178 ++++++++++++++++++++++++++++ _benchmark/commands/generate.md | 24 ++++ _benchmark/commands/index.md | 26 ++++ _benchmark/commands/info.md | 158 ++++++++++++++++++++++++ _benchmark/commands/list.md | 70 +++++++++++ 7 files changed, 628 insertions(+) create mode 100644 _benchmark/commands/compare.md create mode 100644 _benchmark/commands/download.md create mode 100644 _benchmark/commands/execute-test.md create mode 100644 _benchmark/commands/generate.md create mode 100644 _benchmark/commands/index.md create mode 100644 _benchmark/commands/info.md create mode 100644 _benchmark/commands/list.md diff --git a/_benchmark/commands/compare.md b/_benchmark/commands/compare.md new file mode 100644 index 00000000..ef4f3ec9 --- /dev/null +++ b/_benchmark/commands/compare.md @@ -0,0 +1,132 @@ +--- +layout: default +title: compare +nav_order: 55 +parent: Command reference +--- + +# compare + +The `compare` command helps you analyze the difference between two benchmark tests. This can help you analyze the performance impact of changes made from a previous test based on a specific Git revision. + +## Usage + +You can compare two different workload tests using their `TestExecution IDs`. To find a list of tests run from a specific workload, use `opensearch-benchmark list test_executions`. You should receive an output similar to the following: + + +``` + ____ _____ __ ____ __ __ + / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ + / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ +/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< +\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| + /_/ +Recent test-executions: + +Recent test_executions: + +TestExecution ID TestExecution Timestamp Workload Workload Parameters TestProcedure ProvisionConfigInstance User Tags workload Revision Provision Config Revision +------------------------------------ ------------------------- ---------- --------------------- ------------------- ------------------------- ----------- ------------------- --------------------------- +729291a0-ee87-44e5-9b75-cc6d50c89702 20230524T181718Z geonames append-no-conflicts 4gheap 30260cf +f91c33d0-ec93-48e1-975e-37476a5c9fe5 20230524T170134Z geonames append-no-conflicts 4gheap 30260cf +d942b7f9-6506-451d-9dcf-ef502ab3e574 20230524T144827Z geonames append-no-conflicts 4gheap 30260cf +a33845cc-c2e5-4488-a2db-b0670741ff9b 20230523T213145Z geonames append-no-conflicts + +``` + +Then, use `compare` to call a `--baseline` test and a `--contender` test for comparison. + +``` +opensearch-benchmark compare --baseline=417ed42-6671-9i79-11a1-e367636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7 +``` + +You should receive the following response comparing the final benchmark metrics for both tests: + +``` + ____ _____ __ ____ __ __ + / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ + / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ +/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< +\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| + /_/ + +Comparing baseline + TestExecution ID: 729291a0-ee87-44e5-9b75-cc6d50c89702 + TestExecution timestamp: 2023-05-24 18:17:18 + +with contender + TestExecution ID: a33845cc-c2e5-4488-a2db-b0670741ff9b + TestExecution timestamp: 2023-05-23 21:31:45 + + +------------------------------------------------------ + _______ __ _____ + / ____(_)___ ____ _/ / / ___/_________ ________ + / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \ + / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/ +/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/ +------------------------------------------------------ + Metric Baseline Contender Diff +-------------------------------------------------------- ---------- ----------- ----------------- + Min Indexing Throughput [docs/s] 19501 19118 -383.00000 + Median Indexing Throughput [docs/s] 20232 19927.5 -304.45833 + Max Indexing Throughput [docs/s] 21172 20849 -323.00000 + Total indexing time [min] 55.7989 56.335 +0.53603 + Total merge time [min] 12.9766 13.3115 +0.33495 + Total refresh time [min] 5.20067 5.20097 +0.00030 + Total flush time [min] 0.0648667 0.0681833 +0.00332 + Total merge throttle time [min] 0.796417 0.879267 +0.08285 + Query latency term (50.0 percentile) [ms] 2.10049 2.15421 +0.05372 + Query latency term (90.0 percentile) [ms] 2.77537 2.84168 +0.06630 + Query latency term (100.0 percentile) [ms] 4.52081 5.15368 +0.63287 + Query latency country_agg (50.0 percentile) [ms] 112.049 110.385 -1.66392 + Query latency country_agg (90.0 percentile) [ms] 128.426 124.005 -4.42138 + Query latency country_agg (100.0 percentile) [ms] 155.989 133.797 -22.19185 + Query latency scroll (50.0 percentile) [ms] 16.1226 14.4974 -1.62519 + Query latency scroll (90.0 percentile) [ms] 17.2383 15.4079 -1.83043 + Query latency scroll (100.0 percentile) [ms] 18.8419 18.4241 -0.41784 + Query latency country_agg_cached (50.0 percentile) [ms] 1.70223 1.64502 -0.05721 + Query latency country_agg_cached (90.0 percentile) [ms] 2.34819 2.04318 -0.30500 +Query latency country_agg_cached (100.0 percentile) [ms] 3.42547 2.86814 -0.55732 + Query latency default (50.0 percentile) [ms] 5.89058 5.83409 -0.05648 + Query latency default (90.0 percentile) [ms] 6.71282 6.64662 -0.06620 + Query latency default (100.0 percentile) [ms] 7.65307 7.3701 -0.28297 + Query latency phrase (50.0 percentile) [ms] 1.82687 1.83193 +0.00506 + Query latency phrase (90.0 percentile) [ms] 2.63714 2.46286 -0.17428 + Query latency phrase (100.0 percentile) [ms] 5.39892 4.22367 -1.17525 + Median CPU usage (index) [%] 668.025 679.15 +11.12499 + Median CPU usage (stats) [%] 143.75 162.4 +18.64999 + Median CPU usage (search) [%] 223.1 229.2 +6.10000 + Total Young Gen GC time [s] 39.447 40.456 +1.00900 + Total Young Gen GC count 10 11 +1.00000 + Total Old Gen GC time [s] 7.108 7.703 +0.59500 + Total Old Gen GC count 10 11 +1.00000 + Index size [GB] 3.25475 3.25098 -0.00377 + Total written [GB] 17.8434 18.3143 +0.47083 + Heap used for segments [MB] 21.7504 21.5901 -0.16037 + Heap used for doc values [MB] 0.16436 0.13905 -0.02531 + Heap used for terms [MB] 20.0293 19.9159 -0.11345 + Heap used for norms [MB] 0.105469 0.0935669 -0.01190 + Heap used for points [MB] 0.773487 0.772155 -0.00133 + Heap used for points [MB] 0.677795 0.669426 -0.00837 + Segment count 136 121 -15.00000 + Indices Stats(90.0 percentile) [ms] 3.16053 3.21023 +0.04969 + Indices Stats(99.0 percentile) [ms] 5.29526 3.94132 -1.35393 + Indices Stats(100.0 percentile) [ms] 5.64971 7.02374 +1.37403 + Nodes Stats(90.0 percentile) [ms] 3.19611 3.15251 -0.04360 + Nodes Stats(99.0 percentile) [ms] 4.44111 4.87003 +0.42892 + Nodes Stats(100.0 percentile) [ms] 5.22527 5.66977 +0.44450 +``` + +## Options + +You can use the following options to customize the results of your test comparison: + +- `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution. +- `--contender`: The TestExecution ID for the contender being compared to the baseline. +- `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`. +- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. +- `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. +- `--show-in-results`: Determines whether or not to include the comparison in the results file. + + diff --git a/_benchmark/commands/download.md b/_benchmark/commands/download.md new file mode 100644 index 00000000..55cdcbd8 --- /dev/null +++ b/_benchmark/commands/download.md @@ -0,0 +1,40 @@ +--- +layout: default +title: download +nav_order: 60 +parent: Command reference +--- + +# download + +Use the `download` command to select which OpenSearch distribution version to download. + +## Usage + +The following example downloads OpenSearch version 2.7.0: + +``` +opensearch-benchmark download --distribution-version=2.7.0 +``` + +Benchmark then returns the location of the OpenSearch artifact: + +``` +{ + "opensearch": "/Users/.benchmark/benchmarks/distributions/opensearch-2.7.0.tar.gz" +} +``` + +## Options + +Use the following options to customize how OpenSearch Benchmark downloads OpenSearch: + +- `--provision-config-repository`: Defines the repository from which OpenSearch Benchmark loads `provision-configs` and `provision-config-instances`. +- `--provision-config-revision`: Defines a specific Git revision in the `provision-config` that OpenSearch Benchmark should use. +- `--provision-config-path`: Defines the path to the `--provision-config-instance` and any OpenSearch plugin configurations to use. +- `--distribution-version`: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see [Version history](https://opensearch.org/docs/version-history/). +- `--distribution-repository`: Defines the repository from where the OpenSearch distribution should be downloaded. Default is `release`. +- `--provision-config-instance`: Defines the `--provision-config-instance` to use. You can see possible configuration instances using the command `opensearch-benchmark list provision-config-instances`. +- `--provision-config-instance-params`: A comma-separated list of key-value pairs injected verbatim as variables for the `provision-config-instance`. +- `--target-os`: The target operating system (OS) for which the OpenSearch artifact should be downloaded. Default is the current OS. +- `--target-arch`: The name of the CPU architecture for which an artifact should be downloaded. diff --git a/_benchmark/commands/execute-test.md b/_benchmark/commands/execute-test.md new file mode 100644 index 00000000..9a595e76 --- /dev/null +++ b/_benchmark/commands/execute-test.md @@ -0,0 +1,178 @@ +--- +layout: default +title: execute-test +nav_order: 65 +parent: Command reference +--- + +# execute-test + +Whether you're using the included [OpenSearch Benchmark workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) or a [custom workload]({{site.url}}{{site.baseurl}}/benchmark/creating-a-custom-workload/), use the `execute-test` command to gather data about the performance of your OpenSearch cluster according the selected workload. + +## Usage + +The following example executes a test using the `geonames` workload in test mode: + +``` +opensearch-benchmark execute-test --workload=geonames --test-mode +``` + +After the test runs, OpenSearch Benchmark responds with a summary of the benchmark metrics: + +``` +------------------------------------------------------ + _______ __ _____ + / ____(_)___ ____ _/ / / ___/_________ ________ + / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \ + / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/ +/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/ +------------------------------------------------------ + +| Metric | Task | Value | Unit | +|-------------------------------:|---------------------:|----------:|-------:| +| Total indexing time | | 28.0997 | min | +| Total merge time | | 6.84378 | min | +| Total refresh time | | 3.06045 | min | +| Total flush time | | 0.106517 | min | +| Total merge throttle time | | 1.28193 | min | +| Median CPU usage | | 471.6 | % | +| Total Young Gen GC | | 16.237 | s | +| Total Old Gen GC | | 1.796 | s | +| Index size | | 2.60124 | GB | +| Total written | | 11.8144 | GB | +| Heap used for segments | | 14.7326 | MB | +| Heap used for doc values | | 0.115917 | MB | +| Heap used for terms | | 13.3203 | MB | +| Heap used for norms | | 0.0734253 | MB | +| Heap used for points | | 0.5793 | MB | +| Heap used for stored fields | | 0.643608 | MB | +| Segment count | | 97 | | +| Min Throughput | index-append | 31925.2 | docs/s | +| Median Throughput | index-append | 39137.5 | docs/s | +| Max Throughput | index-append | 39633.6 | docs/s | +| 50.0th percentile latency | index-append | 872.513 | ms | +| 90.0th percentile latency | index-append | 1457.13 | ms | +| 99.0th percentile latency | index-append | 1874.89 | ms | +| 100th percentile latency | index-append | 2711.71 | ms | +| 50.0th percentile service time | index-append | 872.513 | ms | +| 90.0th percentile service time | index-append | 1457.13 | ms | +| 99.0th percentile service time | index-append | 1874.89 | ms | +| 100th percentile service time | index-append | 2711.71 | ms | +| ... | ... | ... | ... | +| ... | ... | ... | ... | +| Min Throughput | painless_dynamic | 2.53292 | ops/s | +| Median Throughput | painless_dynamic | 2.53813 | ops/s | +| Max Throughput | painless_dynamic | 2.54401 | ops/s | +| 50.0th percentile latency | painless_dynamic | 172208 | ms | +| 90.0th percentile latency | painless_dynamic | 310401 | ms | +| 99.0th percentile latency | painless_dynamic | 341341 | ms | +| 99.9th percentile latency | painless_dynamic | 344404 | ms | +| 100th percentile latency | painless_dynamic | 344754 | ms | +| 50.0th percentile service time | painless_dynamic | 393.02 | ms | +| 90.0th percentile service time | painless_dynamic | 407.579 | ms | +| 99.0th percentile service time | painless_dynamic | 430.806 | ms | +| 99.9th percentile service time | painless_dynamic | 457.352 | ms | +| 100th percentile service time | painless_dynamic | 459.474 | ms | + +---------------------------------- +[INFO] SUCCESS (took 2634 seconds) +---------------------------------- +``` + +## Options + +Use the following options to customize the `execute-test` command for your use case. Options in this section are categorized by their use case. + +## General settings + +The following options shape how each test runs and how results are output. + +- `--test-mode`: Runs the given workload in test mode, which is useful when checking a workload for errors. +- `--user-tag`: Defines user-specific key-value pairs to be used in metric record as meta information, for example, `intention:baseline-ticket-12345`. +- `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`. +- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. +- `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. +- `--show-in-results`: Determines whether or not to include the comparison in the results file. + + +### Distributions + +The following options set which version OpenSearch and OpenSearch plugins the benchmark test uses. + +- `--distribution-version`: Downloads the specified OpenSearch distribution based on version number. For a list of released OpenSearch versions, see [Version history](https://opensearch.org/docs/version-history/). +- `--distribution-repository`: Defines the repository from where the OpenSearch distribution should be downloaded. Default is `release`. +- `--revision`: Defines the current source code revision to use for running a benchmark test. Default is `current`. + - `current`: Uses the source tree's current revision based on your OpenSearch distribution. + - `latest`: Fetches the latest revision from the main branch of the source tree. + - You can also use a time stamp or commit ID from the source tree. When using a timestamp, specify `@ts` where "ts" is a valid ISO 8601 timestamp, for example `@2013-07-27T10:37:00Z`. +- `--opensearch-plugins`: Defines which [OpenSearch plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) to install. By default, no plugins are installed. +- `--plugin-params:` Defines a comma-separated list of key:value pairs that are injected verbatim into all plugins as variables. +- `--runtime-jdk`: The major version of JDK to use. +- `--client-options`: Defines a comma-separated list of clients to use. All options are passed to the OpenSearch Python client. Default is `timeout:60`. + +### Cluster + +The following option relates to the target cluster of the benchmark. + +- `--target-hosts`: Defines a comma-separated list of host-port pairs which should be targeted if using the pipeline `benchmark-only`. Default is `localhost:9200`. + + +### Distributed workload generation + +The following options help those who want to use multiple hosts to generate load to the benchmark cluster. + +- `--load-worker-coordinator-hosts`: Defines a comma-separated list of hosts that coordinate loads. Default is `localhost`. +- `--enable-worker-coordinator-profiling`: Enables an analysis of the performance of OpenSearch Benchmark's worker coordinator. Default is `false`. + +### Provisioning + +The following options help customize how OpenSearch Benchmark provisions OpenSearch and workloads. + +- `--provision-config-repository`: Defines the repository from which OpenSearch Benchmark loads `provision-configs` and `provision-config-instances`. +- `--provision-config-path`: Defines the path to the `--provision-config-instance` and any OpenSearch plugin configurations to use. +- `--provision-config-revision`: Defines a specific Git revision in the `provision-config` that OpenSearch Benchmark should use. +- `--provision-config-instance`: Defines the `--provision-config-instance` to use. You can see possible configuration instances using the command `opensearch-benchmark list provision-config-instances`. +- `--provision-config-instance-params`: A comma-separated list of key-value pairs injected verbatim as variables for the `provision-config-instance`. + + +### Workload + +The following options determine which workload is used to run the test. + +- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads. +- `--workload-path`: Defines the path to a downloaded or custom workload. +- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use. +- `--workload`: Defines the workload to use based on the workload's name. You can find a list of preloaded workloads using `opensearch-benchmark list workloads`. + +### Test procedures + +The following options define what test procedures the test uses and which operations are contained inside the procedure. + +- `--test-execution-id`: Defines a unique ID for this test run. +- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test-procedures`. +- `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run. +- `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run. +- `--enable-assertions`: Enables assertion checks for tasks. Default is `false`. + +### Pipelines + +The `--pipeline` option selects a pipeline to run. You can find a list of pipelines supported by OpenSearch Benchmark by running `opensearch-benchmark list pipelines`. + + +### Telemetry + +The following options enable telemetry devices on OpenSearch Benchmark. + +- `--telemetry`: Enables the provided telemetry devices when the devices are provided using a comma-separated list. You can find a list of possible telemetry devices by using `opensearch-benchmark list telemetry`. +- `--telemetry-params`: Defines a comma-separated list of key-value pairs that are injected verbatim into the telemetry devices as parameters. + + +### Errors + +The following options set how OpenSearch Benchmark handles errors when running tests. + +- `--on-error`: Controls how OpenSearch Benchmark responds to errors. Default is `continue`. + - `continue`: Continues to run the test despite the error. + - `abort`: Aborts the test when an error occurs. +- `--preserve-install`: Keeps the Benchmark candidate and its index. Default is `false`. +- `--kill-running-processes`: When set to `true`, stops any OpenSearch Benchmark processes currently running and allows OpenSearch Benchmark to continue to run. Default is `false`. diff --git a/_benchmark/commands/generate.md b/_benchmark/commands/generate.md new file mode 100644 index 00000000..29cfce7f --- /dev/null +++ b/_benchmark/commands/generate.md @@ -0,0 +1,24 @@ +--- +layout: default +title: generate +nav_order: 70 +parent: Command reference +--- + +The `generate` command generates visualization based on benchmark results. + +## Usage + +The following example generates a time-series chart, which outputs into the `.benchmark` directory. + +``` +opensearch-benchmark generate --chart-type="time-series" +``` + +## Options + +The following options customize the visualization produced by the `generate` command: + +- `--chart-spec-path`: Sets the path to the JSON files containing chart specifications that can be used to generate charts. +- `--chart-type`: Generates the indicated chart type, either `time-series` or `bar`. Default is `time-series`. +- `--output-path`: The path and name where the chart outputs. Default is `stdout`. diff --git a/_benchmark/commands/index.md b/_benchmark/commands/index.md new file mode 100644 index 00000000..746308b9 --- /dev/null +++ b/_benchmark/commands/index.md @@ -0,0 +1,26 @@ +--- +layout: default +title: Command reference +nav_order: 50 +has_children: true +--- + +# OpenSearch Benchmark command reference + +This section provides a list of commands supported by OpenSearch Benchmark, including commonly used commands such as `execute-test` and `list`. + +- [compare]({{site.url}}{{site.baseurl}}/benchmark/commands/compare/) +- [download]({{site.url}}{{site.baseurl}}/benchmark/commands/download/) +- [execute-test]({{site.url}}{{site.baseurl}}/benchmark/commands/execute-test/) +- [generate]({{site.url}}{{site.baseurl}}/benchmark/commands/generate/) +- [info]({{site.url}}{{site.baseurl}}/benchmark/commands/info/) +- [list]({{site.url}}{{site.baseurl}}/benchmark/commands/list/) + +## List of common options + +All OpenSearch Benchmark commands support the following options: + +- `--h` or `--help`: Provides options and other useful information about each command. +- `--quiet`: Hides as much of the results output as possible. Default is `false`. +- `--offline`: Indicates whether OpenSearch Benchmark has a connection to the Internet. Default is `false`. + diff --git a/_benchmark/commands/info.md b/_benchmark/commands/info.md new file mode 100644 index 00000000..fdea9bd7 --- /dev/null +++ b/_benchmark/commands/info.md @@ -0,0 +1,158 @@ +--- +layout: default +title: info +nav_order: 75 +parent: Command reference +--- + +# info + +The `info` command prints details about a OpenSearch Benchmark component. + +## Usage + +The following example returns information about a workload named `nyc_taxis`. + +``` +opensearch-benchmark info --workload=nyc_taxis +``` + +OpenSearch Benchmark returns information about the workload, as shown in the following example response. + +``` + ____ _____ __ ____ __ __ + / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ + / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ +/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< +\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| + /_/ + +Showing details for workload [nyc_taxis]: + +* Description: Taxi rides in New York in 2015 +* Documents: 165,346,692 +* Compressed Size: 4.5 GB +* Uncompressed Size: 74.3 GB + +=================================== +TestProcedure [searchable-snapshot] +=================================== + +Measuring performance for Searchable Snapshot feature. Based on the default test procedure 'append-no-conflicts'. + +Schedule: +---------- + +1. delete-index +2. create-index +3. check-cluster-health +4. index (8 clients) +5. refresh-after-index +6. force-merge +7. refresh-after-force-merge +8. wait-until-merges-finish +9. create-snapshot-repository +10. delete-snapshot +11. create-snapshot +12. wait-for-snapshot-creation +13. delete-local-index +14. restore-snapshot +15. default +16. range +17. distance_amount_agg +18. autohisto_agg +19. date_histogram_agg + +==================================================== +TestProcedure [append-no-conflicts] (run by default) +==================================================== + +Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only. After that a couple of queries are run. + +Schedule: +---------- + +1. delete-index +2. create-index +3. check-cluster-health +4. index (8 clients) +5. refresh-after-index +6. force-merge +7. refresh-after-force-merge +8. wait-until-merges-finish +9. default +10. range +11. distance_amount_agg +12. autohisto_agg +13. date_histogram_agg + +============================================== +TestProcedure [append-no-conflicts-index-only] +============================================== + +Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only. + +Schedule: +---------- + +1. delete-index +2. create-index +3. check-cluster-health +4. index (8 clients) +5. refresh-after-index +6. force-merge +7. refresh-after-force-merge +8. wait-until-merges-finish + +===================================================== +TestProcedure [append-sorted-no-conflicts-index-only] +===================================================== + +Indexes the whole document corpus in an index sorted by pickup_datetime field in descending order (most recent first) and using a setup that will lead to a larger indexing throughput than the default settings and produce a smaller index (higher compression rate). Document ids are unique so all index operations are append only. + +Schedule: +---------- + +1. delete-index +2. create-index +3. check-cluster-health +4. index (8 clients) +5. refresh-after-index +6. force-merge +7. refresh-after-force-merge +8. wait-until-merges-finish + +====================== +TestProcedure [update] +====================== + +Schedule: +---------- + +1. delete-index +2. create-index +3. check-cluster-health +4. update (8 clients) +5. refresh-after-index +6. force-merge +7. refresh-after-force-merge +8. wait-until-merges-finish + + +------------------------------- +[INFO] SUCCESS (took 2 seconds) +------------------------------- +``` + +## Options + +You can use the following options with the `info` command: + + +- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads. +- `--workload-path`: Defines the path to a downloaded or custom workload. +- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use. +- `--workload`: Defines the workload to use based on the workload's name. You can find a list of preloaded workloads using `opensearch-benchmark list workloads`. +- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test_procedures`. +- `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run. +- `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run. diff --git a/_benchmark/commands/list.md b/_benchmark/commands/list.md new file mode 100644 index 00000000..e1660eeb --- /dev/null +++ b/_benchmark/commands/list.md @@ -0,0 +1,70 @@ +--- +layout: default +title: list +nav_order: 80 +parent: Command reference +--- + +# list + +The `list` command lists the following elements used by OpenSearch Benchmark: + +- `telemetry`: Telemetry devices +- `workloads`: Workloads +- `pipelines`: Pipelines +- `test_executions`: Single run of a workload +- `provision_config_instances`: Provisioned configuration instances +- `opensearch-plugins`: OpenSearch plugins + + +## Usage + +The following example lists any workload test runs and detailed information about each test. + +``` +`opensearch-benchmark list test_executions +``` + +OpenSearch Benchmark returns information about each test. + +``` +benchmark list test_executions + + ____ _____ __ ____ __ __ + / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ + / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ +/ /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< +\____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| + /_/ + + +Recent test_executions: + +TestExecution ID TestExecution Timestamp Workload Workload Parameters TestProcedure ProvisionConfigInstance User Tags workload Revision Provision Config Revision +------------------------------------ ------------------------- ---------- --------------------- ------------------- ------------------------- ----------- ------------------- --------------------------- +729291a0-ee87-44e5-9b75-cc6d50c89702 20230524T181718Z geonames append-no-conflicts 4gheap 30260cf +f91c33d0-ec93-48e1-975e-37476a5c9fe5 20230524T170134Z geonames append-no-conflicts 4gheap 30260cf +d942b7f9-6506-451d-9dcf-ef502ab3e574 20230524T144827Z geonames append-no-conflicts 4gheap 30260cf +a33845cc-c2e5-4488-a2db-b0670741ff9b 20230523T213145Z geonames append-no-conflicts 4gheap 30260cf +ba643ed3-0db5-452e-a680-2b0dc0350cf2 20230522T224450Z geonames append-no-conflicts external 30260cf +8d366ec5-3322-4e09-b041-a4b02e870033 20230519T201514Z geonames append-no-conflicts external 30260cf +4574c13e-8742-41af-a4fa-79480629ecf0 20230519T195617Z geonames append-no-conflicts external 30260cf +3e240d18-fc87-4c49-9712-863196efcef4 20230519T195412Z geonames append-no-conflicts external 30260cf +90f066ae-3d83-41e9-bbeb-17cb0480d578 20230519T194448Z geonames append-no-conflicts external 30260cf +78602e07-0ff8-4f00-9a0e-746fb64e4129 20230519T193258Z geonames append-no-conflicts external 30260cf + +------------------------------- +[INFO] SUCCESS (took 0 seconds) +------------------------------- +``` + +## Options + +You can use the following options with the `test` command: + +- `--limit`: Limits the number of search results for recent test executions. Default is `10`. +- `--workload-repository`: Defines the repository from where OpenSearch Benchmark loads workloads. +- `--workload-path`: Defines the path to a downloaded or custom workload. +- `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use. + +