303 lines
12 KiB
Markdown
303 lines
12 KiB
Markdown
---
|
|
layout: default
|
|
title: Performance Analyzer
|
|
nav_order: 58
|
|
has_children: true
|
|
redirect_from:
|
|
- /monitoring-plugins/pa/
|
|
---
|
|
|
|
# Performance analyzer
|
|
|
|
Performance analyzer is an agent and REST API that allows you to query numerous performance metrics for your cluster, including aggregations of those metrics.
|
|
|
|
The performance analyzer plugin is installed by default in OpenSearch version 2.0 and higher.
|
|
{: .note }
|
|
|
|
## Performance analyzer installation and configuration
|
|
|
|
The following sections provide the steps for installing and configuring the performance analyzer plugin.
|
|
|
|
### Install performance analyzer
|
|
|
|
The performance analyzer plugin is included in the installation for [Docker]({{site.url}}{{site.baseurl}}/opensearch/install/docker/) and [tarball]({{site.url}}{{site.baseurl}}/opensearch/install/tar/). If you need to install the performance analyzer plugin manually, download the plugin from [Maven](https://search.maven.org/search?q=org.opensearch.plugin) and install the plugin using the standard [plugins install]({{site.url}}{{site.baseurl}}/opensearch/install/plugins/) process. Performance analyzer will run on each node in a cluster.
|
|
|
|
To start the performance analyzer root cause analysis (RCA) agent on a tarball installation, run the following command:
|
|
|
|
````bash
|
|
OPENSEARCH_HOME=~/opensearch-2.2.1 OPENSEARCH_JAVA_HOME=~/opensearch-2.2.1/jdk OPENSEARCH_PATH_CONF=~/opensearch-2.2.1/bin ./performance-analyzer-agent-cli
|
|
````
|
|
|
|
The following command enables the performance analyzer plugin and performance analyzer RCA agent:
|
|
|
|
````bash
|
|
curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'
|
|
````
|
|
|
|
To shut down the performance analyzer RCA agent, run the following command:
|
|
|
|
````bash
|
|
kill $(ps aux | grep -i 'PerformanceAnalyzerApp' | grep -v grep | awk '{print $2}')
|
|
````
|
|
|
|
To disable the performance analyzer plugin, run the following command:
|
|
|
|
````bash
|
|
curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": false}'
|
|
````
|
|
|
|
To uninstall the performance analyzer plugin, run the following command:
|
|
|
|
````bash
|
|
bin/opensearch-plugin remove opensearch-performance-analyzer
|
|
````
|
|
|
|
### Configure performance analyzer
|
|
|
|
To configure the performance analyzer plugin, you will need to edit the `performance-analyzer.properties` configuration file in the `config/opensearch-performance-analyzer/` directory. Make sure to uncomment the line `#webservice-bind-host` and set it to `0.0.0.0`. You can reference the following example configuration.
|
|
|
|
````bash
|
|
# ======================== OpenSearch performance analyzer plugin config =========================
|
|
|
|
# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
|
|
|
|
# WebService bind host; default to all interfaces
|
|
webservice-bind-host = 0.0.0.0
|
|
|
|
# Metrics data location
|
|
metrics-location = /dev/shm/performanceanalyzer/
|
|
|
|
# Metrics deletion interval (minutes) for metrics data.
|
|
# Interval should be between 1 to 60.
|
|
metrics-deletion-interval = 1
|
|
|
|
# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
|
|
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
|
|
# the files and wouldn't like for them to be cleaned up.
|
|
cleanup-metrics-db-files = true
|
|
|
|
# WebService exposed by App's port
|
|
webservice-listener-port = 9600
|
|
|
|
# Metric DB File Prefix Path location
|
|
metrics-db-file-prefix-path = /tmp/metricsdb_
|
|
|
|
https-enabled = false
|
|
|
|
#Setup the correct path for certificates
|
|
#certificate-file-path = specify_path
|
|
|
|
#private-key-file-path = specify_path
|
|
|
|
# Plugin Stats Metadata file name, expected to be in the same location
|
|
plugin-stats-metadata = plugin-stats-metadata
|
|
|
|
# Agent Stats Metadata file name, expected to be in the same location
|
|
agent-stats-metadata = agent-stats-metadata
|
|
````
|
|
To start the performance analyzer RCA agent, run the following command.
|
|
|
|
````bash
|
|
OPENSEARCH_HOME=~/opensearch-2.2.1 OPENSEARCH_JAVA_HOME=~/opensearch-2.2.1/jdk OPENSEARCH_PATH_CONF=~/opensearch-2.2.1/bin ./performance-analyzer-agent-cli
|
|
````
|
|
|
|
### Storage
|
|
|
|
Performance analyzer uses `/dev/shm` for temporary storage. During heavy workloads on a cluster, performance analyzer can use up to 1 GB of space.
|
|
|
|
Docker, however, has a default `/dev/shm` size of 64 MB. To change this value, you can use the `docker run --shm-size 1gb` flag or [a similar setting in Docker Compose](https://docs.docker.com/compose/compose-file#shm_size).
|
|
|
|
If you're not using Docker, check the size of `/dev/shm` using `df -h`. The default value is probably plenty, but if you need to change its size, add the following line to `/etc/fstab`:
|
|
|
|
```bash
|
|
tmpfs /dev/shm tmpfs defaults,noexec,nosuid,size=1G 0 0
|
|
```
|
|
|
|
Then remount the file system:
|
|
|
|
```bash
|
|
mount -o remount /dev/shm
|
|
```
|
|
|
|
### Security
|
|
|
|
Performance analyzer supports encryption in transit for requests. It currently does *not* support client or server authentication for requests. To enable encryption in transit, edit `performance-analyzer.properties` in your `$OPENSEARCH_HOME` directory.
|
|
|
|
```bash
|
|
vi $OPENSEARCH_HOME/config/opensearch-performance-analyzer/performance-analyzer.properties
|
|
```
|
|
|
|
Change the following lines to configure encryption in transit. Note that `certificate-file-path` must be a certificate for the server, not a root certificate authority (CA).
|
|
|
|
````bash
|
|
https-enabled = true
|
|
|
|
#Setup the correct path for certificates
|
|
certificate-file-path = specify_path
|
|
|
|
private-key-file-path = specify_path
|
|
````
|
|
|
|
### Enable performance analyzer for RPM/YUM installations
|
|
|
|
If you installed OpenSearch from an RPM distribution, you can start and stop performance analyzer with `systemctl`.
|
|
|
|
```bash
|
|
# Start OpenSearch Performance Analyzer
|
|
sudo systemctl start opensearch-performance-analyzer.service
|
|
# Stop OpenSearch Performance Analyzer
|
|
sudo systemctl stop opensearch-performance-analyzer.service
|
|
```
|
|
|
|
## Example API query and response
|
|
|
|
The following is an example API query:
|
|
|
|
````bash
|
|
GET localhost:9600/_plugins/_performanceanalyzer/metrics/units
|
|
````
|
|
|
|
The following is an example response:
|
|
|
|
````json
|
|
{"Disk_Utilization":"%","Cache_Request_Hit":"count",
|
|
"Refresh_Time":"ms","ThreadPool_QueueLatency":"count",
|
|
"Merge_Time":"ms","ClusterApplierService_Latency":"ms",
|
|
"PublishClusterState_Latency":"ms",
|
|
"Cache_Request_Size":"B","LeaderCheck_Failure":"count",
|
|
"ThreadPool_QueueSize":"count","Sched_Runtime":"s/ctxswitch","Disk_ServiceRate":"MB/s","Heap_AllocRate":"B/s","Indexing_Pressure_Current_Limits":"B",
|
|
"Sched_Waittime":"s/ctxswitch","ShardBulkDocs":"count",
|
|
"Thread_Blocked_Time":"s/event","VersionMap_Memory":"B",
|
|
"Master_Task_Queue_Time":"ms","IO_TotThroughput":"B/s",
|
|
"Indexing_Pressure_Current_Bytes":"B",
|
|
"Indexing_Pressure_Last_Successful_Timestamp":"ms",
|
|
"Net_PacketRate6":"packets/s","Cache_Query_Hit":"count",
|
|
"IO_ReadSyscallRate":"count/s","Net_PacketRate4":"packets/s","Cache_Request_Miss":"count",
|
|
"ThreadPool_RejectedReqs":"count","Net_TCP_TxQ":"segments/flow","Master_Task_Run_Time":"ms",
|
|
"IO_WriteSyscallRate":"count/s","IO_WriteThroughput":"B/s",
|
|
"Refresh_Event":"count","Flush_Time":"ms","Heap_Init":"B",
|
|
"Indexing_Pressure_Rejection_Count":"count",
|
|
"CPU_Utilization":"cores","Cache_Query_Size":"B",
|
|
"Merge_Event":"count","Cache_FieldData_Eviction":"count",
|
|
"IO_TotalSyscallRate":"count/s","Net_Throughput":"B/s",
|
|
"Paging_RSS":"pages",
|
|
"AdmissionControl_ThresholdValue":"count",
|
|
"Indexing_Pressure_Average_Window_Throughput":"count/s",
|
|
"Cache_MaxSize":"B","IndexWriter_Memory":"B",
|
|
"Net_TCP_SSThresh":"B/flow","IO_ReadThroughput":"B/s",
|
|
"LeaderCheck_Latency":"ms","FollowerCheck_Failure":"count",
|
|
"HTTP_RequestDocs":"count","Net_TCP_Lost":"segments/flow",
|
|
"GC_Collection_Event":"count","Sched_CtxRate":"count/s",
|
|
"AdmissionControl_RejectionCount":"count","Heap_Max":"B",
|
|
"ClusterApplierService_Failure":"count",
|
|
"PublishClusterState_Failure":"count",
|
|
"Merge_CurrentEvent":"count","Indexing_Buffer":"B",
|
|
"Bitset_Memory":"B","Net_PacketDropRate4":"packets/s",
|
|
"Heap_Committed":"B","Net_PacketDropRate6":"packets/s",
|
|
"Thread_Blocked_Event":"count","GC_Collection_Time":"ms",
|
|
"Cache_Query_Miss":"count","Latency":"ms",
|
|
"Shard_State":"count","Thread_Waited_Event":"count",
|
|
"CB_ConfiguredSize":"B","ThreadPool_QueueCapacity":"count",
|
|
"CB_TrippedEvents":"count","Disk_WaitTime":"ms",
|
|
"Data_RetryingPendingTasksCount":"count",
|
|
"AdmissionControl_CurrentValue":"count",
|
|
"Flush_Event":"count","Net_TCP_RxQ":"segments/flow",
|
|
"Shard_Size_In_Bytes":"B","Thread_Waited_Time":"s/event",
|
|
"HTTP_TotalRequests":"count",
|
|
"ThreadPool_ActiveThreads":"count",
|
|
"Paging_MinfltRate":"count/s","Net_TCP_SendCWND":"B/flow",
|
|
"Cache_Request_Eviction":"count","Segments_Total":"count",
|
|
"FollowerCheck_Latency":"ms","Heap_Used":"B",
|
|
"Master_ThrottledPendingTasksCount":"count",
|
|
"CB_EstimatedSize":"B","Indexing_ThrottleTime":"ms",
|
|
"Master_PendingQueueSize":"count",
|
|
"Cache_FieldData_Size":"B","Paging_MajfltRate":"count/s",
|
|
"ThreadPool_TotalThreads":"count","ShardEvents":"count",
|
|
"Net_TCP_NumFlows":"count","Election_Term":"count"}
|
|
````
|
|
|
|
## Root cause analysis
|
|
|
|
The [root cause analysis]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/rca/index/) (RCA) framework uses the information from performance analyzer to inform administrators of the root cause of performance and availability issues that their clusters might be experiencing.
|
|
|
|
### Enable the RCA framework
|
|
|
|
To enable the RCA framework, run the following command:
|
|
|
|
```bash
|
|
curl -XPOST http://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'
|
|
```
|
|
|
|
If you encounter the `curl: (52) Empty reply from server` response, run the following command to enable RCA:
|
|
|
|
```bash
|
|
curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k
|
|
```
|
|
|
|
### Example API query and response
|
|
|
|
To request all available RCAs, run the following command:
|
|
|
|
````bash
|
|
GET localhost:9600/_plugins/_performanceanalyzer/rca
|
|
````
|
|
|
|
To request a specific RCA, run the following command:
|
|
|
|
````bash
|
|
GET localhost:9600/_plugins/_performanceanalyzer/rca?name=HighHeapUsageClusterRCA
|
|
````
|
|
|
|
The following is an example response:
|
|
|
|
```json
|
|
{
|
|
"HighHeapUsageClusterRCA": [{
|
|
"RCA_name": "HighHeapUsageClusterRCA",
|
|
"state": "unhealthy",
|
|
"timestamp": 1587426650942,
|
|
"HotClusterSummary": [{
|
|
"number_of_nodes": 2,
|
|
"number_of_unhealthy_nodes": 1,
|
|
"HotNodeSummary": [{
|
|
"host_address": "192.168.144.2",
|
|
"node_id": "JtlEoRowSI6iNpzpjlbp_Q",
|
|
"HotResourceSummary": [{
|
|
"resource_type": "old gen",
|
|
"threshold": 0.65,
|
|
"value": 0.81827232588145373,
|
|
"avg": NaN,
|
|
"max": NaN,
|
|
"min": NaN,
|
|
"unit_type": "heap usage in percentage",
|
|
"time_period_seconds": 600,
|
|
"TopConsumerSummary": [{
|
|
"name": "CACHE_FIELDDATA_SIZE",
|
|
"value": 590702564
|
|
},
|
|
{
|
|
"name": "CACHE_REQUEST_SIZE",
|
|
"value": 28375
|
|
},
|
|
{
|
|
"name": "CACHE_QUERY_SIZE",
|
|
"value": 12687
|
|
}
|
|
],
|
|
}]
|
|
}]
|
|
}]
|
|
}]
|
|
}
|
|
```
|
|
|
|
## Performance analyzer and RCA API references
|
|
|
|
### Related links
|
|
|
|
Further documentation on the use of performance analyzer and RCA can be found at the following links:
|
|
|
|
- [Performance analyzer API guide]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/api/).
|
|
- [Root cause analysis]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/rca/index/).
|
|
- [Root cause analysis API guide]({{site.url}}{{site.baseurl}}/latest/monitoring-plugins/pa/rca/api/).
|
|
- [RFC: Root cause analysis](https://github.com/opensearch-project/performance-analyzer-rca/blob/main/docs/rfc-rca.pdf). |