Add correlation engine to Security Analytics documentation (#3814)
* fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3566 correlation engine Signed-off-by: cwillum <cwmmoore@amazon.com> --------- Signed-off-by: cwillum <cwmmoore@amazon.com>
This commit is contained in:
Chris Moore
GitHub
parent
a4bb42e170
commit
cffae76ac2
|
|
@ -19,6 +19,7 @@ As part of the OpenSearch Project, Security Analytics exists in the open source
|
|||
|
||||
If you would like to leave feedback that could help improve Security Analytics, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
|
||||
|
||||
|
||||
## Components and concepts
|
||||
|
||||
Security Analytics includes a number of tools and features elemental to its operation. The major components that compose the plugin are summarized in the following sections.
|
||||
|
|
@ -66,7 +67,19 @@ When defining a detector, you can specify certain conditions that will trigger a
|
|||
|
||||
For information on setting up alerts, see [Step 3. Set up alerts]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-3-set-up-alerts) in detector creation documentation. For information on managing alerts on the Alerts window, see [Working with alerts]({{site.url}}{{site.baseurl}}/security-analytics/usage/alerts/).
|
||||
|
||||
### Correlation engine
|
||||
|
||||
The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
|
||||
{: .warning }
|
||||
|
||||
The correlation engine gives Security Analytics the ability to compare findings from different log types and draw correlations between them. This facilitates understanding of the relationships between findings from different systems in an infrastructure and increases confidence that an event is meaningful and requires attention.
|
||||
|
||||
The correlation engine uses correlation rules to define threat scenarios involving different log types. It can then perform queries on logs to match relevant findings from those different log sources. To depict relationships between events occurring in different logs, a correlation graph provides a visual representation of findings, their connections, and the proximity of those connections. While the correlation rules define what threat scenarios to look for, the graph provides a visualization that helps you identify the relationships between different findings in a chain of security events.
|
||||
|
||||
To learn more about defining threat scenarios for correlation rules, see [Creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/). To learn more about using the correlation graph, see [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/).
|
||||
|
||||
|
||||
## First steps
|
||||
|
||||
To get started with Security Analytics you need to define detectors, ingest log data, generate findings, and configure alerts. See [Setting up Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/index/) to begin configuring the platform to meet your objectives.
|
||||
To get started with Security Analytics, you need to define detectors, ingest log data, generate findings, define correlation rules, and configure alerts. See [Setting up Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/index/) to begin configuring the platform to meet your objectives.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
layout: default
|
||||
title: Creating correlation rules
|
||||
parent: Setting up Security Analytics
|
||||
nav_order: 16
|
||||
---
|
||||
|
||||
# Creating correlation rules
|
||||
|
||||
The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
|
||||
{: .warning }
|
||||
|
||||
Correlation rules allow you to define threat scenarios involving multiple systems in an infrastructure by matching the signatures of threat events occuring in different log types. Once a rule contains at least two different log sources and the preferred fields and field values that define an intended threat secenario, the correlation engine can query the indexes specified in the correlation rule and identify any correlations between the findings.
|
||||
|
||||
|
||||
## Configuring rules
|
||||
|
||||
Having at least two data sources in the rule configuration is the basis for making connections between different systems in an infrastructure and identifying correlations. Therefore, a minimum of two queries is required for each correlation rule. However, you can include more than two queries to better define a threat scenario and look for correlations between multiple systems. Follow these steps to create a correlation rule:
|
||||
|
||||
1. Begin by selecting **Security Analytics** in the OpenSearch Dashboards main menu. Then select **Correlation rules** from the Security Analytics menu on the left side of the screen. The **Correlation rules** page is displayed, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/create-corr-rule.png" alt="The correlation rules page" width="85%">
|
||||
|
||||
1. Select **Create correlation rule**. The **Create correlation rule** window opens.
|
||||
1. In the **Correlation rule details** field, enter a name for the rule, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-rule-config1.png" alt="The correlation rule name" width="50%">
|
||||
|
||||
1. The **Correlation queries** field contains two dropdown lists. In the **Select index** dropdown list, specify an index or index pattern for the data source. In the **Log type** dropdown list, specify the log type associated with the index, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-rule-config2.png" alt="The data source and log type for the query" width="45%">
|
||||
|
||||
1. In the **Field** dropdown list, specify a log field. In the **Field value** text box, enter a value for the field, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-rule-config3.png" alt="The field and field value for the query" width="45%">
|
||||
|
||||
1. To add more fields to the query, select **Add field**.
|
||||
1. After configuring the first query, repeat the previous step to configure a second query. You can select **Add query** at the bottom of the window to add more queries for the rule, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-rule-config4.png" alt="A second query for the correlation rule" width="50%">
|
||||
|
||||
1. Once the rule is complete, select **Create correlation rule** in the lower-right corner of the window. OpenSearch creates a new rule, the screen returns to the **Correlation rules** window, and the new rule appears in the table of correlation rules.
|
||||
|
||||
|
||||
## Setting a time window
|
||||
|
||||
The Cluster Settings API allows you to correlate findings within a set time window. For example, if your time window is three minutes, the system attempts to correlate findings defined in the threat scenario only when they occur within three minutes of one another. By default, the time window is five minutes. For more information about the Cluster Settings API, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/).
|
||||
|
||||
### Example request
|
||||
|
||||
The following PUT call sets the time window to two minutes:
|
||||
|
||||
```json
|
||||
PUT /_cluster/settings
|
||||
{
|
||||
"transient": {
|
||||
"plugins.security_analytics.correlation_time_window": "2m"
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
|
||||
## Next steps
|
||||
|
||||
After creating detectors and correlation rules, you can use the correlation graph to observe the correlations between findings from different log sources. For information about working with the correlation graph, see [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/).
|
||||
|
||||
|
|
@ -78,7 +78,7 @@ While mapping fields, consider the following:
|
|||
* The **Log source field name** column includes a dropdown list for each of the detector fields. Each dropdown list contains field names extracted from the log index.
|
||||
* To map a detector field name to a log source field name, use the dropdown arrow to open the list of log source fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a mapping field** box, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Field mapping example for pending mappings" width="60%">
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Field mapping example for pending mappings" width="60%">
|
||||
|
||||
* Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes to a green check mark.
|
||||
* Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
layout: default
|
||||
title: Working with alerts
|
||||
parent: Using Security Analytics
|
||||
nav_order: 45
|
||||
nav_order: 46
|
||||
---
|
||||
|
||||
# Working with alerts
|
||||
|
|
|
|||
|
|
@ -0,0 +1,50 @@
|
|||
---
|
||||
layout: default
|
||||
title: Working with the correlation graph
|
||||
parent: Using Security Analytics
|
||||
nav_order: 45
|
||||
---
|
||||
|
||||
# Working with the correlation graph
|
||||
|
||||
The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
|
||||
{: .warning }
|
||||
|
||||
The correlation graph is a security findings knowledge graph. It provides a visualization of information generated by the correlation engine and allows you to focus on specific correlations and inspect them in greater detail. Information on the graph includes findings by log type, the severity levels for the findings, the correlations drawn between findings, and the relevance of the correlations, among other details. You can also manipulate the graph to gain further insight into specific events of interest. This includes filtering findings by date and time, zooming in on the relationship between specific findings and their correlations, and filtering by log type and severity level. Use this section to learn more about using the graph.
|
||||
|
||||
## Acccessing the graph
|
||||
|
||||
Begin by selecting **Security Analytics** in the OpenSearch Dashboards main menu. Then select **Correlations** from the Security Analytics menu on the left side of the screen. The **Correlations** page is displayed, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-graph.png" alt="The correlation graph" width="85%">
|
||||
|
||||
## Interpreting the graph
|
||||
|
||||
The graph displays findings as nodes with colored borders expressing their severity level. A three-letter abbreviation inside the node indicates the log type. The lines that connect the findings represent the correlations between them. A heavy line indicates a strong correlation, while a light line shows a weaker connection.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-graph-detail.png" alt="The correlation graph" width="40%">
|
||||
|
||||
## Using the graph
|
||||
|
||||
You can control which findings are displayed on the graph by filtering by severity level, log type, and time filter. The time filter controls the findings that appear on the graph by setting a date range in which they were generated.
|
||||
* Use the **Severity** dropdown list to select which findings appear on the graph according to their severity level. The number beside the list name indicates how many severity levels are being shown on the graph.
|
||||
* Use the **Log types** dropdown list to select which log types to show on the graph. The number beside the list name indicates how many log types are being shown on the graph.
|
||||
* Select **Reset filters** to return the dropdown lists to their default settings, showing all items.
|
||||
* Use the time filter to set the date range and show only those findings that were generated within that time span. Select **Refresh** to bring the current number of findings up to date.
|
||||
|
||||
You can focus on a particular area of the graph to look at correlations associated with a specific finding by selecting the finding on the graph. The graph then changes to show only the selected finding along with the constellation of findings correlated to it, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-drill-dn.png" alt="Zooming in on a specific finding on the graph" width="40%">
|
||||
|
||||
After narrowing the focus of the graph, informational cards for each of the findings appear on the right-hand side of the screen. The selected finding appears at the top of the cards, and the correlated findings are listed below it in order of their correlation relevance, represented by a correlation score, as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-cards.png" alt="Zooming in on a specific finding on the graph" width="30%">
|
||||
|
||||
You can select one of the correlated findings on the graph to shift the perspective of the correlation relationships. This sends the newly selected finding to the top of the informational cards and displays the other findings as relative correlations.
|
||||
|
||||
The cards display the following details about each finding:
|
||||
* The severity level of the finding: 1, critical; 2, high; 3, medium; 4, low; 5, informational.
|
||||
* A correlation score for correlated findings. The score is based on the proximity of relevant findings in the threat scenario defined by the correlation rule.
|
||||
* The detection rule that generated the finding.
|
||||
* For correlated findings, the correlation rule used to associate it with the selected finding.
|
||||
|
||||
|
|
@ -48,15 +48,17 @@ The **Findings** list displays all findings according to the time of the finding
|
|||
Use the **Rule severity** dropdown list to filter the list of findings by severity. Use the **log type** dropdown list to filter the list by log type.
|
||||
|
||||
The **Actions** column includes two options for each finding:
|
||||
* The diagonal arrow provides a way to open the **Finding details** pane, which describes the finding by parameters defined when creating the detector and includes the document that generated the finding.
|
||||
* The diagonal arrow provides a way to open the [**Finding details**](#finding-details) pane, which describes the finding according to parameters defined when creating the detector and includes the document that generated the finding.
|
||||
* The bell icon allows you to open the **Create detector alert trigger** pane, where you can quickly set up an alert for the specific finding and modify rules and their conditions as required.
|
||||
For details on setting up an alert, see [Step 3. Set up alerts]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-3-set-up-alerts) in detector creation documentation.
|
||||
|
||||
### Finding details
|
||||
|
||||
Each finding in the list also includes a **Finding ID**. In addition to using the diagonal arrow in **Actions**, you can select the ID to open the **Finding details** pane. An example of **Finding details** is shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/findings1.png" alt="Finding details pane" width="60%">
|
||||
|
||||
### Viewing surrounding documents
|
||||
#### Viewing surrounding documents
|
||||
|
||||
The **Finding details** pane contains specific information about the finding, including the document that generated the finding. To investigate the series of events that led to the finding or followed the finding, you can select **View surrounding documents** to open the document in the **Discover** panel and view other documents preceding or following it.
|
||||
|
||||
|
|
@ -73,3 +75,14 @@ The **Finding details** pane contains specific information about the finding, in
|
|||
The **Discover** panel displays the document that generated the finding with a highlighted background. Other documents that came either before or after the event are also displayed.
|
||||
|
||||
For details about working with **Discover** in OpenSearch Dashboards, see [Exploring data]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/).
|
||||
|
||||
#### Viewing correlated findings
|
||||
|
||||
Correlations between findings are generated by the correlation engine, which is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
|
||||
{: .warning }
|
||||
|
||||
To see how the finding is correlated with other findings, select the **Correlations** tab. Correlations are relationships between findings that express a particular threat scenario involving multiple log types. Information in the **Correlated findings** table shows the time at which a correlated finding was generated, a finding's ID, the log type used to generate the finding, its threat severity, and the correlation score---a measure of its proximity to the reference finding---as shown in the following image.
|
||||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/corr-details-findings.png" alt="A table of correlated findings with respect to the reference finding" width="60%">
|
||||
|
||||
You can select **View correlations graph** to visualize correlations between the findings. For more information about using the correlation graph, see [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/).
|
||||
|
|
|
|||
|
|
@ -16,5 +16,6 @@ After creating detectors and generating findings, functionality within the sever
|
|||
* [Working with detectors]({{site.url}}{{site.baseurl}}/security-analytics/usage/detectors/)
|
||||
* [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/)
|
||||
* [Working with rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/)
|
||||
* [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/)
|
||||
* [Working with alerts]({{site.url}}{{site.baseurl}}/security-analytics/usage/alerts/)
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 129 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 269 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
Loading…
Reference in New Issue