Add updates to creating a detector UX (#5176)

* detector draft

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* updated screen shots and fixed links

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* fix links

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* remove strange character

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* Update _security-analytics/index.md

Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

* Update _security-analytics/usage/findings.md

Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

* Update _security-analytics/sec-analytics-config/detectors-config.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

* Update _security-analytics/sec-analytics-config/detectors-config.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

* Update _security-analytics/usage/findings.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

* fixed substep formatting problem

Signed-off-by: Heather Halter <hdhalter@amazon.com>

* Update _security-analytics/sec-analytics-config/detectors-config.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>

---------

Signed-off-by: Heather Halter <hdhalter@amazon.com>
Signed-off-by: Heather Halter <HDHALTER@AMAZON.COM>
Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
This commit is contained in:
Heather Halter 2023-10-13 07:49:08 -07:00 committed by GitHub
parent c4ff0aac04
commit 346c52dd2f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 78 additions and 92 deletions

View File

@ -61,7 +61,7 @@ To learn more about findings, see [Working with findings]({{site.url}}{{site.bas
When defining a detector, you can specify certain conditions that will trigger an alert. When an event triggers an alert, the system sends a notification to a preferred channel, such as Amazon Chime, Slack, or email. The alert can be triggered when the detector matches one or multiple rules. Further conditions can be set by rule severity and tags. You can also create a notification message with a customized subject line and message body.
For information about setting up alerts, see [Step 4. Set up alerts]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-4-set-up-alerts) in the detector creation documentation. For information about managing alerts in the Alerts window, see [Working with alerts]({{site.url}}{{site.baseurl}}/security-analytics/usage/alerts/).
For information about setting up alerts, see [Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/). For information about managing alerts in the Alerts window, see [Working with alerts]({{site.url}}{{site.baseurl}}/security-analytics/usage/alerts/).
### Correlation engine

View File

@ -9,109 +9,50 @@ nav_order: 15
Security Analytics provides the options and functionality to monitor and respond to a wide range of security threats. Detectors are the essential components that determine what to look for and how to respond to those threats. This section covers their creation and configuration.
For information about working with existing detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/detectors/).
---
## Step 1. Define a detector
You can define a new detector by naming the detector and then selecting a data source and detector type. After defining a detector, you can also configure field mappings, create a detector schedule, and set up alerts. Follow the steps in this section to accomplish all of these setup tasks.
You can define a new detector by naming the detector and then selecting a data source and detector type. After defining a detector, you can configure field mappings, create a detector schedule, and set up alerts.
1. On the **Threat detectors** page, choose **Create detector**. The **Define detector** page opens.
1. In **Detector details**, give the detector a name. Adding a description for the detector is optional.
1. In the **Data source** section, select the dropdown arrow and select one or multiple sources for the log data.
When multiple data sources are selected, the logs must be of the same type. We recommend creating separate detectors for different log types.
{: .note }
1. In the **Log types and rules** section, select the log type for the data source from the dropdown list. You can find a list of supported log types in the [Supported log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/log-types/) topic.
To define a detector:
After the log type is selected, the system automatically populates the detection rules associated with the log type.
1. On the **Security Analytics** home page or the **Detectors** page, choose **Create detector**.
1. Give the detector a name and, optionally, a description.
1. In the **Data source** section, select one or more sources for the log data. Use an asterisk (*) to indicate a wildcard pattern. When selecting multiple data sources, their logs must be of the same type. We recommend creating separate detectors for different log types.
For information about creating your own detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules). To create your own log types, see [Creating custom log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/custom-log-type/).
{: .note }
The example in the following image shows the number of rules associated with **network** after this log type is selected.
1. In the **Detection** section, select a log type for the data source. For a list of supported log types, see [Supported log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/log-types/). To create your own log types, see [Creating custom log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/custom-log-type/).
When you select `network`, `cloudtrail`, or `s3` as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).
1. Expand **Detection rules** to show the list of available detection rules for the selected log type. Initially, all rules are selected by default. The following example shows the rules associated with the **windows** log type.
<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-rules.png" alt="Selecting threat detector log type to auto-populate rules" width="85%">
When you select **network**, **cloudtrail**, or **s3** as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).
You can skip the next step for applying select rules if you are satisfied with those automatically populated by the system. Otherwise, go to the next step to select rules individually.
1. Expand **Detection rules** to show the list of available detection rules for the selected log type. Initially, all rules are selected by default. The following image illustrates this.
<img src="{{site.url}}{{site.baseurl}}/images/Security/select_rules.png" alt="Select or deselect rules that the detector will use for findings" width="85%">
<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-rules.png" alt="Selecting threat detector log type to auto-populate rules" width="100%">
You can perform the following actions when viewing rules:
* Use the toggle to the left of **Rule name** to select or deselect rules.
* Use the **Rule severity** and **Source** dropdown lists to filter the rules you want to select from.
* Use the **Search** bar to search for specific rules.
To quickly select one or more known rules and dismiss others, first deselect all rules by turning off the **Rule name** toggle, then search for your target rule names and select each individually by turning its toggle on.
{: .tip }
{: .tip }
1. Review the field mappings. Field mappings allow the system to accurately pass event data from the log to the detector and then use the data to trigger alerts. For more information about field mappings, see the **About field mappings** section later in this topic.
---
## Step 2. Create field mappings
The field mapping step matches field names from the detector rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately pass event data from the log to the detector and then use the data to trigger alerts.
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter, along with the specific detection rules, determines the list of detection field names available for the mapping. Similarly, the selected data source determines the list of log source field names that are available for the mapping.
The system uses prepackaged Sigma rules for detector creation. It can automatically map important fields for a specific log type with the corresponding fields in the Sigma rules. The field mapping step presents a view of automatically mapped fields while also providing the option to customize, change, or add new field mappings. When a detector includes customized rules, you can follow this step to manually map detector rule field names to log source field names.
Because the system has the ability to automatically map field names, this step is optional. However, the more fields that can be mapped between detector fields and log source fields, the greater the accuracy of generated findings.
#### A note on field names
If you choose to perform manual field mapping, you should be familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log source fields in the index, the mapping is typically a straightforward process.
Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the field names are derived from a Sigma rule field standard. To make them easier to identify, however, we have created aliases for the Sigma rule fields based on the open-source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps. They appear in the **Detector field name** column of the mapping tables.
Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder and select the file for the specific log type. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, select the [`windows_logtype.json` file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows_logtype.json). The `raw_field` value in the file represents the Sigma rule field name in the mapping.
#### Amazon Security Lake logs
[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail logs can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
{: .tip }
### Automatically mapped fields
Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Expand **Automatically mapped fields** to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/automatic-mappings.png" alt="Field mapping example for automatic mappings" width="85%">
Although these automatic matches are normally dependable, it's still a good idea to review the mappings in the **Automatically mapped fields** table and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list to search for and select the correct field name. For more on matching field names, see the [Pending field mappings](#pending-field-mappings) section that follows.
### Pending field mappings
The field names that are not automatically mapped appear in the **Pending field mappings** table. In this table you can manually map rule fields to log source fields, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/pending-mappings.png" alt="Field mapping example for pending mappings" width="85%">
While mapping fields, consider the following:
* The **Detector field name** column lists field names based on all of the prepackaged rules associated with the selected log type.
* The **Log source field name** column includes a dropdown list for each of the detector fields. Each dropdown list contains field names extracted from the log index.
* To map a detector field name to a log source field name, use the dropdown arrow to open the list of log source fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a mapping field** box, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Mapping the log field to a detector rule field" width="60%">
* Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes from the alert icon to a check mark.
* Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.
---
## Step 3. Create a detector schedule
1. In the **Detector schedule** section, set how often the detector will run. Specify a unit of time and a corresponding number to set the interval. The following image shows that the detector runs every 3 minutes.
1. In the **Detector schedule** section, create a schedule for how often to run the detector. Specify a unit of time and a corresponding number to set the interval. The following image shows that the detector runs every 3 minutes.
<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-schedule.png" alt="Detector schedule settings to determine how often the detector runs" width="40%">
1. After specifying how often the detector will run, select **Next** in the lower-right corner of the screen. The **Set up alerts** page appears and displays settings for an alert trigger.
1. Select **Next**. The **Set up alerts** page appears and displays settings for an alert trigger.
---
## Step 4. Set up alerts
## Step 2. Set up alerts
The fourth step in creating a detector involves setting up alerts. Alerts are configured to create triggers that, when matched with a set of detection rule criteria, send notifications of possible security events. You can select rule names, rule severity, and tags in any combination to define a trigger. Once a trigger is defined, the alert setup lets you choose the channel on which to be notified and provides options for customizing a message for the notification.
The second step in creating a detector involves setting up alerts. Alerts are configured to create triggers that, when matched with a set of detection rule criteria, send notifications of possible security events. You can select rule names, rule severity, and tags in any combination to define a trigger. Once a trigger is defined, the alert setup lets you choose the channel on which to be notified and provides options for customizing a message for the notification.
At least one alert condition is required before a detector can begin generating findings.
{: .note }
@ -120,7 +61,7 @@ You can also configure alerts from the **Findings** window. To see how to set up
To set up an alert for a detector, continue with the following steps:
1. In the **Trigger name** box, enter a name for the trigger.
1. In the **Trigger name** box, optionally enter a name for the trigger or edit the default name.
1. To define rule matches for the alert, select security rules, severity levels, and tags.
<img src="{{site.url}}{{site.baseurl}}/images/Security/alert_rules.png" alt="Defining an alert" width="70%">
@ -141,9 +82,7 @@ To set up an alert for a detector, continue with the following steps:
* Expand **Show notify message** to show message preferences. The message subject and message body are populated with details about the current alert configuration. You can edit these text fields to customize the message. Beneath the message body text box, you can select **Generate message** to populate more details in the message, such as rule names, rule severity levels, and rule tags.
* Select **Add another alert trigger** to configure an additional alert.
1. After configuring the conditions in the preceding fields, select **Next** in the lower-right corner of the screen. The **Review and create** page opens.
1. Review the specifications for the detector and select **Create detector** in the lower-right corner of the screen. The detector details for the new detector are displayed. When you navigate to the main **Threat detectors** page, the new detector appears in the list.
1. After configuring the conditions in the preceding fields, select **Create detector** in the lower-right corner of the screen.
## Integrated Alerting plugin workflows
@ -154,9 +93,54 @@ You can change the behavior of automatically generated composite monitors by ena
For more information about composite monitors and their workflows, see [Composite monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/composite-monitors/).
---
## About field mappings
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter, along with the specific detection rules, determines the list of detection field names available for the mapping. Similarly, the selected data source determines the list of log source field names that are available for the mapping.
The system uses prepackaged Sigma rules for detector creation. It can automatically map important fields for a specific log type to the corresponding fields in the Sigma rules. The field mapping step presents a view of automatically mapped fields while also providing the option to customize, change, or add new field mappings. When a detector includes customized rules, you can follow this step to manually map detector rule field names to log source field names.
Because the system has the ability to automatically map field names, this step is optional. However, the more fields that can be mapped between detector fields and log source fields, the greater the accuracy of generated findings.
### A note on field names
If you choose to perform manual field mapping, you should be familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log source fields in the index, the mapping is typically a straightforward process.
Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the field names are derived from a Sigma rule field standard. To make them easier to identify, however, we have created aliases for the Sigma rule fields based on the open-source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps. They appear in the **Detector field name** column of the mapping tables.
Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder and select the file for the specific log type. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, select the [`windows_logtype.json` file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows_logtype.json). The `raw_field` value in the file represents the Sigma rule field name in the mapping.
### Amazon Security Lake logs
[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail logs can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
{: .tip }
### Automatically mapped fields
Once you select a data source and log type, the system attempts to automatically map fields between the log and rule fields. Switch to the **Mapped fields** tab to show the list of these mappings. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/automatic-mappings.png" alt="Field mapping example for automatic mappings" width="85%">
Although these automatic matches are normally dependable, it's still a good idea to review the mappings in the **Mapped fields** table and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list to search for and select the correct field name. For more information about matching field names, see the following section.
### Available fields
The field names that are not automatically mapped appear in the **Available fields** table. In this table you can manually map detection rule fields to data source fields, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/pending-mappings.png" alt="Field mapping example for available mappings" width="85%">
While mapping fields, consider the following:
* The **Detection rule field** column lists field names based on all of the prepackaged rules associated with the selected log type.
* The **Data source field** column includes a dropdown list for each of the detector fields. Each dropdown list contains field names extracted from the log index.
* To map a detector field name to a log source field name, use the dropdown arrow to open the list of log source fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a data source field** box.
* Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes from the alert icon to a check mark.
* Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.
## What's next
If you are ready to view findings generated by the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section.
To configure Security Analytics to identify correlations between events happening in different logs throughout your system, see [Creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/).
To configure Security Analytics to identify correlations between events happening in different logs throughout your system, see [Working with correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/).

View File

@ -53,3 +53,5 @@ Threat detector actions allow you to stop and start detectors or delete a detect
1. Select **Delete** in the dropdown list. The Delete detector popup window opens and asks you to verify that you want to delete the detector or detectors.
1. Select **Cancel** to decline the action. Select **Delete detector** to delete the detector or detectors permanently from the list.
## Related articles
[Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/)

View File

@ -52,7 +52,7 @@ Use the **Rule severity** dropdown list to filter the list of findings by severi
The **Actions** column includes two options for each finding:
* The diagonal arrow provides a way to open the [**Finding details**](#finding-details) pane, which describes the finding according to parameters defined when creating the detector and includes the document that generated the finding.
* The bell icon allows you to open the **Create detector alert trigger** pane, where you can quickly set up an alert for the specific finding and modify rules and their conditions as required.
For information about setting up alerts, see [Step 4. Set up alerts]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-4-set-up-alerts) in the detector creation documentation.
For information about setting up alerts, see [Step 2. Set up alerts]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-2-set-up-alerts) in the detector creation documentation.
### Finding details
@ -69,7 +69,7 @@ The **Finding details** pane contains specific information about the finding, in
<img src="{{site.url}}{{site.baseurl}}/images/Security/findings2.png" alt="popup window prompting users to create an index pattern" width="60%">
1. In the **Create index pattern to view documents** window, the index pattern name is automatically populated. Enter the appropriate time field from the log index used to determine the timing for log events. For information on mapping log fields to detector fields, see [Step 2. Create field mappings]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/#step-2-create-field-mappings). Choose **Create index pattern**. The **Create index pattern to view documents** confirmation window opens.
1. In the **Create index pattern to view documents** window, the index pattern name is automatically populated. Enter the appropriate time field from the log index used to determine the timing for log events. Choose **Create index pattern**. The **Create index pattern to view documents** confirmation window opens.
1. Select **View surrounding documents** in the confirmation window. The **Discover** panel opens, as shown in the following image.
<img src="{{site.url}}{{site.baseurl}}/images/Security/findings4.png" alt="Discover panel with surrounding documents" width="85%">

Binary file not shown.

Before

Width:  |  Height:  |  Size: 82 KiB

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 39 KiB

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 111 KiB