Add documentation for log type updates in Security Analytics (#3066)
* fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> * fix#3018-sec-analytics-2.6 Signed-off-by: cwillum <cwmmoore@amazon.com> --------- Signed-off-by: cwillum <cwmmoore@amazon.com>
|
@ -32,14 +32,19 @@ For information on configuring detectors, see [Creating detectors]({{site.url}}{
|
||||||
### Log types
|
### Log types
|
||||||
|
|
||||||
Log types provide the data used to evaluate events occuring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log sources include:
|
Log types provide the data used to evaluate events occuring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log sources include:
|
||||||
* Netflow
|
* Network events
|
||||||
* DNS logs
|
* DNS logs
|
||||||
* Apache access logs
|
* Apache access logs
|
||||||
* Windows logs
|
* Windows logs
|
||||||
* AD/LDAP
|
* AD/LDAP logs
|
||||||
* System logs
|
* System logs
|
||||||
* AWS CloudTrail logs
|
* AWS CloudTrail logs
|
||||||
* Amazon S3 access logs
|
* Amazon S3 access logs
|
||||||
|
* Google Workspace logs
|
||||||
|
* GitHub actions
|
||||||
|
* Microsoft 365 logs
|
||||||
|
* Okta events
|
||||||
|
* Microsoft Azure logs
|
||||||
|
|
||||||
Log types are specified during the creation of detectors, including steps for mapping log fields to the detector. Security Analytics also automatically selects an appropriate set of rules based on a specific log type and populates them for the detector.
|
Log types are specified during the creation of detectors, including steps for mapping log fields to the detector. Security Analytics also automatically selects an appropriate set of rules based on a specific log type and populates them for the detector.
|
||||||
|
|
||||||
|
|
|
@ -13,18 +13,22 @@ Security Analytics provides the options and functionality to monitor and respond
|
||||||
|
|
||||||
You can define a new detector by naming the detector, selecting a data source and detector type, and specifying a detector schedule. After defining a detector, you can also configure field mappings and set up alerts. Follow the steps in this section to accomplish all three of these setup tasks.
|
You can define a new detector by naming the detector, selecting a data source and detector type, and specifying a detector schedule. After defining a detector, you can also configure field mappings and set up alerts. Follow the steps in this section to accomplish all three of these setup tasks.
|
||||||
|
|
||||||
1. On the Detectors page, select the **Create detector** button. The Define detector page opens.
|
1. On the **Threat detectors** page, choose **Create detector**. The **Define detector** page opens.
|
||||||
1. Give the detector a name and, as an option, add a description for the detector.
|
1. In **Detector details**, give the detector a name. Adding a description for the detector is optional.
|
||||||
1. In the Data source section, select the dropdown arrow and select a source for the log data.
|
1. In the **Data source** section, select the dropdown arrow and select one or multiple sources for the log data.
|
||||||
1. In the threat detection type section, select the data type. The Sigma security rules associated with the log data are automatically populated in the Detection rules section below it.
|
|
||||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/detector_rules.png" alt="Selecting threat detector type to auto-populate rules" width="80%">
|
|
||||||
|
|
||||||
You can skip the next step for mapping rules if you are satisfied with those automatically populated by the system. Otherwise, go to the next step to specify select rules.
|
When multiple data sources are selected, the logs must be of the same type. We recommend creating separate detectors for different log types.
|
||||||
{: .note }
|
{: .note }
|
||||||
|
|
||||||
1. In the **Detection rules** section, specify only those rules you want mapped to the detector.
|
1. In the **Log types and rules** section, select the log type for the data source. The Sigma security rules associated with the log data are automatically populated in the **Detection rules** section, as shown in the following image.
|
||||||
<img src="{{site.url}}{{site.baseurl}}/images/Security/select_rules.png" alt="Select or deselect rules that detector will use for findings" width="85%">
|
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/detector_rules.png" alt="Selecting threat detector type to auto-populate rules" width="80%">
|
||||||
* Use the toggle to the left of the rule name to select or deselect rules.
|
|
||||||
|
You can skip the next step for applying select rules if you are satisfied with those automatically populated by the system. Otherwise, go to the next step to select rules individually.
|
||||||
|
{: .note }
|
||||||
|
|
||||||
|
1. In the **Detection rules** section, specify only those rules you want applied to the detector, as shown in the following image.
|
||||||
|
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/select_rules.png" alt="Select or deselect rules that detector will use for findings" width="85%">
|
||||||
|
* Use the toggle to the left of **Rule name** to select or deselect rules.
|
||||||
* Use the **Log type**, **Rule severity**, and **Source** dropdown lists to filter the rules you want to select from.
|
* Use the **Log type**, **Rule severity**, and **Source** dropdown lists to filter the rules you want to select from.
|
||||||
* Use the **Search** bar to search for specific rules.
|
* Use the **Search** bar to search for specific rules.
|
||||||
|
|
||||||
|
@ -32,45 +36,49 @@ You can define a new detector by naming the detector, selecting a data source an
|
||||||
{: .tip }
|
{: .tip }
|
||||||
|
|
||||||
1. In the **Detector schedule** section, set how often the detector will run. Specify a unit of time and a corresponding number to set the interval.
|
1. In the **Detector schedule** section, set how often the detector will run. Specify a unit of time and a corresponding number to set the interval.
|
||||||
1. Select the **Next** button in the lower-right corner of the screen to continue. The **Configure field mapping** page appears.
|
1. Choose **Next** in the lower-right corner of the screen to continue. The **Configure field mapping** page appears.
|
||||||
|
|
||||||
## Step 2. Create field mappings
|
## Step 2. Create field mappings
|
||||||
|
|
||||||
The field mapping step matches field names from the rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately pass event data from the log to the detector and then use the data to trigger alerts.
|
The field mapping step matches field names from the rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately pass event data from the log to the detector and then use the data to trigger alerts.
|
||||||
|
|
||||||
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter, along with the specific detection rules, determines the list of rule field names available for the mapping. Similarly, the selected data source (log index) determines the list of log field names that are available for the mapping.
|
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter, along with the specific detection rules, determines the list of rule field names available for the mapping. Similarly, the selected data source (log index) determines the list of log source field names that are available for the mapping.
|
||||||
|
|
||||||
Because the system uses prepackaged Sigma rules for detector creation, it can automatically map important fields for a specific log type with the corresponding fields in the Sigma rules. The field mapping step presents a view of automatically mapped fields while also providing the option to customize, change, or add new field mappings. When a detector includes custom rules, you can follow this step to manually map rule field names to log field names.
|
Because the system uses prepackaged Sigma rules for detector creation, it can automatically map important fields for a specific log type with the corresponding fields in the Sigma rules. The field mapping step presents a view of automatically mapped fields while also providing the option to customize, change, or add new field mappings. When a detector includes custom rules, you can follow this step to manually map rule field names to log source field names.
|
||||||
|
|
||||||
#### A note on field names
|
#### A note on field names
|
||||||
|
|
||||||
The field mapping process requires that you are familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log fields in the index, the mapping is typically a straightforward process.
|
The field mapping process requires that you are familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log fields in the index, the mapping is typically a straightforward process.
|
||||||
|
|
||||||
Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the rule field names are derived from a Sigma rule field standard. To make them easier to identify, however, we create aliases for the Sigma rule fields based on the open source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps.
|
Security Analytics takes advantage of prepackaged Sigma rules for security event detection. Therefore, the field names are derived from a Sigma rule field standard. To make them easier to identify, however, we have created aliases for the Sigma rule fields based on the open-source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps. They appear in the **Detector field name** column of the mapping tables.
|
||||||
|
|
||||||
Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names with ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, select the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.
|
Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.
|
||||||
|
|
||||||
### Default field mappings
|
### Automatically mapped fields
|
||||||
|
|
||||||
Once you navigate to the **Configure field mapping** page, the system attempts to automatically map fields between the two sources. The **Default mapped fields** table contains mappings that the system created automatically after defining the detector. As shown in the image that follows, when the field names are similar to one another the system can successfully match the two.
|
Once you navigate to the **Configure field mapping** page, the system attempts to automatically map fields between the two sources. The **Automatically mapped fields** table contains mappings that the system created automatically after defining the detector. When the field names are similar to one another, the system can successfully match the two, as shown in the following image.
|
||||||
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/default-mappings.png" alt="Field mapping example for pending mappings" width="85%">
|
|
||||||
|
|
||||||
Although these automatic matches are normally dependable, it's still a good idea to review the mappings in the **Default mapped fields** table and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list as described in the [Pending field mappings](#pending-field-mappings) section that follows to correct the field mapping.
|
<img src="{{site.url}}{{site.baseurl}}/images/Security/automatic-mappings.png" alt="Field mapping example for automatic mappings" width="85%">
|
||||||
|
|
||||||
|
Although these automatic matches are normally dependable, it's still a good idea to review the mappings in the **Automatically mapped fields** table and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list to search for and select the correct field name. For more on matching field names, see the [Pending field mappings](#pending-field-mappings) section that follows.
|
||||||
|
|
||||||
### Pending field mappings
|
### Pending field mappings
|
||||||
|
|
||||||
The field names that are not automatically mapped appear in the **Pending field mappings** table. In this table you can manually map rule fields to log fields, as shown in the following image.
|
The field names that are not automatically mapped appear in the **Pending field mappings** table. In this table you can manually map rule fields to log source fields, as shown in the following image.
|
||||||
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/pending-mappings.png" alt="Field mapping example for pending mappings" width="85%">
|
|
||||||
|
<img src="{{site.url}}{{site.baseurl}}/images/Security/pending-mappings.png" alt="Field mapping example for pending mappings" width="85%">
|
||||||
|
|
||||||
While mapping fields, consider the following:
|
While mapping fields, consider the following:
|
||||||
* The Rule field name column lists field names based on all of the prepackaged rules associated with the selected log type.
|
* The **Detector field name** column lists field names based on all of the prepackaged rules associated with the selected log type.
|
||||||
* The log field name column includes a dropdown list for each of the rule fields. Each dropdown list contains field names extracted from the log index.
|
* The **Log source field name** column includes a dropdown list for each of the detector fields. Each dropdown list contains field names extracted from the log index.
|
||||||
* To map a rule field name to a log field name, use the dropdown arrow to open the list of log fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a mapping field** box.
|
* To map a detector field name to a log source field name, use the dropdown arrow to open the list of log source fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a mapping field** box, as shown in the following image.
|
||||||
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Field mapping example for pending mappings" width="60%">
|
|
||||||
* Once the log field name is selected and mapped to the rule field name, the icon in the Status column to the right changes to a green check mark.
|
|
||||||
* Make as many matches between field names as possible to complete an accurate mapping for rule and log fields.
|
|
||||||
|
|
||||||
After completing the mappings, select the **Next** button in the lower-right corner of the screen. The **Set up alerts** page appears and displays settings for an alert trigger.
|
<img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Field mapping example for pending mappings" width="60%">
|
||||||
|
|
||||||
|
* Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes to a green check mark.
|
||||||
|
* Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.
|
||||||
|
|
||||||
|
After completing the mappings, choose **Next** in the lower-right corner of the screen. The **Set up alerts** page appears and displays settings for an alert trigger.
|
||||||
|
|
||||||
## Step 3. Set up alerts
|
## Step 3. Set up alerts
|
||||||
|
|
||||||
|
@ -95,9 +103,9 @@ To set up an alert for a detector, continue with the following steps:
|
||||||
* Assign a level of severity for the alert to give the recipient an indication of its urgency.
|
* Assign a level of severity for the alert to give the recipient an indication of its urgency.
|
||||||
* Select a channel for the notification. Examples include Slack, Chime, or email. Select the **Manage channels** link to the right of the field to link the notification to a preferred channel.
|
* Select a channel for the notification. Examples include Slack, Chime, or email. Select the **Manage channels** link to the right of the field to link the notification to a preferred channel.
|
||||||
* Select the **Show notify message** label to expand message preferences. You can add a subject for the message and a note to inform recipients of the nature of the message.
|
* Select the **Show notify message** label to expand message preferences. You can add a subject for the message and a note to inform recipients of the nature of the message.
|
||||||
1. After configuring the conditions in the fields above, select the **Next** button in the lower-right corner of the screen. The Review and create page opens.
|
1. After configuring the conditions in the preceding fields, select **Next** in the lower-right corner of the screen. The **Review and create** page opens.
|
||||||
|
|
||||||
After reviewing the specifications for the detector, select the **Create** button in the lower-right corner of the screen to create the detector. The screen returns to the list of all detectors, and the new detector appears in the list.
|
After reviewing the specifications for the detector, select **Create** in the lower-right corner of the screen to create the detector. The screen returns to the list of all detectors, and the new detector appears in the list.
|
||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
|
|
After Width: | Height: | Size: 120 KiB |
Before Width: | Height: | Size: 53 KiB |
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 75 KiB |
Before Width: | Height: | Size: 31 KiB After Width: | Height: | Size: 44 KiB |
Before Width: | Height: | Size: 142 KiB After Width: | Height: | Size: 156 KiB |
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 92 KiB |