Add documentation that supports custom log types (#4969)

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>

_security-analytics/api-tools/index.md

@@ -19,4 +19,5 @@ The APIs for Security Analytics are separated into the following categories:
 * [Mappings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/)
 * [Alerts and findings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/alert-finding-api/)
 * [Correlation engine APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/correlation-eng/)
+* [Log type APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/log-type-api/)
 

_security-analytics/api-tools/log-type-api.md

@@ -0,0 +1,237 @@
+---
+layout: default
+title: Log type APIs
+parent: API tools
+nav_order: 56
+---
+
+# Log type APIs
+
+The log type APIs allow you to create a custom log type, search custom log types, update custom log types, and delete custom log types.
+
+
+## Create log type
+
+Creating a new custom log type involves entering a name and a description and specifying the source as `Custom`.
+
+
+### Example request
+
+```json
+POST /_plugins/_security_analytics/logtype
+{
+  "description": "custom-log-type-desc",
+  "name": "custom-log-type4",
+  "source": "Custom"
+}
+```
+{% include copy-curl.html %}
+
+
+### Example response
+
+```json
+{
+    "_id": "m98uk4kBlb9cbROIpEj2",
+    "_version": 1,
+    "logType": {
+        "name": "custom-log-type4",
+        "description": "custom-log-type-desc",
+        "source": "Custom",
+        "tags": {
+            "correlation_id": 27
+        }
+    }
+}
+```
+
+
+## Search custom log types
+
+This API allows you to search log types in the system.
+
+
+### Example request
+
+```json
+POST /_plugins/_security_analytics/logtype/_search
+{
+    "query": {
+        "match_all": {}
+    }
+}
+```
+{% include copy-curl.html %}
+
+
+### Example response
+
+```json
+{
+    "took": 3,
+    "timed_out": false,
+    "_shards": {
+        "total": 1,
+        "successful": 1,
+        "skipped": 0,
+        "failed": 0
+    },
+    "hits": {
+        "total": {
+            "value": 26,
+            "relation": "eq"
+        },
+        "max_score": 2.0,
+        "hits": [
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "s3",
+                "_score": 2.0,
+                "_source": {
+                    "name": "s3",
+                    "description": "Windows logs",
+                    "source": "Sigma",
+                    "tags": {
+                        "correlation_id": 21
+                    }
+                }
+            },
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "others_compliance",
+                "_score": 2.0,
+                "_source": {
+                    "name": "others_compliance",
+                    "description": "Compliance logs",
+                    "source": "Sigma",
+                    "tags": {
+                        "correlation_id": 4
+                    }
+                }
+            },
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "github",
+                "_score": 2.0,
+                "_source": {
+                    "name": "github",
+                    "description": "Sys logs",
+                    "source": "Sigma",
+                    "tags": {
+                        "correlation_id": 16
+                    }
+                }
+            },
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "others_application",
+                "_score": 2.0,
+                "_source": {
+                    "name": "others_application",
+                    "description": "Application logs",
+                    "source": "Sigma",
+                    "tags": {
+                        "correlation_id": 0
+                    }
+                }
+            },
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "dns",
+                "_score": 2.0,
+                "_source": {
+                    "name": "dns",
+                    "description": "Compliance logs",
+                    "source": "Sigma",
+                    "tags": {
+                        "correlation_id": 15
+                    }
+                }
+            },
+            {
+                "_index": ".opensearch-sap-log-types-config",
+                "_id": "m98uk4kBlb9cbROIpEj2",
+                "_score": 2.0,
+                "_source": {
+                    "name": "custom-log-type-updated4",
+                    "description": "custom-log-type-updated-desc",
+                    "source": "Custom",
+                    "tags": null
+                }
+            }
+        ]
+    }
+}
+```
+
+
+## Update custom log type
+
+This API allows you to update existing custom log types. Use the log type's ID in the route to specify the log type, as shown in the following example:
+
+```json
+PUT /_plugins/_security_analytics/logtype/<log_type_id>
+```
+
+
+### Example request
+
+```json
+PUT /_plugins/_security_analytics/logtype/m98uk4kBlb9cbROIpEj2
+{
+  "name": "custom-log-type4",
+  "description": "custom-log-type-updated-desc",
+  "source": "Custom"
+}
+```
+{% include copy-curl.html %}
+
+
+### Example response
+
+```json
+{
+    "_id": "m98uk4kBlb9cbROIpEj2",
+    "_version": 1,
+    "logType": {
+        "name": "custom-log-type4",
+        "description": "custom-log-type-updated-desc",
+        "source": "Custom",
+        "tags": {
+            "correlation_id": 27
+        }
+    }
+}
+```
+
+
+## Delete custom log type
+
+This API is used to delete a custom log type. Specify the log type's ID in the route to run the operation:
+
+```json
+DELETE /_plugins/_security_analytics/logtype/<log_type_id>
+```
+
+
+### Example request
+
+```json
+DELETE /_plugins/_security_analytics/logtype/m98uk4kBlb9cbROIpEj2
+```
+{% include copy-curl.html %}
+
+
+### Example response
+
+```json
+200 OK
+{
+    "_id": "m98uk4kBlb9cbROIpEj2",
+    "_version": 1
+}
+```
+
+Only custom log types can be deleted. Trying to delete a standard OpenSearch-defined log type results in an error.
+{: .note }
+

_security-analytics/index.md

@@ -33,7 +33,7 @@ For information about configuring detectors, see [Creating detectors]({{site.url
 
 ### Log types
 
-Log types provide the data used to evaluate events occurring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log sources include:
+Log types provide the data used to evaluate events occurring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log types include the following:
 * Network events
 * DNS logs
 * Apache access logs

_security-analytics/sec-analytics-config/custom-log-type.md

@@ -0,0 +1,43 @@
+---
+layout: default
+title: Creating custom log types
+parent: Setting up Security Analytics
+nav_order: 18
+---
+
+
+# Creating custom log types
+
+Log types represent the different sources of data used for threat detection in Security Analytics. In addition to the standard log types supported by Security Analytics, you can create custom log types for your threat detectors. Follow the steps provided here to create a custom log type.
+
+
+## The Log types page
+
+To navigate to the **Log types** page, select **Log types** under **Detectors** in the navigation menu. The following image shows the **Log types** landing page.
+
+<img src="{{site.url}}{{site.baseurl}}/images/Security/c-log-type.png" alt="The Log types landing page." width="85%">
+
+The table that lists the log types provides the name of the log type, its description, and identifies whether it's a standard OpenSearch-defined log type or a custom log type. The following list describes the main features found on the **Log types** page and the actions you can take:
+
+* Select the log type **Name** to open the log type's details page. The **Details** tab is shown by default. This tab includes the log type's ID. You can also select the **Detection rules** tab to show all detection rules associated with the log type.
+* In the **Actions** column, you can select the trash can icon ({::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/alerting/trash-can-icon.png" class="inline-icon" alt="trash can icon"/>{:/}) to delete a custom log type (you cannot delete a standard OpenSearch-defined log type). Follow the prompts to confirm and safely remove the custom log type.
+* Select **Create log type** in the upper-right corner of the screen to begin creating a custom log type. The **Create log type** page opens. Continue with the steps in the section that follows to create a custom log type.
+
+
+## Creating a custom log type
+
+After selecting **Create log type** on the **Log types** page, the **Create log type** page opens and provides the necessary fields for creating a new log type:
+
+1. Enter a name for the log type.
+   
+   The log type name supports characters a--z (lowercase), 0--9, hyphens, and underscores.
+   {: .note }
+   
+1. Enter a description for the log type.
+1. Select **Create log type** in the lower-right corner of the screen. The screen returns to the **Log types** page, and the new log type appears in the list of all log types. Note that the source for the new log type indicates **Custom**.
+
+
+## Log type API
+
+To perform operations for custom log types using the REST API, see [Log type APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/log-type-api/).
+

_security-analytics/sec-analytics-config/detectors-config.md

@@ -21,16 +21,16 @@ You can define a new detector by naming the detector, selecting a data source an
     When multiple data sources are selected, the logs must be of the same type. We recommend creating separate detectors for different log types.
     {: .note }
     
-1. In the **Log types and rules** section, select the log type for the data source. The system automatically populates the Sigma security rules associated with the log type.
+1. In the **Log types and rules** section, select the log type for the data source from the dropdown list. The system automatically populates the detection rules associated with the log type.
    
-   For information about creating your own detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules).
+   For information about creating your own detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules). To create your own log types, see [Creating custom log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/custom-log-type/).
    {: .note }
    
-   The following image shows the number of associated rules populated in the **Detection rules** section.
+   The example in the following image shows the number of rules associated with **network** after this log type is selected.
 
-    <img src="{{site.url}}{{site.baseurl}}/images/Security/detector_rules.png" alt="Selecting threat detector type to auto-populate rules" width="85%">
+    <img src="{{site.url}}{{site.baseurl}}/images/Security/detector-rules.png" alt="Selecting threat detector log type to auto-populate rules" width="85%">
     
-    When you select **Network events**, **CloudTrail logs**, or **S3 access logs** as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).
+    When you select **network**, **cloudtrail**, or **s3** as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).
     
     You can skip the next step for applying select rules if you are satisfied with those automatically populated by the system. Otherwise, go to the next step to select rules individually.
     {: .note }
@@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre
 
 [Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
 
-The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
+The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail logs can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
 
 To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
 {: .tip }
@@ -147,5 +147,7 @@ To set up an alert for a detector, continue with the following steps:
 ---
 ## What's next
 
-If you are ready to view findings for the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section. 
+If you are ready to view findings generated by the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section.
+
+To configure Security Analytics to identify correlations between events happening in different logs throughout your system, see [Creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/).
 

_security-analytics/sec-analytics-config/index.md

@@ -12,9 +12,10 @@ redirect_from:
 
 Before Security Analytics can begin generating findings and sending alerts, administrators must create detectors and make log data available to the system. Once detectors are able to generate findings, you can fine-tune your alerts to focus on specific areas of interest. The following steps outline the basic workflow for setting up components in Security Analytics.
 
-1. Create security detectors and alerts, and ingest log data. See [Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/) for details.
+1. Create threat detectors and alerts, and ingest log data. See [Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/) for more information.
+1. Consider [creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/) to identify connections between events and possible threats occurring in different logs throughout your system.
 1. Inspect findings generated from detector output and create any additional alerts.
-1. If desired, create custom rules by duplicating and then modifying pre-packaged rules. See [Customizing rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#customizing-rules) for details.
+1. If desired, create custom rules to better focus detectors on high-priority concerns in your system. See [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules) for more information.
 
 ## Navigate to Security Analytics
 

_security-analytics/usage/overview.md

@@ -9,8 +9,8 @@ nav_order: 25
 
 When you select **Security Analytics** from the top menu, the Overview page is displayed. The Overview page consists of five sections:
 * Findings and alert count
-* Top recent alerts
+* Recent alerts
-* Top recent findings
+* Recent findings
 * Most frequent detection rules
 * Detectors
 
@@ -35,20 +35,26 @@ You can also select the **Getting started** link to expand the Get started with
 
 The Findings and alert count section provides a graph showing data on the latest findings. Use the **Group by** dropdown list to select either **All findings** or **Log type**.
 
+<img src="{{site.url}}{{site.baseurl}}/images/Security/count.png" alt="A graph showing counts for findings and alerts." width="75%">
+
 ---
 ## Recent alerts
 
 The Recent alerts table displays recent alerts by time, trigger name, and alert severity. Select **View alerts** to go to the Alerts page.
 
+<img src="{{site.url}}{{site.baseurl}}/images/Security/recent-alerts.png" alt="A table showing the most recent alerts." width="50%">
+
 ---
 ## Recent findings
 
 The Recent findings table displays recent findings by time, rule name, rule severity, and detector. Select **View all findings** to go to the Findings page.
 
+<img src="{{site.url}}{{site.baseurl}}/images/Security/recent-findings.png" alt="A table showing the most recent findings." width="50%">
+
 ---
 ## Most frequent detection rules
 
-This section provides a graphical representation of detection rules that trigger findings most often and how they compare to others as a percentage of the whole. The rule names represented by the graph are listed to the right.
+This section provides a graphical representation of detection rules that trigger findings most often and how they compare to others as a percentage of the whole. The rule names represented by the graph are listed to the right. You can hover over each color on the graph to see details about the detection rule it represents.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/rule_graph.png" alt="The detection rule graph on the Overview page" width="50%">
 
@@ -57,3 +63,5 @@ This section provides a graphical representation of detection rules that trigger
 
 The Detectors section displays a list of available detectors by detector name, status (active/inactive), and log type. Select **View all detectors** to go to the Detectors page. Select **Create detector** to go directly to the Define detector page.
 
+<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-overview.png" alt="A table showing available detectors." width="50%">
+