Fix correlation engine documentation by removing all instances of the experimental flag for 2.9 (#4635)

* fix#4631 correlation rule updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4631 correlation rule updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4631 correlation rule updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4631 correlation rule updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4631 correlation rule updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>

_security-analytics/api-tools/alert-finding-api.md

@@ -10,6 +10,7 @@ nav_order: 50
 
 The following APIs can be used for tasks related to alerts and findings.
 
+---
 ## Get Alerts
 
 Provides an option for retrieving alerts related to a specific detector type or detector ID.

_security-analytics/api-tools/correlation-eng.md

@@ -10,7 +10,7 @@ nav_order: 55
 
 Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain time window, and perform other tasks.
 
-
+---
 ## Create correlation rules between log types
 
 This API is used to create correlation rules:
@@ -98,7 +98,7 @@ POST /_plugins/_security_analytics/correlation/rules
 | :--- | :--- |:--- |
 | `_id` | String | The Id for the new rule. |
 
-
+---
 ## List all findings and their correlations within a time window
 
 This API provides a list of all findings and their correlations within a specified time window:
@@ -149,7 +149,7 @@ GET /_plugins/_security_analytics/correlations?start_timestamp=1689289210000&end
 | `logType2` | String | The log type associated with the second finding. |
 | `rules` | Array | A list of correlation rule IDs associated with the correlated findings. |
 
-
+---
 ## List correlations for a finding belonging to a log type
 
 This API is used to list correlations for specific findings and the log types associated with them:

_security-analytics/api-tools/detector-api.md

@@ -9,6 +9,7 @@ nav_order: 35
 
 The following APIs can be used for a number of tasks related to detectors, from creating detectors to updating and searching for detectors. Many API calls use the detector ID in the request, which can be retrieved using the [Search Detector API](#search-detector-api).
 
+---
 ## Create Detector API
 
 Creates a new detector.

_security-analytics/api-tools/mappings-api.md

@@ -9,6 +9,7 @@ nav_order: 45
 
 The following APIs can be used for a number of tasks related to mappings, from creating to getting and updating mappings.
 
+---
 ## Get Mappings View
 
 

_security-analytics/api-tools/rule-api.md

@@ -9,6 +9,7 @@ nav_order: 40
 
 The following APIs can be used for a number of tasks related to rules, from searching for pre-packaged rules to creating and updating custom rules.
 
+---
 ## Create Custom Rule
 
 The Create Custom Rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).

_security-analytics/index.md

@@ -20,7 +20,7 @@ As part of the OpenSearch Project, Security Analytics exists in the open source
 
 If you would like to leave feedback that could help improve Security Analytics, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
 
-
+---
 ## Components and concepts
 
 Security Analytics includes a number of tools and features elemental to its operation. The major components that compose the plugin are summarized in the following sections.
@@ -47,6 +47,7 @@ Log types provide the data used to evaluate events occurring in a system. OpenSe
 * Microsoft 365 logs
 * Okta events
 * Microsoft Azure logs
+* VPC Flow logs
 
 Log types are specified during the creation of detectors, including steps for mapping log fields to the detector. Security Analytics also automatically selects an appropriate set of rules based on a specific log type and populates them for the detector.
 
@@ -70,16 +71,13 @@ For information about setting up alerts, see [Step 4. Set up alerts]({{site.url}
 
 ### Correlation engine
 
-The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
-{: .warning }
-
 The correlation engine gives Security Analytics the ability to compare findings from different log types and draw correlations between them. This facilitates understanding of the relationships between findings from different systems in an infrastructure and increases confidence that an event is meaningful and requires attention.
 
 The correlation engine uses correlation rules to define threat scenarios involving different log types. It can then perform queries on logs to match relevant findings from those different log sources. To depict relationships between events occurring in different logs, a correlation graph provides a visual representation of findings, their connections, and the proximity of those connections. While the correlation rules define what threat scenarios to look for, the graph provides a visualization that helps you identify the relationships between different findings in a chain of security events.
 
 To learn more about defining threat scenarios for correlation rules, see [Creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/). To learn more about using the correlation graph, see [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/).
 
-
+---
 ## First steps
 
 To get started with Security Analytics, you need to define detectors, ingest log data, generate findings, define correlation rules, and configure alerts. See [Setting up Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/index/) to begin configuring the platform to meet your objectives.

_security-analytics/sec-analytics-config/correlation-config.md

@@ -7,12 +7,9 @@ nav_order: 16
 
 # Creating correlation rules
 
-The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
-{: .warning }
-
 Correlation rules allow you to define threat scenarios involving multiple systems in an infrastructure by matching the signatures of threat events occurring in different log types. Once a rule contains at least two different log sources and the preferred fields and field values that define an intended threat scenario, the correlation engine can query the indexes specified in the correlation rule and identify any correlations between the findings.
 
-
+---
 ## Configuring rules
 
 Having at least two data sources in the rule configuration is the basis for making connections between different systems in an infrastructure and identifying correlations. Therefore, a minimum of two queries is required for each correlation rule. However, you can include more than two queries to better define a threat scenario and look for correlations between multiple systems. Follow these steps to create a correlation rule:
@@ -39,9 +36,9 @@ Having at least two data sources in the rule configuration is the basis for maki
   
    <img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-rule-config4.png" alt="A second query for the correlation rule" width="50%">
 
-1. Once the rule is complete, select **Create correlation rule** in the lower-right corner of the window. OpenSearch creates a new rule, the screen returns to the **Correlation rules** window, and the new rule appears in the table of correlation rules.
-
+1. Once the rule is complete, select **Create correlation rule** in the lower-right corner of the window. OpenSearch creates a new rule, the screen returns to the **Correlation rules** window, and the new rule appears in the table of correlation rules. To edit the rule, select the rule name in the **Name** column. The **Edit correlation rule** window opens.
 
+---
 ## Setting a time window
 
 The Cluster Settings API allows you to correlate findings within a set time window. For example, if your time window is three minutes, the system attempts to correlate findings defined in the threat scenario only when they occur within three minutes of one another. By default, the time window is five minutes. For more information about the Cluster Settings API, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/).
@@ -60,7 +57,7 @@ PUT /_cluster/settings
 ```
 {% include copy-curl.html %}
 
-
+---
 ## Next steps
 
 After creating detectors and correlation rules, you can use the correlation graph to observe the correlations between findings from different log sources. For information about working with the correlation graph, see [Working with the correlation graph]({{site.url}}{{site.baseurl}}/security-analytics/usage/correlation-graph/). 

_security-analytics/sec-analytics-config/detectors-config.md

@@ -9,7 +9,7 @@ nav_order: 15
 
 Security Analytics provides the options and functionality to monitor and respond to a wide range of security threats. Detectors are the essential components that determine what to look for and how to respond to those threats. This section covers their creation and configuration. 
 
-
+---
 ## Step 1. Define a detector
 
 You can define a new detector by naming the detector, selecting a data source and detector type, and specifying a detector schedule. After defining a detector, you can also configure field mappings and set up alerts. Follow the steps in this section to accomplish all three of these setup tasks.
@@ -41,7 +41,7 @@ You can define a new detector by naming the detector, selecting a data source an
     To quickly select one or more known rules and dismiss others, first deselect all rules by turning off the **Rule name** toggle, then search for your target rule names and select each individually by turning its toggle on.
     {: .tip }
     
-
+---
 ## Step 2. Create field mappings
 
 The field mapping step matches field names from the detector rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately pass event data from the log to the detector and then use the data to trigger alerts.
@@ -93,7 +93,7 @@ While mapping fields, consider the following:
 * Once the log source field name is selected and mapped to the detector field name, the icon in the **Status** column to the right changes from the alert icon to a check mark.
 * Make as many matches between field names as possible to complete an accurate mapping for the detector and log source fields.
 
-
+---
 ## Step 3. Create a detector schedule
 
 1. In the **Detector schedule** section, set how often the detector will run. Specify a unit of time and a corresponding number to set the interval. The following image shows that the detector runs every 3 minutes.
@@ -102,7 +102,7 @@ While mapping fields, consider the following:
     
 1. After specifying how often the detector will run, select **Next** in the lower-right corner of the screen. The **Set up alerts** page appears and displays settings for an alert trigger.
 
-
+---
 ## Step 4. Set up alerts
 
 The fourth step in creating a detector involves setting up alerts. Alerts are configured to create triggers that, when matched with a set of detection rule criteria, send notifications of possible security events. You can select rule names, rule severity, and tags in any combination to define a trigger. Once a trigger is defined, the alert setup lets you choose the channel on which to be notified and provides options for customizing a message for the notification.
@@ -139,7 +139,7 @@ To set up an alert for a detector, continue with the following steps:
 
 1. Review the specifications for the detector and select **Create detector** in the lower-right corner of the screen. The detector details for the new detector are displayed. When you navigate to the main **Threat detectors** page, the new detector appears in the list.
 
-
+---
 ## What's next
 
 If you are ready to view findings for the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section. 

_security-analytics/security.md

@@ -9,12 +9,12 @@ has_children: false
 
 You can use OpenSearch Security with Security Analytics to assign user permissions and manage the actions that users can and cannot perform. For example, you might want one group of users to be able to create, update, or delete detectors and another group of users to only view detectors. You may want still another group to be able to receive and acknowledge alerts but to be prevented from performing other tasks. The OpenSearch Security framework allows you to control the level of access users have to Security Analytics functionality.
 
-
+---
 ## Security Analytics system indexes
 
 Security Analytics indexes are protected as system indexes and treated differently than other indexes in a cluster. System indexes store configurations and other system settings and, for that reason, cannot be modified using the REST API or the OpenSearch Dashboards interface. Only a user with a TLS [admin certificate]({{site.url}}{{site.baseurl}}/security/configuration/tls/#configuring-admin-certificates) can access system indexes. For more information about working with this type of index, see [System indexes]({{site.url}}{{site.baseurl}}/security/configuration/system-indices/).
 
-
+---
 ## Basic permissions
 
 As an administrator, you can use OpenSearch Dashboards or the Security REST API to assign specific permissions to users based on the specific APIs they need to access. For a list of supported APIs, see [API tools]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/index/).
@@ -23,7 +23,7 @@ OpenSearch Security has three built-in roles that cover most Security Analytics
 
 If these roles don't meet your needs, mix and match individual Security Analytics [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/#security-analytics-permissions) to suit your use case. Each action corresponds to an operation in the REST API. For example, the `cluster:admin/opensearch/securityanalytics/detector/delete` permission allows you to delete detectors.
 
-
+---
 ## (Advanced) Limit access by backend role
 
 You can use backend roles to configure fine-grained access to individual detectors based on roles. For example, backend roles can be assigned to users working in different departments of an organization so that they can view only those detectors owned by the departments in which they work.
@@ -90,7 +90,7 @@ PUT /_plugins/_security/api/rolesmapping/security_analytics_full_access
 
 However, because they have different backend roles, `alice` and `bob` cannot view each other's detectors or their results.
 
-
+---
 ## A note on using fine-grained access control with the plugin
 
 When a trigger generates an alert, the detector configurations, the alert itself, and any notifications that are sent to a channel may include metadata describing the index being queried. By design, the plugin must extract the data and store it as metadata outside of the index. [Document-level security]({{site.url}}{{site.baseurl}}/security/access-control/document-level-security) (DLS) and [field-level security]({{site.url}}{{site.baseurl}}/security/access-control/field-level-security) (FLS) access controls are designed to protect the data in the index. But once the data is stored outside the index as metadata, users with access to the detector and monitor configurations, alerts, and their notifications will be able to view this metadata and possibly infer the contents and quality of data in the index, which would otherwise be concealed by DLS and FLS access control.

_security-analytics/usage/alerts.md

@@ -13,6 +13,7 @@ The Alerts window includes features for viewing and working with alerts. The two
 
 You can select the **Refresh** button at any time to refresh information on the Alerts page.
 
+---
 ## The Alerts graph
 
 The Alerts graph can display alerts by their status or severity. Use the **Group by** dropdown list to specify either Alert status or Alert severity.
@@ -38,6 +39,7 @@ When one of the commonly used windows of time is selected, you can select the **
 
 As one more alternative, you can select an option from the **Recently used date ranges** section to go back to a previous setting.
 
+---
 ## The Alerts list
 
 The Alerts list displays all findings according to the time when the alert was triggered, the alert's trigger name, the detector that triggered the alert, the alert status, and alert severity.

_security-analytics/usage/correlation-graph.md

@@ -7,23 +7,23 @@ nav_order: 45
 
 # Working with the correlation graph
 
-The correlation engine is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
-{: .warning }
-
 The correlation graph is a security findings knowledge graph. It provides a visualization of information generated by the correlation engine and allows you to focus on specific correlations and inspect them in greater detail. Information on the graph includes findings by log type, the severity levels for the findings, the correlations drawn between findings, and the relevance of the correlations, among other details. You can also manipulate the graph to gain further insight into specific events of interest. This includes filtering findings by date and time, zooming in on the relationship between specific findings and their correlations, and filtering by log type and severity level. Use this section to learn more about using the graph. 
 
+---
 ## Acccessing the graph
 
 Begin by selecting **Security Analytics** in the OpenSearch Dashboards main menu. Then select **Correlations** from the Security Analytics menu on the left side of the screen. The **Correlations** page is displayed, as shown in the following image.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-graph.png" alt="The correlation graph" width="85%">
 
+---
 ## Interpreting the graph
 
 The graph displays findings as nodes with colored borders expressing their severity level. A three-letter abbreviation inside the node indicates the log type. The lines that connect the findings represent the correlations between them. A heavy line indicates a strong correlation, while a light line shows a weaker connection.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-graph-detail.png" alt="The correlation graph" width="40%">
 
+---
 ## Using the graph
 
 You can control which findings are displayed on the graph by filtering by severity level, log type, and time filter. The time filter controls the findings that appear on the graph by setting a date range in which they were generated.
@@ -36,7 +36,7 @@ You can focus on a particular area of the graph to look at correlations associat
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-drill-dn.png" alt="Zooming in on a specific finding on the graph" width="40%">
 
-After narrowing the focus of the graph, informational cards for each of the findings appear on the right-hand side of the screen. The selected finding appears at the top of the cards, and the correlated findings are listed below it in order of their correlation relevance, represented by a correlation score, as shown in the following image.
+After narrowing the focus of the graph, informational cards for each of the findings appear on the right-hand side of the screen. The selected finding appears at the top of the cards, and the correlated findings are listed underneath it in order of their correlation relevance, represented by a correlation score, as shown in the following image.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/sec-analytics/corr-cards.png" alt="Zooming in on a specific finding on the graph" width="30%">
 

_security-analytics/usage/detectors.md

@@ -11,6 +11,7 @@ After creating a detector, it appears on the Threat detectors page along with ot
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/threat-detector.png" alt="Threat detector page" width="60%">
 
+---
 ## Threat detector list
 
 The list of threat detectors includes the search bar, the **Status** dropdown list, and the **Log type** dropdown list.
@@ -34,6 +35,7 @@ To edit a detector, begin by selecting the link to the detector in the Detector
 After you select the **Alert triggers** tab, you also have the option to add additional alerts for the detector by selecting **Add another alert condition** at the bottom of the page.
 {: .tip }
 
+---
 ## Detector actions
 
 Threat detector actions allow you to stop and start detectors or delete a detector. To enable actions, first select the checkbox beside one or more detectors in the list.

_security-analytics/usage/findings.md

@@ -13,6 +13,7 @@ The **Findings** window includes features for viewing and working with findings.
 
 You can choose **Refresh** at any time to refresh information on the **Findings** page.
 
+---
 ## The Findings graph
 
 The findings graph can display findings by log type or rule severity. Use the **Group by** dropdown list to specify either log type or rule severity.
@@ -39,6 +40,7 @@ When one of the commonly used windows of time is selected, you can choose **Show
 
 As one more alternative, you can select an option from the **Recently used date ranges** section to go back to a previous setting.
 
+---
 ## The Findings list
 
 The **Findings** list displays all findings according to the time of the finding, the finding ID, the rule name that generated the finding, the detector that captured the finding, and other details, as shown in the following image.
@@ -78,9 +80,6 @@ For details about working with **Discover** in OpenSearch Dashboards, see [Explo
 
 #### Viewing correlated findings
 
-Correlations between findings are generated by the correlation engine, which is an experimental feature released in OpenSearch 2.7. Therefore, we do not recommend using the feature in a production environment at this time. For updates on the progress of the correlation engine, see [Security Analytics Correlation Engine](https://github.com/opensearch-project/security-analytics/issues/369) on GitHub. To share ideas and provide feedback, join the [Security Analytics forum](https://forum.opensearch.org/c/plugins/security-analytics/73).
-{: .warning }
-
 To see how the finding is correlated with other findings, select the **Correlations** tab. Correlations are relationships between findings that express a particular threat scenario involving multiple log types. Information in the **Correlated findings** table shows the time at which a correlated finding was generated, a finding's ID, the log type used to generate the finding, its threat severity, and the correlation score---a measure of its proximity to the reference finding---as shown in the following image.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/corr-details-findings.png" alt="A table of correlated findings with respect to the reference finding" width="60%">

_security-analytics/usage/overview.md

@@ -16,6 +16,7 @@ When you select **Security Analytics** from the top menu, the Overview page is d
 
 Each section provides a summary description for each element of Security Analytics, along with controls that let you take action for each item.
 
+---
 ## Overview and getting started
 
 The upper portion of the Overview page contains two control buttons for refreshing information and getting started with Security Analytics. You can select the **Refresh** button to refresh all of the information on the page. 
@@ -29,24 +30,29 @@ You can also select the **Getting started** link to expand the Get started with
 * In step 3, select **View alerts** to go to the Security alerts page. For details about this page, see [Working with alerts]({{site.url}}{{site.baseurl}}/security-analytics/usage/alerts/).
 * In step 4, select **Manage rules** to go to the Rules page. For more on rules, see [Working with rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/).
 
+---
 ## Findings and alert count
 
 The Findings and alert count section provides a graph showing data on the latest findings. Use the **Group by** dropdown list to select either **All findings** or **Log type**.
 
+---
 ## Recent alerts
 
 The Recent alerts table displays recent alerts by time, trigger name, and alert severity. Select **View alerts** to go to the Alerts page.
 
+---
 ## Recent findings
 
 The Recent findings table displays recent findings by time, rule name, rule severity, and detector. Select **View all findings** to go to the Findings page.
 
+---
 ## Most frequent detection rules
 
 This section provides a graphical representation of detection rules that trigger findings most often and how they compare to others as a percentage of the whole. The rule names represented by the graph are listed to the right.
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/rule_graph.png" alt="The detection rule graph on the Overview page" width="50%">
 
+---
 ## Detectors
 
 The Detectors section displays a list of available detectors by detector name, status (active/inactive), and log type. Select **View all detectors** to go to the Detectors page. Select **Create detector** to go directly to the Define detector page.

_security-analytics/usage/rules.md

@@ -11,6 +11,7 @@ The **Detection rules** window lists all security rules used for detection creat
 
 <img src="{{site.url}}{{site.baseurl}}/images/Security/Rules.png" alt="The Rules page" width="90%">
 
+---
 ## Viewing and filtering rules
 
 When you open the **Detection rules** page, all rules are listed in the table. Use the search bar to search for specific rules by entering a full or partial name and pressing **Return/Enter** on your keyboard. The list is filtered and displays matching results.
@@ -32,11 +33,11 @@ In Visual view, rule details are arranged in fields, and the links are active. S
 * Rule details are formatted as a YAML file according to the Sigma rule specification.
 * To copy the rule, select the copy icon in the upper-right corner of the rule. To quickly create a new, customized rule, you can paste the rule into the YAML editor and make any modifications before saving it. See [Customizing rules](#customizing-rules) for more information.
 
+---
 ## Creating rules
 
 There are multiple ways to create rules on the **Detection rules** page. These methods include manually creating a custom rule, importing a rule, and duplicating an existing rule to customize it. The following sections discuss these methods in detail.  
 
-
 ### Custom rules
 
 The first method of rule creation is to create a custom rule by manually filling in the necessary fields that complete the rule, using either the Visual Editor or the YAML Editor. To do this, select **Create detection rule** in the uppper-right corner of the screen. The **Create detection rule** window opens.
@@ -163,7 +164,6 @@ status: experimental
 
 To assist in rule creation using the **YAML Editor**, you can refer to Sigma's [Rule Creation Guide](https://github.com/SigmaHQ/sigma/wiki/Rule-Creation-Guide) and use the descriptions of each field to learn more about defining the rule.
 
-
 ### Importing rules
 
 Security Analytics also supports the importing of Sigma rules in YAML format. In the **Detection rules** window, follow these steps to import a rule.