OpenSearch/x-pack/docs/en/ml/detector-custom-rules.asciidoc

[role="xpack"]
[[ml-configuring-detector-custom-rules]]
=== Customizing detectors with custom rules

<<ml-rules,Custom rules>> enable you to change the behavior of anomaly
detectors based on domain-specific knowledge.

Custom rules describe _when_ a detector should take a certain _action_ instead
of following its default behavior. To specify the _when_ a rule uses
a `scope` and `conditions`. You can think of `scope` as the categorical
specification of a rule, while `conditions` are the numerical part.
A rule can have a scope, one or more conditions, or a combination of
scope and conditions.

Let us see how those can be configured by examples.

==== Specifying custom rule scope

Let us assume we are configuring a job in order to detect DNS data exfiltration.
Our data contain fields "subdomain" and "highest_registered_domain".
We can use a detector that looks like `high_info_content(subdomain) over highest_registered_domain`.
If we run such a job it is possible that we discover a lot of anomalies on 
frequently used domains that we have reasons to trust. As security analysts, we 
are not interested in such anomalies. Ideally, we could instruct the detector to 
skip results for domains that we consider safe. Using a rule with a scope allows 
us to achieve this.

First, we need to create a list of our safe domains. Those lists are called 
_filters_ in {ml}. Filters can be shared across jobs.

We create our filter using the {ref}/ml-put-filter.html[put filter API]:

[source,js]
----------------------------------
PUT _xpack/ml/filters/safe_domains
{
  "description": "Our list of safe domains",
  "items": ["safe.com", "trusted.com"]
}
----------------------------------
// CONSOLE

Now, we can create our job specifying a scope that uses the `safe_domains` 
filter for the `highest_registered_domain` field:

[source,js]
----------------------------------
PUT _xpack/ml/anomaly_detectors/dns_exfiltration_with_rule
{
  "analysis_config" : {
    "bucket_span":"5m",
    "detectors" :[{
      "function":"high_info_content",
      "field_name": "subdomain",
      "over_field_name": "highest_registered_domain",
      "custom_rules": [{
        "actions": ["skip_result"],
        "scope": {
          "highest_registered_domain": {
            "filter_id": "safe_domains",
            "filter_type": "include"
          }
        }
      }]
    }]
  },
  "data_description" : {
    "time_field":"timestamp"
  }
}
----------------------------------
// CONSOLE

As time advances and we see more data and more results, we might encounter new 
domains that we want to add in the filter. We can do that by using the 
{ref}/ml-update-filter.html[update filter API]:

[source,js]
----------------------------------
POST _xpack/ml/filters/safe_domains/_update
{
  "add_items": ["another-safe.com"]
}
----------------------------------
// CONSOLE
// TEST[setup:ml_filter_safe_domains]

Note that we can use any of the `partition_field_name`, `over_field_name`, or 
`by_field_name` fields in the `scope`.

In the following example we scope multiple fields:

[source,js]
----------------------------------
PUT _xpack/ml/anomaly_detectors/scoping_multiple_fields
{
  "analysis_config" : {
    "bucket_span":"5m",
    "detectors" :[{
      "function":"count",
      "partition_field_name": "my_partition",
      "over_field_name": "my_over",
      "by_field_name": "my_by",
      "custom_rules": [{
        "actions": ["skip_result"],
        "scope": {
          "my_partition": {
            "filter_id": "filter_1"
          },
          "my_over": {
            "filter_id": "filter_2"
          },
          "my_by": {
            "filter_id": "filter_3"
          }
        }
      }]
    }]
  },
  "data_description" : {
    "time_field":"timestamp"
  }
}
----------------------------------
// CONSOLE

Such a detector will skip results when the values of all 3 scoped fields
are included in the referenced filters.

==== Specifying custom rule conditions

Imagine a detector that looks for anomalies in CPU utilization.
Given a machine that is idle for long enough, small movement in CPU could
result in anomalous results where the `actual` value is quite small, for 
example, 0.02. Given our knowledge about how CPU utilization behaves we might 
determine that anomalies with such small actual values are not interesting for 
investigation.

Let us now configure a job with a rule that will skip results where CPU 
utilization is less than 0.20.

[source,js]
----------------------------------
PUT _xpack/ml/anomaly_detectors/cpu_with_rule
{
  "analysis_config" : {
    "bucket_span":"5m",
    "detectors" :[{
      "function":"high_mean",
      "field_name": "cpu_utilization",
      "custom_rules": [{
        "actions": ["skip_result"],
        "conditions": [
          {
            "applies_to": "actual",
            "operator": "lt",
            "value": 0.20
          }
        ]
      }]
    }]
  },
  "data_description" : {
    "time_field":"timestamp"
  }
}
----------------------------------
// CONSOLE

When there are multiple conditions they are combined with a logical `and`.
This is useful when we want the rule to apply to a range. We simply create
a rule with two conditions, one for each end of the desired range.

Here is an example where a count detector will skip results when the count
is greater than 30 and less than 50:

[source,js]
----------------------------------
PUT _xpack/ml/anomaly_detectors/rule_with_range
{
  "analysis_config" : {
    "bucket_span":"5m",
    "detectors" :[{
      "function":"count",
      "custom_rules": [{
        "actions": ["skip_result"],
        "conditions": [
          {
            "applies_to": "actual",
            "operator": "gt",
            "value": 30
          },
          {
            "applies_to": "actual",
            "operator": "lt",
            "value": 50
          }
        ]
      }]
    }]
  },
  "data_description" : {
    "time_field":"timestamp"
  }
}
----------------------------------
// CONSOLE

==== Custom rules in the life-cycle of a job

Custom rules only affect results created after the rules were applied.
Let us imagine that we have configured a job and it has been running
for some time. After observing its results we decide that we can employ
rules in order to get rid of some uninteresting results. We can use
the {ref}/ml-update-job.html[update job API] to do so. However, the rule we 
added will only be in effect for any results created from the moment we added 
the rule onwards. Past results will remain unaffected.

==== Using custom rules VS filtering data

It might appear like using rules is just another way of filtering the data
that feeds into a job. For example, a rule that skips results when the
partition field value is in a filter sounds equivalent to having a query
that filters out such documents. But it is not. There is a fundamental
difference. When the data is filtered before reaching a job it is as if they
never existed for the job. With rules, the data still reaches the job and
affects its behavior (depending on the rule actions).

For example, a rule with the `skip_result` action means all data will still
be modeled. On the other hand, a rule with the `skip_model_update` action means
results will still be created even though the model will not be updated by
data matched by a rule.