[ML][DOCS] Add documentation for detector rules and filters (#32013)
This commit is contained in:
parent
e12e2e0cdd
commit
9a7a649755
|
@ -309,6 +309,16 @@ setups['farequote_datafeed'] = setups['farequote_job'] + '''
|
||||||
"job_id":"farequote",
|
"job_id":"farequote",
|
||||||
"indexes":"farequote"
|
"indexes":"farequote"
|
||||||
}
|
}
|
||||||
|
'''
|
||||||
|
setups['ml_filter_safe_domains'] = '''
|
||||||
|
- do:
|
||||||
|
xpack.ml.put_filter:
|
||||||
|
filter_id: "safe_domains"
|
||||||
|
body: >
|
||||||
|
{
|
||||||
|
"description": "A list of safe domains",
|
||||||
|
"items": ["*.google.com", "wikipedia.org"]
|
||||||
|
}
|
||||||
'''
|
'''
|
||||||
setups['server_metrics_index'] = '''
|
setups['server_metrics_index'] = '''
|
||||||
- do:
|
- do:
|
||||||
|
|
|
@ -47,6 +47,15 @@ The main {ml} resources can be accessed with a variety of endpoints:
|
||||||
* {ref}/ml-delete-calendar-job.html[DELETE /calendars/<calendar_id+++>+++/jobs/<job_id+++>+++]: Disassociate a job from a calendar
|
* {ref}/ml-delete-calendar-job.html[DELETE /calendars/<calendar_id+++>+++/jobs/<job_id+++>+++]: Disassociate a job from a calendar
|
||||||
* {ref}/ml-delete-calendar.html[DELETE /calendars/<calendar_id+++>+++]: Delete a calendar
|
* {ref}/ml-delete-calendar.html[DELETE /calendars/<calendar_id+++>+++]: Delete a calendar
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ml-api-filters]]
|
||||||
|
=== /filters/
|
||||||
|
|
||||||
|
* {ref}/ml-put-filter.html[PUT /filters/<filter_id+++>+++]: Create a filter
|
||||||
|
* {ref}/ml-update-filter.html[POST /filters/<filter_id+++>+++/_update]: Update a filter
|
||||||
|
* {ref}/ml-get-filter.html[GET /filters/<filter_id+++>+++]: List filters
|
||||||
|
* {ref}/ml-delete-filter.html[DELETE /filter/<filter_id+++>+++]: Delete a filter
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-api-datafeeds]]
|
[[ml-api-datafeeds]]
|
||||||
=== /datafeeds/
|
=== /datafeeds/
|
||||||
|
|
|
@ -34,6 +34,7 @@ The scenarios in this section describe some best practices for generating useful
|
||||||
* <<ml-configuring-categories>>
|
* <<ml-configuring-categories>>
|
||||||
* <<ml-configuring-pop>>
|
* <<ml-configuring-pop>>
|
||||||
* <<ml-configuring-transform>>
|
* <<ml-configuring-transform>>
|
||||||
|
* <<ml-configuring-detector-custom-rules>>
|
||||||
|
|
||||||
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/customurl.asciidoc
|
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/customurl.asciidoc
|
||||||
include::customurl.asciidoc[]
|
include::customurl.asciidoc[]
|
||||||
|
@ -49,3 +50,6 @@ include::populations.asciidoc[]
|
||||||
|
|
||||||
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/transforms.asciidoc
|
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/transforms.asciidoc
|
||||||
include::transforms.asciidoc[]
|
include::transforms.asciidoc[]
|
||||||
|
|
||||||
|
:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/ml/detector-custom-rules.asciidoc
|
||||||
|
include::detector-custom-rules.asciidoc[]
|
|
@ -0,0 +1,230 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-configuring-detector-custom-rules]]
|
||||||
|
=== Customizing detectors with rules and filters
|
||||||
|
|
||||||
|
<<ml-rules,Rules and filters>> enable you to change the behavior of anomaly
|
||||||
|
detectors based on domain-specific knowledge.
|
||||||
|
|
||||||
|
Rules describe _when_ a detector should take a certain _action_ instead
|
||||||
|
of following its default behavior. To specify the _when_ a rule uses
|
||||||
|
a `scope` and `conditions`. You can think of `scope` as the categorical
|
||||||
|
specification of a rule, while `conditions` are the numerical part.
|
||||||
|
A rule can have a scope, one or more conditions, or a combination of
|
||||||
|
scope and conditions.
|
||||||
|
|
||||||
|
Let us see how those can be configured by examples.
|
||||||
|
|
||||||
|
==== Specifying rule scope
|
||||||
|
|
||||||
|
Let us assume we are configuring a job in order to DNS data exfiltration.
|
||||||
|
Our data contain fields "subdomain" and "highest_registered_domain".
|
||||||
|
We can use a detector that looks like `high_info_content(subdomain) over highest_registered_domain`.
|
||||||
|
If we run such a job it is possible that we discover a lot of anomalies on
|
||||||
|
frequently used domains that we have reasons to trust. As security analysts, we
|
||||||
|
are not interested in such anomalies. Ideally, we could instruct the detector to
|
||||||
|
skip results for domains that we consider safe. Using a rule with a scope allows
|
||||||
|
us to achieve this.
|
||||||
|
|
||||||
|
First, we need to create a list with our safe domains. Those lists are called
|
||||||
|
`filters` in {ml}. Filters can be shared across jobs.
|
||||||
|
|
||||||
|
We create our filter using the {ref}/ml-put-filter.html[put filter API]:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
PUT _xpack/ml/filters/safe_domains
|
||||||
|
{
|
||||||
|
"description": "Our list of safe domains",
|
||||||
|
"items": ["safe.com", "trusted.com"]
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
Now, we can create our job specifying a scope that uses the filter for the
|
||||||
|
`highest_registered_domain` field:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
PUT _xpack/ml/anomaly_detectors/dns_exfiltration_with_rule
|
||||||
|
{
|
||||||
|
"analysis_config" : {
|
||||||
|
"bucket_span":"5m",
|
||||||
|
"detectors" :[{
|
||||||
|
"function":"high_info_content",
|
||||||
|
"field_name": "subdomain",
|
||||||
|
"over_field_name": "highest_registered_domain",
|
||||||
|
"custom_rules": [{
|
||||||
|
"actions": ["skip_result"],
|
||||||
|
"scope": {
|
||||||
|
"highest_registered_domain": {
|
||||||
|
"filter_id": "safe_domains",
|
||||||
|
"filter_type": "include"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
"data_description" : {
|
||||||
|
"time_field":"timestamp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
As time advances and we see more data and more results, we might encounter new
|
||||||
|
domains that we want to add in the filter. We can do that by using the
|
||||||
|
{ref}/ml-update-filter.html[update filter API]:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
POST _xpack/ml/filters/safe_domains/_update
|
||||||
|
{
|
||||||
|
"add_items": ["another-safe.com"]
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[setup:ml_filter_safe_domains]
|
||||||
|
|
||||||
|
Note that we can provide scope for any of the partition/over/by fields.
|
||||||
|
In the following example we scope multiple fields:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
PUT _xpack/ml/anomaly_detectors/scoping_multiple_fields
|
||||||
|
{
|
||||||
|
"analysis_config" : {
|
||||||
|
"bucket_span":"5m",
|
||||||
|
"detectors" :[{
|
||||||
|
"function":"count",
|
||||||
|
"partition_field_name": "my_partition",
|
||||||
|
"over_field_name": "my_over",
|
||||||
|
"by_field_name": "my_by",
|
||||||
|
"custom_rules": [{
|
||||||
|
"actions": ["skip_result"],
|
||||||
|
"scope": {
|
||||||
|
"my_partition": {
|
||||||
|
"filter_id": "filter_1"
|
||||||
|
},
|
||||||
|
"my_over": {
|
||||||
|
"filter_id": "filter_2"
|
||||||
|
},
|
||||||
|
"my_by": {
|
||||||
|
"filter_id": "filter_3"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
"data_description" : {
|
||||||
|
"time_field":"timestamp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
Such a detector will skip results when the values of all 3 scoped fields
|
||||||
|
are included in the referenced filters.
|
||||||
|
|
||||||
|
==== Specifying rule conditions
|
||||||
|
|
||||||
|
Imagine a detector that looks for anomalies in CPU utilization.
|
||||||
|
Given a machine that is idle for long enough, small movement in CPU could
|
||||||
|
result in anomalous results where the `actual` value is quite small, for
|
||||||
|
example, 0.02. Given our knowledge about how CPU utilization behaves we might
|
||||||
|
determine that anomalies with such small actual values are not interesting for
|
||||||
|
investigation.
|
||||||
|
|
||||||
|
Let us now configure a job with a rule that will skip results where CPU
|
||||||
|
utilization is less than 0.20.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
PUT _xpack/ml/anomaly_detectors/cpu_with_rule
|
||||||
|
{
|
||||||
|
"analysis_config" : {
|
||||||
|
"bucket_span":"5m",
|
||||||
|
"detectors" :[{
|
||||||
|
"function":"high_mean",
|
||||||
|
"field_name": "cpu_utilization",
|
||||||
|
"custom_rules": [{
|
||||||
|
"actions": ["skip_result"],
|
||||||
|
"conditions": [
|
||||||
|
{
|
||||||
|
"applies_to": "actual",
|
||||||
|
"operator": "lt",
|
||||||
|
"value": 0.20
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
"data_description" : {
|
||||||
|
"time_field":"timestamp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
When there are multiple conditions they are combined with a logical `and`.
|
||||||
|
This is useful when we want the rule to apply to a range. We simply create
|
||||||
|
a rule with two conditions, one for each end of the desired range.
|
||||||
|
|
||||||
|
Here is an example where a count detector will skip results when the count
|
||||||
|
is greater than 30 and less than 50:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----------------------------------
|
||||||
|
PUT _xpack/ml/anomaly_detectors/rule_with_range
|
||||||
|
{
|
||||||
|
"analysis_config" : {
|
||||||
|
"bucket_span":"5m",
|
||||||
|
"detectors" :[{
|
||||||
|
"function":"count",
|
||||||
|
"custom_rules": [{
|
||||||
|
"actions": ["skip_result"],
|
||||||
|
"conditions": [
|
||||||
|
{
|
||||||
|
"applies_to": "actual",
|
||||||
|
"operator": "gt",
|
||||||
|
"value": 30
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"applies_to": "actual",
|
||||||
|
"operator": "lt",
|
||||||
|
"value": 50
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
"data_description" : {
|
||||||
|
"time_field":"timestamp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
----------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
==== Rules in the life-cycle of a job
|
||||||
|
|
||||||
|
Rules only affect results created after the rules were applied.
|
||||||
|
Let us imagine that we have configured a job and it has been running
|
||||||
|
for some time. After observing its results we decide that we can employ
|
||||||
|
rules in order to get rid of some uninteresting results. We can use
|
||||||
|
the update-job API to do so. However, the rule we added will only be in effect
|
||||||
|
for any results created from the moment we added the rule onwards. Past results
|
||||||
|
will remain unaffected.
|
||||||
|
|
||||||
|
==== Using rules VS filtering data
|
||||||
|
|
||||||
|
It might appear like using rules is just another way of filtering the data
|
||||||
|
that feeds into a job. For example, a rule that skips results when the
|
||||||
|
partition field value is in a filter sounds equivalent to having a query
|
||||||
|
that filters out such documents. But it is not. There is a fundamental
|
||||||
|
difference. When the data is filtered before reaching a job it is as if they
|
||||||
|
never existed for the job. With rules, the data still reaches the job and
|
||||||
|
affects its behavior (depending on the rule actions).
|
||||||
|
|
||||||
|
For example, a rule with the `skip_result` action means all data will still
|
||||||
|
be modeled. On the other hand, a rule with the `skip_model_update` action means
|
||||||
|
results will still be created even though the model will not be updated by
|
||||||
|
data matched by a rule.
|
|
@ -8,6 +8,8 @@ input data.
|
||||||
The {xpackml} features include the following geographic function: `lat_long`.
|
The {xpackml} features include the following geographic function: `lat_long`.
|
||||||
|
|
||||||
NOTE: You cannot create forecasts for jobs that contain geographic functions.
|
NOTE: You cannot create forecasts for jobs that contain geographic functions.
|
||||||
|
You also cannot add rules with conditions to detectors that use geographic
|
||||||
|
functions.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-lat-long]]
|
[[ml-lat-long]]
|
||||||
|
|
|
@ -15,6 +15,9 @@ The {xpackml} features include the following metric functions:
|
||||||
* <<ml-metric-metric,`metric`>>
|
* <<ml-metric-metric,`metric`>>
|
||||||
* xref:ml-metric-varp[`varp`, `high_varp`, `low_varp`]
|
* xref:ml-metric-varp[`varp`, `high_varp`, `low_varp`]
|
||||||
|
|
||||||
|
NOTE: You cannot add rules with conditions to detectors that use the `metric`
|
||||||
|
function.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-metric-min]]
|
[[ml-metric-min]]
|
||||||
==== Min
|
==== Min
|
||||||
|
@ -221,7 +224,6 @@ mean `responsetime` for each application over time. It detects when the mean
|
||||||
The `metric` function combines `min`, `max`, and `mean` functions. You can use
|
The `metric` function combines `min`, `max`, and `mean` functions. You can use
|
||||||
it as a shorthand for a combined analysis. If you do not specify a function in
|
it as a shorthand for a combined analysis. If you do not specify a function in
|
||||||
a detector, this is the default function.
|
a detector, this is the default function.
|
||||||
//TBD: Is that default behavior still true?
|
|
||||||
|
|
||||||
High- and low-sided functions are not applicable. You cannot use this function
|
High- and low-sided functions are not applicable. You cannot use this function
|
||||||
when a `summary_count_field_name` is specified.
|
when a `summary_count_field_name` is specified.
|
||||||
|
|
|
@ -15,6 +15,8 @@ number of times (frequency) rare values occur.
|
||||||
`exclude_frequent`.
|
`exclude_frequent`.
|
||||||
* You cannot create forecasts for jobs that contain `rare` or `freq_rare`
|
* You cannot create forecasts for jobs that contain `rare` or `freq_rare`
|
||||||
functions.
|
functions.
|
||||||
|
* You cannot add rules with conditions to detectors that use `rare` or
|
||||||
|
`freq_rare` functions.
|
||||||
* Shorter bucket spans (less than 1 hour, for example) are recommended when
|
* Shorter bucket spans (less than 1 hour, for example) are recommended when
|
||||||
looking for rare events. The functions model whether something happens in a
|
looking for rare events. The functions model whether something happens in a
|
||||||
bucket at least once. With longer bucket spans, it is more likely that
|
bucket at least once. With longer bucket spans, it is more likely that
|
||||||
|
|
|
@ -8,6 +8,7 @@ job configuration options.
|
||||||
* <<ml-calendar-resource,Calendars>>
|
* <<ml-calendar-resource,Calendars>>
|
||||||
* <<ml-datafeed-resource,{dfeeds-cap}>>
|
* <<ml-datafeed-resource,{dfeeds-cap}>>
|
||||||
* <<ml-datafeed-counts,{dfeed-cap} counts>>
|
* <<ml-datafeed-counts,{dfeed-cap} counts>>
|
||||||
|
* <<ml-filter-resource,Filters>>
|
||||||
* <<ml-job-resource,Jobs>>
|
* <<ml-job-resource,Jobs>>
|
||||||
* <<ml-jobstats,Job statistics>>
|
* <<ml-jobstats,Job statistics>>
|
||||||
* <<ml-snapshot-resource,Model snapshots>>
|
* <<ml-snapshot-resource,Model snapshots>>
|
||||||
|
@ -19,6 +20,8 @@ include::ml/calendarresource.asciidoc[]
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
include::ml/datafeedresource.asciidoc[]
|
include::ml/datafeedresource.asciidoc[]
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
|
include::ml/filterresource.asciidoc[]
|
||||||
|
[role="xpack"]
|
||||||
include::ml/jobresource.asciidoc[]
|
include::ml/jobresource.asciidoc[]
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
include::ml/jobcounts.asciidoc[]
|
include::ml/jobcounts.asciidoc[]
|
||||||
|
|
|
@ -15,6 +15,14 @@ machine learning APIs and in advanced job configuration options in Kibana.
|
||||||
* <<ml-post-calendar-event,Add scheduled events to calendar>>, <<ml-delete-calendar-event,Delete scheduled events from calendar>>
|
* <<ml-post-calendar-event,Add scheduled events to calendar>>, <<ml-delete-calendar-event,Delete scheduled events from calendar>>
|
||||||
* <<ml-get-calendar,Get calendars>>, <<ml-get-calendar-event,Get scheduled events>>
|
* <<ml-get-calendar,Get calendars>>, <<ml-get-calendar-event,Get scheduled events>>
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ml-api-filter-endpoint]]
|
||||||
|
=== Filters
|
||||||
|
|
||||||
|
* <<ml-put-filter,Create filter>>, <<ml-delete-filter,Delete filter>>
|
||||||
|
* <<ml-update-filter,Update filters>>
|
||||||
|
* <<ml-get-filter,Get filters>>
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-api-datafeed-endpoint]]
|
[[ml-api-datafeed-endpoint]]
|
||||||
=== {dfeeds-cap}
|
=== {dfeeds-cap}
|
||||||
|
@ -69,11 +77,13 @@ include::ml/close-job.asciidoc[]
|
||||||
//CREATE
|
//CREATE
|
||||||
include::ml/put-calendar.asciidoc[]
|
include::ml/put-calendar.asciidoc[]
|
||||||
include::ml/put-datafeed.asciidoc[]
|
include::ml/put-datafeed.asciidoc[]
|
||||||
|
include::ml/put-filter.asciidoc[]
|
||||||
include::ml/put-job.asciidoc[]
|
include::ml/put-job.asciidoc[]
|
||||||
//DELETE
|
//DELETE
|
||||||
include::ml/delete-calendar.asciidoc[]
|
include::ml/delete-calendar.asciidoc[]
|
||||||
include::ml/delete-datafeed.asciidoc[]
|
include::ml/delete-datafeed.asciidoc[]
|
||||||
include::ml/delete-calendar-event.asciidoc[]
|
include::ml/delete-calendar-event.asciidoc[]
|
||||||
|
include::ml/delete-filter.asciidoc[]
|
||||||
include::ml/delete-job.asciidoc[]
|
include::ml/delete-job.asciidoc[]
|
||||||
include::ml/delete-calendar-job.asciidoc[]
|
include::ml/delete-calendar-job.asciidoc[]
|
||||||
include::ml/delete-snapshot.asciidoc[]
|
include::ml/delete-snapshot.asciidoc[]
|
||||||
|
@ -93,6 +103,7 @@ include::ml/get-job.asciidoc[]
|
||||||
include::ml/get-job-stats.asciidoc[]
|
include::ml/get-job-stats.asciidoc[]
|
||||||
include::ml/get-snapshot.asciidoc[]
|
include::ml/get-snapshot.asciidoc[]
|
||||||
include::ml/get-calendar-event.asciidoc[]
|
include::ml/get-calendar-event.asciidoc[]
|
||||||
|
include::ml/get-filter.asciidoc[]
|
||||||
include::ml/get-record.asciidoc[]
|
include::ml/get-record.asciidoc[]
|
||||||
//OPEN
|
//OPEN
|
||||||
include::ml/open-job.asciidoc[]
|
include::ml/open-job.asciidoc[]
|
||||||
|
@ -107,6 +118,7 @@ include::ml/start-datafeed.asciidoc[]
|
||||||
include::ml/stop-datafeed.asciidoc[]
|
include::ml/stop-datafeed.asciidoc[]
|
||||||
//UPDATE
|
//UPDATE
|
||||||
include::ml/update-datafeed.asciidoc[]
|
include::ml/update-datafeed.asciidoc[]
|
||||||
|
include::ml/update-filter.asciidoc[]
|
||||||
include::ml/update-job.asciidoc[]
|
include::ml/update-job.asciidoc[]
|
||||||
include::ml/update-snapshot.asciidoc[]
|
include::ml/update-snapshot.asciidoc[]
|
||||||
//VALIDATE
|
//VALIDATE
|
||||||
|
|
|
@ -0,0 +1,53 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-delete-filter]]
|
||||||
|
=== Delete Filter API
|
||||||
|
++++
|
||||||
|
<titleabbrev>Delete Filter</titleabbrev>
|
||||||
|
++++
|
||||||
|
|
||||||
|
Deletes a filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Request
|
||||||
|
|
||||||
|
`DELETE _xpack/ml/filters/<filter_id>`
|
||||||
|
|
||||||
|
|
||||||
|
==== Description
|
||||||
|
|
||||||
|
This API deletes a {stack-ov}/ml-rules.html[filter].
|
||||||
|
If a {ml} job references the filter, you cannot delete the filter. You must
|
||||||
|
update or delete the job before you can delete the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Path Parameters
|
||||||
|
|
||||||
|
`filter_id` (required)::
|
||||||
|
(string) Identifier for the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Authorization
|
||||||
|
|
||||||
|
You must have `manage_ml`, or `manage` cluster privileges to use this API.
|
||||||
|
For more information, see {xpack-ref}/security-privileges.html[Security Privileges].
|
||||||
|
|
||||||
|
|
||||||
|
==== Examples
|
||||||
|
|
||||||
|
The following example deletes the `safe_domains` filter:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
DELETE _xpack/ml/filters/safe_domains
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[setup:ml_filter_safe_domains]
|
||||||
|
|
||||||
|
When the filter is deleted, you receive the following results:
|
||||||
|
[source,js]
|
||||||
|
----
|
||||||
|
{
|
||||||
|
"acknowledged": true
|
||||||
|
}
|
||||||
|
----
|
||||||
|
//TESTRESPONSE
|
|
@ -0,0 +1,16 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-filter-resource]]
|
||||||
|
=== Filter Resources
|
||||||
|
|
||||||
|
A filter resource has the following properties:
|
||||||
|
|
||||||
|
`filter_id`::
|
||||||
|
(string) A string that uniquely identifies the filter.
|
||||||
|
|
||||||
|
`description`::
|
||||||
|
(array) A description of the filter.
|
||||||
|
|
||||||
|
`items`::
|
||||||
|
(array of strings) An array of strings which is the filter item list.
|
||||||
|
|
||||||
|
For more information, see {stack-ov}/ml-rules.html[Machine learning rules and filters].
|
|
@ -0,0 +1,84 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-get-filter]]
|
||||||
|
=== Get Filters API
|
||||||
|
++++
|
||||||
|
<titleabbrev>Get Filters</titleabbrev>
|
||||||
|
++++
|
||||||
|
|
||||||
|
Retrieves filters.
|
||||||
|
|
||||||
|
|
||||||
|
==== Request
|
||||||
|
|
||||||
|
`GET _xpack/ml/filters/<filter_id>` +
|
||||||
|
|
||||||
|
`GET _xpack/ml/filters/`
|
||||||
|
|
||||||
|
|
||||||
|
===== Description
|
||||||
|
|
||||||
|
You can get a single filter or all filters. For more information, see
|
||||||
|
{stack-ov}/ml-rules.html[Machine learning rules and filters].
|
||||||
|
|
||||||
|
|
||||||
|
==== Path Parameters
|
||||||
|
|
||||||
|
`filter_id`::
|
||||||
|
(string) Identifier for the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Request Body
|
||||||
|
|
||||||
|
`from`:::
|
||||||
|
(integer) Skips the specified number of filters.
|
||||||
|
|
||||||
|
`size`:::
|
||||||
|
(integer) Specifies the maximum number of filters to obtain.
|
||||||
|
|
||||||
|
|
||||||
|
==== Results
|
||||||
|
|
||||||
|
The API returns the following information:
|
||||||
|
|
||||||
|
`filters`::
|
||||||
|
(array) An array of filter resources.
|
||||||
|
For more information, see <<ml-filter-resource>>.
|
||||||
|
|
||||||
|
|
||||||
|
==== Authorization
|
||||||
|
|
||||||
|
You must have `monitor_ml`, `monitor`, `manage_ml`, or `manage` cluster
|
||||||
|
privileges to use this API. For more information, see
|
||||||
|
{xpack-ref}/security-privileges.html[Security Privileges].
|
||||||
|
|
||||||
|
|
||||||
|
==== Examples
|
||||||
|
|
||||||
|
The following example gets configuration information for the `safe_domains`
|
||||||
|
filter:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
GET _xpack/ml/filters/safe_domains
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[setup:ml_filter_safe_domains]
|
||||||
|
|
||||||
|
The API returns the following results:
|
||||||
|
[source,js]
|
||||||
|
----
|
||||||
|
{
|
||||||
|
"count": 1,
|
||||||
|
"filters": [
|
||||||
|
{
|
||||||
|
"filter_id": "safe_domains",
|
||||||
|
"description": "A list of safe domains",
|
||||||
|
"items": [
|
||||||
|
"*.google.com",
|
||||||
|
"wikipedia.org"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
----
|
||||||
|
//TESTRESPONSE
|
|
@ -245,7 +245,7 @@ NOTE: The `field_name` cannot contain double quotes or backslashes.
|
||||||
--
|
--
|
||||||
|
|
||||||
`function`::
|
`function`::
|
||||||
(string) The analysis function that is used.
|
(string) The analysis function that is used.
|
||||||
For example, `count`, `rare`, `mean`, `min`, `max`, and `sum`. For more
|
For example, `count`, `rare`, `mean`, `min`, `max`, and `sum`. For more
|
||||||
information, see {xpack-ref}/ml-functions.html[Function Reference].
|
information, see {xpack-ref}/ml-functions.html[Function Reference].
|
||||||
|
|
||||||
|
@ -262,7 +262,12 @@ NOTE: The `field_name` cannot contain double quotes or backslashes.
|
||||||
|
|
||||||
`use_null`::
|
`use_null`::
|
||||||
(boolean) Defines whether a new series is used as the null series
|
(boolean) Defines whether a new series is used as the null series
|
||||||
when there is no value for the by or partition fields. The default value is `false`. +
|
when there is no value for the by or partition fields. The default value is `false`.
|
||||||
|
|
||||||
|
`custom_rules`::
|
||||||
|
(array) An array of rule objects, which enable customizing how the detector works.
|
||||||
|
For example, a rule may dictate to the detector conditions under which results should be skipped.
|
||||||
|
For more information see <<ml-detector-custom-rule,detector custom rule objects>>. +
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
IMPORTANT: Field names are case sensitive, for example a field named 'Bytes'
|
IMPORTANT: Field names are case sensitive, for example a field named 'Bytes'
|
||||||
|
@ -270,9 +275,9 @@ is different from one named 'bytes'.
|
||||||
|
|
||||||
--
|
--
|
||||||
|
|
||||||
After you create a job, the only property you can change in the detector
|
After you create a job, the only properties you can change in the detector
|
||||||
configuration object is the `detector_description`; all other properties are
|
configuration object are the `detector_description` and the `custom_rules`;
|
||||||
informational.
|
all other properties are informational.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-datadescription]]
|
[[ml-datadescription]]
|
||||||
|
@ -408,6 +413,64 @@ the categorization field value came from.
|
||||||
For more information, see
|
For more information, see
|
||||||
{xpack-ref}/ml-configuring-categories.html[Categorizing Log Messages].
|
{xpack-ref}/ml-configuring-categories.html[Categorizing Log Messages].
|
||||||
|
|
||||||
|
[float]
|
||||||
|
[[ml-detector-custom-rule]]
|
||||||
|
==== Detector Custom Rule
|
||||||
|
|
||||||
|
{stack-ov}/ml-rules.html[Custom rules] enable you to customize the way detectors
|
||||||
|
operate.
|
||||||
|
|
||||||
|
A rule has the following properties:
|
||||||
|
|
||||||
|
`actions`::
|
||||||
|
(array) The set of actions to be triggered when the rule applies.
|
||||||
|
If more than one action is specified the effects of all actions are combined.
|
||||||
|
The available actions include: +
|
||||||
|
`skip_result`::: The result will not be created. This is the default value.
|
||||||
|
Unless you also specify `skip_model_update`, the model will be updated as
|
||||||
|
usual with the corresponding series value.
|
||||||
|
`skip_model_update`::: The value for that series will not be used to update
|
||||||
|
the model. Unless you also specify `skip_result`, the results will be created
|
||||||
|
as usual. This action is suitable when certain values are expected to be
|
||||||
|
consistently anomalous and they affect the model in a way that negatively
|
||||||
|
impacts the rest of the results.
|
||||||
|
`scope`::
|
||||||
|
(object) An optional scope of series where the rule applies. By default the scope
|
||||||
|
includes all series. Scoping is allowed for any of the partition/by/over fields.
|
||||||
|
To add a scope for a field add the field name as a key in the scope object and
|
||||||
|
set its value to an object with properties:
|
||||||
|
`filter_id`::
|
||||||
|
(string) The id of the <<ml-filter-resource,filter>> to be used.
|
||||||
|
`filter_type`::
|
||||||
|
(string) Either `include` (the rule applies for values in the filter)
|
||||||
|
or `exclude` (the rule applies for values not in the filter). Defaults
|
||||||
|
to `include`.
|
||||||
|
|
||||||
|
`conditions`::
|
||||||
|
(array) An optional array of numeric conditions when the rule applies.
|
||||||
|
Multiple conditions are combined together with a logical `AND`.
|
||||||
|
+
|
||||||
|
--
|
||||||
|
NOTE: If your detector uses `lat_long`, `metric`, `rare`, or `freq_rare`
|
||||||
|
functions, you cannot specify `conditions` for your rule.
|
||||||
|
|
||||||
|
|
||||||
|
A condition has the following properties:
|
||||||
|
|
||||||
|
`applies_to`:::
|
||||||
|
(string) Specifies the result property to which the condition applies.
|
||||||
|
The available options are `actual`, `typical`, `diff_from_typical`, `time`.
|
||||||
|
`operator`:::
|
||||||
|
(string) Specifies the condition operator. The available options are
|
||||||
|
`gt` (greater than), `gte` (greater than or equals), `lt` (less than) and `lte` (less than or equals).
|
||||||
|
`value`:::
|
||||||
|
(double) The value that is compared against the `applied_to` field using the `operator`.
|
||||||
|
--
|
||||||
|
|
||||||
|
A rule is required to either have a non-empty scope or at least one condition.
|
||||||
|
For more examples see
|
||||||
|
{stack-ov}/ml-configuring-detector-custom-rules.html[Configuring Detector Custom Rules].
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[ml-apilimits]]
|
[[ml-apilimits]]
|
||||||
==== Analysis Limits
|
==== Analysis Limits
|
||||||
|
|
|
@ -0,0 +1,68 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-put-filter]]
|
||||||
|
=== Create Filter API
|
||||||
|
++++
|
||||||
|
<titleabbrev>Create Filter</titleabbrev>
|
||||||
|
++++
|
||||||
|
|
||||||
|
Instantiates a filter.
|
||||||
|
|
||||||
|
==== Request
|
||||||
|
|
||||||
|
`PUT _xpack/ml/filters/<filter_id>`
|
||||||
|
|
||||||
|
===== Description
|
||||||
|
|
||||||
|
A {stack-ov}/ml-rules.html[filter] contains a list of strings.
|
||||||
|
It can be used by one or more jobs. Specifically, filters are referenced in
|
||||||
|
the `custom_rules` property of <<ml-detectorconfig,detector configuration objects>>.
|
||||||
|
|
||||||
|
==== Path Parameters
|
||||||
|
|
||||||
|
`filter_id` (required)::
|
||||||
|
(string) Identifier for the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Request Body
|
||||||
|
|
||||||
|
`description`::
|
||||||
|
(string) A description of the filter.
|
||||||
|
|
||||||
|
`items`::
|
||||||
|
(array of strings) The items of the filter.
|
||||||
|
A wildcard `*` can be used at the beginning
|
||||||
|
or the end of an item. Up to 10000 items
|
||||||
|
are allowed in each filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Authorization
|
||||||
|
|
||||||
|
You must have `manage_ml`, or `manage` cluster privileges to use this API.
|
||||||
|
For more information, see
|
||||||
|
{xpack-ref}/security-privileges.html[Security Privileges].
|
||||||
|
|
||||||
|
|
||||||
|
==== Examples
|
||||||
|
|
||||||
|
The following example creates the `safe_domains` filter:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _xpack/ml/filters/safe_domains
|
||||||
|
{
|
||||||
|
"description": "A list of safe domains",
|
||||||
|
"items": ["*.google.com", "wikipedia.org"]
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
When the filter is created, you receive the following response:
|
||||||
|
[source,js]
|
||||||
|
----
|
||||||
|
{
|
||||||
|
"filter_id": "safe_domains",
|
||||||
|
"description": "A list of safe domains",
|
||||||
|
"items": ["*.google.com", "wikipedia.org"]
|
||||||
|
}
|
||||||
|
----
|
||||||
|
//TESTRESPONSE
|
|
@ -0,0 +1,67 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[[ml-update-filter]]
|
||||||
|
=== Update Filter API
|
||||||
|
++++
|
||||||
|
<titleabbrev>Update Filter</titleabbrev>
|
||||||
|
++++
|
||||||
|
|
||||||
|
Updates the description of a filter, adds items, or removes items.
|
||||||
|
|
||||||
|
==== Request
|
||||||
|
|
||||||
|
`POST _xpack/ml/filters/<filter_id>/_update`
|
||||||
|
|
||||||
|
//==== Description
|
||||||
|
|
||||||
|
==== Path Parameters
|
||||||
|
|
||||||
|
`filter_id` (required)::
|
||||||
|
(string) Identifier for the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Request Body
|
||||||
|
|
||||||
|
`description`::
|
||||||
|
(string) A description for the filter. See <<ml-filter-resource>>.
|
||||||
|
|
||||||
|
`add_items`::
|
||||||
|
(array of strings) The items to add to the filter.
|
||||||
|
|
||||||
|
`remove_items`::
|
||||||
|
(array of strings) The items to remove from the filter.
|
||||||
|
|
||||||
|
|
||||||
|
==== Authorization
|
||||||
|
|
||||||
|
You must have `manage_ml`, or `manage` cluster privileges to use this API.
|
||||||
|
For more information, see
|
||||||
|
{xpack-ref}/security-privileges.html[Security Privileges].
|
||||||
|
|
||||||
|
|
||||||
|
==== Examples
|
||||||
|
|
||||||
|
You can change the description, add and remove items to the `safe_domains` filter as follows:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
POST _xpack/ml/filters/safe_domains/_update
|
||||||
|
{
|
||||||
|
"description": "Updated list of domains",
|
||||||
|
"add_items": ["*.myorg.com"],
|
||||||
|
"remove_items": ["wikipedia.org"]
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
// TEST[setup:ml_filter_safe_domains]
|
||||||
|
|
||||||
|
The API returns the following results:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
----
|
||||||
|
{
|
||||||
|
"filter_id": "safe_domains",
|
||||||
|
"description": "Updated list of domains",
|
||||||
|
"items": ["*.google.com", "*.myorg.com"]
|
||||||
|
}
|
||||||
|
----
|
||||||
|
//TESTRESPONSE
|
|
@ -35,6 +35,8 @@ each periodic persistence of the model. See <<ml-job-resource>>. | Yes
|
||||||
|
|
||||||
|`description` |A description of the job. See <<ml-job-resource>>. | No
|
|`description` |A description of the job. See <<ml-job-resource>>. | No
|
||||||
|
|
||||||
|
|`detectors` |An array of <<ml-detector-update, detector update objects>>. | No
|
||||||
|
|
||||||
|`groups` |A list of job groups. See <<ml-job-resource>>. | No
|
|`groups` |A list of job groups. See <<ml-job-resource>>. | No
|
||||||
|
|
||||||
|`model_plot_config`: `enabled` |If true, enables calculation and storage of the
|
|`model_plot_config`: `enabled` |If true, enables calculation and storage of the
|
||||||
|
@ -58,12 +60,6 @@ if the job is open when you make the update, you must stop the data feed, close
|
||||||
the job, then restart the data feed and open the job for the changes to take
|
the job, then restart the data feed and open the job for the changes to take
|
||||||
effect.
|
effect.
|
||||||
|
|
||||||
//|`analysis_config`: `detectors`: `detector_index` | A unique identifier of the
|
|
||||||
//detector. Matches the order of detectors returned by
|
|
||||||
//<<ml-get-job,GET job>>, starting from 0. | No
|
|
||||||
//|`analysis_config`: `detectors`: `detector_description` |A description of the
|
|
||||||
//detector. See <<ml-analysisconfig>>. | No
|
|
||||||
|
|
||||||
[NOTE]
|
[NOTE]
|
||||||
--
|
--
|
||||||
* You can update the `analysis_limits` only while the job is closed.
|
* You can update the `analysis_limits` only while the job is closed.
|
||||||
|
@ -73,6 +69,21 @@ of `hard_limit`, this means that it was unable to process some data. You might
|
||||||
want to re-run this job with an increased `model_memory_limit`.
|
want to re-run this job with an increased `model_memory_limit`.
|
||||||
--
|
--
|
||||||
|
|
||||||
|
[[ml-detector-update]]
|
||||||
|
==== Detector Update Objects
|
||||||
|
|
||||||
|
A detector update object has the following properties:
|
||||||
|
|
||||||
|
`detector_index`::
|
||||||
|
(integer) The identifier of the detector to update.
|
||||||
|
|
||||||
|
`description`::
|
||||||
|
(string) The new description for the detector.
|
||||||
|
|
||||||
|
`custom_rules`::
|
||||||
|
(array) The new list of <<ml-detector-custom-rule, rules>> for the detector.
|
||||||
|
|
||||||
|
No other detector property can be updated.
|
||||||
|
|
||||||
==== Authorization
|
==== Authorization
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue