649 lines
17 KiB
Plaintext
649 lines
17 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
|
|
[[eql-search-api]]
|
|
=== EQL search API
|
|
++++
|
|
<titleabbrev>EQL search</titleabbrev>
|
|
++++
|
|
|
|
beta::[]
|
|
|
|
Returns search results for an <<eql,Event Query Language (EQL)>> query.
|
|
|
|
EQL assumes each document in a data stream or index corresponds to an
|
|
event.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
[[eql-search-api-request]]
|
|
==== {api-request-title}
|
|
|
|
`GET /<target>/_eql/search`
|
|
|
|
`POST /<target>/_eql/search`
|
|
|
|
[[eql-search-api-prereqs]]
|
|
==== {api-prereq-title}
|
|
|
|
See <<eql-required-fields>>.
|
|
|
|
[[eql-search-api-limitations]]
|
|
===== Limitations
|
|
|
|
See <<eql-syntax-limitations,EQL limitations>>.
|
|
|
|
[[eql-search-api-path-params]]
|
|
==== {api-path-parms-title}
|
|
|
|
`<target>`::
|
|
(Required, string)
|
|
Comma-separated list of data streams, indices, or <<indices-aliases,index
|
|
aliases>> used to limit the request. Accepts wildcard (`*`) expressions.
|
|
+
|
|
To search all data streams and indices in a cluster, use
|
|
`_all` or `*`.
|
|
|
|
[[eql-search-api-query-params]]
|
|
==== {api-query-parms-title}
|
|
|
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
|
+
|
|
Defaults to `false`.
|
|
|
|
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
|
+
|
|
Defaults to `open`.
|
|
|
|
`ignore_unavailable`::
|
|
(Optional, boolean) If `true`, missing or closed indices are not included in the
|
|
response. Defaults to `true`.
|
|
|
|
`keep_alive`::
|
|
+
|
|
--
|
|
(Optional, <<time-units,time value>>)
|
|
Period for which the search and its results are stored on the cluster. Defaults
|
|
to `5d` (five days).
|
|
|
|
When this period expires, the search and its results are deleted, even if the
|
|
search is still ongoing.
|
|
|
|
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
|
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
|
complete within the period set by the
|
|
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
|
parameter, regardless of this value.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `keep_alive` request body parameter.
|
|
If both parameters are specified, only the query parameter is used.
|
|
====
|
|
--
|
|
|
|
`keep_on_completion`::
|
|
+
|
|
--
|
|
(Optional, boolean)
|
|
If `true`, the search and its results are stored on the cluster.
|
|
|
|
If `false`, the search and its results are stored on the cluster only if the
|
|
request does not complete during the period set by the
|
|
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
|
parameter. Defaults to `false`.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `keep_on_completion` request body
|
|
parameter. If both parameters are specified, only the query parameter is used.
|
|
====
|
|
--
|
|
|
|
`wait_for_completion_timeout`::
|
|
+
|
|
--
|
|
(Optional, <<time-units,time value>>)
|
|
Timeout duration to wait for the request to finish. Defaults to no
|
|
timeout, meaning the request waits for complete search results.
|
|
|
|
If this parameter is specified and the request completes during this period,
|
|
complete search results are returned.
|
|
|
|
If the request does not complete during this period, the search becomes an
|
|
<<eql-search-async,async search>>.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `wait_for_completion_timeout` request
|
|
body parameter. If both parameters are specified, only the query parameter is
|
|
used.
|
|
====
|
|
--
|
|
|
|
[[eql-search-api-request-body]]
|
|
==== {api-request-body-title}
|
|
|
|
`event_category_field`::
|
|
(Required*, string)
|
|
Field containing the event classification, such as `process`, `file`, or
|
|
`network`.
|
|
+
|
|
Defaults to `event.category`, as defined in the {ecs-ref}/ecs-event.html[Elastic
|
|
Common Schema (ECS)]. If a data stream or index does not contain the
|
|
`event.category` field, this value is required.
|
|
+
|
|
The event category field must be mapped as a field type in the
|
|
<<keyword,`keyword`>> family.
|
|
|
|
`fetch_size`::
|
|
(Optional, integer)
|
|
Maximum number of events to search at a time for sequence queries. Defaults to
|
|
`1000`.
|
|
+
|
|
This value must be greater than `2` but cannot exceed the value of the
|
|
<<index-max-result-window,`index.max_result_window`>> setting, which defaults to
|
|
`10000`.
|
|
+
|
|
Internally, a sequence query fetches and paginates sets of events to search for
|
|
matches. This parameter controls the size of those sets. This parameter does not
|
|
limit the total number of events searched or the number of matching events
|
|
returned.
|
|
+
|
|
A greater `fetch_size` value often increases search speed but uses more memory.
|
|
|
|
`filter`::
|
|
(Optional, <<query-dsl,query DSL object>>)
|
|
Query, written in query DSL, used to filter the events on which the EQL query
|
|
runs.
|
|
|
|
`keep_alive`::
|
|
+
|
|
--
|
|
(Optional, <<time-units,time value>>)
|
|
Period for which the search and its results are stored on the cluster. Defaults
|
|
to `5d` (five days).
|
|
|
|
When this period expires, the search and its results are deleted, even if the
|
|
search is still ongoing.
|
|
|
|
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
|
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
|
complete within the period set by the
|
|
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
|
parameter, regardless of this value.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `keep_alive` query parameter.
|
|
If both parameters are specified, only the query parameter is used.
|
|
====
|
|
--
|
|
|
|
[[eql-search-api-keep-on-completion]]
|
|
`keep_on_completion`::
|
|
+
|
|
--
|
|
(Optional, boolean)
|
|
If `true`, the search and its results are stored on the cluster.
|
|
|
|
If `false`, the search and its results are stored on the cluster only if the
|
|
request does not complete during the period set by the
|
|
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
|
parameter. Defaults to `false`.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `keep_on_completion` query parameter.
|
|
If both parameters are specified, only the query parameter is used.
|
|
====
|
|
--
|
|
|
|
[[eql-search-api-request-query-param]]
|
|
`query`::
|
|
(Required, string)
|
|
<<eql-syntax,EQL>> query you wish to run.
|
|
|
|
`size`::
|
|
(Optional, integer or float)
|
|
For <<eql-basic-syntax,basic queries>>, the maximum number of matching events to
|
|
return.
|
|
+
|
|
For <<eql-sequences,sequence queries>>, the maximum number of matching sequences
|
|
to return.
|
|
+
|
|
Defaults to `10`. This value must be greater than `0`.
|
|
+
|
|
NOTE: You cannot use <<eql-pipe-ref,pipes>>, such as `head` or `tail`, to exceed
|
|
this value.
|
|
|
|
[[eql-search-api-tiebreaker-field]]
|
|
`tiebreaker_field`::
|
|
(Optional, string)
|
|
Field used to sort events with the same
|
|
<<eql-search-api-timestamp-field,timestamp field>> value. Defaults to
|
|
`event.sequence`, as defined in the {ecs-ref}/ecs-event.html[Elastic Common
|
|
Schema (ECS)].
|
|
+
|
|
By default, matching events in the search response are sorted by timestamp,
|
|
converted to milliseconds since the {wikipedia}/Unix_time[Unix
|
|
epoch], in ascending order. If two or more events share the same timestamp, this
|
|
field is used to sort the events in ascending, lexicographic order.
|
|
|
|
[[eql-search-api-timestamp-field]]
|
|
`timestamp_field`::
|
|
+
|
|
--
|
|
(Required*, string)
|
|
Field containing event timestamp.
|
|
|
|
Defaults to `@timestamp`, as defined in the
|
|
{ecs-ref}/ecs-event.html[Elastic Common Schema (ECS)]. If a data stream or index
|
|
does not contain the `@timestamp` field, this value is required.
|
|
|
|
Events in the API response are sorted by this field's value, converted to
|
|
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in
|
|
ascending order.
|
|
|
|
The timestamp field should be mapped as a <<date,`date`>>. The
|
|
<<date_nanos,`date_nanos`>> field type is not supported.
|
|
--
|
|
|
|
[[eql-search-api-wait-for-completion-timeout]]
|
|
`wait_for_completion_timeout`::
|
|
+
|
|
--
|
|
(Optional, <<time-units,time value>>)
|
|
Timeout duration to wait for the request to finish. Defaults to no
|
|
timeout, meaning the request waits for complete search results.
|
|
|
|
If this parameter is specified and the request completes during this period,
|
|
complete search results are returned.
|
|
|
|
If the request does not complete during this period, the search becomes an
|
|
<<eql-search-async,async search>>.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
You can also specify this value using the `wait_for_completion_timeout` query
|
|
parameter. If both parameters are specified, only the query parameter is used.
|
|
====
|
|
--
|
|
|
|
[role="child_attributes"]
|
|
[[eql-search-api-response-body]]
|
|
==== {api-response-body-title}
|
|
|
|
[[eql-search-api-response-body-search-id]]
|
|
`id`::
|
|
+
|
|
--
|
|
(string)
|
|
Identifier for the search.
|
|
|
|
This search ID is only provided if one of the following conditions is met:
|
|
|
|
* A search request does not return complete results during the
|
|
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
|
parameter's timeout period, becoming an <<eql-search-async,async search>>.
|
|
|
|
* The search request's <<eql-search-api-keep-on-completion,`keep_on_completion`>>
|
|
parameter is `true`.
|
|
|
|
You can use this ID with the <<get-async-eql-search-api,get async EQL search
|
|
API>> to get the current status and available results for the search.
|
|
--
|
|
|
|
`is_partial`::
|
|
(boolean)
|
|
If `true`, the response does not contain complete search results.
|
|
|
|
`is_running`::
|
|
+
|
|
--
|
|
(boolean)
|
|
If `true`, the search request is still executing.
|
|
|
|
[IMPORTANT]
|
|
====
|
|
If this parameter and the `is_partial` parameter are `true`, the search is an
|
|
<<eql-search-async,ongoing async search>>. If the `keep_alive` period does not
|
|
pass, the complete search results will be available when the search completes.
|
|
|
|
If `is_partial` is `true` but `is_running` is `false`, the search returned
|
|
partial results due to a failure. Only some shards returned results or the node
|
|
coordinating the search failed.
|
|
====
|
|
--
|
|
|
|
`took`::
|
|
+
|
|
--
|
|
(integer)
|
|
Milliseconds it took {es} to execute the request.
|
|
|
|
This value is calculated by measuring the time elapsed
|
|
between receipt of a request on the coordinating node
|
|
and the time at which the coordinating node is ready to send the response.
|
|
|
|
Took time includes:
|
|
|
|
* Communication time between the coordinating node and data nodes
|
|
* Time the request spends in the `search` <<modules-threadpool,thread pool>>,
|
|
queued for execution
|
|
* Actual execution time
|
|
|
|
Took time does *not* include:
|
|
|
|
* Time needed to send the request to {es}
|
|
* Time needed to serialize the JSON response
|
|
* Time needed to send the response to a client
|
|
--
|
|
|
|
`timed_out`::
|
|
(boolean)
|
|
If `true`, the request timed out before completion.
|
|
|
|
`hits`::
|
|
(object)
|
|
Contains matching events and sequences. Also contains related metadata.
|
|
+
|
|
.Properties of `hits`
|
|
[%collapsible%open]
|
|
====
|
|
|
|
`total`::
|
|
(object)
|
|
Metadata about the number of matching events or sequences.
|
|
+
|
|
.Properties of `total`
|
|
[%collapsible%open]
|
|
=====
|
|
|
|
`value`::
|
|
(integer)
|
|
For <<eql-basic-syntax,basic queries>>, the total number of matching events.
|
|
+
|
|
For <<eql-sequences,sequence queries>>, the total number of matching sequences.
|
|
|
|
`relation`::
|
|
+
|
|
--
|
|
(string)
|
|
Indicates whether the number of events or sequences returned is accurate or a
|
|
lower bound.
|
|
|
|
Returned values are:
|
|
|
|
`eq`::: Accurate
|
|
`gte`::: Lower bound, including returned events or sequences
|
|
--
|
|
=====
|
|
|
|
`sequences`::
|
|
(array of objects)
|
|
Contains event sequences matching the query. Each object represents a
|
|
matching sequence. This parameter is only returned for EQL queries containing
|
|
a <<eql-sequences,sequence>>.
|
|
+
|
|
.Properties of `sequences` objects
|
|
[%collapsible%open]
|
|
=====
|
|
`join_keys`::
|
|
(array of values)
|
|
Shared field values used to constrain matches in the sequence. These are defined
|
|
using the <<eql-sequences,`by` keyword>> in the EQL query syntax.
|
|
|
|
`events`::
|
|
(array of objects)
|
|
Contains events matching the query. Each object represents a
|
|
matching event.
|
|
+
|
|
.Properties of `events` objects
|
|
[%collapsible%open]
|
|
======
|
|
`_index`::
|
|
(string)
|
|
Name of the index containing the event.
|
|
|
|
`_id`::
|
|
(string)
|
|
Unique identifier for the event.
|
|
This ID is only unique within the index.
|
|
|
|
`_source`::
|
|
(object)
|
|
Original JSON body passed for the event at index time.
|
|
======
|
|
=====
|
|
|
|
[[eql-search-api-response-events]]
|
|
`events`::
|
|
(array of objects)
|
|
Contains events matching the query. Each object represents a
|
|
matching event.
|
|
+
|
|
.Properties of `events` objects
|
|
[%collapsible%open]
|
|
=====
|
|
`_index`::
|
|
(string)
|
|
Name of the index containing the event.
|
|
|
|
`_id`::
|
|
(string)
|
|
(string)
|
|
Unique identifier for the event.
|
|
This ID is only unique within the index.
|
|
|
|
`_source`::
|
|
(object)
|
|
Original JSON body passed for the event at index time.
|
|
=====
|
|
====
|
|
|
|
[[eql-search-api-example]]
|
|
==== {api-examples-title}
|
|
|
|
[[eql-search-api-basic-query-ex]]
|
|
===== Basic query example
|
|
|
|
The following EQL search request searches for events with an `event.category` of
|
|
`process` that meet the following conditions:
|
|
|
|
* A `process.name` of `cmd.exe`
|
|
* An `process.pid` other than `2013`
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
process where (process.name == "cmd.exe" and process.pid != 2013)
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The API returns the following response. Matching events in the `hits.events`
|
|
property are sorted by <<eql-search-api-timestamp-field,timestamp>>, converted
|
|
to milliseconds since the {wikipedia}/Unix_time[Unix epoch],
|
|
in ascending order.
|
|
|
|
If two or more events share the same timestamp, the
|
|
<<eql-search-api-tiebreaker-field,`tiebreaker_field`>> field is used to sort
|
|
the events in ascending, lexicographic order.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 6,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 2,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "babI3XMBI9IjHuIqU0S_",
|
|
"_source": {
|
|
"@timestamp": "2099-12-06T11:04:05.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "edwCRnyD",
|
|
"sequence": 1
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "cmd.exe",
|
|
"executable": "C:\\Windows\\System32\\cmd.exe"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "b6bI3XMBI9IjHuIqU0S_",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:06:07.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "cMyt5SZ2",
|
|
"sequence": 3
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "cmd.exe",
|
|
"executable": "C:\\Windows\\System32\\cmd.exe"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_id": "babI3XMBI9IjHuIqU0S_"/"_id": $body.hits.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "b6bI3XMBI9IjHuIqU0S_"/"_id": $body.hits.events.1._id/]
|
|
|
|
[[eql-search-api-sequence-ex]]
|
|
===== Sequence query example
|
|
|
|
The following EQL search request matches a <<eql-sequences,sequence>> of events
|
|
that:
|
|
|
|
. Start with an event with:
|
|
+
|
|
--
|
|
* An `event.category` of `file`
|
|
* A `file.name` of `cmd.exe`
|
|
* An `process.pid` other than `2013`
|
|
--
|
|
. Followed by an event with:
|
|
+
|
|
--
|
|
* An `event.category` of `process`
|
|
* A `process.executable` that contains the substring `regsvr32`
|
|
--
|
|
|
|
These events must also share the same `process.pid` value.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
sequence by process.pid
|
|
[ file where file.name == "cmd.exe" and process.pid != 2013 ]
|
|
[ process where stringContains(process.executable, "regsvr32") ]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:sec_logs]
|
|
|
|
The API returns the following response. Matching sequences are included in the
|
|
`hits.sequences` property. The `hits.sequences.join_keys` property contains the
|
|
shared `process.pid` value for each matching event.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 6,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"sequences": [
|
|
{
|
|
"join_keys": [
|
|
2012
|
|
],
|
|
"events": [
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "AtOJ4UjUBAAx3XR5kcCM",
|
|
"_source": {
|
|
"@timestamp": "2099-12-06T11:04:07.000Z",
|
|
"event": {
|
|
"category": "file",
|
|
"id": "dGCHwoeS",
|
|
"sequence": 2
|
|
},
|
|
"file": {
|
|
"accessed": "2099-12-07T11:07:08.000Z",
|
|
"name": "cmd.exe",
|
|
"path": "C:\\Windows\\System32\\cmd.exe",
|
|
"type": "file",
|
|
"size": 16384
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "cmd.exe",
|
|
"executable": "C:\\Windows\\System32\\cmd.exe"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "OQmfCaduce8zoHT93o4H",
|
|
"_source": {
|
|
"@timestamp": "2099-12-07T11:07:09.000Z",
|
|
"event": {
|
|
"category": "process",
|
|
"id": "aR3NWVOs",
|
|
"sequence": 4
|
|
},
|
|
"process": {
|
|
"pid": 2012,
|
|
"name": "regsvr32.exe",
|
|
"command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.sequences.0.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.1._id/]
|