709 lines
19 KiB
Plaintext
709 lines
19 KiB
Plaintext
[role="xpack"]
|
||
[testenv="basic"]
|
||
|
||
[[eql-search-api]]
|
||
=== EQL search API
|
||
++++
|
||
<titleabbrev>EQL search</titleabbrev>
|
||
++++
|
||
|
||
experimental::[]
|
||
|
||
Returns search results for an <<eql,Event Query Language (EQL)>> query.
|
||
|
||
In {es}, EQL assumes each document in a data stream or index corresponds to an
|
||
event.
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
process where process.name = "regsvr32.exe"
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
|
||
[[eql-search-api-request]]
|
||
==== {api-request-title}
|
||
|
||
`GET /<target>/_eql/search`
|
||
|
||
`POST /<target>/_eql/search`
|
||
|
||
[[eql-search-api-prereqs]]
|
||
==== {api-prereq-title}
|
||
|
||
See <<eql-required-fields>>.
|
||
|
||
[[eql-search-api-limitations]]
|
||
===== Limitations
|
||
|
||
See <<eql-syntax-limitations,EQL limitations>>.
|
||
|
||
[[eql-search-api-path-params]]
|
||
==== {api-path-parms-title}
|
||
|
||
`<target>`::
|
||
(Required, string)
|
||
Comma-separated list of data streams, indices, or <<indices-aliases,index
|
||
aliases>> used to limit the request. Accepts wildcard (`*`) expressions.
|
||
+
|
||
To search all data streams and indices in a cluster, use
|
||
`_all` or `*`.
|
||
|
||
[[eql-search-api-query-params]]
|
||
==== {api-query-parms-title}
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
||
+
|
||
Defaults to `false`.
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
||
+
|
||
Defaults to `open`.
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
|
||
|
||
`keep_alive`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Period for which the search and its results are stored on the cluster. Defaults
|
||
to `5d` (five days).
|
||
|
||
When this period expires, the search and its results are deleted, even if the
|
||
search is still ongoing.
|
||
|
||
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
||
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
||
complete within the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter, regardless of this value.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_alive` request body parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
`keep_on_completion`::
|
||
+
|
||
--
|
||
(Optional, boolean)
|
||
If `true`, the search and its results are stored on the cluster.
|
||
|
||
If `false`, the search and its results are stored on the cluster only if the
|
||
request does not complete during the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter. Defaults to `false`.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_on_completion` request body
|
||
parameter. If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
`wait_for_completion_timeout`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Timeout duration to wait for the request to finish. Defaults to no
|
||
timeout, meaning the request waits for complete search results.
|
||
|
||
If this parameter is specified and the request completes during this period,
|
||
complete search results are returned.
|
||
|
||
If the request does not complete during this period, the search becomes an
|
||
<<eql-search-async,async search>>.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `wait_for_completion_timeout` request
|
||
body parameter. If both parameters are specified, only the query parameter is
|
||
used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-request-body]]
|
||
==== {api-request-body-title}
|
||
|
||
`case_sensitive`::
|
||
(Optional, boolean)
|
||
If `true`, matching for the <<eql-search-api-request-query-param,EQL query>> is
|
||
case sensitive. Defaults to `false`.
|
||
|
||
`event_category_field`::
|
||
(Required*, string)
|
||
Field containing the event classification, such as `process`, `file`, or
|
||
`network`.
|
||
+
|
||
Defaults to `event.category`, as defined in the {ecs-ref}/ecs-event.html[Elastic
|
||
Common Schema (ECS)]. If a data stream or index does not contain the
|
||
`event.category` field, this value is required.
|
||
+
|
||
The event category field is typically mapped as a <<keyword,`keyword`>> or
|
||
<<constant-keyword,constant keyword>> field.
|
||
|
||
`fetch_size`::
|
||
(Optional, integer)
|
||
Maximum number of events to search at a time for sequence queries. Defaults to
|
||
`1000`.
|
||
+
|
||
This value must be greater than `2` but cannot exceed the value of the
|
||
<<index-max-result-window,`index.max_result_window`>> setting, which defaults to
|
||
`10000`.
|
||
+
|
||
Internally, a sequence query fetches and paginates sets of events to search for
|
||
matches. This parameter controls the size of those sets. This parameter does not
|
||
limit the total number of events searched or the number of matching events
|
||
returned.
|
||
+
|
||
A greater `fetch_size` value often increases search speed but uses more memory.
|
||
|
||
`filter`::
|
||
(Optional, <<query-dsl,query DSL object>>)
|
||
Query, written in query DSL, used to filter the events on which the EQL query
|
||
runs.
|
||
|
||
`keep_alive`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Period for which the search and its results are stored on the cluster. Defaults
|
||
to `5d` (five days).
|
||
|
||
When this period expires, the search and its results are deleted, even if the
|
||
search is still ongoing.
|
||
|
||
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
||
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
||
complete within the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter, regardless of this value.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_alive` query parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-keep-on-completion]]
|
||
`keep_on_completion`::
|
||
+
|
||
--
|
||
(Optional, boolean)
|
||
If `true`, the search and its results are stored on the cluster.
|
||
|
||
If `false`, the search and its results are stored on the cluster only if the
|
||
request does not complete during the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter. Defaults to `false`.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_on_completion` query parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-request-query-param]]
|
||
`query`::
|
||
(Required, string)
|
||
<<eql-syntax,EQL>> query you wish to run.
|
||
+
|
||
IMPORTANT: This parameter supports a subset of EQL syntax. See
|
||
<<eql-unsupported-syntax>>.
|
||
|
||
`size`::
|
||
(Optional, integer or float)
|
||
For <<eql-basic-syntax,basic queries>>, the maximum number of matching events to
|
||
return.
|
||
+
|
||
For <<eql-sequences,sequence queries>>, the maximum number of matching sequences
|
||
to return.
|
||
+
|
||
Defaults to `10`. This value must be greater than `0`.
|
||
+
|
||
NOTE: You cannot use <<eql-pipe-ref,pipes>>, such as `head` or `tail`, to exceed
|
||
this value.
|
||
|
||
[[eql-search-api-tiebreaker-field]]
|
||
`tiebreaker_field`::
|
||
(Optional, string)
|
||
Field used to sort events with the same
|
||
<<eql-search-api-timestamp-field,timestamp field>> value. Defaults to
|
||
`event.sequence`, as defined in the {ecs-ref}/ecs-event.html[Elastic Common
|
||
Schema (ECS)].
|
||
+
|
||
By default, matching events in the search response are sorted by timestamp,
|
||
converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
|
||
epoch], in ascending order. If two or more events share the same timestamp, this
|
||
field is used to sort the events in ascending, lexicographic order.
|
||
|
||
[[eql-search-api-timestamp-field]]
|
||
`timestamp_field`::
|
||
+
|
||
--
|
||
(Required*, string)
|
||
Field containing event timestamp.
|
||
|
||
Defaults to `@timestamp`, as defined in the
|
||
{ecs-ref}/ecs-event.html[Elastic Common Schema (ECS)]. If a data stream or index
|
||
does not contain the `@timestamp` field, this value is required.
|
||
|
||
Events in the API response are sorted by this field's value, converted to
|
||
milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in
|
||
ascending order.
|
||
|
||
The timestamp field is typically mapped as a <<date,`date`>> or
|
||
<<date_nanos,`date_nanos`>> field.
|
||
--
|
||
|
||
[[eql-search-api-wait-for-completion-timeout]]
|
||
`wait_for_completion_timeout`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Timeout duration to wait for the request to finish. Defaults to no
|
||
timeout, meaning the request waits for complete search results.
|
||
|
||
If this parameter is specified and the request completes during this period,
|
||
complete search results are returned.
|
||
|
||
If the request does not complete during this period, the search becomes an
|
||
<<eql-search-async,async search>>.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `wait_for_completion_timeout` query
|
||
parameter. If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[role="child_attributes"]
|
||
[[eql-search-api-response-body]]
|
||
==== {api-response-body-title}
|
||
|
||
[[eql-search-api-response-body-search-id]]
|
||
`id`::
|
||
+
|
||
--
|
||
(string)
|
||
Identifier for the search.
|
||
|
||
This search ID is only provided if one of the following conditions is met:
|
||
|
||
* A search request does not return complete results during the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter's timeout period, becoming an <<eql-search-async,async search>>.
|
||
|
||
* The search request's <<eql-search-api-keep-on-completion,`keep_on_completion`>>
|
||
parameter is `true`.
|
||
|
||
You can use this ID with the <<get-async-eql-search-api,get async EQL search
|
||
API>> to get the current status and available results for the search.
|
||
--
|
||
|
||
`is_partial`::
|
||
(boolean)
|
||
If `true`, the response does not contain complete search results.
|
||
|
||
`is_running`::
|
||
+
|
||
--
|
||
(boolean)
|
||
If `true`, the search request is still executing.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
If this parameter and the `is_partial` parameter are `true`, the search is an
|
||
<<eql-search-async,ongoing async search>>. If the `keep_alive` period does not
|
||
pass, the complete search results will be available when the search completes.
|
||
|
||
If `is_partial` is `true` but `is_running` is `false`, the search returned
|
||
partial results due to a failure. Only some shards returned results or the node
|
||
coordinating the search failed.
|
||
====
|
||
--
|
||
|
||
`took`::
|
||
+
|
||
--
|
||
(integer)
|
||
Milliseconds it took {es} to execute the request.
|
||
|
||
This value is calculated by measuring the time elapsed
|
||
between receipt of a request on the coordinating node
|
||
and the time at which the coordinating node is ready to send the response.
|
||
|
||
Took time includes:
|
||
|
||
* Communication time between the coordinating node and data nodes
|
||
* Time the request spends in the `search` <<modules-threadpool,thread pool>>,
|
||
queued for execution
|
||
* Actual execution time
|
||
|
||
Took time does *not* include:
|
||
|
||
* Time needed to send the request to {es}
|
||
* Time needed to serialize the JSON response
|
||
* Time needed to send the response to a client
|
||
--
|
||
|
||
`timed_out`::
|
||
(boolean)
|
||
If `true`, the request timed out before completion.
|
||
|
||
`hits`::
|
||
(object)
|
||
Contains matching events and sequences. Also contains related metadata.
|
||
+
|
||
.Properties of `hits`
|
||
[%collapsible%open]
|
||
====
|
||
|
||
`total`::
|
||
(object)
|
||
Metadata about the number of matching events or sequences.
|
||
+
|
||
.Properties of `total`
|
||
[%collapsible%open]
|
||
=====
|
||
|
||
`value`::
|
||
(integer)
|
||
For <<eql-basic-syntax,basic queries>>, the total number of matching events.
|
||
+
|
||
For <<eql-sequences,sequence queries>>, the total number of matching sequences.
|
||
|
||
`relation`::
|
||
+
|
||
--
|
||
(string)
|
||
Indicates whether the number of events or sequences returned is accurate or a
|
||
lower bound.
|
||
|
||
Returned values are:
|
||
|
||
`eq`::: Accurate
|
||
`gte`::: Lower bound, including returned events or sequences
|
||
--
|
||
=====
|
||
|
||
`sequences`::
|
||
(array of objects)
|
||
Contains event sequences matching the query. Each object represents a
|
||
matching sequence. This parameter is only returned for EQL queries containing
|
||
a <<eql-sequences,sequence>>.
|
||
+
|
||
.Properties of `sequences` objects
|
||
[%collapsible%open]
|
||
=====
|
||
`join_keys`::
|
||
(array of strings)
|
||
Shared field values used to constrain matches in the sequence. These are defined
|
||
using the <<eql-sequences,`by` keyword>> in the EQL query syntax.
|
||
|
||
`events`::
|
||
(array of objects)
|
||
Contains events matching the query. Each object represents a
|
||
matching event.
|
||
+
|
||
.Properties of `events` objects
|
||
[%collapsible%open]
|
||
======
|
||
`_index`::
|
||
(string)
|
||
Name of the index containing the event.
|
||
|
||
`_id`::
|
||
(string)
|
||
Unique identifier for the event.
|
||
This ID is only unique within the index.
|
||
|
||
`_version`::
|
||
(integer)
|
||
Version of the document (event). This version is incremented each time the document is
|
||
updated.
|
||
|
||
`_seq_no`::
|
||
(integer)
|
||
Sequence number assigned to the document (event).
|
||
+
|
||
Sequence numbers are used to ensure an older version of a document
|
||
doesn’t overwrite a newer version. See <<optimistic-concurrency-control>>.
|
||
|
||
`_primary_term`::
|
||
(integer)
|
||
Primary term assigned to the document. See <<optimistic-concurrency-control>>.
|
||
|
||
`_score`::
|
||
(float)
|
||
Positive 32-bit floating point number used to determine the relevance of the
|
||
event. See <<relevance-scores>>.
|
||
|
||
`_source`::
|
||
(object)
|
||
Original JSON body passed for the event at index time.
|
||
======
|
||
=====
|
||
|
||
[[eql-search-api-response-events]]
|
||
`events`::
|
||
(array of objects)
|
||
Contains events matching the query. Each object represents a
|
||
matching event.
|
||
+
|
||
.Properties of `events` objects
|
||
[%collapsible%open]
|
||
=====
|
||
`_index`::
|
||
(string)
|
||
Name of the index containing the event.
|
||
|
||
`_id`::
|
||
(string)
|
||
(string)
|
||
Unique identifier for the event.
|
||
This ID is only unique within the index.
|
||
|
||
`_score`::
|
||
(float)
|
||
Positive 32-bit floating point number used to determine the relevance of the
|
||
event. See <<relevance-scores>>.
|
||
|
||
`_source`::
|
||
(object)
|
||
Original JSON body passed for the event at index time.
|
||
=====
|
||
====
|
||
|
||
[[eql-search-api-example]]
|
||
==== {api-examples-title}
|
||
|
||
[[eql-search-api-basic-query-ex]]
|
||
===== Basic query example
|
||
|
||
The following EQL search request searches for events with an `event.category` of
|
||
`file` that meet the following conditions:
|
||
|
||
* A `file.name` of `cmd.exe`
|
||
* An `process.pid` other than `2013`
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
file where (file.name == "cmd.exe" and process.pid != 2013)
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
// TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/]
|
||
|
||
The API returns the following response. Matching events in the `hits.events`
|
||
property are sorted by <<eql-search-api-timestamp-field,timestamp>>, converted
|
||
to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch],
|
||
in ascending order.
|
||
|
||
If two or more events share the same timestamp, the
|
||
<<eql-search-api-tiebreaker-field,`tiebreaker_field`>> field is used to sort
|
||
the events in ascending, lexicographic order.
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"is_partial": false,
|
||
"is_running": false,
|
||
"took": 6,
|
||
"timed_out": false,
|
||
"hits": {
|
||
"total": {
|
||
"value": 2,
|
||
"relation": "eq"
|
||
},
|
||
"events": [
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_type": "_doc",
|
||
"_id": "fwGeywNsBl8Y9Ys1x51b",
|
||
"_score": null,
|
||
"_source": {
|
||
"@timestamp": "2020-12-06T11:04:07.000Z",
|
||
"event": {
|
||
"category": "file",
|
||
"id": "dGCHwoeS",
|
||
"sequence": 2,
|
||
},
|
||
"file": {
|
||
"accessed": "2020-12-07T11:07:08.000Z",
|
||
"name": "cmd.exe",
|
||
"path": "C:\\Windows\\System32\\cmd.exe",
|
||
"type": "file",
|
||
"size": 16384
|
||
},
|
||
"process": {
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe",
|
||
"pid": 2012
|
||
}
|
||
}
|
||
},
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_type": "_doc",
|
||
"_id": "AtOJ4UjUBAAx3XR5kcCM",
|
||
"_score": null,
|
||
"_source": {
|
||
"@timestamp": "2020-12-07T11:07:08.000Z",
|
||
"event": {
|
||
"category": "file",
|
||
"id": "bYA7gPay",
|
||
"sequence": 4
|
||
},
|
||
"file": {
|
||
"accessed": "2020-12-07T11:07:08.000Z",
|
||
"name": "cmd.exe",
|
||
"path": "C:\\Windows\\System32\\cmd.exe",
|
||
"type": "file",
|
||
"size": 16384
|
||
},
|
||
"process": {
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe",
|
||
"pid": 2012
|
||
}
|
||
}
|
||
}
|
||
]
|
||
}
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
||
// TESTRESPONSE[s/"_id": "fwGeywNsBl8Y9Ys1x51b"/"_id": $body.hits.events.0._id/]
|
||
// TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.events.1._id/]
|
||
|
||
[[eql-search-api-sequence-ex]]
|
||
===== Sequence query example
|
||
|
||
The following EQL search request matches a <<eql-sequences,sequence>> of events
|
||
that:
|
||
|
||
. Start with an event with:
|
||
+
|
||
--
|
||
* An `event.category` of `file`
|
||
* A `file.name` of `cmd.exe`
|
||
* An `process.pid` other than `2013`
|
||
--
|
||
. Followed by an event with:
|
||
+
|
||
--
|
||
* An `event.category` of `process`
|
||
* A `process.executable` that contains the substring `regsvr32`
|
||
--
|
||
|
||
These events must also share the same `process.pid` value.
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
sequence by process.pid
|
||
[ file where file.name == "cmd.exe" and process.pid != 2013 ]
|
||
[ process where stringContains(process.executable, "regsvr32") ]
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
|
||
The API returns the following response. Matching sequences are included in the
|
||
`hits.sequences` property. The `hits.sequences.join_keys` property contains the
|
||
shared `process.pid` value for each matching event.
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"is_partial": false,
|
||
"is_running": false,
|
||
"took": 6,
|
||
"timed_out": false,
|
||
"hits": {
|
||
"total": {
|
||
"value": 1,
|
||
"relation": "eq"
|
||
},
|
||
"sequences": [
|
||
{
|
||
"join_keys": [
|
||
"2012"
|
||
],
|
||
"events": [
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_type": "_doc",
|
||
"_id": "AtOJ4UjUBAAx3XR5kcCM",
|
||
"_version": 1,
|
||
"_seq_no": 3,
|
||
"_primary_term": 1,
|
||
"_score": null,
|
||
"_source": {
|
||
"@timestamp": "2020-12-07T11:07:08.000Z",
|
||
"event": {
|
||
"category": "file",
|
||
"id": "bYA7gPay",
|
||
"sequence": 4
|
||
},
|
||
"file": {
|
||
"accessed": "2020-12-07T11:07:08.000Z",
|
||
"name": "cmd.exe",
|
||
"path": "C:\\Windows\\System32\\cmd.exe",
|
||
"type": "file",
|
||
"size": 16384
|
||
},
|
||
"process": {
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe",
|
||
"pid": 2012
|
||
}
|
||
}
|
||
},
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_type": "_doc",
|
||
"_id": "yDwnGIJouOYGBzP0ZE9n",
|
||
"_version": 1,
|
||
"_seq_no": 4,
|
||
"_primary_term": 1,
|
||
"_score": null,
|
||
"_source": {
|
||
"@timestamp": "2020-12-07T11:07:09.000Z",
|
||
"event": {
|
||
"category": "process",
|
||
"id": "aR3NWVOs",
|
||
"sequence": 5
|
||
},
|
||
"process": {
|
||
"name": "regsvr32.exe",
|
||
"executable": "C:\\Windows\\System32\\regsvr32.exe",
|
||
"pid": 2012
|
||
}
|
||
}
|
||
}
|
||
]
|
||
}
|
||
]
|
||
}
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
||
// TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.sequences.0.events.0._id/]
|
||
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
|