mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-10 23:15:04 +00:00
bff3c7470e
The building block of the eql response is currently the SearchHit. This is a problem since it is tied to an actual search, and thus has scoring, highlighting, shard information and a lot of other things that are not relevant for EQL. This becomes a problem when doing sequence queries since the response is not generated from one search query and thus there are no SearchHits to speak of. Emulating one is not just conceptually incorrect but also problematic since most of the data is missed or made-up. As such this PR introduces a simple class, Event, that maps nicely to the terminology while hiding the ES internals (the use of SearchHit or GetResult/GetResponse depending on the API used). Fix #59764 Fix #59779 Co-authored-by: Igor Motov <igor@motovs.org> (cherry picked from commit 997376fbe6ef2894038968842f5e0635731ede65)
672 lines
18 KiB
Plaintext
672 lines
18 KiB
Plaintext
[role="xpack"]
|
||
[testenv="basic"]
|
||
|
||
[[eql-search-api]]
|
||
=== EQL search API
|
||
++++
|
||
<titleabbrev>EQL search</titleabbrev>
|
||
++++
|
||
|
||
experimental::[]
|
||
|
||
Returns search results for an <<eql,Event Query Language (EQL)>> query.
|
||
|
||
In {es}, EQL assumes each document in a data stream or index corresponds to an
|
||
event.
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
process where process.name = "regsvr32.exe"
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
|
||
[[eql-search-api-request]]
|
||
==== {api-request-title}
|
||
|
||
`GET /<target>/_eql/search`
|
||
|
||
`POST /<target>/_eql/search`
|
||
|
||
[[eql-search-api-prereqs]]
|
||
==== {api-prereq-title}
|
||
|
||
See <<eql-required-fields>>.
|
||
|
||
[[eql-search-api-limitations]]
|
||
===== Limitations
|
||
|
||
See <<eql-syntax-limitations,EQL limitations>>.
|
||
|
||
[[eql-search-api-path-params]]
|
||
==== {api-path-parms-title}
|
||
|
||
`<target>`::
|
||
(Required, string)
|
||
Comma-separated list of data streams, indices, or <<indices-aliases,index
|
||
aliases>> used to limit the request. Accepts wildcard (`*`) expressions.
|
||
+
|
||
To search all data streams and indices in a cluster, use
|
||
`_all` or `*`.
|
||
|
||
[[eql-search-api-query-params]]
|
||
==== {api-query-parms-title}
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
||
+
|
||
Defaults to `false`.
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
||
+
|
||
Defaults to `open`.
|
||
|
||
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
|
||
|
||
`keep_alive`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Period for which the search and its results are stored on the cluster. Defaults
|
||
to `5d` (five days).
|
||
|
||
When this period expires, the search and its results are deleted, even if the
|
||
search is still ongoing.
|
||
|
||
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
||
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
||
complete within the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter, regardless of this value.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_alive` request body parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
`keep_on_completion`::
|
||
+
|
||
--
|
||
(Optional, boolean)
|
||
If `true`, the search and its results are stored on the cluster.
|
||
|
||
If `false`, the search and its results are stored on the cluster only if the
|
||
request does not complete during the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter. Defaults to `false`.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_on_completion` request body
|
||
parameter. If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
`wait_for_completion_timeout`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Timeout duration to wait for the request to finish. Defaults to no
|
||
timeout, meaning the request waits for complete search results.
|
||
|
||
If this parameter is specified and the request completes during this period,
|
||
complete search results are returned.
|
||
|
||
If the request does not complete during this period, the search becomes an
|
||
<<eql-search-async,async search>>.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `wait_for_completion_timeout` request
|
||
body parameter. If both parameters are specified, only the query parameter is
|
||
used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-request-body]]
|
||
==== {api-request-body-title}
|
||
|
||
`case_sensitive`::
|
||
(Optional, boolean)
|
||
If `true`, matching for the <<eql-search-api-request-query-param,EQL query>> is
|
||
case sensitive. Defaults to `false`.
|
||
|
||
`event_category_field`::
|
||
(Required*, string)
|
||
Field containing the event classification, such as `process`, `file`, or
|
||
`network`.
|
||
+
|
||
Defaults to `event.category`, as defined in the {ecs-ref}/ecs-event.html[Elastic
|
||
Common Schema (ECS)]. If a data stream or index does not contain the
|
||
`event.category` field, this value is required.
|
||
+
|
||
The event category field is typically mapped as a <<keyword,`keyword`>> or
|
||
<<constant-keyword,constant keyword>> field.
|
||
|
||
`fetch_size`::
|
||
(Optional, integer)
|
||
Maximum number of events to search at a time for sequence queries. Defaults to
|
||
`1000`.
|
||
+
|
||
This value must be greater than `2` but cannot exceed the value of the
|
||
<<index-max-result-window,`index.max_result_window`>> setting, which defaults to
|
||
`10000`.
|
||
+
|
||
Internally, a sequence query fetches and paginates sets of events to search for
|
||
matches. This parameter controls the size of those sets. This parameter does not
|
||
limit the total number of events searched or the number of matching events
|
||
returned.
|
||
+
|
||
A greater `fetch_size` value often increases search speed but uses more memory.
|
||
|
||
`filter`::
|
||
(Optional, <<query-dsl,query DSL object>>)
|
||
Query, written in query DSL, used to filter the events on which the EQL query
|
||
runs.
|
||
|
||
`keep_alive`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Period for which the search and its results are stored on the cluster. Defaults
|
||
to `5d` (five days).
|
||
|
||
When this period expires, the search and its results are deleted, even if the
|
||
search is still ongoing.
|
||
|
||
If the <<eql-search-api-keep-on-completion,`keep_on_completion`>> parameter is
|
||
`false`, {es} only stores <<eql-search-async,async searches>> that do not
|
||
complete within the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter, regardless of this value.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_alive` query parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-keep-on-completion]]
|
||
`keep_on_completion`::
|
||
+
|
||
--
|
||
(Optional, boolean)
|
||
If `true`, the search and its results are stored on the cluster.
|
||
|
||
If `false`, the search and its results are stored on the cluster only if the
|
||
request does not complete during the period set by the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter. Defaults to `false`.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `keep_on_completion` query parameter.
|
||
If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[[eql-search-api-request-query-param]]
|
||
`query`::
|
||
(Required, string)
|
||
<<eql-syntax,EQL>> query you wish to run.
|
||
+
|
||
IMPORTANT: This parameter supports a subset of EQL syntax. See
|
||
<<eql-unsupported-syntax>>.
|
||
|
||
`size`::
|
||
(Optional, integer or float)
|
||
For <<eql-basic-syntax,basic queries>>, the maximum number of matching events to
|
||
return.
|
||
+
|
||
For <<eql-sequences,sequence queries>>, the maximum number of matching sequences
|
||
to return.
|
||
+
|
||
Defaults to `10`. This value must be greater than `0`.
|
||
+
|
||
NOTE: You cannot use <<eql-pipe-ref,pipes>>, such as `head` or `tail`, to exceed
|
||
this value.
|
||
|
||
[[eql-search-api-tiebreaker-field]]
|
||
`tiebreaker_field`::
|
||
(Optional, string)
|
||
Field used to sort events with the same
|
||
<<eql-search-api-timestamp-field,timestamp field>> value. Defaults to
|
||
`event.sequence`, as defined in the {ecs-ref}/ecs-event.html[Elastic Common
|
||
Schema (ECS)].
|
||
+
|
||
By default, matching events in the search response are sorted by timestamp,
|
||
converted to milliseconds since the {wikipedia}/Unix_time[Unix
|
||
epoch], in ascending order. If two or more events share the same timestamp, this
|
||
field is used to sort the events in ascending, lexicographic order.
|
||
|
||
[[eql-search-api-timestamp-field]]
|
||
`timestamp_field`::
|
||
+
|
||
--
|
||
(Required*, string)
|
||
Field containing event timestamp.
|
||
|
||
Defaults to `@timestamp`, as defined in the
|
||
{ecs-ref}/ecs-event.html[Elastic Common Schema (ECS)]. If a data stream or index
|
||
does not contain the `@timestamp` field, this value is required.
|
||
|
||
Events in the API response are sorted by this field's value, converted to
|
||
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in
|
||
ascending order.
|
||
|
||
The timestamp field is typically mapped as a <<date,`date`>> or
|
||
<<date_nanos,`date_nanos`>> field.
|
||
--
|
||
|
||
[[eql-search-api-wait-for-completion-timeout]]
|
||
`wait_for_completion_timeout`::
|
||
+
|
||
--
|
||
(Optional, <<time-units,time value>>)
|
||
Timeout duration to wait for the request to finish. Defaults to no
|
||
timeout, meaning the request waits for complete search results.
|
||
|
||
If this parameter is specified and the request completes during this period,
|
||
complete search results are returned.
|
||
|
||
If the request does not complete during this period, the search becomes an
|
||
<<eql-search-async,async search>>.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
You can also specify this value using the `wait_for_completion_timeout` query
|
||
parameter. If both parameters are specified, only the query parameter is used.
|
||
====
|
||
--
|
||
|
||
[role="child_attributes"]
|
||
[[eql-search-api-response-body]]
|
||
==== {api-response-body-title}
|
||
|
||
[[eql-search-api-response-body-search-id]]
|
||
`id`::
|
||
+
|
||
--
|
||
(string)
|
||
Identifier for the search.
|
||
|
||
This search ID is only provided if one of the following conditions is met:
|
||
|
||
* A search request does not return complete results during the
|
||
<<eql-search-api-wait-for-completion-timeout,`wait_for_completion_timeout`>>
|
||
parameter's timeout period, becoming an <<eql-search-async,async search>>.
|
||
|
||
* The search request's <<eql-search-api-keep-on-completion,`keep_on_completion`>>
|
||
parameter is `true`.
|
||
|
||
You can use this ID with the <<get-async-eql-search-api,get async EQL search
|
||
API>> to get the current status and available results for the search.
|
||
--
|
||
|
||
`is_partial`::
|
||
(boolean)
|
||
If `true`, the response does not contain complete search results.
|
||
|
||
`is_running`::
|
||
+
|
||
--
|
||
(boolean)
|
||
If `true`, the search request is still executing.
|
||
|
||
[IMPORTANT]
|
||
====
|
||
If this parameter and the `is_partial` parameter are `true`, the search is an
|
||
<<eql-search-async,ongoing async search>>. If the `keep_alive` period does not
|
||
pass, the complete search results will be available when the search completes.
|
||
|
||
If `is_partial` is `true` but `is_running` is `false`, the search returned
|
||
partial results due to a failure. Only some shards returned results or the node
|
||
coordinating the search failed.
|
||
====
|
||
--
|
||
|
||
`took`::
|
||
+
|
||
--
|
||
(integer)
|
||
Milliseconds it took {es} to execute the request.
|
||
|
||
This value is calculated by measuring the time elapsed
|
||
between receipt of a request on the coordinating node
|
||
and the time at which the coordinating node is ready to send the response.
|
||
|
||
Took time includes:
|
||
|
||
* Communication time between the coordinating node and data nodes
|
||
* Time the request spends in the `search` <<modules-threadpool,thread pool>>,
|
||
queued for execution
|
||
* Actual execution time
|
||
|
||
Took time does *not* include:
|
||
|
||
* Time needed to send the request to {es}
|
||
* Time needed to serialize the JSON response
|
||
* Time needed to send the response to a client
|
||
--
|
||
|
||
`timed_out`::
|
||
(boolean)
|
||
If `true`, the request timed out before completion.
|
||
|
||
`hits`::
|
||
(object)
|
||
Contains matching events and sequences. Also contains related metadata.
|
||
+
|
||
.Properties of `hits`
|
||
[%collapsible%open]
|
||
====
|
||
|
||
`total`::
|
||
(object)
|
||
Metadata about the number of matching events or sequences.
|
||
+
|
||
.Properties of `total`
|
||
[%collapsible%open]
|
||
=====
|
||
|
||
`value`::
|
||
(integer)
|
||
For <<eql-basic-syntax,basic queries>>, the total number of matching events.
|
||
+
|
||
For <<eql-sequences,sequence queries>>, the total number of matching sequences.
|
||
|
||
`relation`::
|
||
+
|
||
--
|
||
(string)
|
||
Indicates whether the number of events or sequences returned is accurate or a
|
||
lower bound.
|
||
|
||
Returned values are:
|
||
|
||
`eq`::: Accurate
|
||
`gte`::: Lower bound, including returned events or sequences
|
||
--
|
||
=====
|
||
|
||
`sequences`::
|
||
(array of objects)
|
||
Contains event sequences matching the query. Each object represents a
|
||
matching sequence. This parameter is only returned for EQL queries containing
|
||
a <<eql-sequences,sequence>>.
|
||
+
|
||
.Properties of `sequences` objects
|
||
[%collapsible%open]
|
||
=====
|
||
`join_keys`::
|
||
(array of values)
|
||
Shared field values used to constrain matches in the sequence. These are defined
|
||
using the <<eql-sequences,`by` keyword>> in the EQL query syntax.
|
||
|
||
`events`::
|
||
(array of objects)
|
||
Contains events matching the query. Each object represents a
|
||
matching event.
|
||
+
|
||
.Properties of `events` objects
|
||
[%collapsible%open]
|
||
======
|
||
`_index`::
|
||
(string)
|
||
Name of the index containing the event.
|
||
|
||
`_id`::
|
||
(string)
|
||
Unique identifier for the event.
|
||
This ID is only unique within the index.
|
||
|
||
`_version`::
|
||
(integer)
|
||
Version of the document (event). This version is incremented each time the document is
|
||
updated.
|
||
|
||
`_seq_no`::
|
||
(integer)
|
||
Sequence number assigned to the document (event).
|
||
+
|
||
Sequence numbers are used to ensure an older version of a document
|
||
doesn’t overwrite a newer version. See <<optimistic-concurrency-control>>.
|
||
|
||
`_primary_term`::
|
||
(integer)
|
||
Primary term assigned to the document. See <<optimistic-concurrency-control>>.
|
||
|
||
`_source`::
|
||
(object)
|
||
Original JSON body passed for the event at index time.
|
||
======
|
||
=====
|
||
|
||
[[eql-search-api-response-events]]
|
||
`events`::
|
||
(array of objects)
|
||
Contains events matching the query. Each object represents a
|
||
matching event.
|
||
+
|
||
.Properties of `events` objects
|
||
[%collapsible%open]
|
||
=====
|
||
`_index`::
|
||
(string)
|
||
Name of the index containing the event.
|
||
|
||
`_id`::
|
||
(string)
|
||
(string)
|
||
Unique identifier for the event.
|
||
This ID is only unique within the index.
|
||
|
||
`_source`::
|
||
(object)
|
||
Original JSON body passed for the event at index time.
|
||
=====
|
||
====
|
||
|
||
[[eql-search-api-example]]
|
||
==== {api-examples-title}
|
||
|
||
[[eql-search-api-basic-query-ex]]
|
||
===== Basic query example
|
||
|
||
The following EQL search request searches for events with an `event.category` of
|
||
`process` that meet the following conditions:
|
||
|
||
* A `process.name` of `cmd.exe`
|
||
* An `process.pid` other than `2013`
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
process where (process.name == "cmd.exe" and process.pid != 2013)
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
// TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/]
|
||
|
||
The API returns the following response. Matching events in the `hits.events`
|
||
property are sorted by <<eql-search-api-timestamp-field,timestamp>>, converted
|
||
to milliseconds since the {wikipedia}/Unix_time[Unix epoch],
|
||
in ascending order.
|
||
|
||
If two or more events share the same timestamp, the
|
||
<<eql-search-api-tiebreaker-field,`tiebreaker_field`>> field is used to sort
|
||
the events in ascending, lexicographic order.
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"is_partial": false,
|
||
"is_running": false,
|
||
"took": 6,
|
||
"timed_out": false,
|
||
"hits": {
|
||
"total": {
|
||
"value": 2,
|
||
"relation": "eq"
|
||
},
|
||
"events": [
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_id": "babI3XMBI9IjHuIqU0S_",
|
||
"_source": {
|
||
"@timestamp": "2099-12-06T11:04:05.000Z",
|
||
"event": {
|
||
"category": "process",
|
||
"id": "edwCRnyD",
|
||
"sequence": 1
|
||
},
|
||
"process": {
|
||
"pid": 2012,
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe"
|
||
}
|
||
}
|
||
},
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_id": "b6bI3XMBI9IjHuIqU0S_",
|
||
"_source": {
|
||
"@timestamp": "2099-12-07T11:06:07.000Z",
|
||
"event": {
|
||
"category": "process",
|
||
"id": "cMyt5SZ2",
|
||
"sequence": 3
|
||
},
|
||
"process": {
|
||
"pid": 2012,
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe"
|
||
}
|
||
}
|
||
}
|
||
]
|
||
}
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
||
// TESTRESPONSE[s/"_id": "babI3XMBI9IjHuIqU0S_"/"_id": $body.hits.events.0._id/]
|
||
// TESTRESPONSE[s/"_id": "b6bI3XMBI9IjHuIqU0S_"/"_id": $body.hits.events.1._id/]
|
||
|
||
[[eql-search-api-sequence-ex]]
|
||
===== Sequence query example
|
||
|
||
The following EQL search request matches a <<eql-sequences,sequence>> of events
|
||
that:
|
||
|
||
. Start with an event with:
|
||
+
|
||
--
|
||
* An `event.category` of `file`
|
||
* A `file.name` of `cmd.exe`
|
||
* An `process.pid` other than `2013`
|
||
--
|
||
. Followed by an event with:
|
||
+
|
||
--
|
||
* An `event.category` of `process`
|
||
* A `process.executable` that contains the substring `regsvr32`
|
||
--
|
||
|
||
These events must also share the same `process.pid` value.
|
||
|
||
[source,console]
|
||
----
|
||
GET /my-index-000001/_eql/search
|
||
{
|
||
"query": """
|
||
sequence by process.pid
|
||
[ file where file.name == "cmd.exe" and process.pid != 2013 ]
|
||
[ process where stringContains(process.executable, "regsvr32") ]
|
||
"""
|
||
}
|
||
----
|
||
// TEST[setup:sec_logs]
|
||
|
||
The API returns the following response. Matching sequences are included in the
|
||
`hits.sequences` property. The `hits.sequences.join_keys` property contains the
|
||
shared `process.pid` value for each matching event.
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"is_partial": false,
|
||
"is_running": false,
|
||
"took": 6,
|
||
"timed_out": false,
|
||
"hits": {
|
||
"total": {
|
||
"value": 1,
|
||
"relation": "eq"
|
||
},
|
||
"sequences": [
|
||
{
|
||
"join_keys": [
|
||
2012
|
||
],
|
||
"events": [
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_id": "AtOJ4UjUBAAx3XR5kcCM",
|
||
"_source": {
|
||
"@timestamp": "2099-12-06T11:04:07.000Z",
|
||
"event": {
|
||
"category": "file",
|
||
"id": "dGCHwoeS",
|
||
"sequence": 2
|
||
},
|
||
"file": {
|
||
"accessed": "2099-12-07T11:07:08.000Z",
|
||
"name": "cmd.exe",
|
||
"path": "C:\\Windows\\System32\\cmd.exe",
|
||
"type": "file",
|
||
"size": 16384
|
||
},
|
||
"process": {
|
||
"pid": 2012,
|
||
"name": "cmd.exe",
|
||
"executable": "C:\\Windows\\System32\\cmd.exe"
|
||
}
|
||
}
|
||
},
|
||
{
|
||
"_index": "my-index-000001",
|
||
"_id": "OQmfCaduce8zoHT93o4H",
|
||
"_source": {
|
||
"@timestamp": "2099-12-07T11:07:09.000Z",
|
||
"event": {
|
||
"category": "process",
|
||
"id": "aR3NWVOs",
|
||
"sequence": 4
|
||
},
|
||
"process": {
|
||
"pid": 2012,
|
||
"name": "regsvr32.exe",
|
||
"command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
|
||
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
||
}
|
||
}
|
||
}
|
||
]
|
||
}
|
||
]
|
||
}
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/"took": 6/"took": $body.took/]
|
||
// TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.sequences.0.events.0._id/]
|
||
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.1._id/]
|