honeymoose/OpenSearch
1
0
Fork 0
mirror of https://github.com/honeymoose/OpenSearch.git synced 2025-02-10 06:55:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/reference/eql/search.asciidoc
Costin Leau 3c81b91474 EQL: Add Head/Tail pipe support (#58536) 
Introduce pipe support, in particular head and tail
(which can also be chained).

(cherry picked from commit 4521ca3367147d4d6531cf0ab975d8d705f400ea)
(cherry picked from commit d6731d659d012c96b19879d13cfc9e1eaf4745a4)
2020-06-27 09:49:14 +03:00

882 lines
25 KiB
Plaintext
Raw Blame History

[role="xpack"]
[testenv="basic"]
[[eql-search]]
== Run an EQL search
dev::[]
To start using EQL in {es}, first ensure your event data meets
<<eql-requirements,EQL requirements>>. You can then use the <<eql-search-api,EQL
search API>> to search event data stored in one or more {es} indices.
.*Example*
[%collapsible]
====
To get started, ingest or add the data to an {es} index.
The following <<docs-bulk,bulk API>> request adds some example log data to the
`sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema
(ECS)].
[source,console]
----
PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD","sequence": 1 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "dGCHwoeS", "sequence": 2 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP
[TIP]
=====
You also can set up {beats-ref}/getting-started.html[{beats}], such as
{auditbeat-ref}/auditbeat-getting-started.html[{auditbeat}] or
{winlogbeat-ref}/winlogbeat-getting-started.html[{winlogbeat}], to automatically
send and index your event data in {es}. See
{beats-ref}/getting-started.html[Getting started with {beats}].
=====
You can now use the EQL search API to search this index using an EQL query.
The following request searches the `sec_logs` index using the EQL query
specified in the `query` parameter. The EQL query matches events with an
`event.category` of `process` that have a `process.name` of `cmd.exe`.
[source,console]
----
GET /sec_logs/_eql/search
{
"query": """
process where process.name == "cmd.exe"
"""
}
----
Because the `sec_log` index follows the ECS, you don't need to specify the
required <<eql-required-fields,event category or timestamp>> fields. The request
uses the `event.category` and `@timestamp` fields by default.
The API returns the following response containing the matching events. Events
in the response are sorted by timestamp, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 60,
"timed_out": false,
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"events": [
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"@timestamp": "2020-12-06T11:04:05.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "edwCRnyD",
"sequence": 1
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"fields": {
"@timestamp": [
"1607252645000"
]
},
"sort": [
1607252645000
]
},
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "3",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:06:07.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "cMyt5SZ2",
"sequence": 3
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"fields": {
"@timestamp": [
"1607339167000"
]
},
"sort": [
1607339167000
]
}
]
}
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====
[discrete]
[[eql-search-sequence]]
=== Search for a sequence of events
Many query languages allow you to match single events. However, EQL's
<<eql-sequences,sequence syntax>> lets you match an ordered series of events.
.*Example*
[%collapsible]
====
The following EQL search request matches a sequence that:
. Starts with an event with:
+
--
* An `event.category` of `file`
* A `file.name` of `cmd.exe`
--
. Followed by an event with:
+
--
* An `event.category` of `process`
* A `process.name` that contains the substring `regsvr32`
--
[source,console]
----
GET /sec_logs/_eql/search
{
"query": """
sequence
[ file where file.name == "cmd.exe" ]
[ process where stringContains(process.name, "regsvr32") ]
"""
}
----
// TEST[s/search/search\?filter_path\=\-\*\.sequences\.events\.\*fields/]
The API returns the following response. Matching events in
the `hits.sequences.events` property are sorted by
<<eql-search-api-timestamp-field,timestamp>>, converted to milliseconds since
the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 60,
"timed_out": false,
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"sequences": [
{
"events": [
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "4",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:07:08.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe",
"type": "file",
"size": 16384
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"sort": [
1607339228000
]
},
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "5",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:07:09.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "aR3NWVOs",
"sequence": 5
},
"process": {
"name": "regsvr32.exe",
"path": "C:\\Windows\\System32\\regsvr32.exe"
}
},
"sort": [
1607339229000
]
}
]
}
]
}
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
You can further constrain matching event sequences using the `by` keyword.
The following EQL search request adds `by agent.id` to each event item. This
ensures events matching the sequence share the same `agent.id` field value.
[source,console]
----
GET /sec_logs/_eql/search
{
"query": """
sequence
[ file where file.name == "cmd.exe" ] by agent.id
[ process where stringContains(process.name, "regsvr32") ] by agent.id
"""
}
----
Because the `agent.id` field is shared across all events in the sequence, it
can be included using `sequence by`. The following query is equivalent to the
prior one.
[source,console]
----
GET /sec_logs/_eql/search
{
"query": """
sequence by agent.id
[ file where file.name == "cmd.exe" ]
[ process where stringContains(process.name, "regsvr32") ]
"""
}
----
// TEST[s/search/search\?filter_path\=\-\*\.sequences\.events\.\*fields/]
The API returns the following response. The `hits.sequences.join_keys` property
contains the shared `agent.id` value for each matching event.
[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 60,
"timed_out": false,
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"sequences": [
{
"join_keys": [
"8a4f500d"
],
"events": [
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "4",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:07:08.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "file",
"id": "bYA7gPay",
"sequence": 4
},
"file": {
"accessed": "2020-12-07T11:07:08.000Z",
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe",
"type": "file",
"size": 16384
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"sort": [
1607339228000
]
},
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "5",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:07:09.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "aR3NWVOs",
"sequence": 5
},
"process": {
"name": "regsvr32.exe",
"path": "C:\\Windows\\System32\\regsvr32.exe"
}
},
"sort": [
1607339229000
]
}
]
}
]
}
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====
[discrete]
[[eql-search-specify-event-category-field]]
=== Specify an event category field
The EQL search API uses `event.category` as the required
<<eql-required-fields,event category field>> by default. You can use the
`event_category_field` parameter to specify another event category field.
.*Example*
[%collapsible]
====
The following request specifies `file.type` as the event category
field.
[source,console]
----
GET /sec_logs/_eql/search
{
"event_category_field": "file.type",
"query": """
file where agent.id == "8a4f500d"
"""
}
----
====
[discrete]
[[eql-search-specify-timestamp-field]]
=== Specify a timestamp field
The EQL search API uses `@timestamp` as the required <<eql-required-fields,event
timestamp field>> by default. You can use the `timestamp_field` parameter to
specify another timestamp field.
.*Example*
[%collapsible]
====
The following request specifies `file.accessed` as the event
timestamp field.
[source,console]
----
GET /sec_logs/_eql/search
{
"timestamp_field": "file.accessed",
"query": """
file where (file.size > 1 and file.type == "file")
"""
}
----
====
[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker
By default, the EQL search API sorts matching events in the search response by
timestamp. However, if two or more events share the same timestamp, a tiebreaker
field is used to sort the events in ascending, lexicographic order.
The EQL search API uses `event.sequence` as the default tiebreaker field. You
can use the `tiebreaker_field` parameter to specify another field.
.*Example*
[%collapsible]
====
The following request specifies `event.start` as the tiebreaker field.
[source,console]
----
GET /sec_logs/_eql/search
{
"tiebreaker_field": "event.id",
"query": """
process where process.name == "cmd.exe" and stringContains(process.path, "System32")
"""
}
----
The API returns the following response. Note the `sort` property of each
matching event contains an array of two items:
* The first item is the event's <<eql-search-api-timestamp-field,timestamp>>,
converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
epoch].
* The second item is the event's `event.id` value. This value is used as a sort
tiebreaker for events with the same timestamp.
[source,console-result]
----
{
"is_partial": false,
"is_running": false,
"took": 34,
"timed_out": false,
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"events": [
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "1",
"_score": null,
"_source": {
"@timestamp": "2020-12-06T11:04:05.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "edwCRnyD",
"sequence": 1
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"fields": {
"@timestamp": [
"1607252645000"
]
},
"sort": [
1607252645000, <1>
"edwCRnyD" <2>
]
},
{
"_index": "sec_logs",
"_type": "_doc",
"_id": "3",
"_score": null,
"_source": {
"@timestamp": "2020-12-07T11:06:07.000Z",
"agent": {
"id": "8a4f500d"
},
"event": {
"category": "process",
"id": "cMyt5SZ2",
"sequence": 3
},
"process": {
"name": "cmd.exe",
"path": "C:\\Windows\\System32\\cmd.exe"
}
},
"fields": {
"@timestamp": [
"1607339167000"
]
},
"sort": [
1607339167000, <1>
"cMyt5SZ2" <2>
]
}
]
}
}
----
// TESTRESPONSE[s/"took": 34/"took": $body.took/]
<1> The event's <<eql-search-api-timestamp-field,timestamp>>, converted to
milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix
epoch]
<2> The event's `event.id` value.
====
[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using query DSL
You can use the EQL search API's `filter` parameter to specify an additional
query using <<query-dsl,query DSL>>. This query filters the documents on which
the EQL query runs.
.*Example*
[%collapsible]
====
The following request uses a `range` query to filter the `sec_logs`
index down to only documents with a `file.size` value greater than `1` but less
than `1000000` bytes. The EQL query in `query` parameter then runs on these
filtered documents.
[source,console]
----
GET /sec_logs/_eql/search
{
"filter": {
"range" : {
"file.size" : {
"gte" : 1,
"lte" : 1000000
}
}
},
"query": """
file where (file.type == "file" and file.name == "cmd.exe")
"""
}
----
====
[discrete]
[[eql-search-async]]
=== Run an async EQL search
EQL searches in {es} are designed to run on large volumes of data quickly,
often returning results in milliseconds. Because of this, the EQL search API
runs _synchronous_ searches by default. This means the search request waits for
complete results before returning a response.
However, complete results can take longer for searches across:
* <<frozen-indices,Frozen indices>>
* <<modules-cross-cluster-search,Multiple clusters>>
* Many shards
To avoid long waits, you can use the EQL search API's
`wait_for_completion_timeout` parameter to run an _asynchronous_, or _async_,
search.
Set the `wait_for_completion_timeout` parameter to a duration you'd like to wait
for complete search results. If the search request does not finish within this
period, the search becomes an async search. The EQL search
API returns a response that includes:
* A search ID, which can be used to monitor the progress of the async search and
retrieve complete results when it finishes.
* An `is_partial` value of `true`, indicating the response does not contain
complete search results.
* An `is_running` value of `true`, indicating the search is async and ongoing.
The async search continues to run in the background without blocking
other requests.
[%collapsible]
.*Example*
====
The following request searches the `frozen_sec_logs` index, which has been
<<frozen-indices,frozen>> for storage and is rarely searched.
Because searches on frozen indices are expected to take longer to complete, the
request contains a `wait_for_completion_timeout` parameter value of `2s`
(two seconds).
If the request does not return complete results in two seconds, the search
becomes an async search and a search ID is returned.
[source,console]
----
GET /frozen_sec_logs/_eql/search
{
"wait_for_completion_timeout": "2s",
"query": """
process where process.name == "cmd.exe"
"""
}
----
// TEST[s/frozen_sec_logs/sec_logs/]
After two seconds, the request returns the following response. Note the
`is_partial` and `is_running` properties are `true`, indicating an ongoing async
search.
[source,console-result]
----
{
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
"is_partial": true,
"is_running": true,
"took": 2000,
"timed_out": false,
"hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
// TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
====
You can use the the returned search ID and the <<get-async-eql-search-api,get
async EQL search API>> to check the progress of an ongoing async search.
The get async EQL search API also accepts a `wait_for_completion_timeout` query
parameter. Set the `wait_for_completion_timeout` parameter to a duration you'd
like to wait for complete search results. If the request does not complete
during this period, the response returns an `is_partial` value of `true` and no
search results.
[%collapsible]
.*Example*
====
The following get async EQL search API request checks the progress of the
previous async EQL search. The request specifies a `wait_for_completion_timeout`
query parameter value of `2s` (two seconds).
[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
----
// TEST[skip: no access to search ID]
The request returns the following response. Note the `is_partial` and
`is_running` properties are `false`, indicating the async EQL search has
finished and the search results in the `hits` property are complete.
[source,console-result]
----
{
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
"is_partial": false,
"is_running": false,
"took": 2000,
"timed_out": false,
"hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"_index": "frozen_sec_logs"/"_index": "sec_logs"/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
====
[discrete]
[[eql-search-store-async-eql-search]]
=== Change the search retention period
By default, the EQL search API only stores async searches and their results for
five days. After this period, any ongoing searches or saved results are deleted.
You can use the EQL search API's `keep_alive` parameter to change the duration
of this period.
.*Example*
[%collapsible]
====
In the following EQL search API request, the `keep_alive` parameter is `2d` (two
days). This means that if the search becomes async, its results
are stored on the cluster for two days. After two days, the async
search and its results are deleted, even if it's still ongoing.
[source,console]
----
GET /sec_logs/_eql/search
{
"keep_alive": "2d",
"wait_for_completion_timeout": "2s",
"query": """
process where process.name == "cmd.exe"
"""
}
----
====
You can use the <<get-async-eql-search-api,get async EQL search API>>'s
`keep_alive` query parameter to later change the retention period. The new
retention period starts after the get async EQL search API request executes.
.*Example*
[%collapsible]
====
The following get async EQL search API request sets the `keep_alive` query
parameter to `5d` (five days). The async search and its results are deleted five
days after the get async EQL search API request executes.
[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]
====
You can use the <<delete-async-eql-search-api,delete async EQL search API>> to
manually delete an async EQL search before the `keep_alive` period ends. If the
search is still ongoing, this cancels the search request.
.*Example*
[%collapsible]
====
The following delete async EQL search API request deletes an async EQL search
and its results.
[source,console]
----
DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]
====
[discrete]
[[eql-search-store-sync-eql-search]]
=== Store synchronous EQL searches
By default, the EQL search API only stores async searches that cannot be
completed within the period set by the `wait_for_completion_timeout` parameter.
To save the results of searches that complete during this period, set the
`keep_on_completion` parameter to `true`.
[%collapsible]
.*Example*
====
In the following EQL search API request, the `keep_on_completion` parameter is
`true`. This means the search results are stored on the cluster, even if
the search completes within the `2s` (two-second) period set by the
`wait_for_completion_timeout` parameter.
[source,console]
----
GET /sec_logs/_eql/search
{
"keep_on_completion": true,
"wait_for_completion_timeout": "2s",
"query": """
process where process.name == "cmd.exe"
"""
}
----
The API returns the following response. Note that a search ID is provided in the
`id` property. The `is_partial` and `is_running` properties are `false`,
indicating the EQL search was synchronous and returned complete search results.
[source,console-result]
----
{
"id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
"is_partial": false,
"is_running": false,
"took": 52,
"timed_out": false,
"hits": ...
}
----
// TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
// TESTRESPONSE[s/"took": 52/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
You can use the search ID and the <<get-async-eql-search-api,get async EQL
search API>> to retrieve the same results later.
[source,console]
----
GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
----
// TEST[skip: no access to search ID]
====
Saved synchronous searches are still subject to the storage retention period set
by the `keep_alive` parameter. After this period, the search and its saved
results are deleted.
You can also manually delete saved synchronous searches using the
<<delete-async-eql-search-api,delete async EQL search API>>.
[discrete]
[[eql-search-case-sensitive]]
=== Run a case-sensitive EQL search
By default, matching for EQL queries is case-insensitive. You can use the EQL
search API's `case_sensitive` parameter to toggle case sensitivity on or off.
.*Example*
[%collapsible]
====
The following search request contains a query that matches `process` events
with a `process.path` containing `System32`.
Because the `case_sensitive` parameter is `true`, this query only matches
`process.path` values containing `System32` with the exact same capitalization.
A `process.path` value containing `system32` or `SYSTEM32` would not match this
query.
[source,console]
----
GET /sec_logs/_eql/search
{
"keep_on_completion": true,
"case_sensitive": true,
"query": """
process where stringContains(process.path, "System32")
"""
}
----
====
Reference in New Issue View Git Blame Copy Permalink