[role="xpack"] [testenv="basic"] [[eql-search]] == Run an EQL search experimental::[] To start using EQL in {es}, first ensure your event data meets <>. You can then use the <> to search event data stored in one or more {es} data streams or indices. The API requires a query written in {es}'s supported <>. To get started, ingest or add the data to an {es} data stream or index. The following <> request adds some example log data to the `sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema (ECS)]. [source,console] ---- PUT /sec_logs/_bulk?refresh {"index":{ }} { "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD", "sequence": 1 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } {"index":{ }} { "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "dGCHwoeS", "sequence": 2 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } {"index":{ }} { "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } {"index":{ }} { "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } {"index":{ }} { "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } {"index":{ }} { "@timestamp": "2020-12-07T11:07:10.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "GTSmSqgz0U", "sequence": 6, "type": "termination" }, "process": { "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } ---- // TESTSETUP [TIP] ===== You also can set up {beats-ref}/getting-started.html[{beats}], such as {auditbeat-ref}/auditbeat-installation-configuration.html[{auditbeat}] or {winlogbeat-ref}/winlogbeat-installation-configuration.html[{winlogbeat}], to automatically send and index your event data in {es}. See {beats-ref}/getting-started.html[Getting started with {beats}]. ===== You can now use the EQL search API to search this index using an EQL query. The following request searches the `sec_logs` index using the EQL query specified in the `query` parameter. The EQL query matches events with an `event.category` of `process` that have a `process.name` of `cmd.exe`. [source,console] ---- GET /sec_logs/_eql/search { "query": """ process where process.name == "cmd.exe" """ } ---- // TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/] Because the `sec_log` index follows the ECS, you don't need to specify the required <> fields. The request uses the `event.category` and `@timestamp` fields by default. The API returns the following response containing the matching events. Events in the response are sorted by timestamp, converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. [source,console-result] ---- { "is_partial": false, "is_running": false, "took": 60, "timed_out": false, "hits": { "total": { "value": 2, "relation": "eq" }, "events": [ { "_index": "sec_logs", "_type": "_doc", "_id": "OQmfCaduce8zoHT93o4H", "_score": null, "_source": { "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD", "sequence": 1 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } }, { "_index": "sec_logs", "_type": "_doc", "_id": "xLkCaj4EujzdNSxfYLbO", "_score": null, "_source": { "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } } ] } } ---- // TESTRESPONSE[s/"took": 60/"took": $body.took/] // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/] // TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/] [discrete] [[eql-search-sequence]] === Search for a sequence of events Many query languages allow you to match single events. However, EQL's <> lets you match an ordered series of events. The following EQL search request matches a sequence that: . Starts with an event with: + -- * An `event.category` of `file` * A `file.name` of `cmd.exe` -- . Followed by an event with: + -- * An `event.category` of `process` * A `process.name` that contains the substring `regsvr32` -- [source,console] ---- GET /sec_logs/_eql/search { "query": """ sequence [ file where file.name == "cmd.exe" ] [ process where stringContains(process.name, "regsvr32") ] """ } ---- The API returns the following response. Matching events in the `hits.sequences.events` property are sorted by <>, converted to milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order. [source,console-result] ---- { "is_partial": false, "is_running": false, "took": 60, "timed_out": false, "hits": { "total": { "value": 1, "relation": "eq" }, "sequences": [ { "events": [ { "_index": "sec_logs", "_type": "_doc", "_id": "AtOJ4UjUBAAx3XR5kcCM", "_version" : 1, "_seq_no" : 3, "_primary_term" : 1, "_score": null, "_source": { "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } }, { "_index": "sec_logs", "_type": "_doc", "_id": "yDwnGIJouOYGBzP0ZE9n", "_version" : 1, "_seq_no" : 4, "_primary_term" : 1, "_score": null, "_source": { "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } } ] } ] } } ---- // TESTRESPONSE[s/"took": 60/"took": $body.took/] // TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.sequences.0.events.0._id/] // TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/] You can use the <> to constrain a sequence to a specified timespan. The following EQL search request adds `with maxspan=1h` to the previous query. This ensures all events in a matching sequence occur within one hour (`1h`) of the first event's timestamp. [source,console] ---- GET /sec_logs/_eql/search { "query": """ sequence with maxspan=1h [ file where file.name == "cmd.exe" ] [ process where stringContains(process.name, "regsvr32") ] """ } ---- You can further constrain matching event sequences using the <>. The following EQL search request adds `by agent.id` to each event item. This ensures events matching the sequence share the same `agent.id` field value. [source,console] ---- GET /sec_logs/_eql/search { "query": """ sequence with maxspan=1h [ file where file.name == "cmd.exe" ] by agent.id [ process where stringContains(process.name, "regsvr32") ] by agent.id """ } ---- Because the `agent.id` field is shared across all events in the sequence, it can be included using `sequence by`. The following query is equivalent to the prior one. [source,console] ---- GET /sec_logs/_eql/search { "query": """ sequence by agent.id with maxspan=1h [ file where file.name == "cmd.exe" ] [ process where stringContains(process.name, "regsvr32") ] """ } ---- The API returns the following response. The `hits.sequences.join_keys` property contains the shared `agent.id` value for each matching event. [source,console-result] ---- { "is_partial": false, "is_running": false, "took": 60, "timed_out": false, "hits": { "total": { "value": 1, "relation": "eq" }, "sequences": [ { "join_keys": [ "8a4f500d" ], "events": [ { "_index": "sec_logs", "_type": "_doc", "_id": "AtOJ4UjUBAAx3XR5kcCM", "_version": 1, "_seq_no": 3, "_primary_term": 1, "_score": null, "_source": { "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } }, { "_index": "sec_logs", "_type": "_doc", "_id": "yDwnGIJouOYGBzP0ZE9n", "_version": 1, "_seq_no": 4, "_primary_term": 1, "_score": null, "_source": { "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "executable": "C:\\Windows\\System32\\regsvr32.exe" } } } ] } ] } } ---- // TESTRESPONSE[s/"took": 60/"took": $body.took/] // TESTRESPONSE[s/"_id": "AtOJ4UjUBAAx3XR5kcCM"/"_id": $body.hits.sequences.0.events.0._id/] // TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/] You can use the <> to specify an expiration event for sequences. Matching sequences must end before this event. The following request adds `until [ process where event.type == "termination" ]` to the previous EQL query. This ensures matching sequences end before a process termination event. [source,console] ---- GET /sec_logs/_eql/search { "query": """ sequence by agent.id with maxspan=1h [ file where file.name == "cmd.exe" ] [ process where stringContains(process.name, "regsvr32") ] until [ process where event.type == "termination" ] """ } ---- [discrete] [[eql-search-specify-event-category-field]] === Specify an event category field By default, the EQL search API uses `event.category` as the <>. You can use the `event_category_field` parameter to specify another event category field. The following request specifies `file.type` as the event category field. [source,console] ---- GET /sec_logs/_eql/search { "event_category_field": "file.type", "query": """ file where agent.id == "8a4f500d" """ } ---- [discrete] [[eql-search-specify-timestamp-field]] === Specify a timestamp field By default, EQL searches use `@timestamp` as the <>. You can use the EQL search API's `timestamp_field` parameter to specify another timestamp field. The following request specifies `file.accessed` as the event timestamp field. [source,console] ---- GET /sec_logs/_eql/search { "timestamp_field": "file.accessed", "query": """ file where (file.size > 1 and file.type == "file") """ } ---- [discrete] [[eql-search-specify-a-sort-tiebreaker]] === Specify a sort tiebreaker By default, the EQL search API sorts matching events in the search response by timestamp. However, if two or more events share the same timestamp, a tiebreaker field is used to sort the events in ascending, lexicographic order. The EQL search API uses `event.sequence` as the default tiebreaker field. You can use the `tiebreaker_field` parameter to specify another field. The following request specifies `event.start` as the tiebreaker field. [source,console] ---- GET /sec_logs/_eql/search { "tiebreaker_field": "event.id", "query": """ process where process.name == "cmd.exe" and stringContains(process.executable, "System32") """ } ---- // TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/] The API returns the following response. [source,console-result] ---- { "is_partial": false, "is_running": false, "took": 34, "timed_out": false, "hits": { "total": { "value": 2, "relation": "eq" }, "events": [ { "_index": "sec_logs", "_type": "_doc", "_id": "OQmfCaduce8zoHT93o4H", "_score": null, "_source": { "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD", "sequence": 1 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } }, { "_index": "sec_logs", "_type": "_doc", "_id": "xLkCaj4EujzdNSxfYLbO", "_score": null, "_source": { "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "executable": "C:\\Windows\\System32\\cmd.exe" } } } ] } } ---- // TESTRESPONSE[s/"took": 34/"took": $body.took/] // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/] // TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/] [discrete] [[eql-search-filter-query-dsl]] === Filter using query DSL You can use the EQL search API's `filter` parameter to specify an additional query using <>. This query filters the documents on which the EQL query runs. The following request uses a `range` query to filter the `sec_logs` index down to only documents with a `file.size` value greater than `1` but less than `1000000` bytes. The EQL query in `query` parameter then runs on these filtered documents. [source,console] ---- GET /sec_logs/_eql/search { "filter": { "range" : { "file.size" : { "gte" : 1, "lte" : 1000000 } } }, "query": """ file where (file.type == "file" and file.name == "cmd.exe") """ } ---- [discrete] [[eql-search-async]] === Run an async EQL search EQL searches in {es} are designed to run on large volumes of data quickly, often returning results in milliseconds. Because of this, the EQL search API runs _synchronous_ searches by default. This means the search request waits for complete results before returning a response. However, complete results can take longer for searches across: * <> * <> * Many shards To avoid long waits, you can use the EQL search API's `wait_for_completion_timeout` parameter to run an _asynchronous_, or _async_, search. Set the `wait_for_completion_timeout` parameter to a duration you'd like to wait for complete search results. If the search request does not finish within this period, the search becomes an async search. The EQL search API returns a response that includes: * A search ID, which can be used to monitor the progress of the async search and retrieve complete results when it finishes. * An `is_partial` value of `true`, indicating the response does not contain complete search results. * An `is_running` value of `true`, indicating the search is async and ongoing. The async search continues to run in the background without blocking other requests. The following request searches the `frozen_sec_logs` index, which has been <> for storage and is rarely searched. Because searches on frozen indices are expected to take longer to complete, the request contains a `wait_for_completion_timeout` parameter value of `2s` (two seconds). If the request does not return complete results in two seconds, the search becomes an async search and a search ID is returned. [source,console] ---- GET /frozen_sec_logs/_eql/search { "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ } ---- // TEST[s/frozen_sec_logs/sec_logs/] After two seconds, the request returns the following response. Note the `is_partial` and `is_running` properties are `true`, indicating an ongoing async search. [source,console-result] ---- { "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", "is_partial": true, "is_running": true, "took": 2000, "timed_out": false, "hits": ... } ---- // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] // TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/] // TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/] // TESTRESPONSE[s/"took": 2000/"took": $body.took/] // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/] You can use the the returned search ID and the <> to check the progress of an ongoing async search. The get async EQL search API also accepts a `wait_for_completion_timeout` query parameter. Set the `wait_for_completion_timeout` parameter to a duration you'd like to wait for complete search results. If the request does not complete during this period, the response returns an `is_partial` value of `true` and no search results. The following get async EQL search API request checks the progress of the previous async EQL search. The request specifies a `wait_for_completion_timeout` query parameter value of `2s` (two seconds). [source,console] ---- GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s ---- // TEST[skip: no access to search ID] The request returns the following response. Note the `is_partial` and `is_running` properties are `false`, indicating the async EQL search has finished and the search results in the `hits` property are complete. [source,console-result] ---- { "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=", "is_partial": false, "is_running": false, "took": 2000, "timed_out": false, "hits": ... } ---- // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/] // TESTRESPONSE[s/"took": 2000/"took": $body.took/] // TESTRESPONSE[s/"_index": "frozen_sec_logs"/"_index": "sec_logs"/] // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/] [discrete] [[eql-search-store-async-eql-search]] === Change the search retention period By default, the EQL search API only stores async searches and their results for five days. After this period, any ongoing searches or saved results are deleted. You can use the EQL search API's `keep_alive` parameter to change the duration of this period. In the following EQL search API request, the `keep_alive` parameter is `2d` (two days). This means that if the search becomes async, its results are stored on the cluster for two days. After two days, the async search and its results are deleted, even if it's still ongoing. [source,console] ---- GET /sec_logs/_eql/search { "keep_alive": "2d", "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ } ---- You can use the <>'s `keep_alive` query parameter to later change the retention period. The new retention period starts after the get async EQL search API request executes. The following get async EQL search API request sets the `keep_alive` query parameter to `5d` (five days). The async search and its results are deleted five days after the get async EQL search API request executes. [source,console] ---- GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d ---- // TEST[skip: no access to search ID] You can use the <> to manually delete an async EQL search before the `keep_alive` period ends. If the search is still ongoing, this cancels the search request. The following delete async EQL search API request deletes an async EQL search and its results. [source,console] ---- DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d ---- // TEST[skip: no access to search ID] [discrete] [[eql-search-store-sync-eql-search]] === Store synchronous EQL searches By default, the EQL search API only stores async searches that cannot be completed within the period set by the `wait_for_completion_timeout` parameter. To save the results of searches that complete during this period, set the `keep_on_completion` parameter to `true`. In the following EQL search API request, the `keep_on_completion` parameter is `true`. This means the search results are stored on the cluster, even if the search completes within the `2s` (two-second) period set by the `wait_for_completion_timeout` parameter. [source,console] ---- GET /sec_logs/_eql/search { "keep_on_completion": true, "wait_for_completion_timeout": "2s", "query": """ process where process.name == "cmd.exe" """ } ---- The API returns the following response. Note that a search ID is provided in the `id` property. The `is_partial` and `is_running` properties are `false`, indicating the EQL search was synchronous and returned complete search results. [source,console-result] ---- { "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=", "is_partial": false, "is_running": false, "took": 52, "timed_out": false, "hits": ... } ---- // TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/] // TESTRESPONSE[s/"took": 52/"took": $body.took/] // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/] You can use the search ID and the <> to retrieve the same results later. [source,console] ---- GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY= ---- // TEST[skip: no access to search ID] Saved synchronous searches are still subject to the storage retention period set by the `keep_alive` parameter. After this period, the search and its saved results are deleted. You can also manually delete saved synchronous searches using the <>. [discrete] [[eql-search-case-sensitive]] === Run a case-sensitive EQL search By default, matching for EQL queries is case-insensitive. You can use the EQL search API's `case_sensitive` parameter to toggle case sensitivity on or off. The following search request contains a query that matches `process` events with a `process.executable` containing `System32`. Because the `case_sensitive` parameter is `true`, this query only matches `process.executable` values containing `System32` with the exact same capitalization. A `process.executable` value containing `system32` or `SYSTEM32` would not match this query. [source,console] ---- GET /sec_logs/_eql/search { "keep_on_completion": true, "case_sensitive": true, "query": """ process where stringContains(process.executable, "System32") """ } ----