OpenSearch/docs/reference/eql/search.asciidoc

[role="xpack"]
[testenv="basic"]
[[eql-search]]
== Run an EQL search

dev::[]

To start using EQL in {es}, first ensure your event data meets
<<eql-requirements,EQL requirements>>. You can then use the <<eql-search-api,EQL
search API>> to search event data stored in one or more {es} indices.

.*Example*
[%collapsible]
====
To get started, ingest or add the data to an {es} index.

The following <<docs-bulk,bulk API>> request adds some example log data to the
`sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema
(ECS)].

[source,console]
----
PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP

[TIP]
=====
You also can set up {beats-ref}/getting-started.html[{beats}], such as
{auditbeat-ref}/auditbeat-getting-started.html[{auditbeat}] or
{winlogbeat-ref}/winlogbeat-getting-started.html[{winlogbeat}], to automatically
send and index your event data in {es}. See
{beats-ref}/getting-started.html[Getting started with {beats}].
=====

You can now use the EQL search API to search this index using an EQL query.

The following request searches the `sec_logs` index using the EQL query
specified in the `query` parameter. The EQL query matches events with an
`event.category` of `process` that have a `process.name` of `cmd.exe`.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    process where process.name == "cmd.exe"
  """
}
----

Because the `sec_log` index follows the ECS, you don't need to specify the
required <<eql-required-fields,event category or timestamp>> fields. The request
uses the `event.category` and `@timestamp` fields by default.

The API returns the following response containing the matching events. Events
in the response are sorted by timestamp, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "sec_logs",
        "_type": "_doc",
        "_id": "1",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-06T11:04:05.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607252645000
        ]
      },
      {
        "_index": "sec_logs",
        "_type": "_doc",
        "_id": "3",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-07T11:06:07.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607339167000
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====

[discrete]
[[eql-search-sequence]]
=== Search for a sequence of events

Many query languages allow you to match single events. However, EQL's
<<eql-sequences,sequence syntax>> lets you match an ordered series of events.

.*Example*
[%collapsible]
====
The following EQL search request matches a sequence that:

. Starts with an event with:
+
--
* An `event.category` of `file`
* A `file.name` of `cmd.exe`
--
. Followed by an event with:
+
--
* An `event.category` of `process`
* A `process.name` that contains the substring `regsvr32`
--

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}
----

The API returns the following response. Matching events in
the `hits.sequences.events` property are sorted by
<<eql-search-api-timestamp-field,timestamp>>, converted to milliseconds since
the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "sec_logs",
            "_type": "_doc",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_type": "_doc",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]

You can further constrain matching event sequences using the `by` keyword.

The following EQL search request adds `by agent.id` to each event item. This
ensures events matching the sequence share the same `agent.id` field value.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ] by agent.id
      [ process where stringContains(process.name, "regsvr32") ] by agent.id
  """
}
----

Because the `agent.id` field is shared across all events in the sequence, it
can be included using `sequence by`. The following query is equivalent to the
prior one.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence by agent.id
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}
----

The API returns the following response. The `hits.sequences.join_keys` property
contains the shared `agent.id` value for each matching event.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          "8a4f500d"
        ],
        "events": [
          {
            "_index": "sec_logs",
            "_type": "_doc",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_type": "_doc",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====

[discrete]
[[eql-search-specify-event-category-field]]
=== Specify an event category field

The EQL search API uses `event.category` as the required
<<eql-required-fields,event category field>> by default. You can use the
`event_category_field` parameter to specify another event category field.

.*Example*
[%collapsible]
====
The following request specifies `file.type` as the event category
field.

[source,console]
----
GET /sec_logs/_eql/search
{
  "event_category_field": "file.type",
  "query": """
    file where agent.id == "8a4f500d"
  """
}
----
====

[discrete]
[[eql-search-specify-timestamp-field]]
=== Specify a timestamp field

The EQL search API uses `@timestamp` as the required <<eql-required-fields,event
timestamp field>> by default. You can use the `timestamp_field` parameter to
specify another timestamp field.

.*Example*
[%collapsible]
====
The following request specifies `file.accessed` as the event
timestamp field.

[source,console]
----
GET /sec_logs/_eql/search
{
  "timestamp_field": "file.accessed",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}
----
====

[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using query DSL

You can use the EQL search API's `filter` parameter to specify an additional
query using <<query-dsl,query DSL>>. This query filters the documents on which
the EQL query runs.

.*Example*
[%collapsible]
====
The following request uses a `range` query to filter the `sec_logs`
index down to only documents with a `file.size` value greater than `1` but less
than `1000000` bytes. The EQL query in `query` parameter then runs on these
filtered documents.

[source,console]
----
GET /sec_logs/_eql/search
{
  "filter": {
    "range" : {
      "file.size" : {
        "gte" : 1,
        "lte" : 1000000
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}
----
====

[discrete]
[[eql-search-case-sensitive]]
=== Run a case-sensitive EQL search

By default, matching for EQL queries is case-insensitive. You can use the EQL
search API's `case_sensitive` parameter to toggle case sensitivity on or off.

.*Example*
[%collapsible]
====
The following search request contains a query that matches `process` events
with a `process.path` containing `System32`.

Because the `case_sensitive` parameter is `true`, this query only matches
`process.path` values containing `System32` with the exact same capitalization.
A `process.path` value containing `system32` or `SYSTEM32` would not match this
query.

[source,console]
----
GET /sec_logs/_eql/search
{
  "case_sensitive": true,
  "query": """
    process where stringContains(process.path, "System32")
  """
}
----
====