honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[DOCS] Tighten EQL copy (#64081) (#64104)

This commit is contained in:
James Rodewig 2020-10-24 11:09:51 -04:00 committed by GitHub
parent 0f5d98e40c
commit d6143bb49d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 116 additions and 218 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

124
docs/reference/eql/detect-threats-with-eql.asciidoc
View File

@ -11,10 +11,10 @@ https://attack.mitre.org/techniques/T1218/010/[regsvr32 misuse] in Windows event
logs.
`regsvr32.exe` is a built-in command-line utility used to register `.dll`
libraries in Windows. As a native tool, `regsvr32.exe` has a trusted status in
Windows, letting it bypass most allowlist software and script blockers.
libraries in Windows. As a native tool, `regsvr32.exe` has a trusted status,
letting it bypass most allowlist software and script blockers.
Attackers with access to a user's command line can use `regsvr32.exe` to run
malicious scripts using `.dll` libraries, even on machines that otherwise
malicious scripts via `.dll` libraries, even on machines that otherwise
disallow such scripts.
One common variant of regsvr32 misuse is a
@ -31,19 +31,16 @@ register and run a remote script. These commands often look like this:
[[eql-ex-threat-detection-setup]]
=== Setup
This tutorial uses a test dataset for regsvr32 misuse from
https://github.com/redcanaryco/atomic-red-team[Atomic Red Team]. The dataset has
been normalized and mapped to use fields from the {ecs-ref}[Elastic Common
Schema (ECS)], including the `@timestamp` and `event.category` fields. The
dataset includes events that imitate behaviors of a Squiblydoo attack, as
documented in the https://attack.mitre.org[MITRE ATT&CK®] knowledge base.
This tutorial uses a test dataset from
https://github.com/redcanaryco/atomic-red-team[Atomic Red Team] that includes
events imitating a Squiblydoo attack. The data has been mapped to
{ecs-ref}[Elastic Common Schema (ECS)] fields.
To get started, download and index the dataset:
To get started:
. Download the https://raw.githubusercontent.com/elastic/elasticsearch/{branch}/docs/src/test/resources/normalized-T1117-AtomicRed-regsvr32.json[`normalized-T1117-AtomicRed-regsvr32.json`] dataset.
. Download https://raw.githubusercontent.com/elastic/elasticsearch/{branch}/docs/src/test/resources/normalized-T1117-AtomicRed-regsvr32.json[`normalized-T1117-AtomicRed-regsvr32.json`].
. Index the data into `my-index-000001` with the following <<docs-bulk,bulk
API>> request:
. Use the <<docs-bulk,bulk API>> to index the data:
+
[source,sh]
----
@ -51,8 +48,7 @@ curl -H "Content-Type: application/json" -XPOST "localhost:9200/my-index-000001/
----
// NOTCONSOLE
. Use the <<cat-indices,cat indices API>> to verify the data was successfully
indexed.
. Use the <<cat-indices,cat indices API>> to verify the data was indexed:
+
[source,console]
----
@ -60,8 +56,7 @@ GET /_cat/indices/my-index-000001?v&h=health,status,index,docs.count
----
// TEST[setup:atomic_red_regsvr32]
+
The API response should show a `docs.count` value of `150`, indicating 150
documents were indexed.
The response should show a `docs.count` of `150`.
+
[source,txt]
----
@ -74,14 +69,7 @@ yellow open my-index-000001 150
[[eql-ex-get-a-count-of-regsvr32-events]]
=== Get a count of regsvr32 events
Since you're looking for regsvr32 misuse, start by getting a count of any
events associated with a `regsvr32.exe` process.
The following <<eql-search-api,EQL search API>> request uses an EQL query to
retrieve a count of events with a `process.name` of `regsvr32.exe`. The query
starts with the <<eql-syntax-match-any-event-category,`any where` keywords>>,
meaning the query can match events of any <<eql-required-fields,event
category>>.
First, get a count of events associated with a `regsvr32.exe` process:
[source,console]
----
@ -95,15 +83,13 @@ GET /my-index-000001/_eql/search?filter_path=-hits.events <1>
----
// TEST[setup:atomic_red_regsvr32]
<1> Uses the `?filter_path=-hits.events` query parameter to exclude the
`hits.events` property from the response. The `hits.events` property contains
the document source for any matching events. This request is intended to
retrieve a count of events only.
<2> Uses an EQL query to match events with a `process.name` of `regsvr32.exe`.
<3> Returns up to 200 events or sequences matching the EQL query.
<1> `?filter_path=-hits.events` excludes the `hits.events` property from the
response. This search is only intended to get an event count, not a list of
matching events.
<2> Matches any event with a `process.name` of `regsvr32.exe`.
<3> Returns up to 200 hits for matching events.
The request returns the following response, indicating that 143 events match the
query.
The response returns 143 related events.
[source,console-result]
----
@ -126,21 +112,9 @@ query.
[[eql-ex-check-for-command-line-artifacts]]
=== Check for command line artifacts
Based on your previous query, you know regsvr32 processes were associated with
143 events. But how was `regsvr32.exe` first called? And who called it?
`regsvr32.exe` is a command-line utility so it may help to narrow your results
to processes where the command line was used.
Update the previous EQL query as follows:
* Change the `any` keyword to `process`. This limits matches to events with an
`event.category` of `process`.
* Add the `and process.command_line.keyword != null` condition to match only
events with a command line value.
You'll also need to remove the `filter_path=-hits.events` query parameter. This
lets you retrieve the document source for any matching events.
`regsvr32.exe` processes were associated with 143 events. But how was
`regsvr32.exe` first called? And who called it? `regsvr32.exe` is a command-line
utility. Narrow your results to processes where the command line was used:
[source,console]
----
@ -153,16 +127,10 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:atomic_red_regsvr32]
The query matches one process event. The event has an `event.type` of
`creation`, indicating the start of a `regsvr32.exe` process.
Based on the `process.command_line` value in the response, `regsvr32.exe` used
`scrobj.dll` to register a script, `RegSvr32.sct`. This fits the behavior of a
Squiblydoo attack.
The response also includes other valuable information about how the
`regsvr32.exe` process started, such as the `@timestamp`, the associated
`user.id`, and the `process.parent.name`.
The query matches one event with an `event.type` of `creation`, indicating the
start of a `regsvr32.exe` process. Based on the event's `process.command_line`
value, `regsvr32.exe` used `scrobj.dll` to register a script, `RegSvr32.sct`.
This fits the behavior of a Squiblydoo attack.
[source,console-result]
----
@ -218,15 +186,7 @@ The response also includes other valuable information about how the
[[eql-ex-check-for-malicious-script-loads]]
=== Check for malicious script loads
You now know that a `regsvr32.exe` process was used to register a potentially
malicious script, `RegSvr32.sct`. Next, see if `regsvr32.exe` later loads the
`scrobj.dll` library.
Modify the previous EQL query as follows:
* Change the `process` keyword to `library`.
* Replace the `process.command_line.keyword != null` condition with
`dll.name == "scrobj.dll`.
Check if `regsvr32.exe` later loads the `scrobj.dll` library:
[source,console]
----
@ -239,8 +199,7 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:atomic_red_regsvr32]
The query matches an event, confirming `scrobj.dll` was later loaded by
`regsvr32.exe`.
The query matches an event, confirming `scrobj.dll` was loaded.
[source,console-result]
----
@ -283,28 +242,21 @@ The query matches an event, confirming `scrobj.dll` was later loaded by
// TESTRESPONSE[s/"_id": "ol5MJXMBMk1dGnErnBW8"/"_id": $body.hits.events.0._id/]
[discrete]
[[eql-ex-detemine-likelihood-of-sucess]]
[[eql-ex-detemine-likelihood-of-success]]
=== Determine the likelihood of success
In many cases, malicious scripts are used to connect to remote servers or
download other files. If this occurred, the attack might have succeeded.
In many cases, attackers use malicious scripts to connect to remote servers or
download other files. Use an <<eql-sequences,EQL sequence query>> to check
for the following series of events:
Use an <<eql-sequences,EQL sequence query>> to check for the following series of
events, in order:
. A `regsvr32.exe` process, which could have been used to register malicious
scripts as `scrobj.dll`
. A `regsvr32.exe` process
. A load of the `scrobj.dll` library by the same process
. Any network event by the same process, which could indicate the download of a
remote file
. Any network event by the same process
To match, each event in the sequence must share the same process ID, recorded in
the `process.pid` field.
Based on the command line value seen in the previous result, you can expect to
find a match. However, the sequence query isn't designed for that specific
command. Instead, it looks for a pattern of suspicious behavior while still
being generic enough to detect similar threats in the future.
Based on the command line value seen in the previous response, you can expect to
find a match. However, this query isn't designed for that specific command.
Instead, it looks for a pattern of suspicious behavior that's generic enough to
detect similar threats.
[source,console]
----

210
docs/reference/eql/eql.asciidoc
View File

@ -8,55 +8,47 @@
beta::[]
Event Query Language (EQL) is a query language for event-based, time series
data, such as logs.
Event Query Language (EQL) is a query language for event-based time series
data, such as logs, metrics, and traces.
[discrete]
[[eql-advantages]]
== Advantages of EQL
* *EQL lets you express relationships between events.* +
Many query languages allow you to match only single events. EQL lets you match a
Many query languages allow you to match single events. EQL lets you match a
sequence of events across different event categories and time spans.
* *EQL has a low learning curve.* +
<<eql-syntax,EQL syntax>> looks like other common query languages, such as SQL.
It lets you write and read queries intuitively, which makes for quick, iterative
searching.
EQL lets you write and read queries intuitively, which makes for quick,
iterative searching.
* *EQL is designed for security use cases.* +
While you can use EQL for any event-based data, we created EQL for threat
hunting. EQL not only supports indicator of compromise (IOC) searching but
makes it easy to describe activity that goes beyond IOCs.
While you can use it for any event-based data, we created EQL for threat
hunting. EQL not only supports indicator of compromise (IOC) searches but can
describe activity that goes beyond IOCs.
[discrete]
[[eql-required-fields]]
== Required fields
To run an EQL search, the searched data stream or index must contain a
_timestamp_ and _event category_ field. By default, EQL uses the `@timestamp`
and `event.category` fields from the {ecs-ref}[Elastic Common Schema
(ECS)]. To use a different timestamp or event category field, see
<<specify-a-timestamp-or-event-category-field>>.
TIP: While no schema is required to use EQL, we recommend using the
{ecs-ref}[Elastic Common Schema (ECS)]. EQL searches are designed to work with
core ECS fields by default.
EQL assumes each document in a data stream or index corresponds to an event. To
run an EQL search, each document must contain a _timestamp_ and _event category_
field.
EQL uses the `@timestamp` and `event.category` fields from the {ecs-ref}[ECS] as
the default timestamp and event category fields. If your documents use a
different timestamp or event category field, you must specify it in the search
request. See <<specify-a-timestamp-or-event-category-field>>.
{ecs-ref}[ECS]. EQL searches are designed to work with core ECS fields by
default.
[discrete]
[[run-an-eql-search]]
== Run an EQL search
You can use the <<eql-search-api,EQL search API>> to run an EQL search. For
supported query syntax, see <<eql-syntax>>.
The following request searches `my-index-000001` for events with an
`event.category` of `process` and a `process.name` of `regsvr32.exe`. Each
document in `my-index-000001` includes a `@timestamp` and `event.category`
field.
Use the <<eql-search-api,EQL search API>> to run a <<eql-basic-syntax,basic
EQL query>>:
[source,console]
----
@ -69,10 +61,9 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:sec_logs]
By default, EQL searches return only the top 10 matching hits. For basic EQL
queries, these hits are matching events and are included in the `hits.events`
property. Matching events are sorted by timestamp, converted to milliseconds
since the {wikipedia}/Unix_time[Unix epoch], in ascending order.
By default, basic EQL queries return the top 10 matching events in the
`hits.events` property. These hits are sorted by timestamp, converted to
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in ascending order.
[source,console-result]
----
@ -131,8 +122,7 @@ since the {wikipedia}/Unix_time[Unix epoch], in ascending order.
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]
You can use the `size` request body parameter to get a larger or smaller set of
hits. For example, the following request retrieves up to `50` matching hits.
Use the `size` parameter to get a smaller or larger set of hits:
[source,console]
----
@ -150,23 +140,9 @@ GET /my-index-000001/_eql/search
[[eql-search-sequence]]
=== Search for a sequence of events
You can use EQL's <<eql-sequences,sequence syntax>> to search for an ordered
series of events.
The following EQL search request matches a sequence that:
. Starts with an event with:
+
--
* An `event.category` of `process`
* A `process.name` of `regsvr32.exe`
--
. Followed by an event with:
+
--
* An `event.category` of `file`
* A `file.name` that contains the substring `scrobj.dll`
--
Use EQL's <<eql-sequences,sequence syntax>> to search for a series of
ordered events. List the event items in ascending chronological order,
with the most recent event listed last:
[source,console]
----
@ -247,12 +223,8 @@ Matching sequences are returned in the `hits.sequences` property.
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
You can use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to
constrain a sequence to a specified timespan.
The following EQL search request adds `with maxspan=1h` to the previous query.
This ensures all events in a matching sequence occur within `1h` (one hour) of
the first event's timestamp.
Use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to constrain
matching sequences to a timespan:
[source,console]
----
@ -267,11 +239,8 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:sec_logs]
You can further constrain matching event sequences using the
<<eql-by-keyword,`by` keyword>>.
The following EQL search request adds `by process.pid` to each event item. This
ensures events matching the sequence share the same `process.pid` field value.
Use the <<eql-by-keyword,`by` keyword>> to match events that share the
same field values:
[source,console]
----
@ -286,9 +255,8 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:sec_logs]
Because the `process.pid` field is shared across all events in the sequence, it
can be included using `sequence by`. The following query is equivalent to the
previous one.
If a field value should be shared across all events, use the `sequence by`
keyword. The following query is equivalent to the previous one.
[source,console]
----
@ -303,8 +271,7 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:sec_logs]
The API returns the following response. The `hits.sequences.join_keys` property
contains the shared `process.pid` value for each matching event.
The `hits.sequences.join_keys` property contains the shared field values.
[source,console-result]
----
@ -373,13 +340,9 @@ contains the shared `process.pid` value for each matching event.
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
You can use the <<eql-until-keyword,`until` keyword>> to specify an expiration
Use the <<eql-until-keyword,`until` keyword>> to specify an expiration
event for sequences. Matching sequences must end before this event.
The following request adds `until [ process where event.type == "termination" ]`
to the previous query. This ensures matching sequences end before a `process`
event with an `event.type` of `termination`.
[source,console]
----
GET /my-index-000001/_eql/search
@ -398,25 +361,9 @@ GET /my-index-000001/_eql/search
[[specify-a-timestamp-or-event-category-field]]
=== Specify a timestamp or event category field
To run an EQL search, each searched document must contain a timestamp and event
category field. The EQL search API uses the `@timestamp` and `event.category`
fields from the {ecs-ref}[Elastic Common Schema (ECS)] by default. If your
documents use a different timestamp or event category field, you must specify it
in the search request using the `timestamp_field` or `event_category_field`
parameters.
The event category field must be mapped as a field type in the
<<keyword,`keyword`>> family. The timestamp field should be mapped as a
<<date,`date`>> field type. <<date_nanos,`date_nanos`>> timestamp fields are not
supported.
NOTE: You cannot use a <<nested,`nested`>> field or the sub-fields of a `nested`
field as the timestamp or event category field. See <<eql-nested-fields>>.
The following request uses the `timestamp_field` parameter to specify
`file.accessed` as the timestamp field. The request also uses the
`event_category_field` parameter to specify `file.type` as the event category
field.
The EQL search API uses the `@timestamp` and `event.category` fields from the
{ecs-ref}[ECS] by default. To specify different fields, use the
`timestamp_field` and `event_category_field` parameters:
[source,console]
----
@ -431,42 +378,41 @@ GET /my-index-000001/_eql/search
----
// TEST[setup:sec_logs]
[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker
The event category field must be mapped as a <<keyword,`keyword`>> family field
type. The timestamp field should be mapped as a <<date,`date`>> field type.
<<date_nanos,`date_nanos`>> timestamp fields are not supported. You cannot use a
<<nested,`nested`>> field or the sub-fields of a `nested` field as the timestamp
or event category field.
By default, the EQL search API sorts matching hits in the search response by
timestamp. However, if two or more events share the same timestamp, you can use
a tiebreaker field to sort the events in ascending, lexicographic order.
[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker
The EQL search API uses `event.sequence` as the default tiebreaker field. You
can use the `tiebreaker_field` parameter to specify another field.
By default, the EQL search API returns matching events by timestamp. If two or
more events share the same timestamp, {es} uses a tiebreaker field value to sort
the events in ascending, lexicographic order.
The following request specifies `event.id` as the tiebreaker field.
`event.sequence` is the default tiebreaker field. To specify another tiebreaker
field, use the `tiebreaker_field` parameter:
[source,console]
----
GET /my-index-000001/_eql/search
{
"tiebreaker_field": "event.id",
"query": """
process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
"""
}
----
[source,console]
----
GET /my-index-000001/_eql/search
{
"tiebreaker_field": "event.id",
"query": """
process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
"""
}
----
// TEST[setup:sec_logs]
[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using query DSL
You can use the `filter` parameter to specify an additional query using
<<query-dsl,query DSL>>. This query filters the documents on which the EQL query
runs.
The following request uses a `range` query to filter `my-index-000001` to only
documents with a `file.size` value greater than `1` but less than `1000000`
bytes. The EQL query in `query` parameter then runs on these filtered documents.
The `filter` parameter uses <<query-dsl,query DSL>> to limit the documents on
which an EQL query runs.
[source,console]
----
@ -494,7 +440,7 @@ GET /my-index-000001/_eql/search
EQL searches are designed to run on large volumes of data quickly, often
returning results in milliseconds. For this reason, EQL searches are
_synchronous_ by default. The search request waits for complete results before
returning a response.
returning a response.
However, complete results can take longer for searches across:
@ -502,8 +448,8 @@ However, complete results can take longer for searches across:
* <<modules-cross-cluster-search,Multiple clusters>>
* Many shards
To avoid long waits, you can use the `wait_for_completion_timeout` parameter to
run an _asynchronous_, or _async_, EQL search.
To avoid long waits, use the `wait_for_completion_timeout` parameter to run an
_asynchronous_, or _async_, EQL search.
Set `wait_for_completion_timeout` to a duration you'd like to wait
for complete search results. If the search request does not finish within this
@ -558,11 +504,11 @@ and `is_running` properties are `true`, indicating an async search.
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
You can use the the search ID and the <<get-async-eql-search-api,get async EQL
search API>> to check the progress of an async search.
Use the <<get-async-eql-search-api,get async EQL search API>> to check the progress
of an async search.
The get async EQL search API also accepts a `wait_for_completion_timeout`
parameter. If ongoing search does not complete during this period, the response
parameter. If an ongoing search does not complete during this period, the response
returns an `is_partial` value of `true` and no search results.
The following get async EQL search API request checks the progress of the
@ -599,7 +545,7 @@ in the `hits` property are complete.
=== Change the search retention period
By default, the EQL search API stores async searches for five days. After this
period, any searches and their results are deleted. You can use the `keep_alive`
period, any searches and their results are deleted. Use the `keep_alive`
parameter to change this retention period.
In the following EQL search request, the `keep_alive` parameter is `2d` (two
@ -621,8 +567,8 @@ GET /my-index-000001/_eql/search
// TEST[setup:sec_logs]
You can use the <<get-async-eql-search-api,get async EQL search API>>'s
`keep_alive`parameter to later change the retention period. The new
retention period starts after the get request executes.
`keep_alive` parameter to later change the retention period. The new retention
period starts after the get request runs.
The following request sets the `keep_alive` query parameter to `5d` (five days).
The async search and its results are deleted five days after the get request
@ -634,9 +580,9 @@ GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTo
----
// TEST[skip: no access to search ID]
You can use the <<delete-async-eql-search-api,delete async EQL search API>> to
Use the <<delete-async-eql-search-api,delete async EQL search API>> to
manually delete an async EQL search before the `keep_alive` period ends. If the
search is still ongoing, this cancels the search request.
search is still ongoing, {es} cancels the search request.
The following request deletes an async EQL search and its results.
@ -656,9 +602,9 @@ completed within the period set by `wait_for_completion_timeout`.
To save the results of searches that complete during this period, set the
`keep_on_completion` parameter to `true`.
In the following search request, `keep_on_completion` is `true`. This means the
search results are stored on the cluster, even if the search completes within
the `2s` (two-second) period set by the `wait_for_completion_timeout` parameter.
In the following search request, `keep_on_completion` is `true`. Search results
are stored on the cluster, even if the search completes within the `2s`
(two-second) period set by the `wait_for_completion_timeout` parameter.
[source,console]
----
@ -692,8 +638,8 @@ was synchronous and returned complete results in `hits`.
// TESTRESPONSE[s/"took": 52/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
You can use the search ID and the <<get-async-eql-search-api,get async EQL
search API>> to retrieve the same results later.
Use the search ID and the <<get-async-eql-search-api,get async EQL search API>>
to retrieve the same results later.
[source,console]
----
@ -711,4 +657,4 @@ You can also manually delete saved synchronous searches using the
include::syntax.asciidoc[]
include::functions.asciidoc[]
include::pipes.asciidoc[]
include::detect-threats-with-eql.asciidoc[]
include::detect-threats-with-eql.asciidoc[]