[DOCS] EQL: Add search/index speed tip for functions (#54346) (#54575)

EQL functions are an easy way for users to transform indexed data
at search time. However, using multiple functions can make
queries difficult to write and slows search speeds.

Users can circumvent this by indexing fields containing the transformed
data, but that usually slows index speeds.

This adds a related tip and example covering these tradeoffs.
This commit is contained in:
James Rodewig 2020-04-01 08:39:04 -04:00 committed by GitHub
parent e9c201b446
commit 92d570d6f3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 47 additions and 1 deletions

View File

@ -294,4 +294,50 @@ dots (`.`), hyphens (`-`), or spaces, must be escaped using backticks (+++`+++).
{es} supports several of EQL's built-in functions. You can use these functions
to convert data types, perform math, manipulate strings, and more.
For a list of supported functions, see <<eql-function-ref>>.
For a list of supported functions, see <<eql-function-ref>>.
[TIP]
====
Using functions in EQL queries can result in slower search speeds. If you
often use functions to transform indexed data, you can speed up search by making
these changes during indexing instead. However, that often means slower index
speeds.
.*Example*
[%collapsible]
=====
An index contains the `file.path` field. `file.path` contains the full path to a
file, including the file extension.
When running EQL searches, users often use the `endsWith` function with the
`file.path` field to match file extensions:
[source,eql]
----
file where endsWith(file.path,".exe") or endsWith(file.path,".dll")
----
While this works, it can be repetitive to write and can slow search speeds. To
speed up search, you can do the following instead:
. <<indices-put-mapping,Add a new field>>, `file.extension`, to the index. The
`file.extension` field will contain only the file extension from the
`file.path` field.
. Use an <<ingest,ingest pipeline>> containing the <<grok-processor,`grok`>>
processor or another preprocessor tool to extract the file extension from the
`file.path` field before indexing.
. Index the extracted file extension to the `file.extension` field.
These changes may slow indexing but allow for faster searches. Users
can use the `file.extension` field instead of multiple `endsWith` function
calls:
[source,eql]
----
file where file.extension in ("exe", "dll")
----
=====
We recommend testing and benchmarking any indexing changes before deploying them
in production. See <<tune-for-indexing-speed>> and <<tune-for-search-speed>>.
====