26 Commits

Author SHA1 Message Date
Costin Leau
ff0ea62cb8 EQL: Fix casing for tiebreaker field (#57943)
Use tiebreaker instead of tieBreaker

(cherry picked from commit 3c774948a5d5e10fac267cb9a54f5d0559a00c1d)
2020-06-11 00:10:19 +03:00
Costin Leau
439205d1ea EQL: Introduce tie breaker support (#57787)
Allow a field inside the data to be used as a tie breaker for events
that have the same timestamp.
The field is optional by default.
If used, the tie-breaker always requires a non-null value since it is
used inside `search_after` which requires a non-null value.

Fix #56824

(cherry picked from commit e5719ecb474b32730d93afdbb6834a32b0b2df8b)
2020-06-09 22:50:19 +03:00
Bogdan Pintea
74b2c8a770 Change error message for comp against fields (#57126)
Change the error message wording for comparisons against fields in
filtering (s/variables/fields).

(cherry picked from commit d9a1cb50940d0a98fd75b9c0123ca6e1d862f65d)
2020-05-26 17:57:51 +02:00
Costin Leau
6f4af43405 EQL: Skip execution for filters with empty results (#56718)
Optimize away events queries and joins/sequence that cannot match any
results without having to query the backend.

(cherry picked from commit 69c8ef8cfefd8fc6dcb6d1a566bfcd537068e3e4)
2020-05-14 22:38:23 +03:00
Aleksandr Maus
87a10806ab
EQL: Fix cidrMatch function fails to match when used in scripts (#56246) (#56735)
EQL: Fix cidrMatch function fails to match when used in scripts (#56246)

Addresses https://github.com/elastic/elasticsearch/issues/55709
2020-05-13 22:41:24 -04:00
Ross Wolf
61e2cf89b5
EQL: Add number function (#55084)
* EQL: Add number function
* EQL: Fix the locale used for number for deterministic functionality
* EQL: Add more ToNumber tests
* EQL: Add more number ToNumberProcessor unit tests
* EQL: Remove unnecessary overrides, fix processor methods
* EQL: Remove additional unnecessary overrides
* EQL: Lint fixes for ToNumber
* EQL: ToNumber renames from PR feedback
* EQL: Remove NumberFormat locale handling
* EQL: Removed NumberFormat from ToNumber
* EQL: Add number function tests
* EQL: ToNumberProcessorTests formatting
* EQL: Remove newline in ToNumberProcessorTests
* EQL: Add number(..., null) test
* EQL: Create expression.function.scalar.math package
* EQL: Remove painless whitespace for ToNumber.asScript
* EQL: Add Long support
2020-05-13 14:09:06 -06:00
Costin Leau
9f1ecd52eb EQL: Introduce support for sequences (#56300)
Initial support for EQL sequences
The current algorithm is focused on correctness and does not contain
any optimization which is left for the future.

The current implementation uses a state machine approach which moves
ascending and runs each query one after the other working on computing
sequences as the data comes in.
For each result, the key and its timestamp are being extracted which are
then used for matching/building a sequence.

(cherry picked from commit 4f3e18c894a1841d333022361ad9d1fdf1477dc3)
2020-05-13 15:42:31 +03:00
Marios Trivyzas
cbbbd499bf
SQL/EQL: Add support for scalars within LIKE/RLIKE (#56495) (#56674)
- Add support for scalar functions on the field of SQL's LIKE/RLIKE
- Add support for scalar functions on the field of EQL's match/matchLite

Closes: #55058
(cherry picked from commit 51c14e2dbb7fb29004a23369c449d425b3ac8fe2)
2020-05-13 13:40:24 +02:00
Ross Wolf
389082033e
EQL: Add concat function (#55193)
* EQL: Add concat function
* EQL: for loop spacing for concat
* EQL: return unresolved arguments to concat early
* EQL: Add concat integration tests
* EQL: Fix concat query fail test
* EQL: Add class for concat function testing
* EQL: Add concat integration tests
* EQL: Update concat() null behavior
2020-05-05 12:53:34 -06:00
Ross Wolf
6da686c7e0
EQL: Add match function implementation (#55182)
* EQL: Add Match function
* EQL: Add note about character classes
* EQL: QueryFolderFailTests.java
* EQL: Add match() fail tests
* EQL: Add match tests and fix alias
* EQL: Add match verifier failure tests
* EQL: Reorder query folder fail tests
2020-05-04 09:34:20 -06:00
Aleksandr Maus
ad54cca823
EQL: implement math functions: add, divide, module, multiply, subtract (#55137) (#55737)
* EQL: implement math functions: add, divide, module, multiply, subtract
2020-04-24 15:52:27 -04:00
Ross Wolf
96a903b17f
EQL: Add string function (#54470)
* EQL: Add string() function
* EQL: Reorder queryfolder_tests
* EQL: Add test queries
* EQL: Fix InternalEqlScriptUtils.string and test case
* EQL: Fix testStringFunctionWithText error message
* EQL: Flatten ToStringFunctionPipe.equals
* EQL: Reorder painless whitelist
* EQL: Address feedback and remove string(null) handling
* EQL: Move string(pid) test over
* EQL: Rename source -> value
2020-04-10 09:48:29 -06:00
Andrei Stefan
85f129a50a
EQL: indexOf function implementation (#54543) (#54989)
(cherry picked from commit a4b1d6e52d9ba22d541dd86d69861b1efee83604)
2020-04-09 02:41:01 +03:00
Aleksandr Maus
d02f774cb6
EQL: implement cidrMatch function (#54186) (#54928)
Related to https://github.com/elastic/elasticsearch/issues/54132
2020-04-07 22:07:28 -04:00
Aleksandr Maus
868798e4db
EQL: implement between function (#54277) (#54913) 2020-04-07 16:52:30 -04:00
Ross Wolf
022f829d84
EQL: Add wildcard function (#54020)
* EQL: Add wildcard function
* EQL: Cleanup Wildcard.getArguments
* EQL: Cleanup Wildcard and rearrange methods
* EQL: Wildcard newline lint
* EQL: Make StringUtils function final
* EQL: Make Wildcard.asLikes return ScalarFunction
* QL: Restore BinaryLogic.java
* EQL: Add Wildcard PR feedback
* EQL: Add Wildcard verification tests
* EQL: Switch wildcard to isFoldable test
* EQL: Change wildcard test to numeric field
* EQL: Remove Wildcard.get_arguments
2020-04-03 10:15:43 -06:00
Andrei Stefan
977302e46c
EQL: startsWith and endsWith functions implementation (#54504)
* EQL: startsWith function implementation (#54400)

(cherry picked from commit 666719fcfc40f6fc0535609577791369123320ab)

* EQL: endsWith function implementation (#54442)

(cherry picked from commit 554a4c8ef04b67eed107d29b57185e9af25d9d4f)
2020-03-31 18:06:03 +03:00
Costin Leau
68f74cf593
EQL: Fix custom scripting for functions (#53935) (#54114)
Improve separation of scripting between EQL and SQL by delegating common
methods to QL. The context detection is determined based on the package
to avoid having repetitive class hierarchies.
The Painless whitelists have been improved so that the declaring class
is used instead of the inherited one.

Relates #53688

(cherry picked from commit 6d46033e736c64ac9255c5d6964600d2a931430a)

EQL: Add Substring function with Python semantics (#53688)

Does not reuse substring from SQL due to the difference in semantics and
the accepted arguments.
Currently it is missing full integration tests as, due to the usage of
scripting, requires an actual integration test against a proper cluster
(and likely its own QA project).

(cherry picked from commit f58680bad33d5ce4139157a69a4d9f5f286bc3c4)
2020-03-24 20:54:19 +02:00
Aleksandr Maus
fd0cdde38c
EQL: EqlActionIT improvements (#53780) (#53888)
Related to https://github.com/elastic/elasticsearch/issues/53598
2020-03-20 17:28:15 -04:00
Aleksandr Maus
d064846416
EQL: Test infrastructure improvements (#53253) (#53297)
Update CommonEqlRestTestCase code to simplify making changes as requested.
Update EqlActionIT to simplify the test code as requested.
Replace Jackson parser with XContent in EqlActionIT.
Whitelist more EQL tests specs that are now supported.
2020-03-09 14:11:54 -04:00
Aleksandr Maus
b47bffba24
EQL: consistent naming for event type vs event category (#53073) (#53090)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Aleksandr Maus
89ed857c79
EQL: Change request parameter query to filter and rule to query (#52971) (#53006)
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
Igor Motov
e5b21a3fc6
Add HLRC for EQL search (#52550)
Adds EQL HLRC client with the search method.

Relates to #51961
2020-02-21 08:44:08 -05:00
Aleksandr Maus
d4f6f38150
EQL: Fix #51541: [CI] unknown setting [xpack.eql.enabled] in release-tests (#51699) (#51770)
Fixes #51541
Co-authored-by: Igor Motov <igor@motovs.org>
2020-01-31 15:14:27 -05:00
Aleksandr Maus
d715176c00 Add more Eql REST API validation integration tests, clean up request implementation (#50822) 2020-01-27 15:12:48 -05:00
Aleksandr Maus
79875ce4d9 Initial EQL rest API implementation (#49768) 2020-01-27 15:11:41 -05:00