Commit Graph

44 Commits

Author SHA1 Message Date
Mayya Sharipova bf4857d9e0
Search hit refactoring (#41656) (#54584)
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue #24422 to remove
dependency on MapperService to check if a field is metafield.

Relates to PR: #38373
Relates to issue #24422

Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
2020-04-01 15:19:00 -04:00
Andrei Stefan 977302e46c
EQL: startsWith and endsWith functions implementation (#54504)
* EQL: startsWith function implementation (#54400)

(cherry picked from commit 666719fcfc40f6fc0535609577791369123320ab)

* EQL: endsWith function implementation (#54442)

(cherry picked from commit 554a4c8ef04b67eed107d29b57185e9af25d9d4f)
2020-03-31 18:06:03 +03:00
Andrei Stefan 364ea0a3c0
EQL: Length function implementation (#54209) (#54490)
(cherry picked from commit 18493467e55e014be2c9e0ebdf734e9d7fc4beaa)
2020-03-31 16:49:18 +03:00
Ross Wolf d11e977b1f
EQL: Use In from QL (#53244)
* EQL: Use In from QL
* EQL: Add more In tests
* EQL: Test In duplicates
* EQL: Add test for In mixed types
* EQL: Copy In translation to QL
* SQL: Use InComparisons from QL
* EQL: Remove boost checks from QueryFolderOkTests
* QL: Add TranslatorHandler.convert
2020-03-30 15:19:23 -06:00
Ross Wolf 627ca03c72
EQL: Remove parser handling for functions (#54028)
* EQL: Remove parser handling for functions
* EQL: Comment out array functions in queries-unsupported.eql
2020-03-24 14:03:02 -06:00
Costin Leau 68f74cf593
EQL: Fix custom scripting for functions (#53935) (#54114)
Improve separation of scripting between EQL and SQL by delegating common
methods to QL. The context detection is determined based on the package
to avoid having repetitive class hierarchies.
The Painless whitelists have been improved so that the declaring class
is used instead of the inherited one.

Relates #53688

(cherry picked from commit 6d46033e736c64ac9255c5d6964600d2a931430a)

EQL: Add Substring function with Python semantics (#53688)

Does not reuse substring from SQL due to the difference in semantics and
the accepted arguments.
Currently it is missing full integration tests as, due to the usage of
scripting, requires an actual integration test against a proper cluster
(and likely its own QA project).

(cherry picked from commit f58680bad33d5ce4139157a69a4d9f5f286bc3c4)
2020-03-24 20:54:19 +02:00
Aleksandr Maus fd0cdde38c
EQL: EqlActionIT improvements (#53780) (#53888)
Related to https://github.com/elastic/elasticsearch/issues/53598
2020-03-20 17:28:15 -04:00
Aleksandr Maus 83bef862e0
EQL: Extract query folder tests definitions into resources (#53802) (#53869) 2020-03-20 10:39:35 -04:00
Aleksandr Maus 31d45b3c95
EQL: Improve query folder test suite (#53187) (#53476)
Related to https://github.com/elastic/elasticsearch/issues/52775
2020-03-12 10:58:07 -04:00
Aleksandr Maus d064846416
EQL: Test infrastructure improvements (#53253) (#53297)
Update CommonEqlRestTestCase code to simplify making changes as requested.
Update EqlActionIT to simplify the test code as requested.
Replace Jackson parser with XContent in EqlActionIT.
Whitelist more EQL tests specs that are now supported.
2020-03-09 14:11:54 -04:00
Ross Wolf f5f922c6f6
EQL: Add IsNull/IsNotNull checks (#52791)
* EQL: Add IsNull/IsNotNull checks
* EQL: Simplify IsNull/IsNotNull optimization
* EQL: Split string tests over multiple lines
2020-03-09 10:41:04 -06:00
Ross Wolf d6813cb348
EQL: Convert wildcards to LIKE in analyzer (#51901)
* EQL: Convert wildcard comparisons to Like
* EQL: Simplify wildcard handling, update tests
* EQL: Lint fixes for Optimizer.java
2020-03-06 13:13:07 -07:00
Aleksandr Maus 2dc872f052
EQL: Add HLRC for EQL stats (#53043) (#53148) 2020-03-05 09:20:38 -05:00
Ross Wolf a5e82d7fd6
EQL: Add explicit 'any where ...' handling (#52526) 2020-03-04 10:11:03 -07:00
Aleksandr Maus b47bffba24
EQL: consistent naming for event type vs event category (#53073) (#53090)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Costin Leau 712e0c05cd EQL: Add implicit ordering on timestamp (#53004)
QL: Move Sort base class from SQL to QL
(cherry picked from commit 798015b7bbd565e9c4222724614baeb432c7c2b3)
2020-03-02 22:41:36 +02:00
Aleksandr Maus 89ed857c79
EQL: Change request parameter query to filter and rule to query (#52971) (#53006)
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
Costin Leau a674085903 EQL: Disable field extraction for returned events (#52884)
Return the whole source of matching events

(cherry picked from commit 79ca586ab1d89d645fb58142b82202f14ce5d361)
2020-02-28 13:48:15 +02:00
Costin Leau 40bc06f6ad EQL: Hook engine to Elasticsearch (#52828)
Add query execution and return actual results returned from
Elasticsearch inside the tests

(cherry picked from commit 3e039282bf991af87604a6d4f8eada19d5e33842)
2020-02-27 11:22:22 +02:00
Costin Leau a8911802d3 EQL: transform query AST into queryDSL (#52432)
(cherry picked from commit 94cef29df259319dfe2a3bf92d3f1a42d7e45781)
2020-02-25 17:53:59 +02:00
Aleksandr Maus a6f5b4bb78
Unmute EqlActionIT (#52757)
Related to https://github.com/elastic/elasticsearch/issues/52737
2020-02-25 10:22:07 -05:00
Aleksandr Maus b2cb38ccf5
EQL: Expand verification tests (#52664) (#52725)
* EQL: Expand verification tests (#52664)

Expand verification tests
Fix some error messaging consistency in EqlParser

Related to https://github.com/elastic/elasticsearch/issues/51873

* Adjust for 7.x compatibility
2020-02-25 07:19:33 -05:00
Mark Vieira 025352f0a4
Mute EqlActionIT 2020-02-24 16:06:30 -08:00
Aleksandr Maus a7bdb0b456
EQL: Add integration tests harness to test EQL feature parity with original implementation (#52248) (#52675)
The tests use the original test queries from
https://github.com/endgameinc/eql/blob/master/eql/etc/test_queries.toml
for EQL implementation correctness validation.
The file test_queries_unsupported.toml serves as a "blacklist" for the
queries that we do not support. Currently all of the queries are
blacklisted. Over the time the expectation is to eventually have an
empty "blacklist" when all of the queries are fully supported.

The tests use the original test vector from
https://raw.githubusercontent.com/endgameinc/eql/master/eql/etc/test_data.json.

Only one EQL and the response is stubbed for now to match the expected
output from that query. This part would need some tweaking after EQL is
fully wired.

Related to https://github.com/elastic/elasticsearch/issues/49581
2020-02-24 12:46:59 -05:00
Jay Modi f3f6ff97ee
Single instance of the IndexNameExpressionResolver (#52604)
This commit modifies the codebase so that our production code uses a
single instance of the IndexNameExpressionResolver class. This change
is being made in preparation for allowing name expression resolution
to be augmented by a plugin.

In order to remove some instances of IndexNameExpressionResolver, the
single instance is added as a parameter of Plugin#createComponents and
PersistentTaskPlugin#getPersistentTasksExecutor.

Backport of #52596
2020-02-21 07:50:02 -07:00
Igor Motov e5b21a3fc6
Add HLRC for EQL search (#52550)
Adds EQL HLRC client with the search method.

Relates to #51961
2020-02-21 08:44:08 -05:00
Costin Leau 5373a77fb9 QL: Extract common Failure class (#52281)
Shared across SQL and EQL

(cherry picked from commit 1aeda20d3ec3d6c885de03c6043dd1e8eab9f230)
2020-02-13 14:35:15 +02:00
Costin Leau 26900bfb05 EQL: Add infra for planning and query folding (#52065)
Actual folding not yet in place (TBD)

(cherry picked from commit d52b96f273a94c90e475a5035cd57baa086fb0c0)
2020-02-12 18:51:42 +02:00
Andrei Stefan a3ebacfcf3
52169 & 52172 7x backport (#52256)
* Extract common optimizer tests (#52169)

(cherry picked from commit e5ad72bc22e9ec0686ab582195f0032efcb880bf)

* Hook in the optimizer rules (#52172)

(cherry picked from commit 1f90d8cc56052fbf2af604e72f9f5ca73f5e75d5)
2020-02-12 11:20:03 +02:00
James Rodewig d68a4ec82e
[7.x] Permit EQL feature flag in release builds (#52201) (#52214)
7.x backport of #52201

Provides a path to set register the EQL feature flag in release builds.
This enables EQL in release builds so that release docs tests pass.

Release docs tests do not have infrastructure in place to only register
snippets from included portions of the docs, they instead include all
docs snippets.

Since EQL can not be enabled in release builds, this meant that the EQL
snippets fail in the release docs tests.

This adds the ability to enable EQL in the release docs tests. This
system property will be removed when EQL is ready for release.
2020-02-11 11:49:49 -05:00
Andrei Stefan 2f1631d9d0
Telemetry data initial implementation (#51715) (#52175)
(cherry picked from commit f1d1cceacaacf226fcd2459f34689843b822fe4b)
2020-02-11 09:15:47 +02:00
Jay Modi 3edadfefd0 RestHandlers declare handled routes (#52123)
This commit changes how RestHandlers are registered with the
RestController so that a RestHandler no longer needs to register itself
with the RestController. Instead the RestHandler interface has new
methods which when called provide information about the routes
(method and path combinations) that are handled by the handler
including any deprecated and/or replaced combinations.

This change also makes the publication of RestHandlers safe since they
no longer publish a reference to themselves within their constructors.

Closes #51622

Co-authored-by: Jason Tedor <jason@tedor.me>

Backport of #51950
2020-02-09 22:48:32 -07:00
Costin Leau 6ff0e411a8
EQL: backport updates to 7.x (#51940) 2020-02-05 16:45:58 +02:00
Ryan Ernst 21224caeaf Remove comparison to true for booleans (#51723)
While we use `== false` as a more visible form of boolean negation
(instead of `!`), the true case is implied and the true value does not
need to explicitly checked. This commit converts cases that have slipped
into the code checking for `== true`.
2020-01-31 16:35:43 -08:00
Aleksandr Maus eb1ed2a35f Compilation fixes for 7.x 2020-01-27 16:23:36 -05:00
Aleksandr Maus d8f1735e39 Add xpack.eql.enabled feature flag, disabled by default. Enabled only for integration tests. (#51370)
Related to https://github.com/elastic/elasticsearch/issues/49581
2020-01-27 15:15:22 -05:00
Costin Leau d049de5b72 EQL: import QL into EQL (#50904)
Link QL into the new build file
Remove duplicate classes and use the new ql package
Update Exception hierarchy on top of QlException
2020-01-27 15:13:22 -05:00
Igor Motov c184411456 EQL: Replace EqlSearchResponse.Hits parser with ObjectParser (#50925)
Replaces the existing hand-build Hits parser with a
ConstructingObjectParser version.

Relates to #49581
2020-01-27 15:13:09 -05:00
Igor Motov 88cc30c0d8 EQL: Remove list classes from EqlSearchResponse (#50870)
Removes unnecessary classes from EqlSearchResponse that just represent
lists of other elements.

Relates to #49581
2020-01-27 15:13:00 -05:00
Aleksandr Maus d715176c00 Add more Eql REST API validation integration tests, clean up request implementation (#50822) 2020-01-27 15:12:48 -05:00
Igor Motov 628083183f EQL: Make EqlSearchResponse immutable (#50810)
Refactors EqlSearchResponse to make it immutable

Relates to #49581
2020-01-27 15:12:07 -05:00
Aleksandr Maus 31d2d01e25 Correct search_after handling (#50629) 2020-01-27 15:11:51 -05:00
Aleksandr Maus 79875ce4d9 Initial EQL rest API implementation (#49768) 2020-01-27 15:11:41 -05:00
Costin Leau 10a16d15d1 Add draft EQL grammar and expression tree 2020-01-27 15:11:18 -05:00