Commit Graph

170 Commits

Author SHA1 Message Date
Andrei Stefan 93abbb9057
Add data streams wildcard pattern yml test () ()
(cherry picked from commit e13a365eeb6d8c6a7c9a91f94f0e8e78e3fe4773)
2020-08-18 19:38:07 +03:00
Andrei Stefan 5de0f19cc3
EQL: Return sequence join keys in the original type () ()
(cherry picked from commit d54957d61faa0d502387656e3cace594017b6ea0)
2020-08-18 19:37:15 +03:00
Andrei Stefan 90e116738e
QL: add filtering query dsl support to IndexResolver () ()
(cherry picked from commit 7b3635d796be26af9f87d19963a8ed4ab4bbf13f)
2020-08-17 17:59:58 +03:00
Jake Landis bcb9d06bb6
[7.x] Cleanup xpack build.gradle () ()
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
2020-08-03 13:11:43 -05:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api () ()
- Replace immediate task creations by using task avoidance api
- One step closer to 
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Dan Hermann fe12217c7f
[7.x] Move REST specs for data streams () 2020-07-23 08:10:54 -05:00
Costin Leau 9cc80621c3 EQL: Fix matching of tail/desc queries ()
When dealing with tail queries, data is returned descending for the base
criterion yet the rest of the queries are ascending. This caused a
problem during insertion since while in a page, the data is ASC, between
pages the blocks of data is DESC.
This caused incorrectly sorting inside a SequenceGroup which led to
incorrect results.

Further more in case of limit, since the data in a page is ASC, early
return is not possible neither is desc matching. Thus the page needs to
be consumed first before finding the final results.
A future improvement could be to keep only the top N results dropping
the rest during insertion time.

(cherry picked from commit 77c88da054a1ce662a264f72cde5986d4ce37e3a)
2020-07-19 00:49:16 +03:00
Andrei Stefan d513e1090f
Do not create the index, if it's already there () ()
(cherry picked from commit d097447d257efdf0a36b1157e1f177aed86ecca1)
2020-07-17 11:38:30 +03:00
Costin Leau 5f2285a8b3 EQL: Fix bug in returning results ()
Using serialization/deserialization when dealing with non-trivial
documents causes the process to get stuck not to mention it is expensive.
Use a much more simple approach at the expense of losing information
(we're just interested in the source after all).

(cherry picked from commit e1659822db7ce1390ba9bbfb21768e24a0907dff)
2020-07-16 01:01:13 +03:00
Costin Leau 6b75525efb EQL: Improve testing spec ()
Case sensitivity is incorporated as a test dimension - instead of
running the same test twice, two different tests are created.
Clean-up the test invocation by removing unused parameters.

Fix 

(cherry picked from commit 72c8a3582d8e8a4a663d82814a17a1a3d2757292)
2020-07-15 18:07:24 +03:00
Igor Motov b5ab447b3e
EQL: Fix async EQL Rest test () ()
Unfortunately, we cannot guarantee that the execution will be truly
async even with 0ms timeout since we cannot block the execution. So, we need
to modify the test to work in both async and non-async mode.

Closes 
2020-07-15 11:02:33 -04:00
Costin Leau 679619c798 EQL: Improve retrieval of results ()
Instead of retrieving an entire SearchHit, get just a reference and
postpone the document retrieval when assembling the final results.
Remove sort information from results to make them consistent.
Move TumblingWindow under the sequence package.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
(cherry picked from commit bccfbcd81f2f1d3552e95e4a9ee2618fb3059bd9)
2020-07-14 23:53:57 +03:00
Andrei Stefan 1fd16ffb70
Add license header to EqlStatsIT.java () 2020-07-14 18:45:13 +03:00
Andrei Stefan cf752992d6
Add telemetry metrics () 2020-07-14 16:25:24 +03:00
Andrei Dan 7dcdaeae49
Default to @timestamp in composable template datastream definition () ()
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.

(cherry picked from commit 5609353c5d164e15a636c22019c9c17fa98aac30)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-14 12:36:54 +01:00
Costin Leau 5580eb61ed EQL: Improve sequence limiting ()
Improve the way limit (in particular offset) is being applied to handle
the case where the matches are less than the offset and absolute limit.

Combine Matcher and SequenceStateMachine into one class since the two
have evolved beyond their original name and structure.

(cherry picked from commit 63d3c62cdfc33dea03f21d5565b9c8ea104003eb)
2020-07-14 13:19:09 +03:00
Igor Motov 1acb4aeba9
EQL: Prepare for release () ()
Enables eql setting in release builds.

Relates 
2020-07-13 11:54:32 -04:00
Dan Hermann b9fb12924b
Data stream support for EQL search 2020-07-09 13:10:44 -05:00
Costin Leau d9c1e531db EQL: Introduce until functionality ()
Sequences now support until conditional, which prevents a match from
occurring if the until matches a document while doing look-ups.
Thus a sequence must complete before the until condition matches - if
any document within the sequence occurs at, or after, the until hit, the
sequence is discarded.

(cherry picked from commit 1ba1b9f0661aee655aa48cf9475ac61aaee2bfda)
2020-07-09 17:12:01 +03:00
Andrei Stefan d187b531ed
EQL: Give a name to all toml tests and enforce the naming of new tests () ()
(cherry picked from commit c8ffe3c9237d3cdd90331795b8e37517155b7e91)
2020-07-09 16:20:29 +03:00
Andrei Stefan c0e0bca84c
Remove search_after and implicit_join_key_field () ()
(cherry picked from commit 6ede6c59eff321b9fedad30e19508b9e4f788b54)
2020-07-09 12:34:01 +03:00
Costin Leau 3e32d060bf EQL: Fix bug in skipping window ()
Corrected condition that caused a sequence window to be skipped when a query
returns no results by checking not just the current stage but also following
ones as they can match with in-flight sequences.
Improve logging
Fix NPE when emptying a SequenceGroup
Increase randomization in testing
Make maxspan inclusive (up to and equal to value vs just up to)

(cherry picked from commit ad32c488688cb350c2934dfca03af86045e997b0)
2020-07-08 14:36:39 +03:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task () ()
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in  since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Costin Leau f9c15d0fec EQL: Introduce sequencing fetch size ()
The current internal sequence algorithm relies on fetching multiple results and then paginating through the dataset. Depending on the dataset and memory, setting a larger page size can yield better performance at the expense of memory.
This PR makes this behavior explicit by decoupling the fetch size from size, the maximum number of results desired.
As such, use in testing a minimum fetch size which exposed a number of bugs:

Jumping across data across queries causing valid data to be seen as a gap.
Incorrectly resuming searching across pages (again causing data to be discarded).
which have been addressed.

(cherry picked from commit 2f389a7724790d7b0bda67264d6eafcfa8b2116e)
2020-07-06 19:14:26 +03:00
Costin Leau fe775a315f EQL: Obey size request parameter ()
While at it, change the default size to 10 (to align it with the search
API defaults).

(cherry picked from commit 45795939b277e736a9e4f2f008d1c3f406239075)
2020-07-06 19:14:25 +03:00
Costin Leau 965f77fa44 EQL: Introduce sequence internal paging ()
Refactor sequence matching classes in order to decouple querying from
results consumption (and matching).
Rename some classes to better convey their intent.

Introduce internal pagination of sequence algorithm, that is getting the
data in slices and, if needed, moving forward in order to find more
matches until either the dataset is consumer or the number of results
desired is found.

(cherry picked from commit bcf2c1141302f3f98c85e82d2c501aa02c8540e9)
2020-07-02 13:44:21 +03:00
Andrei Stefan b904a60275
EQL: Add case handling to stringContains () ()
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
(cherry picked from commit 1a58776d3aa563beb364b067a1db46497122306f)
2020-07-01 13:51:45 +03:00
Andrei Stefan 470bcee5bf
EQL: Integrate TOML tests for function folding () ()
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
(cherry picked from commit e9b1fa58cf8d510a4b4afb14f66b0d5f9c603ebb)
2020-07-01 13:50:54 +03:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) ()
* Replace compile configuration usage with api ()

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
Andrei Stefan 7b80ea7218
Fix release tests () ()
(cherry picked from commit 7816c100612168bf46595c4813fe374bca2e7259)
2020-06-30 13:42:32 +03:00
Costin Leau 3a546f1f51 EQL: Introduce support for sequence maxspan ()
EQL sequences can specify now a maximum time allowed for their span
(computed between the first and the last matching event).

(cherry picked from commit 747c3592244192a2e25a092f62aec91a899afc83)
2020-06-29 21:31:00 +03:00
Igor Motov 773f3574a9
Removes debug logging from RestEqlCancellationIT ()
The test didn't fail since the fix in . So, it's time to remove debug
logging and close the issue.

Closes 
2020-06-29 13:15:01 -04:00
Andrei Stefan 3cb8f54f28
EQL: case sensitivity aware integration testing () ()
* EQL: case sensitivity aware integration testing ()

* Add DataLoader
* Rewrite case sensitivity settings:
NULL -> run both case sensitive and insensitive tests
TRUE -> run case sensitive test only
FALSE -> run case insensitive test only
* Rename test_queries_supported
* Add more toml tests from the Python client

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
(cherry picked from commit 34d383421599f060a5c083b40df35f135de49e39)
2020-06-29 18:40:07 +03:00
Costin Leau 3c81b91474 EQL: Add Head/Tail pipe support ()
Introduce pipe support, in particular head and tail
(which can also be chained).

(cherry picked from commit 4521ca3367147d4d6531cf0ab975d8d705f400ea)
(cherry picked from commit d6731d659d012c96b19879d13cfc9e1eaf4745a4)
2020-06-27 09:49:14 +03:00
Igor Motov 20af856abd
[7.x] EQL: Adds an ability to execute an asynchronous EQL search ()
Adds async support to EQL searches

Closes 

Co-authored-by: James Rodewig james.rodewig@elastic.co
2020-06-25 14:11:57 -04:00
Andrei Stefan 69f73d948b
EQL: code cleanup and further tests () ()
Add FunctionPipe tests to all functions. Cleanup functions code.

(cherry picked from commit 0f83d5799841fe99d8aeaf46e50dd11aa6bf8a57)
2020-06-24 17:38:56 +03:00
Rene Groeschke 01e9126588
Remove deprecated usage of testCompile configuration () ()
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-14 22:30:44 +02:00
Costin Leau ff0ea62cb8 EQL: Fix casing for tiebreaker field ()
Use tiebreaker instead of tieBreaker

(cherry picked from commit 3c774948a5d5e10fac267cb9a54f5d0559a00c1d)
2020-06-11 00:10:19 +03:00
Aleksandr Maus ec60335496
EQL: implement case sensitivity for indexOf and endsWith string functions () ()
* EQL: implement case sensitivity for indexOf and endsWith string functions
2020-06-10 08:55:49 -04:00
Costin Leau 439205d1ea EQL: Introduce tie breaker support ()
Allow a field inside the data to be used as a tie breaker for events
that have the same timestamp.
The field is optional by default.
If used, the tie-breaker always requires a non-null value since it is
used inside `search_after` which requires a non-null value.

Fix 

(cherry picked from commit e5719ecb474b32730d93afdbb6834a32b0b2df8b)
2020-06-09 22:50:19 +03:00
Bogdan Pintea 74b2c8a770 Change error message for comp against fields ()
Change the error message wording for comparisons against fields in
filtering (s/variables/fields).

(cherry picked from commit d9a1cb50940d0a98fd75b9c0123ca6e1d862f65d)
2020-05-26 17:57:51 +02:00
Costin Leau 6f4af43405 EQL: Skip execution for filters with empty results ()
Optimize away events queries and joins/sequence that cannot match any
results without having to query the backend.

(cherry picked from commit 69c8ef8cfefd8fc6dcb6d1a566bfcd537068e3e4)
2020-05-14 22:38:23 +03:00
Andrei Stefan ddf4e47e86
EQL: fix QueryFolderOkTests () ()
(cherry picked from commit 8b21ccd0eac3b3d0fbd090152b3dff6ae5217b52)
2020-05-14 10:58:25 +03:00
Aleksandr Maus 87a10806ab
EQL: Fix cidrMatch function fails to match when used in scripts () ()
EQL: Fix cidrMatch function fails to match when used in scripts ()

Addresses https://github.com/elastic/elasticsearch/issues/55709
2020-05-13 22:41:24 -04:00
Ross Wolf 61e2cf89b5
EQL: Add number function ()
* EQL: Add number function
* EQL: Fix the locale used for number for deterministic functionality
* EQL: Add more ToNumber tests
* EQL: Add more number ToNumberProcessor unit tests
* EQL: Remove unnecessary overrides, fix processor methods
* EQL: Remove additional unnecessary overrides
* EQL: Lint fixes for ToNumber
* EQL: ToNumber renames from PR feedback
* EQL: Remove NumberFormat locale handling
* EQL: Removed NumberFormat from ToNumber
* EQL: Add number function tests
* EQL: ToNumberProcessorTests formatting
* EQL: Remove newline in ToNumberProcessorTests
* EQL: Add number(..., null) test
* EQL: Create expression.function.scalar.math package
* EQL: Remove painless whitespace for ToNumber.asScript
* EQL: Add Long support
2020-05-13 14:09:06 -06:00
Costin Leau 9f1ecd52eb EQL: Introduce support for sequences ()
Initial support for EQL sequences
The current algorithm is focused on correctness and does not contain
any optimization which is left for the future.

The current implementation uses a state machine approach which moves
ascending and runs each query one after the other working on computing
sequences as the data comes in.
For each result, the key and its timestamp are being extracted which are
then used for matching/building a sequence.

(cherry picked from commit 4f3e18c894a1841d333022361ad9d1fdf1477dc3)
2020-05-13 15:42:31 +03:00
Marios Trivyzas cbbbd499bf
SQL/EQL: Add support for scalars within LIKE/RLIKE () ()
- Add support for scalar functions on the field of SQL's LIKE/RLIKE
- Add support for scalar functions on the field of EQL's match/matchLite

Closes: 
(cherry picked from commit 51c14e2dbb7fb29004a23369c449d425b3ac8fe2)
2020-05-13 13:40:24 +02:00
Andrei Stefan f0074e93a0
QL: case sensitive support in EQL () ()
* QL: case sensitive support in EQL ()
* adds a generic startsWith function to QL
* modifies the existent EQL startsWith function to be case sensitive
aware
* improves the existent EQL startsWith function to use a prefix query
when the function is used in a case sensitive context. Same improvement
is used in SQL's newly added STARTS_WITH function.
* adds case sensitivity to EQL configuration through a case_sensitive
parameter in the eql request, as established in .
The case_sensitive parameter can be specified when running queries
(default is case insensitive)

(cherry picked from commit ee5a09ea840167566e34c28c8225dc38bc6a7ae8)
2020-05-12 16:56:18 +03:00
Ryan Ernst 902fc546bd
Migrate remaining ESIntegTestCases to internalClusterTest () ()
This commit migrates the ESIntegTestCase tests in x-pack to the
internalClusterTest source set.
2020-05-11 21:06:04 -07:00
Andrei Stefan 980f175222
EQL: simplify equals/not-equals TRUE/FALSE expressions () ()
* Simplify equals/not-equals TRUE/FALSE expressions, by returning them
as is (TRUE variant) or negating them (FALSE variant)

(cherry picked from commit 17858afbe6da5fa0b3ecfc537cabb337e4baaffe)
2020-05-07 03:02:04 +03:00
Ross Wolf 389082033e
EQL: Add concat function ()
* EQL: Add concat function
* EQL: for loop spacing for concat
* EQL: return unresolved arguments to concat early
* EQL: Add concat integration tests
* EQL: Fix concat query fail test
* EQL: Add class for concat function testing
* EQL: Add concat integration tests
* EQL: Update concat() null behavior
2020-05-05 12:53:34 -06:00
William Brafford 3499fa917c
Deprecated xpack "enable" settings should be no-ops () ()
The following settings are now no-ops:

* xpack.flattened.enabled
* xpack.logstash.enabled
* xpack.rollup.enabled
* xpack.slm.enabled
* xpack.sql.enabled
* xpack.transform.enabled
* xpack.vectors.enabled

Since these settings no longer need to be checked, we can remove settings
parameters from a number of constructors and methods, and do so in this
commit.

We also update documentation to remove references to these settings.
2020-05-05 10:40:49 -04:00
Marios Trivyzas cc21468559
SQL: Fix issue with date range queries and timezone () ()
Previously, the timezone parameter was not passed to the RangeQuery
and as a results queries that use the ES date math notation (now,
now-1d, now/d, now/h, now+2h, etc.) were using the UTC timezone and
not the one passed through the "timezone"/"time_zone" JDBC/REST params.
As a consequence, the date math defined dates were always considered in
UTC and possibly led to incorrect results for queries like:
```
SELECT * FROM t WHERE date BETWEEN now-1d/d AND now/d
```

Fixes: 
(cherry picked from commit 300f010c0b18ed0f10a41d5e1606466ba0a3088f)
2020-05-05 10:54:23 +02:00
Ross Wolf 6da686c7e0
EQL: Add match function implementation ()
* EQL: Add Match function
* EQL: Add note about character classes
* EQL: QueryFolderFailTests.java
* EQL: Add match() fail tests
* EQL: Add match tests and fix alias
* EQL: Add match verifier failure tests
* EQL: Reorder query folder fail tests
2020-05-04 09:34:20 -06:00
Andrei Stefan 5d1bc6c89c
EQL: reject queries that use a nested field or a sub-field of a nested field ()
* Reject queries that act on nested fields or fields with nested field types in their hierarchy ()

(cherry picked from commit 2a024461cd9da821112953d4c6e565ea622c678b)
2020-05-04 15:50:31 +03:00
William Brafford d53c941c41
Make xpack.monitoring.enabled setting a no-op () ()
* Make xpack.monitoring.enabled setting a no-op

This commit turns xpack.monitoring.enabled into a no-op. Mostly, this involved
removing the setting from the setup for integration tests. Monitoring may
introduce some complexity for test setup and teardown, so we should keep an eye
out for turbulence and failures

* Docs for making deprecated setting a no-op
2020-05-01 16:42:11 -04:00
Ryan Ernst 52b9d8d15e
Convert remaining license methods to isAllowed () ()
This commit converts the remaining isXXXAllowed methods to instead of
use isAllowed with a Feature value. There are a couple other methods
that are static, as well as some licensed features that check the
license directly, but those will be dealt with in other followups.
2020-04-30 15:52:22 -07:00
Aleksandr Maus ad54cca823
EQL: implement math functions: add, divide, module, multiply, subtract () ()
* EQL: implement math functions: add, divide, module, multiply, subtract
2020-04-24 15:52:27 -04:00
David Turner 7941f4a47e Add RepositoriesService to createComponents() args ()
Today we pass the `RepositoriesService` to the searchable snapshots plugin
during the initialization of the `RepositoryModule`, forcing the plugin to be a
`RepositoryPlugin` even though it does not implement any repositories.

After discussion we decided it best for now to pass this in via
`Plugin#createComponents` instead, pending some future work in which plugins
can depend on services more dynamically.
2020-04-16 16:27:36 +01:00
Igor Motov 8a669dc9b7
EQL: Add cascading search cancellation ()
EQL search cancellation now propagates cancellation to underlying search
operations.

Relates to 
2020-04-14 08:06:02 -04:00
Ross Wolf 96a903b17f
EQL: Add string function ()
* EQL: Add string() function
* EQL: Reorder queryfolder_tests
* EQL: Add test queries
* EQL: Fix InternalEqlScriptUtils.string and test case
* EQL: Fix testStringFunctionWithText error message
* EQL: Flatten ToStringFunctionPipe.equals
* EQL: Reorder painless whitelist
* EQL: Address feedback and remove string(null) handling
* EQL: Move string(pid) test over
* EQL: Rename source -> value
2020-04-10 09:48:29 -06:00
Costin Leau a7e4f79e8f EQL: Deprecate lenient sequence declaration ()
Deprecate alternative sequence parameter declaration (with then by)
Disallow lack of time units inside maxspan

Fix 
Relate 

(cherry picked from commit 201adafba9def1de4bf843760defb9def3394f63)
2020-04-10 10:30:07 +03:00
Andrei Stefan 85f129a50a
EQL: indexOf function implementation () ()
(cherry picked from commit a4b1d6e52d9ba22d541dd86d69861b1efee83604)
2020-04-09 02:41:01 +03:00
Aleksandr Maus d02f774cb6
EQL: implement cidrMatch function () ()
Related to https://github.com/elastic/elasticsearch/issues/54132
2020-04-07 22:07:28 -04:00
Aleksandr Maus de381271f1
EQL: implement stringContains function () () 2020-04-07 17:55:13 -04:00
Aleksandr Maus 868798e4db
EQL: implement between function () () 2020-04-07 16:52:30 -04:00
Costin Leau 8b1e87cb61 EQL: Change query folding spec from new lines to ; ()
The usage of blank lines as separator between tests can be tricky to
deal with in case of merges where such lines can be added by accident.
Further more counting non-consecutive lines is non-intuitive.
The tests have been aligned to use ; at the end of the query and
exceptions so that the presence or absence of empty lines is irrelevant.
The parsing of the spec has been changed to perform validation to not
allow invalid/incomplete specs to cause exceptions.

(cherry picked from commit 192ad88d3a51e1e1f1f82830526518720ec88217)
2020-04-07 21:57:06 +03:00
Costin Leau 99846f47b7 QL: Introduce infrastructure for surrogate functions ()
Some functions act as shortcuts for more verbose declarations (sometimes
with certain constraints). This PR removes the boilerplate around
declaring such functions as well as a dedicated rule for the optimizer
to perform the actual substitution.

Fix 

(cherry picked from commit 3231d01b0c583deb89252fafe84db48878da3246)
2020-04-07 00:46:50 +03:00
Costin Leau 36121117f0 EQL: Sequence/Join parsing and model ()
Add parsing and (logical) domain model for sequence and join

(cherry picked from commit 9e9632d41a39877256c68634ab18e441f4b67fe8)
2020-04-06 23:15:35 +03:00
Igor Motov 1aa87cd4a9
EQL: Make EQL search task cancellable ()
First step towards async search execution. At the moment we don't try to cancel
the underlying search requests, and just check if the task is canceled before
performing network operation (such as field caps and search)

Relates to 
2020-04-06 13:38:03 -04:00
Ross Wolf 022f829d84
EQL: Add wildcard function ()
* EQL: Add wildcard function
* EQL: Cleanup Wildcard.getArguments
* EQL: Cleanup Wildcard and rearrange methods
* EQL: Wildcard newline lint
* EQL: Make StringUtils function final
* EQL: Make Wildcard.asLikes return ScalarFunction
* QL: Restore BinaryLogic.java
* EQL: Add Wildcard PR feedback
* EQL: Add Wildcard verification tests
* EQL: Switch wildcard to isFoldable test
* EQL: Change wildcard test to numeric field
* EQL: Remove Wildcard.get_arguments
2020-04-03 10:15:43 -06:00
Mayya Sharipova bf4857d9e0
Search hit refactoring () ()
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue  to remove
dependency on MapperService to check if a field is metafield.

Relates to PR: 
Relates to issue 

Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
2020-04-01 15:19:00 -04:00
Andrei Stefan 977302e46c
EQL: startsWith and endsWith functions implementation ()
* EQL: startsWith function implementation ()

(cherry picked from commit 666719fcfc40f6fc0535609577791369123320ab)

* EQL: endsWith function implementation ()

(cherry picked from commit 554a4c8ef04b67eed107d29b57185e9af25d9d4f)
2020-03-31 18:06:03 +03:00
Andrei Stefan 364ea0a3c0
EQL: Length function implementation () ()
(cherry picked from commit 18493467e55e014be2c9e0ebdf734e9d7fc4beaa)
2020-03-31 16:49:18 +03:00
Ross Wolf d11e977b1f
EQL: Use In from QL ()
* EQL: Use In from QL
* EQL: Add more In tests
* EQL: Test In duplicates
* EQL: Add test for In mixed types
* EQL: Copy In translation to QL
* SQL: Use InComparisons from QL
* EQL: Remove boost checks from QueryFolderOkTests
* QL: Add TranslatorHandler.convert
2020-03-30 15:19:23 -06:00
Ross Wolf 627ca03c72
EQL: Remove parser handling for functions ()
* EQL: Remove parser handling for functions
* EQL: Comment out array functions in queries-unsupported.eql
2020-03-24 14:03:02 -06:00
Costin Leau 68f74cf593
EQL: Fix custom scripting for functions () ()
Improve separation of scripting between EQL and SQL by delegating common
methods to QL. The context detection is determined based on the package
to avoid having repetitive class hierarchies.
The Painless whitelists have been improved so that the declaring class
is used instead of the inherited one.

Relates 

(cherry picked from commit 6d46033e736c64ac9255c5d6964600d2a931430a)

EQL: Add Substring function with Python semantics ()

Does not reuse substring from SQL due to the difference in semantics and
the accepted arguments.
Currently it is missing full integration tests as, due to the usage of
scripting, requires an actual integration test against a proper cluster
(and likely its own QA project).

(cherry picked from commit f58680bad33d5ce4139157a69a4d9f5f286bc3c4)
2020-03-24 20:54:19 +02:00
Aleksandr Maus fd0cdde38c
EQL: EqlActionIT improvements () ()
Related to https://github.com/elastic/elasticsearch/issues/53598
2020-03-20 17:28:15 -04:00
Aleksandr Maus 83bef862e0
EQL: Extract query folder tests definitions into resources () () 2020-03-20 10:39:35 -04:00
Jake Landis db3420d757
[7.x] Optimize which Rest resources are used by the Rest tests… ()
This should help with Gradle's incremental compile such that projects
only depend upon the resources they use.

related 
2020-03-19 12:28:59 -05:00
Aleksandr Maus 31d45b3c95
EQL: Improve query folder test suite () ()
Related to https://github.com/elastic/elasticsearch/issues/52775
2020-03-12 10:58:07 -04:00
Aleksandr Maus d064846416
EQL: Test infrastructure improvements () ()
Update CommonEqlRestTestCase code to simplify making changes as requested.
Update EqlActionIT to simplify the test code as requested.
Replace Jackson parser with XContent in EqlActionIT.
Whitelist more EQL tests specs that are now supported.
2020-03-09 14:11:54 -04:00
Ross Wolf f5f922c6f6
EQL: Add IsNull/IsNotNull checks ()
* EQL: Add IsNull/IsNotNull checks
* EQL: Simplify IsNull/IsNotNull optimization
* EQL: Split string tests over multiple lines
2020-03-09 10:41:04 -06:00
Ross Wolf d6813cb348
EQL: Convert wildcards to LIKE in analyzer ()
* EQL: Convert wildcard comparisons to Like
* EQL: Simplify wildcard handling, update tests
* EQL: Lint fixes for Optimizer.java
2020-03-06 13:13:07 -07:00
Aleksandr Maus 2dc872f052
EQL: Add HLRC for EQL stats () () 2020-03-05 09:20:38 -05:00
Ross Wolf a5e82d7fd6
EQL: Add explicit 'any where ...' handling () 2020-03-04 10:11:03 -07:00
Aleksandr Maus b47bffba24
EQL: consistent naming for event type vs event category () ()
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-04 08:02:38 -05:00
Costin Leau 712e0c05cd EQL: Add implicit ordering on timestamp ()
QL: Move Sort base class from SQL to QL
(cherry picked from commit 798015b7bbd565e9c4222724614baeb432c7c2b3)
2020-03-02 22:41:36 +02:00
Aleksandr Maus 89ed857c79
EQL: Change request parameter query to filter and rule to query () ()
Related to https://github.com/elastic/elasticsearch/issues/52911
2020-03-02 09:26:23 -05:00
Costin Leau a674085903 EQL: Disable field extraction for returned events ()
Return the whole source of matching events

(cherry picked from commit 79ca586ab1d89d645fb58142b82202f14ce5d361)
2020-02-28 13:48:15 +02:00
Costin Leau 40bc06f6ad EQL: Hook engine to Elasticsearch ()
Add query execution and return actual results returned from
Elasticsearch inside the tests

(cherry picked from commit 3e039282bf991af87604a6d4f8eada19d5e33842)
2020-02-27 11:22:22 +02:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests () ()
* Smarter copying of the rest specs and tests ()

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
Costin Leau a8911802d3 EQL: transform query AST into queryDSL ()
(cherry picked from commit 94cef29df259319dfe2a3bf92d3f1a42d7e45781)
2020-02-25 17:53:59 +02:00
Aleksandr Maus a6f5b4bb78
Unmute EqlActionIT ()
Related to https://github.com/elastic/elasticsearch/issues/52737
2020-02-25 10:22:07 -05:00
Aleksandr Maus b2cb38ccf5
EQL: Expand verification tests () ()
* EQL: Expand verification tests ()

Expand verification tests
Fix some error messaging consistency in EqlParser

Related to https://github.com/elastic/elasticsearch/issues/51873

* Adjust for 7.x compatibility
2020-02-25 07:19:33 -05:00
Mark Vieira 025352f0a4
Mute EqlActionIT 2020-02-24 16:06:30 -08:00
Aleksandr Maus a7bdb0b456
EQL: Add integration tests harness to test EQL feature parity with original implementation () ()
The tests use the original test queries from
https://github.com/endgameinc/eql/blob/master/eql/etc/test_queries.toml
for EQL implementation correctness validation.
The file test_queries_unsupported.toml serves as a "blacklist" for the
queries that we do not support. Currently all of the queries are
blacklisted. Over the time the expectation is to eventually have an
empty "blacklist" when all of the queries are fully supported.

The tests use the original test vector from
https://raw.githubusercontent.com/endgameinc/eql/master/eql/etc/test_data.json.

Only one EQL and the response is stubbed for now to match the expected
output from that query. This part would need some tweaking after EQL is
fully wired.

Related to https://github.com/elastic/elasticsearch/issues/49581
2020-02-24 12:46:59 -05:00
Jay Modi f3f6ff97ee
Single instance of the IndexNameExpressionResolver ()
This commit modifies the codebase so that our production code uses a
single instance of the IndexNameExpressionResolver class. This change
is being made in preparation for allowing name expression resolution
to be augmented by a plugin.

In order to remove some instances of IndexNameExpressionResolver, the
single instance is added as a parameter of Plugin#createComponents and
PersistentTaskPlugin#getPersistentTasksExecutor.

Backport of 
2020-02-21 07:50:02 -07:00
Igor Motov e5b21a3fc6
Add HLRC for EQL search ()
Adds EQL HLRC client with the search method.

Relates to 
2020-02-21 08:44:08 -05:00
Costin Leau 5373a77fb9 QL: Extract common Failure class ()
Shared across SQL and EQL

(cherry picked from commit 1aeda20d3ec3d6c885de03c6043dd1e8eab9f230)
2020-02-13 14:35:15 +02:00