AutoFollowIT tests are regularly failing on CI because they rely
on how cluster state updates are processed within the integration
clusters. We tried to limit this in #49141 by moving to latches
instead of waiting for assertions to pass but there are still some
places were it still need to wait for the cluster state updates to
be processed and auto-follow stats to be updated.
This commit gives more time to assertBusy() that verifies the
AutoFollowStats (up to 60 seconds) and also always log the
auto-follow stats in case the assertions failed.
Closes#48982
* EQL: Plug query params into the AstBuilder (#51886)
As the eventType is customizable, plug that into the parser based on the
given request.
(cherry picked from commit 5b4a3a3c07eacbc339cbd4c05a3621d056cc8d60)
* EQL: Add field resolution and verification (#51872)
Add basic field resolution inside the Analyzer and a basic Verifier to
check for any unresolved fields.
(cherry picked from commit 7087358ae2fb212811d480ec8641a46167946c82)
* EQL: Introduce basic execution pipeline (#51809)
Add main classes that form the 'execution' pipeline are added - most of
them have no functionality; the purpose of this PR is to add flesh out
the contract between the various moving parts so that work can start on
them independently.
(cherry picked from commit 9a1bae50a49af7fe8467b74b154c0d82c6bb9a19)
* EQL: Add AstBuilder to convert to QL tree (#51558)
* EQL: Add AstBuilder visitors
* EQL: Add tests for wildcards and sets
* EQL: Fix licensing
* EQL: Fix ExpressionTests.java license
* EQL: Cleanup imports
* EQL: PR feedback and remove LiteralBuilder
* EQL: Split off logical plan from expressions
* EQL: Remove stray import
* EQL: Add predicate handling for set checks
* EQL: Remove commented out dead code
* EQL: Remove wildcard test, wait until analyzer
(cherry picked from commit a462700f9c8e1fb977d62d42eb0077403b8fa98b)
* EQL grammar updates and tests (#49658)
* EQL: Additional tests and grammar updates
* EQL: Add backtick escaped identifiers
* EQL: Adding keywords to language
* EQL: Add checks for unsupported syntax
* EQL: Testing updates and PR feedback
* EQL: Add string escapes
* EQL: Cleanup grammar for identifier
* EQL: Remove tabs from .eql tests
(cherry picked from commit 6f1890bf2d52cabdfd1e7848fb481cf54b895f25)
There is an open JDK bug that is causing an assertion in the JDK's
http server to trip if we don't drain the request body before sending response headers.
See https://bugs.openjdk.java.net/browse/JDK-8180754
Working around this issue here by always draining the request at the beginning of the handler.
Fixes#51446
Adds documentation for basic EQL syntax.
Joins, sequences, and other syntax to be added as its supported
in future development.
Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Add unit and integration tests where literals are SELECTed
in combination with GROUP BY and possibly aggregate functions.
Relates to #41411 and #34583
which have been fixed.
(cherry picked from commit b97f1ca12675d6ea4772c60578922fe1cc2409ee)
This patch supplements #51792 and #51535 where the type of the "slices" parameter has been fixed.
(cherry picked from commit 2ed9e95100474f3dfbeb7efb0529e237b8f61e53)
We only drop ilm/slm policies on teardown only if the running docs tests
are ilm/slm related.
This updates the test name pattern to match the ilm/slm related tests
when running on windows
(eg.`reference\ilm/update-lifecycle-policy/line_29`).
(cherry picked from commit 4bb5bbd52eee59bd3eee6d766a9efc159822d9b9)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
* [DOCS] Align with ILM API docs (#48705)
* [DOCS] Reconciled with Snapshot/Restore reorg
* [DOCS] Split off ILM overview to a separate topic. (#51287)
* [DOCS} Split off overview to a separate topic.
* [DOCS] Incorporated feedback from @jrodewig.
* [DOCS] Edit ILM GS tutorial (#51513)
* [DOCS] Edit ILM GS tutorial
* [DOCS] Incorporated review feedback from @andreidan.
* [DOCS] Removed test link & fixed anchor & title.
* Update docs/reference/ilm/getting-started-ilm.asciidoc
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
* Fixed glossary merge error.
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Adding best_compression (#49974)
This commit adds a `codec` parameter to the ILM `forcemerge` action. When setting the codec to `best_compression` ILM will close the index, then update the codec setting, re-open the index, and finally perform a force merge.
* Fix ForceMergeAction toSteps construction (#51825)
There was a duplicate force merge step and the test continued to fail. This commit clarifies the
`toStep` method and changes the `assertBestCompression` method for better readability.
Resolves#51822
* Update version constants
Co-authored-by: Sivagurunathan Velayutham <sivadeva.93@gmail.com>
Currently, the same class `FieldCapabilities` is used both to represent the
capabilities for one index, and also the merged capabilities across indices. To
help clarify the logic, this PR proposes to create a separate class
`IndexFieldCapabilities` for the capabilities in one index. The refactor will
also help when adding `source_path` information in #49264, since the merged
source path field will have a different structure from the field for a single index.
Individual changes:
* Add a new class IndexFieldCapabilities.
* Remove extra constructor from FieldCapabilities.
* Combine the add and merge methods in FieldCapabilities.Builder.
Currently, a mappings update request, where dynamic_mappings is an object
instead of an array, results in a http response with a 500 code. This PR checks
for this condition and throws a MapperParsingException like we do for other
malformed mapping cases.
Closes#51486
The work to switch file upload over to treating delimited files
like semi-structured text and using the ingest pipeline for CSV
parsing makes the multi-line start pattern used for delimited
files much more critical than it used to be.
Previously it was always based on the time field, even if that
was towards the end of the columns, and no multi-line pattern
was created if no timestamp was detected.
This change improves the multi-line start pattern by:
1. Never creating a multi-line pattern if the sample contained
only single line records. This improves the import
efficiency in a common case.
2. Choosing the leftmost field that has a well-defined pattern,
whether that be the time field or a boolean/numeric field.
This reduces the risk of a field with newlines occurring
earlier, and also means the algorithm doesn't automatically
fail for data without a timestamp.
* Use standard format for reload settings API
The reload-secure-settings API page was not reorganized for the standard
API format, so this commit is reorganizing the page and adding some
links to the page in related documentation.
* Fix broken links
* Reorder examples to correctly check API response
* Note that only certain settings are reloadable
* [DOCS] Edits layout
* [DOCS] Removes unnecessary callouts
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Due to of a typo in the version regex pattern only the last digit of a major
version number is taken into consideration. So docker's version 17.0.1 is parsed
as 7.0.1.
Adds a secure and reloadable SECURE_AUTH_PASSWORD setting to allow keystore entries in the form "xpack.monitoring.exporters.*.auth.secure_password" to securely supply passwords for monitoring HTTP exporters. Also deprecates the insecure `AUTH_PASSWORD` setting.
ActionListener.completeWith would catch exceptions from
listener.onResponse and deliver them to lister.onFailure, essentially
double notifying the listener. Instead we now assert that listeners do
not throw when using ActionListener.completeWith.
Relates #50886
when a timezone is not provided Ingest logic should consider a time to be in a timezone provided as a parameter.
When a timezone is provided Ingest should recalculate a time to the timezone provided as a parameter
closes#51108
backport(#51215)
With the new mechanism for storing cluster state in lucene, we store
index metadata in multiple data paths too. This causes cluster state
publish to timeout too frequently with a 1s timeout, so increasing it to
5s. Also increasing follower check timeout to 5s since it also sometimes
has fsync in its timeout path and leader check for symmetry.
Closes#51329
We suspect the flakiness could’ve come from the fact that the rollover
step used to create the new index and roll the write alias to the new
index in separate cluster state updates. So the assertion that the
rolled index exists could’ve passed in the test but, before the
alias was rolled over to the new index, the subsequent write we execute
in the test (namely
`indexDocs("test_user", "x-pack-test-password", "foo_alias", 1)`)
would’ve sent the new document to the source index (ie. foo-logs-000001)
This would see the source index containing 3 documents and the rolled
index (foo-logs-000002) 0 documents.
However, we fixed this and the rollover step executes the “create index
and roll alias” in one single cluster update, so this situation should
not occur anymore.
(cherry picked from commit 834261c4fe7dd93f437eeec43c00d01ff2279f86)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Adds more tests for the new "proxy" remote cluster connection mode, using a Docker-based
setup, as well as testing SNI-based routing using HAProxy.
- Checks that the new proxy mode can work in situations where the publish host of the nodes in
the remote cluster are not routable.
- Checks that the new proxy mode can work with clusters where nodes are hidden behind
HAProxy.
- Checks that the new proxy mode can work with clusters where nodes are hidden behind
HAProxy, using SNI to identify the nodes/cluster behind HAProxy.
Relates #49067
The previous patch in c1d9966d35d incorrectly set the `type` to `number|auto`,
which is incorrect — the "polymorphic" type, denoted with the `|` sign,
should contain only other types, ie. number, string, bool, etc.
Fixes#51535
(cherry picked from commit 68db7fc611622ca0e418f454249e376e01f80587)
* REST: Test: Fix the `accept_enterprise` parameter for Get License API (#51527)
The Get License API specifies the `accept_enterprise` parameter as a `boolean`:
0ca5cb8cb6/x-pack/plugin/src/test/resources/rest-api-spec/api/license.get.json (L22-L27)
In the test, a `string` is passed however, which makes the test compilation fail in the Go client.
(cherry picked from commit e2a2169b3d44592057c143253bb56375ed3e4268)
* Fix the SQL API documentation in REST specification (#51534)
This patch fixes the SQL REST API documentation to conform to the current schema.
(cherry picked from commit c8b6a849852699883086a6ada42279f2f68d7e07)
* Fix the "slices" parameter for the Delete By Query API in the REST specification (#51535)
This patch updates the `type` parameter in the Delete By Query API: according to
[the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html#docs-delete-by-query-slice),
it can be set to "auto", but the type in the documentation allows only numerical values.
This prevents people from setting the parameter to "auto" eg. in the Go client,
which generates source from the specification, and sets the corresponding Go
type as number.
The patch uses the `|` notation, which we have discussed previously for encoding
a "polymorphic" parameter like this.
Related: https://github.com/elastic/go-elasticsearch/issues/77
* Fix the Enrich API documentation in REST specification (#51528)
This patch fixes the REST API documentation for the Enrich APIs to conform to the current schema.
(cherry picked from commit 59f28f4f2feeba3f6d2f0b632410577eacb28121)
do index refresh of the internal transform index with the system user
instead of using the calling user which does not have sufficient rights
if security is enabled
fixes#51728
While we use `== false` as a more visible form of boolean negation
(instead of `!`), the true case is implied and the true value does not
need to explicitly checked. This commit converts cases that have slipped
into the code checking for `== true`.
LoggingOutputStream reads a stream and breaks on newlines. This commit
fixes the behavior to account for windows newlines also containing `\r`.
closes#51532
* Fix SnapshotLifecycleRestIT.testFullPolicySnapshot
This previously was missing some key information in the output of the failure. This captures that
information and adds logging at each step so we can determine the cause *if* it fails again.
Resolves#50358
Henning Andersen
Tanguy Leroux
Costin Leau
Costin Leau
Armin Braun
Benjamin Trent
James Rodewig
Martijn van Groningen
Marios Trivyzas
Ignacio Vera
David Kyle
Karel Minarik
Yannick Welsch
Adrien Grand
Andrei Dan
debadair
Lee Hinman
Julie Tibshirani
Maria Ralli
feifeiiiiiiiiii
Hendrik Muhs
Florian Kelbert
Dan Hermann
baifan
Grzegorz Banasiak
David Roberts
William Brafford
Igor Motov
Jonathan Budzenski
Jochem Wichers Hoeth
Przemyslaw Gomulka
Ryan Ernst