Commit Graph

4593 Commits

Author SHA1 Message Date
Armin Braun aae93a7578
Allow Repository Plugins to Filter Metadata on Create (#51472) (#51542)
* Allow Repository Plugins to Filter Metadata on Create

Add a hook that allows repository plugins to filter the repository metadata
before it gets written to the cluster state.
2020-01-28 18:33:26 +01:00
Gordon Brown 89c2834b24
Deprecate creation of dot-prefixed index names except for hidden and system indices (#49959)
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered by
a plugin that extends SystemIndexPlugin. This is the first step
towards more thorough protections for system indices.

This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register .tasks as a system index.
2020-01-28 10:01:16 -07:00
Yannick Welsch f6686345c9 Avoid unnecessary setup and teardown in docs tests (#51430)
The docs tests have recently been running much slower than before (see #49753).

The gist here is that with ILM/SLM we do a lot of unnecessary setup / teardown work on each
test. Compounded with the slightly slower cluster state storage mechanism, this causes the
tests to run much slower.

In particular, on RAMDisk, docs:check is taking

ES 7.4: 6:55 minutes
ES master: 16:09 minutes
ES with this commit: 6:52 minutes

on SSD, docs:check is taking

ES 7.4: ??? minutes
ES master: 32:20 minutes
ES with this commit: 11:21 minutes
2020-01-28 16:52:23 +01:00
David Roberts 550254ec7f [ML] Use CSV ingest processor in find_file_structure ingest pipeline (#51492)
Changes the find_file_structure response to include a CSV
ingest processor in the ingest pipeline it suggests.

Previously the Kibana file upload functionality parsed CSV
in the browser, but by parsing CSV in the ingest pipeline
it makes the Kibana file upload functionality more easily
interchangable with Filebeat such that the configurations
it creates can more easily be used to import data with the
same structure repeatedly in production.
2020-01-28 14:38:43 +00:00
Aleksandr Maus a8bd4d08e3 Merge branch 'feature/eql_backport' into 7.x 2020-01-28 09:19:39 -05:00
Hendrik Muhs 53e4d1ef07 [Transform] fix TransformRobustnessIT intermittent test failures part 2 (#51523)
add wait for completion in transform robustness test to avoid occasional test failures during cleanup

fixes #51347
2020-01-28 13:37:01 +01:00
William Brafford 9efa5be60e
Password-protected Keystore Feature Branch PR (#51123) (#51510)
* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
2020-01-28 05:32:32 -05:00
Hendrik Muhs 2239ba8c6e
[Transform] avoid mapping problems with index templates (#51368) (#51519)
insert explict mappings for objects in nested output to avoid clashes with index templates

fixes #51321
2020-01-28 11:31:07 +01:00
Hendrik Muhs 61663b495e add an integration test using date_nanos as timestamp (#51477)
add a test for using date_nanos as timestamp field in a continuous transform
2020-01-28 10:10:23 +01:00
Hendrik Muhs bebce4b190 audit index creation after it the index has been created (#51479)
moves audit message for index creation after the index has been successfully created. This has
been confusing for a user where index creation failed but audit reported index creation.
2020-01-28 10:06:46 +01:00
Ioannis Kakavas 4f3548fbd7
Disable diagnostic trust manager in tests (#51501)
This commit sets `xpack.security.ssl.diagnose.trust` to false in all
of our tests when running in FIPS 140 mode and when settings objects
are used to create an instance of the SSLService. This is needed
in 7.x because setting xpack.security.ssl.diagnose.trust to true
wraps SunJSSE TrustManager with our own DiagnosticTrustManager and
this is not allowed when SunJSSE is in FIPS mode.
An alternative would be to set xpack.security.fips.enabled to
true which would also implicitly disable
xpack.security.ssl.diagnose.trust but would have additional effects
(would require that we set PBKDF2 for password hashing algorithm in
all test clusters, would prohibit using JKS keystores in nodes even
if relevant tests have been muted in FIPS mode etc.)

Relates: #49900
Resolves: #51268
2020-01-28 10:17:35 +02:00
Przemko Robakowski 919083decd
Don't overwrite target field with SetSecurityUserProcessor (#51454) (#51506)
* Don't overwrite target field with SetSecurityUserProcessor

This change fix problem with `SetSecurityUserProcessor` which was overwriting
whole target field and not only fields really filled by the processor.

Closes #51428

* Unused imports removed
2020-01-28 02:12:09 +01:00
Jason Tedor 92b611ece1
Formalize build snapshot (#51484)
Today we are repeatedly checking if the current build is a snapshot
build or not by reading the system property build.snapshot. This commit
formalizes this by adding a build parameter to indicate whether or not
the current build is a snapshot build.
2020-01-27 16:56:31 -05:00
Aleksandr Maus eb1ed2a35f Compilation fixes for 7.x 2020-01-27 16:23:36 -05:00
Aleksandr Maus d8f1735e39 Add xpack.eql.enabled feature flag, disabled by default. Enabled only for integration tests. (#51370)
Related to https://github.com/elastic/elasticsearch/issues/49581
2020-01-27 15:15:22 -05:00
Costin Leau d049de5b72 EQL: import QL into EQL (#50904)
Link QL into the new build file
Remove duplicate classes and use the new ql package
Update Exception hierarchy on top of QlException
2020-01-27 15:13:22 -05:00
Igor Motov c184411456 EQL: Replace EqlSearchResponse.Hits parser with ObjectParser (#50925)
Replaces the existing hand-build Hits parser with a
ConstructingObjectParser version.

Relates to #49581
2020-01-27 15:13:09 -05:00
Igor Motov 88cc30c0d8 EQL: Remove list classes from EqlSearchResponse (#50870)
Removes unnecessary classes from EqlSearchResponse that just represent
lists of other elements.

Relates to #49581
2020-01-27 15:13:00 -05:00
Aleksandr Maus d715176c00 Add more Eql REST API validation integration tests, clean up request implementation (#50822) 2020-01-27 15:12:48 -05:00
Igor Motov 628083183f EQL: Make EqlSearchResponse immutable (#50810)
Refactors EqlSearchResponse to make it immutable

Relates to #49581
2020-01-27 15:12:07 -05:00
Aleksandr Maus 31d2d01e25 Correct search_after handling (#50629) 2020-01-27 15:11:51 -05:00
Aleksandr Maus 79875ce4d9 Initial EQL rest API implementation (#49768) 2020-01-27 15:11:41 -05:00
Costin Leau 10a16d15d1 Add draft EQL grammar and expression tree 2020-01-27 15:11:18 -05:00
Costin Leau e22f501018
QL: Backport project to 7.x (#51497)
* Introduce reusable QL plugin for SQL and EQL (#50815)

Extract reusable functionality from SQL into its own dedicated project QL.
Implemented as a plugin, it provides common components across SQL and the upcoming EQL.

While this commit is fairly large, for the most part it's just a big file move from sql package to the newly introduced ql.

(cherry picked from commit ec1ac0d463bfa12a02c8174afbcdd6984345e8b4)

* SQL: Fix incomplete registration of geo NamedWritables

(cherry picked from commit e295763686f9592976e551e504fdad1d2a3a566d)

* QL: Extend NodeSubclass to read classes from jars (#50866)

As the test classes are spread across more than one project, the Gradle
classpath contains not just folders but also jars.
This commit allows the test class to explore the archive content and
load matching classes from said source.

(cherry picked from commit 25ad74928afcbf286dc58f7d430491b0af662f04)

* QL: Remove implicit conversion inside Literal (#50962)

Literal constructor makes an implicit conversion for each value given
which turns out has some subtle side-effects.
Improve MathProcessors to preserve numeric type where possible
Fix bug on issue compatibility between date and intervals
Preserve the source when folding inside the Optimizer

(cherry picked from commit 9b73e225b0aa07a23859550fb117bae571a2b672)

* QL: Refactor DataType for pluggability (#51328)

Change DataType from enum to class
Break DataType enums into QL (default) and SQL types
Make data type conversion pluggable so that new types can be introduced

As part of the process:
- static type conversion in QL package (such as Literal) has been
removed
- several utility classes have been broken into base (QL) and extended
(SQL) parts based on type awareness
- operators (+,-,/,*) are
- due to extensibility, serialization of arithmetic operation has been
slightly changed and pushed down to the operator executor itself

(cherry picked from commit aebda81b30e1563b877a8896309fd50633e0b663)

* Compilation fixes for 7.x
2020-01-27 22:03:58 +02:00
Ryan Ernst 6ee1baf2ed
Migrate cron eval bats test to java (#50940) (#51007)
This commit migrates the simple test of the cron eval tool from bats to
java packaging tests.

relates #46005
2020-01-27 10:49:01 -08:00
Nik Everett 4ff314a9d5
Begin moving date_histogram to offset rounding (take two) (#51271) (#51495)
We added a new rounding in #50609 that handles offsets to the start and
end of the rounding so that we could support `offset` in the `composite`
aggregation. This starts moving `date_histogram` to that new offset.

This is a redo of #50873 with more integration tests.

This reverts commit d114c9db3e1d1a766f9f48f846eed0466125ce83.
2020-01-27 13:40:54 -05:00
David Roberts 3c223ceea1 [ML] Fix 2 digit year regex in find_file_structure (#51469)
The DATE and DATESTAMP Grok patterns match 2 digit years
as well as 4 digit years.  The pattern determination in
find_file_structure worked correctly in this case, but
the regex used to create a multi-line start pattern was
assuming a 4 digit year.  Also, the quick rule-out
patterns did not always correctly consider 2 digit years,
meaning that detection was inconsistent.

This change fixes both problems, and also extends the
tests for DATE and DATESTAMP to check both 2 and 4 digit
years.
2020-01-27 17:23:18 +00:00
Benjamin Trent 8559ff7cee
[ML][Inference] fixing pattern compilation + unnecessary string copy (#51483) (#51487) 2020-01-27 12:12:34 -05:00
Martijn van Groningen 8b851bfc33
Removed more unchecked suppress warnings.
See #48381
2020-01-27 14:51:49 +01:00
Martijn van Groningen 716904fab7
Unmuted test with more logging and
removed unchecked suppress warnings.

See #48381
2020-01-27 14:10:43 +01:00
Hendrik Muhs b233e93014
[Transform] refactor naming leftovers and apply code formating (#51465) (#51470)
refactor renaming leftovers: "data frame transform" to "transforms", touch only internals (variable
names, non-public API's, doc strings, ...) and apply code-formatting (spotless). No logical changes.
2020-01-27 14:04:57 +01:00
Andrei Dan 977cce002e
Preserve slm-history-ilm-policy between test runs (#51442) (#51468)
(cherry picked from commit 4e95c8a94fa700d44ac31ef17547512748ab1885)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-01-27 10:40:40 +00:00
Andrei Dan d872db278a
Fix TimeSeriesLifecycleActionsIT.testShrinkAction (#51431) (#51467)
* Fix TimeSeriesLifecycleActionsIT.testShrinkAction

Shrinking a 6 shard index to 3 shards can be quite time consuming and
assertBusy probes the conditions at exponentially growing intervals.

This separates the one assertion that was used for all the conditions
into multiple assertBusy statements and increases the timeout for waiting
for the shrink to complete.

* Allow more time for shrink to complete

This commit allows more time for the shrink operation to complete in
testRetryFailedShrinkAction (separating the assertBusy calls too) and
testMoveToRolloverStep.

* Shrink to no more than 2 shards in tests

(cherry picked from commit 5fe780148fa3536915d61475b087896a5b9ace82)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-01-27 10:40:29 +00:00
Martijn van Groningen d289c1d5f1
Wrong bug url in @AwaitsFix
See #48381
2020-01-27 10:38:03 +01:00
Martijn van Groningen e253b7e73d
Retry response exceptions in the test.
Relates to #30777
2020-01-27 10:32:38 +01:00
Martijn van Groningen 7e0f73e035
Muted watcher bwc restart test #30777 2020-01-27 10:32:37 +01:00
Ioannis Kakavas ee202a642f
Enable tests in FIPS 140 in JDK 11 (#49485)
This change changes the way to run our test suites in 
JVMs configured in FIPS 140 approved mode. It does so by:

- Configuring any given runtime Java in FIPS mode with the bundled
policy and security properties files, setting the system
properties java.security.properties and java.security.policy
with the == operator that overrides the default JVM properties
and policy.

- When runtime java is 11 and higher, using BouncyCastle FIPS 
Cryptographic provider and BCJSSE in FIPS mode. These are 
used as testRuntime dependencies for unit
tests and internal clusters, and copied (relevant jars)
explicitly to the lib directory for testclusters used in REST tests

- When runtime java is 8, using BouncyCastle FIPS 
Cryptographic provider and SunJSSE in FIPS mode. 

Running the tests in FIPS 140 approved mode doesn't require an
additional configuration either in CI workers or locally and is
controlled by specifying -Dtests.fips.enabled=true
2020-01-27 11:14:52 +02:00
Przemysław Witek dd3e2f1e18
[7.x] Update quantiles document in the index the document belongs to (#51135) (#51415) 2020-01-27 10:13:02 +01:00
Przemko Robakowski fbec19c022
Centralize mocks initialization in ILM steps tests (#51384) (#51453)
* Centralize mocks initialization in ILM steps tests

This change centralizes initialization of `Client`, `AdminClient`
and `IndicesAdminClient` for all classes extending `AbstractStepTestCase`.
This removes a lot of code duplication and make it easier to write tests.
This also removes need for `AsyncActionStep#setClient`

* Unused imports removed

* Added missed tests

* Fix OpenFollowerIndexStepTests
2020-01-25 01:19:55 +01:00
Lee Hinman 8560847dd9
[7.x] Check all snapshots in SnapshotLifecycleRestIT.testFullP… (#51448)
* Check all snapshots in SnapshotLifecycleRestIT.testFullPolicy

Rather than check the first returned snapshot for a snapshot starting with `snap-` in
SnapshotLifecycleRestIT.testFullPolicy, this commit changes the test to find any snapshots starting
with `snap-`.

In the event that there are no snapshots (the failure case), this also exposes the full results map
so we can diagnose why a failure occurred.

Relates to #50358

* Use a more imperative style for checking
2020-01-24 14:30:42 -07:00
Lee Hinman bdb8b6aa0d
[7.x] Separate aliases used for tests in TimeSeriesLifecycleAc… (#51432)
* Separate aliases used for tests in TimeSeriesLifecycleActionsIT

This is related to #51375 and hopes to help illuminate why some of those tests are failing. This
commit switches the aliases used in the test to use a random alias name every time (since there were
some complaints in the tests about aliases having more than one write index). With this we hope to
determine the actual cause of the failure in the test.

This also adds additional information to the exception returned when calling move-to-step with the
incorrect current step.

* Fix rest test
2020-01-24 11:05:19 -07:00
Benjamin Trent bf53ca3380
[7.x] [ML] Add _cat/ml/anomaly_detectors API (#51364) (#51408)
[ML] Add _cat/ml/anomaly_detectors API (#51364)
2020-01-24 11:54:22 -05:00
Benjamin Trent fc994d9ce1
[ML][Inference] Adds validations for model PUT (#51376) (#51409)
Adds validations making sure that

* `input.field_names` is not empty
* `ensemble.trained_models` is not empty
* `tree.feature_names` is not empty

closes https://github.com/elastic/elasticsearch/issues/51354
2020-01-24 09:29:12 -05:00
Hendrik Muhs d177747f66 fix TransformRobustnessIT intermittent test failures
ensure the cluster is not in some intermediate state when cleaning up.

fixes #51347
2020-01-24 15:19:11 +01:00
Martijn van Groningen 36b460060c
Unmuted watcher security smoke tests on 7 dot x branch.
Also removed the usage of types in watcher's index action and
added more logging in case this test fails again.

Relates to #30777
2020-01-24 14:51:07 +01:00
Martijn van Groningen 7af0474101
Add more logging when failing watch history entry fails. (#50931)
Relates to #30777
2020-01-24 14:49:57 +01:00
Benjamin Trent 76660a5a4f
[7.x] [ML][Inference] add tags url param to GET (#51330) (#51404)
* [ML][Inference] add tags url param to GET (#51330)

Adds a new URL parameter, `tags` to the GET _ml/inference/<model_id> endpoint.

This parameter allows the list of models to be further reduced to those who contain all the provided tags.
2020-01-24 08:26:58 -05:00
Martijn van Groningen d3078c5b40
Re-enable FullClusterRestartIT#testWatcher test (#50463)
Previously this test failed waiting for yellow:
https://gradle-enterprise.elastic.co/s/fv55holsa36tg/console-log#L2676

Oddly cluster health returned red status, but there were no unassigned, relocating or initializing shards.
Placed the waiting for green in a try-catch block, so that when this fails again then cluster state gets printed.

Relates to #48381
2020-01-24 14:07:09 +01:00
Martijn van Groningen 53ac28e398
Update smoke test watcher test suite with the changes in master branch.
Relates to #32299
2020-01-24 14:02:55 +01:00
Hendrik Muhs ded7407b4d
[Transform] Adapt tests for error message to 7.x format
adapt messages to 7.x format (#51398)

fixes #51360
2020-01-24 12:17:32 +01:00