Commit Graph

49749 Commits

Author SHA1 Message Date
Lee Hinman e395cf3419
Guard against null settings in CCRIndexLifecycleIT (#51008) (#51054)
It's possible that the index could return no settings and thus throw a
`NullPointerException`.

I wasn't able to reproduce the original issue, but this should guard
against in the future.

Resolves #50646

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-01-15 11:21:18 -07:00
Christoph Büscher d291f189a8
Fix hardcoded version replacement in put-dfanalytics.asciidoc #51053
The version replacement for the code snippet should replace 7.6 with the current version,
but doesn't match because of a missing whitespace.

Closes #51052
2020-01-15 18:09:37 +01:00
taku333 65af0a0f0a [DOCS] Add 7.5.1 link to release notes overview (#51022) 2020-01-15 11:53:26 -05:00
Lee Hinman ad60f0015e
Address failures in SnapshotLifecycleRestIT.testFullPolicySnapshot (#51013)
This test failed a couple of different ways, related to timing, as well
as concurrent snapshots, and also naming.

This commit splits the giant `assertBusy` into separate parts so that we don't
perform ~5 different requests and tests in the same loop. It also gives each
test a unique repository so that no other test can accidentally re-use
snapshots.

Resolves #50358 (hopefully!)
2020-01-15 09:47:41 -07:00
Lee Hinman b14a949fa9 Add blurb about ILM-injected unfollow action (#51009)
These injected actions are harmless and safe to ignore for non-CCR indices.

Resolves #50548
2020-01-15 09:46:57 -07:00
Robin Clarke 5ea18cb4af [Docs] Fix sub-heading in start-stop-ilm.asciidoc (#51045)
Removed superfluous `=`.
2020-01-15 16:16:22 +01:00
Zachary Tong 9eab87062c Bump version to 7.7.0 2020-01-15 10:12:14 -05:00
Yannick Welsch dc47b380c8 Block too many concurrent mapping updates (#51038)
Ensures that there are not too many concurrent dynamic mapping updates going out from the
data nodes to the master.

Closes #50670
2020-01-15 16:02:11 +01:00
Przemysław Witek b4a631277a
Add missing docs for new evaluation metrics (#50967) (#51041) 2020-01-15 15:53:42 +01:00
Armin Braun dd09dc7af6
Fix MultiVersionRepository BwC Tests (#51006)
* Fix MultiVersionRepository BwC Tests

The HLRC doesn't like what its getting back from some older 6.x versions for the restore status
so I moved that request to the low level client.
Closes #50819
2020-01-15 15:19:08 +01:00
Rory Hunter 2f069d8f3f
Tweak formatter config for long generic lines (#51027)
Backport of #50909. The current formatting config allows some long
generic declarations to break the 140 character limit. Tweak the config
to wrap such lines.
2020-01-15 13:17:37 +00:00
Alan Woodward 0257de8c26
Emit warnings when index templates have multiple mappings (#50982)
Index templates created in the 5x line can still be present in the cluster state
through multiple upgrades, and may have more than one mapping defined. 8x
will stop supporting templates with multiple mappings, and we should emit
deprecation warnings in 7x clusters to give users a chance to update their
templates before upgrading.
2020-01-15 11:59:27 +00:00
David Roberts 1536c3e622 [TEST] Increase ML distributed test job open timeout (#50998)
There have been occasional failures, presumably due to
too many tests running in parallel, caused by jobs taking
around 15 seconds to open.  (You can see the job open
successfully during the cleanup phase shortly after the
failure of the test in these cases.)  This change increases
the wait time from 10 seconds to 20 seconds to reduce the
risk of this happening.
2020-01-15 08:58:55 +00:00
Martijn van Groningen e76c3d4d32
Tidy up enrich processors: (#50957)
* Fix generics usages.
* Sealed match processor class.
2020-01-15 08:51:22 +01:00
Tomas Della Vedova 5b6fa79fd8
[ML] Removed key value from the catch regex test (#50977) (#51021) 2020-01-15 08:50:59 +01:00
István Zoltán Szabó b570f417c2 [DOCS] Describes the relationship of the time-related settings in anomaly detection docs (#50959)
Co-Authored-By: David Roberts <dave.roberts@elastic.co>
2020-01-15 08:46:04 +01:00
Tim Vernum e41c0b1224
Deprecating kibana_user and kibana_dashboard_only_user roles (#50963)
This change adds a new `kibana_admin` role, and deprecates
the old `kibana_user` and`kibana_dashboard_only_user`roles.

The deprecation is implemented via a new reserved metadata
attribute, which can be consumed from the API and also triggers
deprecation logging when used (by a user authenticating to
Elasticsearch).

Some docs have been updated to avoid references to these
deprecated roles.

Backport of: #46456

Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
2020-01-15 11:07:19 +11:00
Gordon Brown f207e5bde9
Add generated benchmark files to gitignore (#51000)
When building IntelliJ generates several source files related to the
benchmarks. This commit adds that path to the gitignore so these don't
get accidentally committed.
2020-01-14 16:18:28 -07:00
Nik Everett fc5fde7950
Add "did you mean" to ObjectParser (#50938) (#50985)
Check it out:
```
$ curl -u elastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_update/foo?pretty -d'{
  "dac": {}
}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
  },
  "status" : 400
}
```

The tricky thing about implementing this is that x-content doesn't
depend on Lucene. So this works by creating an extension point for the
error message using SPI. Elasticsearch's server module provides the
"spell checking" implementation.
s
2020-01-14 17:53:41 -05:00
Ryan Ernst 4bdab0e985 Fix windows chown to work with single file (#51004)
The chown utility for packaging tests works on windows when the given
path is a directory, but would fail if the path was a single file. This
commit fixes it to handle both cases.

relates #50825
2020-01-14 14:40:04 -08:00
Yannick Welsch 4b0581f182 Remove custom metadata tool (#50813)
Adds a command-line tool to remove broken custom metadata from the cluster state.

Relates to #48701
2020-01-14 23:08:33 +01:00
James Rodewig a290762df1 [DOCS] Document `breakers`, `script`, and `discovery` node stats (#50509)
Documents the `breakers`, `script`, and `discovery` parameters returned
by the `_nodes/stats` API.
2020-01-14 16:51:50 -05:00
Nik Everett a8aca6b2a0
Switch AggregationSpec to ContextParser (#50871) (#50980)
We seem to have settled on the `ContextParser` interface for parsing
stuff, mostly because `ObjectParser` implements it. We don't really
*need* the old `Aggregator.Parser` interface any more because it
duplicates `ContextParser` but with the arguments reversed.

This adds support to `AggregationSpec` to declare aggregation parsers
using `ContextParser`. This should integrate cleanly with
`ObjectParser`. It doesn't drop support for `Aggregator.Parser` or
change the plugin intrface at all so it *should* be safe to backport to
7.x. And we can remove `Aggregator.Parser` in a follow up which is only
targeted to 8.0.
2020-01-14 16:50:52 -05:00
Nik Everett 9a3d4db840
Begin moving date_histogram to offset rounding (backport of #50873) (#50978)
We added a new rounding in #50609 that handles offsets to the start and
end of the rounding so that we could support `offset` in the `composite`
aggregation. This starts moving `date_histogram` to that new offset.
2020-01-14 16:50:27 -05:00
Jason Tedor d5623c8f09
Report progress of multiple plugin installs (#51001)
When installing multiple plugins at once, this commit changes the
behavior to report installed plugins as we go. In the case of failure,
we emit a message that we are rolling back any plugins that were
installed successfully, and also that they were successfully rolled
back. In the case a plugin is not successfully rolled back, we report
this clearly too, alerting the user that there might still be state on
disk they would have to clean up.
2020-01-14 16:24:41 -05:00
Christoph Büscher 2f13751bad
Deprecate and remove camel-case nGram and edgeNGram tokenizers (#50862) (#50991)
We deprecated and removed the camel-case versions of the nGram and edgeNGram
filters a while ago and we should do the same with the nGram and edgeNGram tokenizers.
This PR deprecates the use of these names in favour of ngram and edge_ngram in
7. Usage will be disallowed on new indices starting with 8 then.
2020-01-14 21:42:34 +01:00
lcawl 6848dee84b [DOCS] Fixes typo in keystore command 2020-01-14 11:57:02 -08:00
Tal Levy 9ee2e11181
[7.x] Adds support for geo-bounds filtering in geogrid aggregations (#50996)
* Adds support for geo-bounds filtering in geogrid aggregations (#50002)

It is fairly common to filter the geo point candidates in
geohash_grid and geotile_grid aggregations according to some
viewable bounding box. This change introduces the option of
specifying this filter directly in the tiling aggregation.

This is even more relevant to `geo_shape` where the bounds will restrict
the shape to be within the bounds

this optional `bounds` parameter is parsed in an equivalent fashion to
the bounds specified in the geo_bounding_box query.
2020-01-14 11:18:46 -08:00
Mark Vieira 2de2a3634e
Add matrix job params as build scan tag
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-01-14 09:54:10 -08:00
Benjamin Trent 72c270946f
[ML][Inference] Adding classification_weights to ensemble models (#50874) (#50994)
* [ML][Inference] Adding classification_weights to ensemble models

classification_weights are a way to allow models to
prefer specific classification results over others
this might be advantageous if classification value
probabilities are a known quantity and can improve
model error rates.
2020-01-14 12:40:25 -05:00
Tom Veasey de5713fa4b
[ML] Disable invalid assertion (#50988)
Backport #50986.
2020-01-14 17:35:00 +00:00
Jason Tedor ca9ca68cbe
Allow installing multiple plugins as a transaction (#50924)
This commit allows the plugin installer to install multiple plugins in a
single invocation. The installation will be treated as a transaction, so
that all of the plugins are install successfully, or none of the plugins
are installed.
2020-01-14 12:20:54 -05:00
Armin Braun 16c07472e5
Track Snapshot Version in RepositoryData (#50930) (#50989)
* Track Snapshot Version in RepositoryData (#50930)

Add tracking of snapshot versions to RepositoryData to make BwC logic more efficient.
Follow up to #50853
2020-01-14 18:15:07 +01:00
David Kyle 7f309a18f1
[7.x][ML] Explicitly require a OriginSettingClient in ML results iterators (#50981)
In classes where the client is used directly rather than through a call to 
executeAsyncWithOrigin explicitly require the client to be OriginSettingClient 
rather than using the Client interface. 

Also remove calls to deprecated ClientHelper.clientWithOrigin() method.
2020-01-14 17:14:39 +00:00
Lisa Cawley a5a8b60d78 [DOCS] Fix realm chains example (#50568) 2020-01-14 09:01:45 -08:00
Tim Brooks 6e7478b846
Allow proxy mode server name to be configured (#50951)
Currently, proxy mode allows a remote cluster connection to be setup by
expecting all open connections to be routed through an intermediate
proxy. The proxy must use some logic to ensure that the connections end
up on the correct remote cluster. One mechanism provided is that the
default distribution TLS implementations will forward the host component
of the configured address to the remote connection using the SNI
extension. This is limiting as it requires that the proxy be configured
in a way that always uses a valid hostname as the proxy address.

Instead, this commit adds an additional setting to allow the server_name
to be configured independently. This allows the proxy address to be
specified as a IP literal, but the server_name specified as an arbitrary
string which still must be a valid hostname. It also decouples the
server_name from the requirement of being a DNS resolvable domain.
2020-01-14 10:57:44 -06:00
Armin Braun 1fe2d76a91
Fix S3 3rd Party Tests (#50983)
Only load fixtures plugin in snapshot-tool tests if we're actually going to use a fixture
because otherwise configuration fails.

Closes #50971
2020-01-14 17:46:47 +01:00
Armin Braun 6e8ea7aaa2
Work around JVM Bug in LongGCDisruptionTests (#50731) (#50974)
There is a JVM bug causing `Thread#suspend` calls to randomly take
multiple seconds breaking these tests that call the method numerous times
in a loop. Increasing the timeout would will not work since we may call
`suspend` tens if not hundreds of times and even a small number of them
experiencing the blocking will lead to multiple minutes of waiting.

This PR detects the specific issue by timing the `Thread#suspend` calls and
skips the remainder of the test if it timed out because of the JVM bug.

Closes #50047
2020-01-14 17:13:21 +01:00
Tim Brooks d8510be3d9
Revert "Send cluster name and discovery node in handshake (#48916)" (#50944)
This reverts commit 0645ee88e2.
2020-01-14 09:53:13 -06:00
Alan Woodward 4974f56b25 Fix analysis BWC tests - warnings now emitted on index creation 2020-01-14 14:48:40 +00:00
Dimitris Athanasiou 1d8cb3c741
[7.x][ML] Add num_top_feature_importance_values param to regression and classi… (#50914) (#50976)
Adds a new parameter to regression and classification that enables computation
of importance for the top most important features. The computation of the importance
is based on SHAP (SHapley Additive exPlanations) method.

Backport of #50914
2020-01-14 16:46:09 +02:00
Hendrik Muhs 0178c7c5d0
[7.x][Transform] correctly retrieve checkpoints from remote indices (#50903) (#50969)
uses remote client(s) to correctly retrieve index checkpoints from remote clusters
2020-01-14 15:09:14 +01:00
Yannick Welsch f1c5031766 Fix queuing in AsyncLucenePersistedState (#50958)
The logic in AsyncLucenePersistedState was flawed, unexpectedly queuing up two update tasks
in parallel.
2020-01-14 15:04:28 +01:00
Yannick Welsch 91d7b446a0 Warn on slow metadata performance (#50956)
Has the new cluster state storage layer emit warnings in case metadata performance is very
slow.

Relates #48701
2020-01-14 15:04:28 +01:00
Alan Woodward 8c16725a0d Check for deprecations when analyzers are built (#50908)
Generally speaking, deprecated analysis components in elasticsearch will issue deprecation
warnings when they are first used. However, this means that no warnings are emitted when
indexes are created with deprecated components, and users have to actually index a document
to see warnings. This makes it much harder to see these warnings and act on them at
appropriate times.

This is worse in the case where components throw exceptions on upgrade. In this case, users
will not be aware of a problem until a document is indexed, instead of at index creation time.

This commit adds a new check that pushes an empty string through all user-defined analyzers
and normalizers when an IndexAnalyzers object is built for each index; deprecation warnings
and exceptions are now emitted when indexes are created or opened.

Fixes #42349
2020-01-14 13:52:02 +00:00
Przemysław Witek 9c6ffdc2be
[7.x] Handle nested and aliased fields correctly when copying mapping. (#50918) (#50968) 2020-01-14 14:43:39 +01:00
James Rodewig f028ab08d1 [DOCS] Use `s` parameter in cat API overview example (#50616)
Updates a snippet to use the `s` query string parameter rather than
piping the output to a separate `sort` command.

This ensures the snippet is tested and available in clients other than
curl (Kibana console, etc.).

Issue was originally raised by @hackaholic in #40926.
2020-01-14 08:22:07 -05:00
David Kyle 69a3626ee1 Mute SnapshotLifecycleRestIT testFullPolicySnapshot
Relates to #50358
2020-01-14 13:46:37 +01:00
Florian Kelbert 277798606b
[DOCS] Correctly read total hits inside watcher config
Relates to #50611 and #50612
2020-01-14 12:58:52 +01:00
Daniel Mitterdorfer 263083b882
Mute HttpCertificateCommandTests.testTextFileSubstitutions (#50965) (#50966)
Relates #50964
2020-01-14 12:40:34 +01:00