Commit Graph

53404 Commits

Author SHA1 Message Date
Dimitris Athanasiou 07ab0beea0
[7.x][ML] Improve handling of exception while starting DFA process (#61838) (#61847)
While starting the data frame analytics process it is possible
to get an exception before the process crash handler is in place.
In addition, right after starting the process, we check the process
is alive to ensure we capture a failed process. However, those exceptions
are unhandled.

This commit catches any exception thrown while starting the process
and sets the task to failed with the root cause error message.

I have also taken the chance to remove some unused parameters
in `NativeAnalyticsProcessFactory`.

Relates #61704

Backport of #61838
2020-09-02 16:32:45 +03:00
Costin Leau e6dc8054a5 EQL: Introduce filter pipe (#61805)
Allow filtering through a pipe, across events and sequences.
Filter pipes are pushed down to base queries.
For now filtering after limit (head/tail) is forbidden as the
semantics are still up for debate.

Fix #59763

(cherry picked from commit 80569a388b76cecb5f55037fe989c8b6f140761b)
2020-09-02 15:48:51 +03:00
Jim Ferenczi a0e4331c49 Cleanup usages of QueryPhaseResultConsumer (#61713)
This commit generalizes how QueryPhaseResultConsumer is initialized.
The query phase always uses this consumer so it doesn't need to be hidden behind
an abstract class.
2020-09-02 14:41:02 +02:00
Rory Hunter a8bbdd937e Fix deprecation logging config in Docker (#61840)
PR #61474 reworked deprecation logging to rely more heavily on log4j. Unfortunately,
the changes required to log4j's configuration were not applied to the version we ship
with the Docker image.
2020-09-02 12:51:05 +01:00
David Roberts 89599ba0a3
[ML] Update ML mappings upgrade test and extend to config index (#61830)
The ML mappings upgrade test had become useless as it was
checking a field that has been the same since 6.5. This
commit switches to a field that was changed in 7.9.

Additionally, the test only used to check the results index
mappings.  This commit also adds checking for the config
index.

Backport of #61340
2020-09-02 12:23:59 +01:00
David Kyle d268540f20
[ML] Check and install the latest template in the DFA executor (#61589) (#61842)
During a rolling upgrade it is possible that a worker node will be upgraded before
the master in which case the DFA templates will not have been installed.
Before a DFA task starts check that the latest template is installed and install it if necessary.
2020-09-02 12:16:29 +01:00
Alan Woodward d59343b4ba
Allow [null] values in [null_value] (#61798) (#61807)
Several field mappers have a null_value parameter, that allows you to specify a placeholder
value to insert into a document if the incoming value for that field is null. The default value
for this is always null, meaning "add no placeholder". However, we explicitly bar users from
setting this parameter directly to null (done in #7978, in order to fix an NPE).

This exclusion means that if a mapper is serialized with include_defaults, then we either need
to special-case null_value to ensure that it is not output when it holds the default value, or
we find that the resulting serialized form cannot be used to create a mapping. This stops us
doing some useful generic testing of mappers.

This commit permits null as a parameter value for null_value, and changes the tests to check
that it is a) permissible and b) applied without throwing errors. As part of the testing changes,
a new base class MapperServiceTestCase is refactored from MapperTestCase, holding
the various helper methods related to building mappings but not the single-mapper specific
abstract methods.

Closes #58823
2020-09-02 10:42:19 +01:00
Andrei Dan 1d83aca520
Drop assertion that rest client header warnings conform to RFC 7234 (#61365) (#61799)
This removes the assertion that the header warnings we parse in the
rest client reponses conform to RFC 7234	because we are not in full control
of the warnings that could be present in the responses (ie. proxies might
emit warnings that don't comply).

We still maintain this assertion on the ES side (see `HeaderWarning#addWarning`)
for the warnings we emit.

(cherry picked from commit 1259a46cbe84d32e85cd1a7455012d177b809702)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-09-02 09:57:51 +01:00
Julie Tibshirani 9ee5f20ebc Link to the keyword family page from the field types docs. (#61819)
We now link to the top-level keyword type family page instead of its individual
subsections. This better fits the page format, where each type name is a link.
2020-09-01 16:23:49 -07:00
James Rodewig 129b233156
[DOCS] Document dynamic cluster settings (#61760) (#61817)
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-09-01 16:04:23 -04:00
Igor Motov 48e53cca94
Fix wrong NaN comparison (#61795) (#61811)
Fixes wrong NaN comparison in error message generator in GeoPolygonDecomposer and PolygonBuilder.

Supersedes #48207

Co-authored-by: Pedro Luiz Cabral Salomon Prado <pedroprado010@users.noreply.github.com>
2020-09-01 15:50:38 -04:00
James Rodewig 8613bde780
[DOCS] Combine keyword family docs (#61662) (#61813) 2020-09-01 15:32:56 -04:00
Rene Groeschke 41613c16e9
Rework output normalization in build-tools func tests (#61706) (#61721) 2020-09-01 21:00:27 +02:00
Tim Brooks e573fa9abc
Add data.path fast path for FilePermission (#61302)
The recursive data.path FilePermission check is an extremely hot
codepath in Elasticsearch. Unfortunately the FilePermission check in
Java is extremely allocation heavy. As it iterates through different
file permissions, it allocates byte arrays for each Path component that
must be compared. This PR improves the situation by adding the recursive
data.path FilePermission it its own PermissionsCollection object which
is checked first.
2020-09-01 12:03:22 -06:00
James Rodewig fd976e668c
[DOCS] EQL: Clarify until keyword docs (#61794) (#61808) 2020-09-01 13:56:51 -04:00
Rene Groeschke 77ecea6e3b
Fix jvm options expansion on windows (#61743) (#61789)
* Fix jvm options expansion on windows
* Avoid long paths on windows
* Relativize paths in jvmoptions against working dir
2020-09-01 19:49:19 +02:00
Armin Braun 28710c985d
Dry up Settings from Map Construction (#61778) (#61803)
We used the same hack all over the place. At least drying it up to a single place.

Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>
2020-09-01 19:46:10 +02:00
Nik Everett f8158bdb2d Skip failing test
Tracked by https://github.com/elastic/elasticsearch/issues/61561
2020-09-01 13:44:31 -04:00
Tanguy Leroux 6e944d9e21
Throws IndexNotFoundException in TransportGetAction for unknown System indices (#61785) (#61791)
The change #57936 introduced a dedicated thread pool for reads in system indices. 
It also introduced a potential NPE in the case the index to read in not yet present in 
the cluster state. This commit fixes that bug by using the getIndexSafe() instead of 
just index() method when retrieving the index's metadata so that an INFE is thrown 
if the index does not exist.
2020-09-01 17:41:57 +02:00
Dimitris Athanasiou 2547cfbe54
[7.x][ML] Persist progress when setting DFA task to failed (#61782) (#61792)
When an error occurs and we set the task to failed via
the `DataFrameAnalyticsTask.setFailed` method we do not
persist progress. If the job is later restarted, this means
we do not correctly restore from where we can but instead
we start the job from scratch and have to redo the reindexing
phase.

This commit solves this bug by persisting the progress before
setting the task to failed.

Backport of #61782
2020-09-01 18:33:07 +03:00
Dan Hermann d52ee17054
Adjust BWC after backport of #60818 2020-09-01 08:30:32 -05:00
Dan Hermann 88a448f1cd
Fix wrong result when executing bulk requests with and without pipeline (#60818) (#61777) 2020-09-01 07:05:25 -05:00
Armin Braun 3fd25bfa87
Fix Concurrent Snapshot Create+Delete + Delete Index (#61770) (#61773)
We had a bug here were we put a `null` value into the shard
assignment mapping when reassigning work after a snapshot delete
had gone through. This only affects partial snaphots but essentially
dead-locks the snapshot process.

Closes #61762
2020-09-01 13:20:25 +02:00
Rory Hunter 50119bb08f Make Docker test robust when removing a temp dir (#61639)
Closes #61614. Closes #61553.

In a particular Docker test, ensure is shut down before removing the
data directory. Also add the force flag to `rm`.
2020-09-01 12:03:41 +01:00
Tanguy Leroux d94d6b5b70 Also account for state not recovered in BlobStoreCacheService
Following #61726 after a test failure
2020-09-01 12:10:04 +02:00
Ioannis Kakavas ced2c140fe
Unmute TokenAuthIntegTests test (#61715)
@ywangd made an awesome analysis on why this test is failing, over
at https://github.com/elastic/elasticsearch/issues/55816#issuecomment-620913282

This change makes it so that we use the same client to perform a
refresh of a token, as we use to subsequently attempt to authenticate
with the refreshed token. This ensures the tests are failing and is
a good approximation of how we expect the same client doing the
refresh, to also perform the subsequent authentication in real life
uses.

The errors we were seeing from users have disappeared after #55114
so we deem our behavior safe.
2020-09-01 13:06:11 +03:00
Tanguy Leroux 787dfda4c1
Prevent snapshots to be mounted as system indices (#61517) (#61727)
System indices can be snapshotted and are therefore potential candidates 
to be mounted as searchable snapshot indices. As of today nothing 
prevents a snapshot to be mounted under an index name starting with . 
and this can lead to conflicting situations because searchable snapshot 
indices are read-only and Elasticsearch expects some system indices 
to be writable; because searchable snapshot indices will soon use an 
internal system index (#60522) to speed up recoveries and we should
prevent the system index to be itself a searchable snapshot index 
(leading to some deadlock situation for recovery).

This commit introduces a changes to prevent snapshots to be mounted 
as a system index.
2020-09-01 11:13:28 +02:00
Tanguy Leroux 92eb6e7844
Remove cluster state listener in BlobStoreCacheService (#61726) (#61769)
BlobStoreCacheService implements ClusterStateListener in order to
 maintain a ready flag that can be used to know when the snapshot 
blob cache should be queries or not.

Now the getAsync() method correctly handles the various exceptions 
that can be thrown when the .snapshot-blob-cache index is not 
available(in isExpectedCacheGetException()) and logs as DEBUG 
we can safely remove the ready flag.
2020-09-01 11:12:52 +02:00
Boice Huang 8fdd3d158b Remove redundant symbol in msearch tests (#61353) 2020-09-01 10:58:22 +02:00
Lisa Cawley d5f1223343 [DOCS] Clarify enabling monitoring features (#61758) 2020-08-31 13:16:23 -07:00
Lisa Cawley 0e4303433b
[DOCS] Document static monitoring settings (#61748) (#61756) 2020-08-31 13:03:17 -07:00
Nik Everett fb84c1f73e
Calculate precise cardinality upper bounds (#61529) (#61754)
This reworks `CardinalityUpperBound` to support precise estimates while
maintaining most of the public API. This will allow us to make more
informed choices about the data structures that we use in aggregations.
None of those interesting choices come as part of this change, but they
are more possible with it.
2020-08-31 15:10:02 -04:00
James Rodewig f39a9bbe19
[DOCS] Document static ILM settings (#61745) (#61749) 2020-08-31 14:02:10 -04:00
Benjamin Trent 7dabaad7d9
[ML] refactor ml job node selection into its own class (#61521) (#61747)
This is a minor refactor where the job node load logic (node availability, etc.) is refactored into its own class.

This will allow future things (i.e. autoscaling decisions) to use the same node load detection class.
backport of #61521
2020-08-31 14:00:23 -04:00
Benjamin Trent 8b33d8813a
[ML] binary classification per-class feature importance for model inference (#61597) (#61746)
This commit addresses two issues:

- per class feature importance is now written out for binary classification (logistic regression)
- The `class_name` in per class feature importance now matches what is written in the `top_classes` array.

backport of https://github.com/elastic/elasticsearch/pull/61597
2020-08-31 13:57:00 -04:00
Dan Hermann 2858e1efc4
Document new stats in _cat/nodes (#60445) (#61742) 2020-08-31 12:40:21 -05:00
James Rodewig 130a7cea78
[DOCS] Add placeholder for 7.9.1 release notes (#61652) 2020-08-31 12:25:59 -04:00
Adam Locke 5723b928d7
Remove Outdated Snapshot Docs (#61684) (#61728)
Removing some now outdated statements that refer to a time
when snapshot operations could not run concurrently.

Closes #61680
2020-08-31 12:04:27 -04:00
Rene Groeschke 4aaeb1e4e3
Address review feedback
- avoid ! when negating boolean expressions
2020-08-31 17:55:16 +02:00
James Rodewig caa1a9024c
[DOCS] Add jump list to breaking changes page (#61598) 2020-08-31 11:25:57 -04:00
James Rodewig f47363074e
[DOCS] Remove placeholder for 7.8.2 release notes (#61653) 2020-08-31 11:22:14 -04:00
James Rodewig 1f24fc03a0
[DOCS] Document dynamic cluster-lvl shard alloc settings (#61338) (#61735) 2020-08-31 11:19:57 -04:00
James Rodewig 8228cdad67
[DOCS] Fix typo in range query docs (#61722) (#61731) 2020-08-31 11:03:11 -04:00
James Rodewig f94999bb2f
[DOCS] Add force merge to hot phase list (#61725) (#61729) 2020-08-31 11:02:41 -04:00
James Rodewig ccbe2938c8
[DOCS] Fix Gsub processor snippet (#61720) (#61723) 2020-08-31 10:43:26 -04:00
Jason Tedor 43cb7c48bd
Adjust Lucene versions for 7.9.1
This commit adjusts the Lucene versions for 7.9.1 after the backporting
of upgrading the 7.9 branch to Lucene 8.6.2.
2020-08-31 10:30:39 -04:00
Mayya Sharipova fe9c66096c Small refactoring of AsyncExecutionId (#61640)
- don't do encoding of asynchExecutionId if it is already provided in
the encoded form
- create a new instance of AsyncExecutionId after checks for
correctness are done
2020-08-31 10:24:36 -04:00
James Rodewig 054a64d66f
[DOCS] Fix old NodeSelector field in Low Level REST Client (#61551) (#61718)
Co-authored-by: Manabu Matsuzaki <matsumana@users.noreply.github.com>
2020-08-31 10:07:58 -04:00
Nhat Nguyen e37ce561c7 Set timeout of auto put-follow request to unbounded (#61679)
If the master node of the follower cluster is busy, then the 
auto-follower will fail to initialize the following process. This also
occurs when an auto-follow pattern matches multiple indices. We should
set the timeout of put-follow requests issued by the auto-follower to
unbounded to avoid this problem.

Closes #56891
2020-08-31 09:58:19 -04:00
Jason Tedor 64cd229b35
Upgrade to Lucene 8.6.2 (#61688)
This commit upgrades the Lucene dependencies to 8.6.2.
2020-08-31 09:54:07 -04:00