Commit Graph

49922 Commits

Author SHA1 Message Date
Benjamin Trent e372854d43
[ML][Inference] Fix model pagination with models as resources () ()
This adds logic to handle paging problems when the ID pattern + tags reference models stored as resources. 

Most of the complexity comes from the issue where a model stored as a resource could be at the start, or the end of a page or when we are on the last page.
2020-01-31 07:52:19 -05:00
István Zoltán Szabó dfc9f2330c [DOCS] Adds PUT inference API docs ()
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-31 13:13:34 +01:00
István Zoltán Szabó 9600ab4f57 [DOCS] Adds recommendation on dedicated master-eligible nodes ()
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-01-31 12:56:58 +01:00
Dominic Page d7e1215e42
Backport of to 7.x ()
* Refactor GeoShape tests to GeoShape and GeoPoint ()

Backport to 7.x
2020-01-31 11:55:10 +01:00
Adrien Grand 915a931e93
Bucket aggregation circuit breaker optimization. () ()
Co-authored-by: Howard <danielhuang@tencent.com>
2020-01-31 11:30:51 +01:00
Sven Schliesing dc99acee66 [Docs] Fix typo in node-tool.asciidoc () 2020-01-31 10:36:21 +01:00
Ioannis Kakavas 1dc965f03f
Mute ec2 test in FIPS 140 mode () ()
as it needs an extra permission, until we can figure out how to
grant the permission in FIPS 140 mode too.

See: https://github.com/elastic/elasticsearch/issues/51685
2020-01-31 09:35:20 +02:00
Henning Andersen 282ae8fd8c Increase log level for failing AbstractDisruptionIT tests ()
Increase log level for two failing tests to include trace logging for
PersistedClusterStateService.

Relates 
2020-01-31 08:30:09 +01:00
Ioannis Kakavas 46ffc57abe Fix test compilation error 2020-01-31 09:23:02 +02:00
David Turner 72ae0ca73f Log exceptions in TcpTransport at DEBUG level ()
When running Elasticsearch on a flaky network, we may see nodes leaving the
cluster with reason `disconnected`. It may be useful to the cluster
administrator to see the full exception that caused the disconnection, but this
is only available with `TRACE` level logging which commingles the details of
the problem with other messages that are not useful to end users.

This commit promotes logging of exceptions in `TcpTransport` from `TRACE` to
`DEBUG` to separate them from the truly `TRACE`-level messages.
2020-01-31 01:36:58 +00:00
Yang Wang 77b00fc0c0
Add warnings for invalid realm order config () ()
The changes are to help users prepare for migration to next major
release (v8.0.0) regarding to the break change of realm order config.

Warnings are added for when:
* A realm does not have an order config
* Multiple realms have the same order config

The warning messages are added to both deprecation API and loggings.
The main reasons for doing this are: 1) there is currently no automatic relay
between the two; 2) deprecation API is under basic and we need logging
for OSS.
2020-01-31 12:32:37 +11:00
Mark Vieira 11e86b13b4
Disable build cache for packaging tests () 2020-01-30 17:23:49 -08:00
Mark Vieira 6c879ac63c
Test cluster throttle should never be less than 1 () 2020-01-30 17:07:29 -08:00
Mark Vieira 6b18cab826
Avoid NPE in Gradle build when max-workers is set to 1 () 2020-01-30 17:07:16 -08:00
Lisa Cawley 1a40ebfa67 [DOCS] Adds missing testenv attribute () 2020-01-30 16:15:17 -08:00
Gordon Brown 10c8179351
Use exclusions list instead of fake system indices ()
This commit switches the strategy for managing dot-prefixed indices that
should be hidden indices from using "fake" system indices to an explicit
exclusions list that must be updated when those indices are converted to
hidden indices.
2020-01-30 16:31:27 -07:00
Mark Vieira 8d2370bf00
Always use bundled JDK for external cluster nodes when BWC testing () () 2020-01-30 14:35:43 -08:00
Bogdan Pintea f1173aaa48
SQL: Add optimisations for not-equalities () ()
* Optimize not-equalities in con-/disjunctions

This commit adds optimisations of not-equalities in conjunctions and
disjunctions:
* for conjunctions, the not-equality can be optimized away when applied
together with a range or inequality, in case the not-equality point
falls outside the domain of the later condition; if its on the boarder,
it will modify the bound, to simply exclude the equality, if present;
otherwise no optimisation can be applied;
* for disjunctions, the not-equals could filter away the ranges and
inequalities, unless these include an equality on the bound, in which
case the entire condition becomes always true, but this would influence
the score() function, so it's been omitted;

* fix aggregations of inequalities in ranges

This commit fixes the loop that aggregates inequalities into ranges:
- it won't advance the outer loop index in case of a merge, since the
current element is removed;
- it will break the inner loop, since comparision against the element
selected in the outer loop can't continue, as it had been removed.



(cherry picked from commit 789724ac2cc726de603849b4eeb8194da7528bcc)
2020-01-30 23:29:39 +01:00
Lee Hinman b9faa0733d
[7.x] Rename ILM history index enablement setting () ()
* Rename ILM history index enablement setting

The previous setting was `index.lifecycle.history_index_enabled`, this commit changes it to
`indices.lifecycle.history_index_enabled` to indicate this is not an index-level setting (it's node
level).
2020-01-30 15:27:44 -07:00
Benjamin Trent 1380dd439a
[7.x] [ML][Inference] Fix weighted mode definition () ()
* [ML][Inference] Fix weighted mode definition ()

Weighted mode inaccurately assumed that the "max value" of the input values would be the maximum class value. This does not make sense. 

Weighted Mode should know how many classes there are. Hence the new parameter `num_classes`. This indicates what the maximum class value to be expected.
2020-01-30 15:33:25 -05:00
Nhat Nguyen 69ef9b05cd Increase shard inactive time to 1h in upgrade tests ()
testRecovery relies on the fact that shards are not flushed on inactive. 
Our CI recently was too slow. It took more than 20 minutes to complete
the full cluster restart suite. This slowness caused some shards of
testRecovery were flushed on inactive.

This commit increases the inactive time to 1h to reduce this noise.

Closes 
2020-01-30 14:48:56 -05:00
Nhat Nguyen 1cba5d7c4b Force flush in FrozenEngine#testSearchers ()
We need to force flush to make the last commit safe; otherwise, we might 
fail to open FrozenEngine. Note that we force flush before closing a
shard.

Closes 
2020-01-30 14:48:45 -05:00
Ioannis Kakavas 72c5bc9630
Disable diagnose trust mananer only when needed ()
Do not try to set xpack.ssl.security.diagnose.trust
when it is not registered
2020-01-30 21:35:03 +02:00
Mark Vieira bb963b996c
Support downloading JDKs with legacy version format ()
(cherry picked from commit b70f925ccb735dc84d59598de06df6bf35bd4bdc)
2020-01-30 10:56:27 -08:00
James Rodewig 36b2663e98 [DOCS] Add attribute for Lucene analysis links ()
Adds a `lucene-analysis-docs` attribute for the Lucene `/analysis/`
javadocs directory. This should prevent typos and keep the docs DRY.
2020-01-30 11:24:01 -05:00
Benjamin Trent 2a2a0941af
[ML][Inference] stream inflate to parser + throw when byte limit is reached () ()
Three fixes for when the `compressed_definition` is utilized on PUT

* Update the inflate byte limit to be the minimum of 10% the max heap, or 1GB (what it was previously)
* Stream data directly to the JSON parser, so if it is invalid, we don't have to inflate the whole stream to find out
* Throw when the maximum bytes are reach indicating that is why the request was rejected
2020-01-30 10:16:14 -05:00
Armin Braun 9c7a63214c
Fix InternalEngineTests.testSeqNoAndCheckpoints () ()
* Fix InternalEngineTests.testSeqNoAndCheckpoints

If we force flush while possibly triggering a merge the local checkpoint may change
from the expectation from the loop that just increments on every operation.

Closes 
2020-01-30 15:42:48 +01:00
James Rodewig 4fcf5a9de4 [DOCS] Rewrite analysis intro ()
* [DOCS] Rewrite analysis intro. Move index/search analysis content.

* Rewrites 'Text analysis' page intro as high-level definition.
  Adds guidance on when users should configure text analysis
* Rewrites and splits index/search analysis content:
  * Conceptual content -> 'Index and search analysis' under 'Concepts'
  * Task-based content -> 'Specify an analyzer' under 'Configure...'
* Adds detailed examples for when to use the same index/search analyzer
  and when not.
* Adds new example snippets for specifying search analyzers

* clarifications

* Add toc. Decrement headings.

* Reword 'When to configure' section

* Remove sentence from tip
2020-01-30 09:32:16 -05:00
Marios Trivyzas f373020349 SQL: Fix ORDER BY YEAR() function ()
Previously, if YEAR() was used as and ORDER BY argument without being
wrapped with another scalar (e.g. YEAR(birth_date) + 10), no script
ordering was used but instead the underlying field (e.g. birth_date)
was used instead as a performance optimisation. This works correctly if
YEAR() is the only ORDER BY arg but if further args are used as tie
breakers for the ordering wrong results are produced. This is because
2 rows with the different birth_date but on the same year are not tied
as the underlying ordering is on birth_date and not on the
YEAR(birth_date), and the following ORDER BY args are ignored.

Remove this optimisation for YEAR() to avoid incorrect results in
such cases.

As a consequence another bug is revealed: scalar functions on top
of nested fields produce scripted sorting/filtering which is not yet
supported. In such cases no error was thrown but instead all values for
such nested fields were null and were passed to the script implementing
the sorting/filtering, producing incorrect results.

Detect such cases and throw a validation exception.

Fixes: 
(cherry picked from commit f41efd6753dc3650a7eabb3e07b02b3b32c5704c)
2020-01-30 15:29:36 +01:00
Nhat Nguyen f0fad5b622
Deprecate translog retention settings () ()
This change deprecates the translog retention settings as they are
effectively ignored since 7.4.

Relates 
Relates 
2020-01-30 09:03:10 -05:00
Armin Braun 1064009e9d
Allow Parallel Snapshot Restore And Delete () ()
There is no reason not to allow deletes in parallel to restores
if they're dealing with different snapshots.
A delete will not remove any files related to the snapshot that
is being restored if it is different from the deleted snapshot
because those files will still be referenced by the restoring
snapshot.
Loading RepositoryData concurrently to modifying it is concurrency
safe nowadays as well since the repo generation is tracked in the
cluster state.

Closes 
2020-01-30 14:27:05 +01:00
Henning Andersen ca8601373a [DOCS] Task management API experimental status issue ()
Add issue reference to documentation.

Relates 
2020-01-30 14:15:47 +01:00
Martijn van Groningen f7e2082378
Backport: unmute rolling upgrade watcher tests and ()
set watcher logger to debug level.

These tests haven't run in such a long time,
we first need to get a better picture how/if
these tests fail today.

Backport of 
See 
2020-01-30 14:01:30 +01:00
Marios Trivyzas 285a167c34 SQL: Verify Full-Text Search functions not allowed in SELECT ()
Add a verification that full-text search functions `MATCH()` and `QUERY()`
are not allowed in the SELECT clause, so that a nice error message is
returned to the user early instead of an "ugly" exception.

Fixes: 
2020-01-30 13:14:38 +01:00
Henning Andersen 2e8a2c4baf
Fix ActionListener.map exception handling () ()
ActionListener.map would call listener.onFailure for exceptions from
listener.onResponse, but this means we could double trigger some
listeners which is generally unexpected. Instead, we should assume that
a listener's onResponse (and onFailure) implementation is responsible
for its own exception handling.
2020-01-30 12:54:55 +01:00
Hendrik Muhs b5106aa59d dump audit index to logs for better debugging ()
The audit index is re-created for every testrun and therefore potential useful debug information
gets lost. This change reads out the audit index and logs the results, which makes them available
for debugging CI issues.

relates 
2020-01-30 11:14:56 +01:00
Albert Zaharovits f25b6cc2eb
Add new 'maintenance' index privilege
This commit creates a new index privilege named `maintenance`.
The privilege grants the following actions: `refresh`, `flush` (also synced-`flush`),
and `force-merge`. Previously the actions were only under the `manage` privilege
which in some situations was too permissive.

Co-authored-by: Amir H Movahed <arhd83@gmail.com>
2020-01-30 11:59:11 +02:00
Peter Dyson b5a2ee5be2 [DOCS] Fix minor typo affecting formatting () 2020-01-29 23:44:09 -08:00
Henning Andersen 149b68d850 [ML] Fix possible race condition starting datafeed ()
Datafeeds being closed while starting could result in and NPE. This was
handled as any other failure, masking out the NPE. However, this
conflicts with the changes in .

Related to  and 
2020-01-30 08:23:45 +01:00
Lisa Cawley 28f2f3dd02 [DOCS] Minor fixes in transform documentation () 2020-01-29 16:58:18 -08:00
Ryan Ernst cf5a2269a5 Fix stderr to also be captured by log4j ()
In  we redirected stdout and stderr to log4j, to capture jdk
and external library messages. However, a typo in the method name used
to redirect the stream in java means stdout is currently being
duplicated twice, and stderr not captured. This commit corrects that
mistake. Unfortunately this is at a level that cannot really be tested,
thus we are still missing tests for this behavior.
2020-01-29 16:37:56 -08:00
Mark Vieira 4f214d20ab
Limit max concurrency of test cluster nodes to a function of max workers ()
(cherry picked from commit 9a0238c7166e70e467ca61c1353157979fd1598b)
2020-01-29 14:51:08 -08:00
Julie Tibshirani 40f4f2d267 Avoid processing search profile results twice. ()
Just a small clean-up, not motivated by performance.
2020-01-29 14:37:39 -08:00
Aleksandr Maus 0d21d9e2c5
EQL: Enable QA/rest integration tests for snapshot builds only () ()
* Related to : [CI] unknown setting [xpack.eql.enabled] in release-tests
2020-01-29 16:38:52 -05:00
Julie Tibshirani 9dcc3ef7e6
Always use one shard in vector REST tests. ()
This PR tries to address the intermittent vector test failures on 7.x by making
sure we create indices with one shard.

The fix is based on this theory as to what's happening:
* On 7.x, the default number of shards is 1, but in REST tests we randomly use
2 in order to cover the multiple shards case. In the failing test run, we use 2
shards and all documents end up on only one shard.
* During a search, the response from the empty shard doesn't produce
deprecation warnings because  we never try to execute the script. If not all
shard responses contain the warning headers, then certain deprecation warnings
can be lost (due to the bug described in ).

Addresses .
Relates to .
2020-01-29 12:24:41 -08:00
Lisa Cawley fdf74f6ae4 [DOCS] Removes beta qualifiers from transform documentation () 2020-01-29 08:41:54 -08:00
Lisa Cawley 3f4156e95a
[DOCS] Adds release highlight for transforms () 2020-01-29 08:35:02 -08:00
Przemysław Witek 683170b007
Increase the number of indexed documents to increase a chance that there are at least 2 training rows. () () 2020-01-29 17:17:19 +01:00
Armin Braun 74e3694234
Optimize GCS Repo Uploads () ()
For small uploads (that can still be up to 5MB!) we needlessly
reading the `InputStream` into a BAOS which entailed allocating
the `byte[]` for the stream contents twice (because to `toByteArray` on the BAOS copies).

Also, for resumeable uploads we were needlessly wrapping the output channel and running each individual write in its own privileged context when we could just wrap the whole upload in a single privileged context.

Relates 
2020-01-29 16:07:30 +01:00
James Rodewig 078e13b1fd [DOCS] Enable EQL on docs integ tests () 2020-01-29 09:05:14 -05:00