Commit Graph

43517 Commits

Author SHA1 Message Date
Dimitris Athanasiou b04b3173db
[ML][TEST] Clean up max_model_memory_limit cluster setting (#37101)
Removes the `xpack.ml.max_model_memory_limit` cluster setting
at the teardown of the `ml_info.yml` tests to ensure the setting
does not trip other tests.
2019-01-03 15:15:31 +02:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg (#36933)
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: #36903
2019-01-03 13:55:09 +02:00
Christoph Büscher 046f86f274
Deprecate use of type in reindex request body (#36823)
Types can be used both in the source and dest section of the body which will
be translated to search and index requests respectively. Adding a deprecation warning
for those cases and removing examples using more than one type in reindex since
support for this is going to be removed.
2019-01-03 10:29:14 +01:00
Christoph Büscher e21054d176
Remove two unused methods in Iterables (#37075)
These helper methods are unused in the rest of the codebase.
2019-01-03 10:28:47 +01:00
Nhat Nguyen b7f6ee72a6 Adjust Lucene version for 6.7
Relates #37088
2019-01-03 04:20:47 -05:00
Jim Ferenczi 78ba1889cf
Replace the TreeMap in the composite aggregation (#36675)
The `composite` aggregation uses a TreeMap to keep track of the best buckets.
This ensures a log(n) time cost to insert new buckets but also to retrieve buckets
that are already present in the map. In order to speed up the retrieval of buckets
this change replaces the TreeMap with a priority queue and a HashMap. The insertion
cost is still log(n) but the retrieval of buckets through the HashMap is now done in constant
time. This optimization can bring significant improvement since each document needs
to check if its associated buckets are already present in the current best buckets.
2019-01-03 09:51:35 +01:00
David Turner a40f0545e6
Make ILM examples larger (#37039)
There are a handful of examples in the ILM documentation that could result in
rolling over indices more quickly than we might normally recommend,
contributing to over-sharding in cases where the examples are copied without
modification. This change makes some numbers bigger to try and avoid this.
2019-01-03 08:28:49 +00:00
Daniel Mitterdorfer 75f3443c62
Rename setting to enable mmap
With this commit we rename `node.store.allow_mmapfs` to
`node.store.allow_mmap`. Previously this setting has controlled whether
`mmapfs` could be used as a store type. With the introduction of
`hybridfs` which also relies on memory-mapping,
`node.store.allow_mmapfs` also applies to `hybridfs` and thus we rename
it in order to convey that it is actually used to allow memory-mapping
but not a specific store type.

Relates #36668
Relates #37070
2019-01-03 07:10:34 +01:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Ioannis Kakavas f0cd19f39b [TEST] Lower deleteInterval in TokenAuthIntegTests
We run subsequent token invalidation requests and we still want to
trigger the deletion of expired tokens so we need to lower the
deleteInterval parameter significantly. Especially now that the
bwc expiration logic is removed and the invalidation process is
much shorter

Resolves #37063
2019-01-02 23:54:02 +02:00
David Findley d4e7660248 Fix weighted_avg parser not found for RestHighLevelClient (#37027)
Add integration test for weighted avg sub aggregation
Add weighted avg parser to DefaultNamedXContents

Fixes #36861
2019-01-02 15:53:21 -06:00
Ke Li 62ece69b92 Remove system property es.enforce_max_shards_per_node (#36968)
The system property es.enforce_max_shards_per_node was not needed any more,
because we always enforce cluster-wide shard limit now.
2019-01-02 14:01:12 -07:00
Alan Woodward 7a0047744d
`query_string` should use indexed prefixes (#36895)
The QueryStringQueryBuilder does not currently delegate to the field mapper's prefixQuery
method, so does not use indexed prefixes. This commit corrects this.

It also fixes a bug where a query a* would not match the word a if indexed prefixes were used with
a minchar setting of 2.
2019-01-02 20:12:24 +00:00
Julie Tibshirani 265fdce312 Fix FullClusterRestartIT#testRollupIDSchemeAfterRestart by adding a missing expected warning. 2019-01-02 10:48:32 -08:00
David Kyle 42bb2bae21
[ML] Order GET job stats response by job id (#36841) 2019-01-02 16:52:20 +00:00
Luca Cavanna 42ea644903
Remove single shard optimization when suggesting shard_size (#37041)
When executing terms aggregations we set the shard_size, meaning the
number of buckets to collect on each shard, to a value that's higher than
the number of requested buckets, to guarantee some basic level of
precision. We have an optimization in place so that we leave shard_size
set to size whenever we are searching against a single shard, in which
case maximum precision is guaranteed by definition.

Such optimization requires us access to the total number of shards that
the search is executing against. In the context of cross-cluster search,
once we will introduce multiple reduction steps (one per cluster) each
cluster will only know the number of local shards, which is problematic
as we should only optimize if we are searching against a single shard in a
single cluster. It could be that we are searching against one shard per cluster
in which case the current code would optimize number of terms causing
a loss of precision.

While discussing how to address the CCS scenario, we decided that we do
not want to introduce further complexity caused by this single shard
optimization, as it benefits only a minority of cases, especially when
the benefits are not so great.

This commit removes the single shard optimization, meaning that we will
always have heuristic enabled on how many number of buckets to collect
on the shards, even when searching against a single shard.

This will cause more buckets to be collected when searching against a single
shard compared to before. If that becomes a problem for some users, they
can work around that by setting the shard_size equal to the size.

Relates to #32125
2019-01-02 17:45:49 +01:00
Josh Soref e0a677b033 Clarify what to run for gradle idea (#37058) 2019-01-02 17:40:11 +01:00
Josh Soref 02aa9f51ee Spelling: correct wrong spellings of similar (#37054) 2019-01-02 17:38:43 +01:00
Josh Soref f6e4b9a014 Spelling: correct wrong spellings of likelihood (#37052) 2019-01-02 17:37:03 +01:00
Josh Soref 5d5d5c26bc Spelling: replace interruptable with interruptible (#37049) 2019-01-02 17:35:53 +01:00
Josh Soref c04d5c5468 Spelling: correct wrong spellings of "input" (#37048) 2019-01-02 17:34:37 +01:00
Josh Soref 1df66d21fe Spelling: replace uknown with unknown (#37056) 2019-01-02 17:33:02 +01:00
Przemyslaw Gomulka 15c4d5b184
Fix line length in org.elasticsearch.get (#37071)
Remove the line suppression for this package and fix offedning lines
relates #34844
2019-01-02 17:22:15 +01:00
Luca Cavanna 450d3014f6 [TEST] Fix testLimitConcurrentShardRequests failure
With #36221 we introduced shards counting to address a rare failure.
This caused a worse problem in this test when replicas were allocated
and shards failures were randomly returned. The latch has to take into
account additional attempts caused by the shard failures, which means
that in order for run to be called, performPhaseOnShard will be called
(numShards + numFailures) times.

To address this, we need to decide upfront which shard is going to fail,
making sure that at least one shards is successful otherwise the whole
request fails.

Closes #37074
2019-01-02 16:46:41 +01:00
Luca Cavanna b10a33054e Mute failing testLimitConcurrentShardRequests
relates to #37074
2019-01-02 16:35:24 +01:00
Christoph Büscher 8e6f68f6b0
Fix Eclipse specific type casting issue (#37072)
The eclipse compiler needs some help in infering the functions return values.
2019-01-02 15:58:20 +01:00
Josh Soref 51cb63f934 Spelling: replace substract with subtract (#37055) 2019-01-02 14:15:21 +01:00
Josh Soref 565a95c382 Spelling: replace respositories with repositories (#37053) 2019-01-02 14:13:54 +01:00
Josh Soref d3e98278c3 Spelling: replace cachable with cacheable (#37047) 2019-01-02 14:10:30 +01:00
Abdullah DURSUN 8a02bacf76 Fix typo in multi-search.asciidoc (#37060) 2019-01-02 10:32:42 +01:00
Daniel Mitterdorfer f0052b1a7a
Add hybridfs store type
With this commit we introduce a new store type `hybridfs` that is a
hybrid between `mmapfs` and `niofs`. This store type chooses different
strategies to read Lucene files based on the read access pattern (random
or linear) in order to optimize performance.

This store type has been available in earlier versions of Elasticsearch
as `default_fs`. We have chosen a different name now in order to convey
the intent of the store type instead of tying it to the fact whether it
is the default choice.

Relates #36668
2019-01-02 10:10:32 +01:00
Luca Cavanna 9e70696628
[TEST] Address rejected execution in SearchAsyncActionTests (#37028)
SearchAsyncActionTests may fail with RejectedExecutionException as InitialSearchPhase may try to execute a runnable after the test has successfully completed, and the corresponding executor was already shut down. The latch was located in getNextPhase that is almost correct, but does not cover for the last finishAndRunNext round that gets executed after onShardResult is invoked.

This commit moves the latch to count the number of shards, and allowing the test to count down later, after finishAndRunNext has been potentially forked. This way nothing else will be executed once the executor is shut down at the end of the tests.

Closes #36221
Closes #33699
2019-01-02 09:47:13 +01:00
Alexis Wilke 35c09adbe1 Replaced the word 'shards' with 'replicas' in an error message. (#36234) (#36275)
Closes #36234
2019-01-01 16:37:15 +01:00
Peter Lyons 794303af73 [Docs] Fix URL path to aliases (#36852)
This API URL contains /aliases/aliases and that seems to be a duplicate.
2018-12-31 17:58:49 +01:00
Andrei Stefan 9c283aed0a
SQL: Move the SQL ODBC client applications pages from the ODBC section into the general ES SQL Client Applications page (#37004) 2018-12-31 14:07:39 +02:00
Andrei Stefan 9a6c61015c
Use the same format when comparing the error messages containing date fields (#37006)
Test fix: make sure the format in which we check the date is the same as the one we used to format the initial JDBC date field.
2018-12-31 13:41:54 +02:00
Luca Cavanna fd7cde88db Mute failing RolloverIT#testRolloverWithDateMath
Relates to #37037
2018-12-31 11:51:50 +01:00
Armin Braun 85be9d6a89
SNAPSHOT: Deterministic ClusterState Tests (#36644)
* Use `DeterministicTaskQueue` infrastructure to test `SnapshotsService`
2018-12-31 11:17:21 +01:00
Luca Cavanna adb957b5aa Mute failing DateMathExpressionResolverTests tests
Relates to #37037
2018-12-31 10:37:10 +01:00
Hendrik Muhs 632c7fbed2
[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698
2018-12-31 08:30:08 +01:00
Luca Cavanna d3f1fe46d3 Increase await timeouts in RemoteClusterServiceTests
Closes #33852
2018-12-28 17:03:40 +01:00
Armin Braun 4ac8fc6906
Force Refresh Listeners when Acquiring all Operation Permits (#36835)
* Fixes the issue reproduced in the added tests:
   * When having open index requests on a shard that are waiting for a refresh, relocating that shard
becomes blocked until that refresh happens (which could be never as in the test scenario).
2018-12-28 16:42:51 +01:00
Luca Cavanna c1beb95aa1 Mute LocalIndexFollowingIT#testRemoveRemoteConnection
Relates to #37014
2018-12-28 16:39:36 +01:00
Luca Cavanna cb6bac3f88
Skip final reduction if SearchRequest holds a cluster alias (#37000)
With #36997 we added the ability to provide a cluster alias with a
SearchRequest.

The next step is to disable the final reduction whenever a cluster alias
is provided with the SearchRequest. A cluster alias will be provided
when executing a cross-cluster search request with alternate execution
mode, where each cluster does its own reduction locally. In order for
the CCS node to be able to later perform an additional reduction of the
results, we need to make sure that all the needed info stays available.
This means that terms aggregations can be reduced but not pruned, and
pipeline aggs should not be executed. The final reduction will happen
later in the CCS coordinating node.

Relates to #36997 & #32125
2018-12-28 14:58:20 +01:00
Armin Braun 34d22f378d TESTS: Mute testSnapshotCanceledOnRemovedShard
* relates #37005
2018-12-28 14:20:45 +01:00
Luca Cavanna 51fe20e0c3
Add support for local cluster alias to SearchRequest (#36997)
With the upcoming cross-cluster search alternate execution mode, the CCS
 node will be able to split a CCS request into multiple search requests,
 one per remote cluster involved. In order to do that, the CCS node has
to be able to signal to each remote cluster that such sub-requests are
part of a CCS request. Each cluster does not know about the other
clusters involved, and does not know either what alias it is given in
the CCS node, hence the CCS coordinating node needs to be able to provide
the alias as part of the search request so that it is used as index prefix
in the returned search hits.

The cluster alias is a notion that's already supported in the search shards
iterator and search shard target, but it is currently used in CCS as both
 index prefix and connection lookup key when fanning out to all the shards.
With CCS alternate execution mode the provided cluster alias needs to be
used only as index prefix, as shards are local to each cluster hence no
cluster alias should be used for connection lookups.

The local cluster alias can be set to the SearchRequest at the transport layer
only, and its constructor/getter methods are package private.

Relates to #32125
2018-12-28 12:43:25 +01:00
Ioannis Kakavas 0cae979dfe
Remove bwc logic for token invalidation (#36893)
- Removes bwc invalidation logic from the TokenService
- Removes bwc serialization for InvalidateTokenResponse objects as
    old nodes in supported mixed clusters during upgrade will be 6.7 and
    thus will know of the new format
- Removes the created field from the TokensInvalidationResult and the
    InvalidateTokenResponse as it is no longer useful in > 7.0
2018-12-28 13:09:42 +02:00
Ioannis Kakavas 44bd7db59e [TEST] Pass a clientSSLContext in LdapTestCase
If we don't explicitly sett the client SSLSocketFactory when
creating an InMemoryDirectoryServer and setting its SSL config, it
will result in using a TrustAllTrustManager(that extends
X509TrustManager) which is not allowed in a FIPS 140 JVM.
Instead, we get the SSLSocketFactory from the existing SSLContext
and pass that to be used.

Resolves #37013
2018-12-28 11:19:58 +02:00
Yannick Welsch 935c2e98b0
Zen2: Turn to follower on follower check when no state accepted yet from new leader (#37003)
Improves on #36449 which did not cover the situation where a node had bumped its term during
the election, and not when receiving the first follower check. This was uncovered while refactoring
NodeJoinTests so that they don't need to access to an internal field of Coordinator anymore (which
can now be made private).
2018-12-28 08:37:04 +01:00
Tim Vernum e16fd4ebd6
Change missing authn message to not mention tokens (#36750)
The phrase "missing authentication token" is historic and is based
around the use of "AuthenticationToken" objects inside the Realm code.

However, now that we have a TokenService and token API, this message
would sometimes lead people in the wrong direction and they would try
and generate a "token" for authentication purposes when they would
typically just need a username:password Basic Auth header.

This change replaces the word "token" with "credentials".
2018-12-28 15:26:46 +11:00