Commit Graph

44755 Commits

Author SHA1 Message Date
Christoph Büscher 9f6c77fad4
Fix FullClusterRestartIT#testSnapshotRestore ()
This test failed on 7.1 when running full cluster restart tests against pre-7.0
clusters (e.g. 6.6 clusters). The fixes the expected type in the
templates after the cluster restart.
2019-02-15 20:12:26 +01:00
Lisa Cawley 339a15bb09 [DOCS] Edits warning in put watch API () 2019-02-15 09:40:12 -08:00
Lisa Cawley d300048cd5
[DOCS] Updates methods for upgrading machine learning () () 2019-02-15 09:29:45 -08:00
Martijn van Groningen 03b67b3ee1
Introduced class reuses follow parameter code between ShardFollowTasks ()
and AutoFollowPattern classes.

The ImmutableFollowParameters is like the already existing FollowParameters,
but all of its fields are final.
2019-02-15 18:26:15 +01:00
Alan Woodward 176013e23c Avoid double term construction in DfsPhase ()
DfsPhase captures terms used for scoring a query in order to build global term statistics across
multiple shards for more accurate scoring. It currently does this by building the query's `Weight`
and calling `extractTerms` on it to collect terms, and then calling `IndexSearcher.termStatistics()`
for each collected term. This duplicates work, however, as the various `Weight` implementations 
will already have collected these statistics at construction time.

This commit replaces this round-about way of collecting stats, instead using a delegating
IndexSearcher that collects the term contexts and statistics when `IndexSearcher.termStatistics()`
is called from the Weight.

It also fixes a bug when using rescorers, where a `QueryRescorer` would calculate distributed term
statistics, but ignore field statistics.  `Rescorer.extractTerms` has been removed, and replaced with
a new method on `RescoreContext` that returns any queries used by the rescore implementation.
The delegating IndexSearcher then collects term contexts and statistics in the same way described
above for each Query.
2019-02-15 16:00:38 +00:00
Hannes Van De Vreken 27cf7e27e7 Fix typo in DateRange docs (yyy → yyyy) () 2019-02-15 10:19:54 -05:00
iverase b19b778cbb [CI] Muting method testFollowIndex in IndexFollowingIT
Relates to 
2019-02-15 16:07:45 +01:00
David Pilato 78541dfb2a
Update Lucene snapshot repo for 7.0.0-beta1 ()
This commit updates the documentation for the Lucene snapshot repo.
2019-02-15 08:57:44 -06:00
Daniel Mitterdorfer fcc7f553f5
Also mmap cfs files for hybridfs () ()
With this commit we add the `.cfs` file extension to the list of file
types that are memory-mapped by hybridfs. `.cfs` files combine all files
of a Lucene segment into a single file in order to save file handles. As
this strategy is only used for "small" segments (less than 10% of the
shard size), it is benefical to memory-map them instead of accessing
them via NIO.

Relates 
2019-02-15 15:34:40 +01:00
Costin Leau 9357c17288 SQL: Doc on syntax (identifiers in particular) ()
Add section on syntax, identifiers and literals and on single vs double quotes.

(cherry picked from commit aafdb598082e451f36294bd174d0887a276d8c7f)
2019-02-15 15:24:33 +02:00
Alpar Torok eb7da4b90d Upgrade to Gradle 5.2.1 () 2019-02-15 15:15:43 +02:00
Marios Trivyzas 6181d2f4f4
Build: Fix issue with test status logging () ()
Handle the case of `Description` being null which is a valid case as
described in the `HeartBeatEvent`'s javadoc, which previously resulted
in exceptions that "pollute" the build output.

Follows: 
Backport: 
2019-02-15 15:06:49 +02:00
Yogesh Gaikwad 36c274867e
Fix intermittent failure in ApiKeyIntegTests () ()
Few tests failed intermittently and most of the
times due to invalidated or expired keys that were
deleted were still reported in search results.
This commit removes the test and adds enhancements
to other tests testing different scenario's.

When ExpiredApiKeysRemover is triggered, the tests
did not await its termination thereby sometimes
the results would be wrong for a search operation.

DELETE_INTERVAL setting has been further reduced to
100ms so we can trigger ExpiredApiKeysRemover faster.

Closes 
2019-02-15 23:01:35 +11:00
Martijn van Groningen 60cc04ed13
Migrate muted auto follow rolling upgrade test and unmute this test ()
The rest of `CCRIT` is now no longer relevant, because the remaining
test tests the same of the index following test in the rolling upgrade
multi cluster module.

Added `tests.upgrade_from_version` version to test. It is not needed
in this branch, but is in 6.7 branch.

Closes 
2019-02-15 11:25:13 +01:00
David Turner 578514e892 Recover peers from translog, ignoring soft deletes ()
Today if soft deletes are enabled then we read the operations needed for peer
recovery from Lucene. However we do not currently make any attempt to retain
history in Lucene specifically for peer recoveries so we may discard it and
fall back to a more expensive file-based recovery. Yet we still retain
sufficient history in the translog to perform an operations-based peer
recovery.

In the long run we would like to fix this by retaining more history in Lucene,
possibly using shard history retention leases (). For now, however, this
commit reverts to performing peer recoveries using the history retained in the
translog regardless of whether soft deletes are enabled or not.
2019-02-15 10:45:15 +01:00
Henning Andersen a211e51343 ShardBulkAction ignore primary response on primary ()
Previously, if a version conflict occurred and a previous primary
response was present, the original primary response would be used both
for sending to replica and back to client. This was made in the past as an
attempt to fix issues with conflicts after relocations where a bulk request
would experience a closed shard half way through and thus have to retry
on the new primary. It could then fail on its own update.

With sequence numbers, this leads to an issue, since if a primary is
demoted (network partitions), it will send along the original response
in the request. In case of a conflict on the new primary, the old
response is sent to the replica. That data could be stale, leading to
inconsistency between primary and replica.

Relocations now do an explicit hand-off from old to new primary and
ensures that no operations are active while doing this. Above is thus no
longer necessary. This change removes the special handling of conflicts
and ignores primary responses when executing shard bulk requests on the
primary.
2019-02-15 10:13:11 +01:00
Yannick Welsch d55e52223f Smarter CCR concurrent file chunk fetching ()
The previous logic for concurrent file chunk fetching did not allow for multiple chunks from the same
file to be fetched in parallel. The parallelism only allowed to fetch chunks from different files in
parallel. This required complex logic on the follower to be aware from which file it was already
fetching information, in order to ensure that chunks for the same file would be fetched in sequential
order. During benchmarking, this exhibited throughput issues when recovery came towards the end,
where it would only be sequentially fetching chunks for the same largest segment file, with
throughput considerably going down in a high-latency network as there was no parallelism anymore.

The new logic here follows the peer recovery model more closely, and sends multiple requests for
the same file in parallel, and then reorders the results as necessary. Benchmarks show that this
leads to better overall throughput and the implementation is also simpler.
2019-02-15 07:51:58 +01:00
Shaunak Kashyap 1f74ba2d33
[Monitoring] Remove `include_type_name` parameter from GET _template request ()
Backport of  to `7.x`. Original description:

The HTTP exporter code in the Monitoring plugin makes `GET _template` requests to check for existence of templates. These requests don't need to pass the `include_type_name` query parameter so this PR removes it from the request. This should remove the following deprecation log entries on the Monitoring cluster in 7.0.0 onwards:

```
[types removal] Specifying include_type_name in get index template requests is deprecated.
```
2019-02-14 16:09:52 -08:00
Jay Modi 5d06226507
Fix writing of SecurityFeatureSetUsage to pre-7.1 ()
This change makes the writing of new usage data conditional based on
the version that is being written to. A test has also been added to
ensure serialization works as expected to an older version.

Relates , 
2019-02-14 16:28:52 -07:00
Lee Hinman 7d449c5f65
Check that delete index request succeeded in test teardown () ()
Backport of 

When tearing down from `ESSingleNodeTestCase` we perform a delete on "*"
indices, it some cases, however, those indices are not fully deleted. Rather
than have a failure occur later down the change (see:
https://github.com/elastic/elasticsearch/issues/30290#issuecomment-463589008 )
the failure should occurr immediately so it can be diagnosed more easily.
2019-02-14 13:46:17 -07:00
Jason Tedor 00cb8d0be8
Mark coordinator test as awaits fix
This test is failing frequently so this commit mutes it.

Relates 
2019-02-14 12:43:31 -05:00
Lee Hinman 0c733c04be Remove immediate operation retry after mapping update ()
Prior to this commit, when an indexing operation resulted in an
`Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction
immediately retries the indexing operation to see if it succeeds. In the event
that it succeeds the context does not wait until the mapping update has
propagated through the cluster state before finishing the indexing.

In some of our tests we rely on mappings being available as soon as they've been
introduced in a document that indexed correctly. By removing the immediate retry
we always wait for this to be the case.

Resolves 
Supercedes 
Relates to 
2019-02-14 09:31:08 -07:00
debadair d9c255dbbf [DOCS] Added include and reference to beta1 RNs () 2019-02-14 07:43:54 -08:00
Christoph Büscher 52a4ca5962 Remove mentioning of types from bulk API docs ()
The docs on master still mention types in the context of conflicts with
documents and also mentions the deprecated endpoint including types.
2019-02-14 16:32:57 +01:00
Alpar Torok 99551001cd Skip BWC tests in checkPart1 and checkPart2 ()
Don't run bwc tests for check part 1 and 2
2019-02-14 17:29:07 +02:00
Christoph Büscher 6c5cec4ff4 Enable silent FollowersCheckerTest ()
One of the test methods wasn't run because it was private. Making this method
public and fixing some issues around mocking the threadpool that otherwise would
lead to an NPE.
2019-02-14 16:16:48 +01:00
Lee Hinman cf0bdf3b28 Update TESTING.asciidoc with platform specific instructions ()
This adds the instructions for building a platform-specific distribution.
2019-02-14 08:13:52 -07:00
taku333 8bb2a2a405 SQL: change JDBC setup URL in the documentation ()
(cherry picked from commit 103786ea27da72b2fccd3cf511b3143dae0fc530)
2019-02-14 17:12:08 +02:00
Jay Modi e59b7b696a
Use consistent view of realms for authentication ()
This change updates the authentication service to use a consistent view
of the realms based on the license state at the start of
authentication. Without this, the license can change during
authentication of a request and it will result in a failure if the
realm that extracted the token is no longer in the realm list. This
manifests in some tests as an authentication failure that should never
really happen; one example would be the test framework's transport
client user should always have a succesful authentication but in the
LicensingTests this can fail and will show up as a
NoNodeAvailableException.

Additionally, the licensing tests have been updated to ensure that
there is consistency when changing the license. The license is changed
by modifying the internal xpack license state on each node, which has
no protection against be changed by some pending cluster action. The
methods to disable and enable now ensure we have a green cluster and
that the cluster is consistent before returning.

Closes 
2019-02-14 07:49:14 -07:00
Albert Zaharovits 6243a9797f _cat/indices with Security, hide names when wildcard ()
This changes the output of the `_cat/indices` API with `Security` enabled.

It is possible to only display the index name (and possibly the index
health, depending on the request options) but not its stats (doc count, merges,
size, etc). This is the case for closed indices which have index metadata in the
cluster state but no associated shards, hence no shard stats.
However, when `Security` is enabled, and the request contains wildcards,
**open** indices without stats are a common occurrence. This is because the
index names in the response table are picked up directly from the cluster state
which is not filtered by `Security`'s _indexNameExpressionResolver_, unlike the
stats data which is populated by the indices stats API which does go through the
index name resolver.
This is a bug, because it is circumventing `Security`'s function to hide
unauthorized indices.

This has been fixed by displaying the index names as they are resolved by the indices
stats API. The outputs of these two APIs is now very similar: same index names,
similar data but different format.

Closes 
2019-02-14 15:09:17 +02:00
Andrei Stefan 7d78f4641b SQL: fall back to using the field name for column label ()
(cherry picked from commit 0567bf24957be477e7649cff94872b0e7dc4d284)
2019-02-14 14:10:59 +02:00
Yogesh Gaikwad 335cf91bb9
Add enabled status for token and api key service () ()
Right now there is no way to determine whether the
token service or API key service is enabled or not.
This commit adds support for the enabled status of
token and API key service to the security feature set
usage API `/_xpack/usage`.

Closes 
2019-02-14 23:08:52 +11:00
Martijn van Groningen 96e7d71948
Handle the fact that `ShardStats` instance may have no commit or seqno stats ()
The should fix the following NPE:

```
[2019-02-11T23:27:48,452][WARN ][o.e.p.PersistentTasksNodeService] [node_s_0] task kD8YzUhHTK6uKNBNQI-1ZQ-0 failed with an exception
  1> java.lang.NullPointerException: null
  1>    at org.elasticsearch.xpack.ccr.action.ShardFollowTasksExecutor.lambda$fetchFollowerShardInfo$7(ShardFollowTasksExecutor.java:305) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1108) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1189) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1169) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:417) [elasticsearch-8.0.0-SNAP
SHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:391) [elasticsearch-8.0.0-SNAP
SHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
  1>    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
```

Relates to 
2019-02-14 13:05:21 +01:00
David Roberts 6ea483a663 Mute DedicatedClusterSnapshotRestoreIT testRestoreShrinkIndex
Due to https://github.com/elastic/elasticsearch/issues/38845
2019-02-14 11:46:22 +00:00
Dimitris Athanasiou 21f76aba28
[ML] Extract base class for integ tests with native processes () () 2019-02-14 12:15:00 +02:00
Luca Cavanna 7456117019 [TEST] address testCollectNodes rare failure ()
 changed the expected exception for "no such cluster" error from
`IllegalStateException` to a dedicated `NoSuchRemoteClusterException`.
An assertion in `testCollectNodes` needs to be updated accordingly.
2019-02-14 10:57:14 +01:00
Alexander Reelsen 8e5e48319e
Add documentation about breaking java time changes ()
In addition remove joda time mentions across the docs, make 
sure links are updated to java time javadocs.

Forward port of 
2019-02-14 10:18:12 +01:00
Martijn van Groningen 88489a3f3a
Backport rolling upgrade multi cluster module ()
* Add rolling upgrade multi cluster test module ()

This test starts 2 clusters, each with 3 nodes.
First the leader cluster is started and tests are run against it and
then the follower cluster is started and tests execute against this two cluster.

Then the follower cluster is upgraded, one node at a time.
After that the leader cluster is upgraded, one node at a time.
Every time a node is upgraded tests are ran while both clusters are online.
(and either leader cluster has mixed node versions or the follower cluster)

This commit only tests CCR index following, but could be used for CCS tests as well.
In particular for CCR, unidirectional index following is tested during a rolling upgrade.
During the test several indices are created and followed in the leader cluster before or
while the follower cluster is being upgraded.

This tests also verifies that attempting to follow an index in the upgraded cluster
from the not upgraded cluster fails. After both clusters are upgraded following the
index that previously failed should succeed.

Relates to  and 

* Filter out upgraded version index settings when starting index following ()

The `index.version.upgraded` and `index.version.upgraded_string` are likely
to be different between leader and follower index. In the event that
a follower index gets restored on a upgraded node while the leader index
is still on non-upgraded nodes.

Closes 
2019-02-14 08:12:14 +01:00
Lee Hinman 60c1dcde88 Only flush Watcher's bulk processor if Watcher is enabled ()
When shutting down Watcher, the `bulkProcessor` is null if watcher has been
disabled in the configuration. This protects the flush and close calls with a
check for watcher enabled to avoid a NullPointerException

Resolves 
2019-02-13 16:13:53 -07:00
Julie Tibshirani 4ad4bc7f5f Update the removal of types docs with the new 6.7 behavior. ()
Follow-up to , where we made a tweak to the deprecation behavior.
2019-02-13 14:45:17 -08:00
Nhat Nguyen 5d22e45990 Copy retention leases when trim unsafe commits ()
When a primary shard is recovered from its store, we trim the last
commit (when it's unsafe). If that primary crashes before the recovery
completes, we will lose the committed retention leases because they are
baked in the last commit. With this change, we copy the retention leases
from the last commit to the safe commit when trimming unsafe commits.

Relates 
2019-02-13 17:27:48 -05:00
Tim Brooks ec08581319
Improve CcrRepositoryIT mappings tests ()
Currently we index documents concurrently to attempt to ensure that we
update mappings during the restore process. However, this does not
actually test that the mapping will be correct and is dangerous as it
can lead to a misalignment between the max sequence number and the local
checkpoint. If these are not aligned, peer recovery cannot be completed
without initiating following which this test does not do. That causes
teardown assertions to fail.

This commit removes the concurrent indexing and flushes after the
documents are indexed. Additionally it modifies the mapping specific
test to ensure that there is a mapping update when the restore session
is initiated. This mapping update is picked up at the end of the restore
by the follower.
2019-02-13 13:47:10 -07:00
Julie Tibshirani e769cb4efd Perform precise check for types warnings in cluster restart tests. ()
Instead of using `WarningsHandler.PERMISSIVE`, we only match warnings
that are due to types removal.

This PR also renames `allowTypeRemovalWarnings` to `allowTypesRemovalWarnings`.

Relates to .
2019-02-13 11:28:58 -08:00
Jason Tedor 062eea8fcc
Fix excessive increments in soft delete policy ()
In this case, we were incrementing the policy too much. This means on
every iteration we actually keep increasing the minimum retained
sequence number, even with leases in place. It was a bug from when the
soft deletes policy had retention leases incorporated into it. This
commit fixes this bug by ensuring we only increment in the proper
places, and adds careful tests for the various situations.
2019-02-13 14:04:45 -05:00
Benjamin Trent d2ac05e249
ML allow aliased .ml-anomalies* index on PUT Job () () 2019-02-13 10:58:55 -06:00
Jake Landis 46bb663a09
Make 7.x like 6.7 user agent ecs, but default to true ()
Forward port of https://github.com/elastic/elasticsearch/pull/38757

This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object ()"
This reverts commit 5b008a34aa.

* Revert "Add ECS schema for user-agent ingest processor () ()"
This reverts commit cac6b8e06f.

* cherry-pick 5dfe1935345da3799931fd4a3ebe0b6aa9c17f57 
Add ECS schema for user-agent ingest processor ()

* cherry-pick ec8ddc890a34853ee8db6af66f608b0ad0cd1099 
Ingest node - user agent, move device to an object () ()
  
* cherry-pick f63cbdb9b426ba24ee4d987ca767ca05a22f2fbb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor ()

* make true the default for the ecs option, and update 7.0 references and tests
2019-02-13 10:28:01 -06:00
Przemyslaw Gomulka 7404882105
Fix line separators in JSON logging tests backport#38771
The hardcoded '\n' in string will not work in Windows where there is a
different line separator. A System.lineSeparator should be used to make
it work on all platforms
closes  
backport 
2019-02-13 13:34:33 +01:00
Darren Meiss 34b6383c47 Edits to text of Profile API documentation ()
Minor edits of text.
2019-02-13 10:08:50 +01:00
Jim Ferenczi 251ced79ee Add release notes for 7.0.0-beta1 ()
Add release notes for 7.0.0-beta1
2019-02-13 09:54:34 +01:00
Dimitrios Liappis 470ac0ec95
Don't fail init script if `/proc/.../max_map_count` absent ()
Currently init scripts fail when `/proc/sys/vm/max_map_count` is not present
with `-bash: [: too many arguments`.

Fix conditional logic to avoid trying to set the `max_map_count` sysctl if not
present.

Backport of: 
Relates: 
2019-02-13 09:56:47 +02:00