c7edaba4e8
Two tests were periodically failing. What both tests are doing is starting a relocation of a shard from one node to another. Once the recovery process is started, the test blocks cluster state processing on the relocation target using the BlockClusterStateProcessing disruption. The test then indefinitely waits for the relocation to complete. The stack dump shows that the relocation is stuck in the PeerRecoveryTargetService.waitForClusterState method, waiting for the relocation target node to have at least the same cluster state version as the relocation source. The reason why it gets stuck is the following race: 1) The test code executes a reroute command that relocates a shard from one node to another 2) Relocation target node starts applying the clusterstate with relocation info, starting the recovery process. 4) Recovery is super fast and quickly goes to the waitForClusterState method, which wants to ensure that the cluster state that is applied on the relocation target is at least as new as the one on the relocation source. The relocation source has already applied the cluster state but the relocation target is still in the process of applying it. The waitForClusterState method thus uses a ClusterObserver to wait for the next cluster state. Internally this means submitting a task with priority HIGH to the cluster service. 5) Cluster state application is about to finish on the relocation target. As one of the last steps, it acks to the master which makes the reroute command return successfully. 6) The test code then blocks cluster state processing on the relocation target by submitting a cluster state update task (with priority IMMEDIATE) that blocks execution. If the task that is submitted in step 6 is handled before the one in step 4 by ClusterService's thread executor, cluster state processing becomes blocked and prevents the cluster state observer from observing the applied cluster state. |
||
|---|---|---|
| .. | ||
| licenses | ||
| src | ||
| build.gradle | ||