From 7dff8fbb1d0b7af05ca6007ca031f6e5fbdfa737 Mon Sep 17 00:00:00 2001 From: Yannick Welsch Date: Fri, 8 Jul 2016 17:30:46 +0200 Subject: [PATCH] Update resiliency docs (#19303) Adds clarifications about Jepsen tests and new section on issues with versioning. --- docs/resiliency/index.asciidoc | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/docs/resiliency/index.asciidoc b/docs/resiliency/index.asciidoc index 802c380b4a7..9f8a22c7337 100644 --- a/docs/resiliency/index.asciidoc +++ b/docs/resiliency/index.asciidoc @@ -59,9 +59,9 @@ We are committed to tracking down and fixing all the issues that are posted. ==== Jepsen Tests The Jepsen platform is specifically designed to test distributed systems. It is not a single test and is regularly adapted -to create new scenarios. We have ported all published scenarios to our testing infrastructure. Of course -as the system evolves, new scenarios can come up that are not yet covered. We are committed to investigating all new scenarios and will -report issues that we find on this page and in our GitHub repository. +to create new scenarios. We have currently ported all published Jepsen scenarios that deal with loss of acknowledged writes to our testing +framework. As the Jepsen tests evolve, we will continue porting new scenarios that are not covered yet. We are committed to investigating +all new scenarios and will report issues that we find on this page and in our GitHub repository. [float] === Better request retry mechanism when nodes are disconnected (STATUS: ONGOING) @@ -102,6 +102,19 @@ space. The following issues have been identified: Other safeguards are tracked in the meta-issue {GIT}11511[#11511]. +[float] +=== The _version field may not uniquely identify document content during a network partition (STATUS: ONGOING) + +When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue +indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is +partitioned away. It won't acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. +The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to +step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from +the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted +writes for the same document (see {GIT}19269[#19269]). + +We are currently implementing Sequence numbers {GIT}10708[#10708] which better track primary changes. Sequence numbers thus provide a basis +for uniquely identifying writes even in the presence of network partitions and will replace `_version` in operations that require this. [float] === Relocating shards omitted by reporting infrastructure (STATUS: ONGOING) @@ -119,20 +132,22 @@ in the case of each type of failure. The plan is to have a test case that valida [float] === Run Jepsen (STATUS: ONGOING) -We have ported all of the known scenarios in the Jepsen blogs to our testing infrastructure. The new tests are run continuously in our -testing farm and are passing. We are also working on running Jepsen independently to verify that no failures are found. +We have ported the known scenarios in the Jepsen blogs that check loss of acknowledged writes to our testing infrastructure. +The new tests are run continuously in our testing farm and are passing. We are also working on running Jepsen independently to verify +that no failures are found. == Unreleased [float] -=== Port Jepsen tests to our testing framework (STATUS: UNRELEASED, V5.0.0) +=== Port Jepsen tests dealing with loss of acknowledged writes to our testing framework (STATUS: UNRELEASED, V5.0.0) -We have increased our test coverage to include scenarios tested by Jepsen, as described in the Elasticsearch related blogs. We make heavy -use of randomization to expand on the scenarios that can be tested and to introduce new error conditions. +We have increased our test coverage to include scenarios tested by Jepsen that demonstrate loss of acknowledged writes, as described in +the Elasticsearch related blogs. We make heavy use of randomization to expand on the scenarios that can be tested and to introduce +new error conditions. You can follow the work on the master branch of the https://github.com/elastic/elasticsearch/blob/master/core/src/test/java/org/elasticsearch/discovery/DiscoveryWithServiceDisruptionsIT.java[`DiscoveryWithServiceDisruptionsIT` class], -where the `testAckedIndexing` test was specifically added to cover known Jepsen related scenarios. +where the `testAckedIndexing` test was specifically added to check that we don't lose acknowledged writes in various failure scenarios. [float]