honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add resiliency note on replica divergence (#36960)

This commit is contained in:
David Turner 2018-12-23 09:50:40 +00:00 committed by GitHub
parent 7562768bd6
commit e2e82039d9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 40 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
docs/resiliency/index.asciidoc
View File

@ -170,6 +170,46 @@ shard.
== Completed == Completed
[float]
=== Divergence between primary and replica shard copies when documents deleted (STATUS: DONE, V6.3.0)
Certain combinations of delays in performing activities related to the deletion
of a document could result in the operations on that document being interpreted
differently on different shard copies. This could lead to a divergence in the
number of documents held in each copy.
Deleting an unacknowledged document that was concurrently being inserted using
an auto-generated ID was erroneously sensitive to the order in which those
operations were processed on each shard copy. Thanks to the introduction of
sequence numbers ({GIT}10708[#10708]) it is now possible to detect these
out-of-order operations, and this issue was fixed in {GIT}28787[#28787].
Re-creating a document a specific interval after it was deleted could result in
that document's tombstone having being cleaned up on some, but not all, copies
when processing the indexing operation that re-creates it. This resulted in
varying behaviour across the shard copies. The problematic interval was set by
the `index.gc_deletes` setting, which is 60 seconds by default. Again, sequence
numbers ({GIT}10708[#10708]) gives us the machinery to detect these conflicting
activities, and this issue was fixed in {GIT}28790[#28790].
Under certain rare circumstances a replica might erroneously interpret a stale
tombstone for a document as fresh, resulting in a concurrent indexing operation
for that same document behaving differently on this replica than on the
primary. This is fixed in {GIT}29619[#29619]. Triggering this issue required
the following activities all to occur in a short time window, in a specific
order on the primary and a different specific order on the replica:
* a document is deleted twice
* another document is indexed with the same ID as this first document
* another document is indexed with a completely different, auto-generated, ID
* two refreshes
We found the first two of these issues by empirical testing, and then we built
https://github.com/elastic/elasticsearch-formal-models/blob/master/ReplicaEngine/tla/ReplicaEngine.tla[a
formal model of the replica's behaviour] using TLA+. Running the TLC model
checker on this model found all three issues. We then applied the proposed
fixes to the model and validated that the fixed design behaved as expected.
[float] [float]
=== Port Jepsen tests dealing with loss of acknowledged writes to our testing framework (STATUS: DONE, V5.0.0) === Port Jepsen tests dealing with loss of acknowledged writes to our testing framework (STATUS: DONE, V5.0.0)