Engine - do not index operations with seq# lower than the local checkpoint into lucene (#25827)
When a replica processes out of order operations, it can drop some due to version comparisons. In the past that would have resulted in a VersionConflictException being thrown and the operation was totally ignored. With the seq# push, we started storing these operations in the translog (but not indexing them into lucene) in order to have complete op histories to facilitate ops based recoveries. This in turn had the undesired effect that deleted docs may be resurrected during recovery in some extreme edge situation (see a complete explanation below). This PR contains a simple fix, which is also an optimization for the recovery process, incoming operation that have a seq# lower than the current local checkpoint (i.e., have already been processed) should not be indexed into lucene. Note that sometimes we can also skip storing them in the translog, but this is not required for the fix and is more complicated. This is the equivalent of #25592 ## More details on resurrected ops Consider two operations: - Index d1, seq no 1 - Delete d1, seq no 3 On a replica they come out of order: - Translog gen 1 contains: - delete (seqNo 3) - Translog gen 2 contains: - index (seqNo 1) (wasn't indexed into lucene, but put into the translog) - another operation (seqNo 10) - Translog gen 3 - another op (seqNo 9) - Engine commits with: - local checkpoint 9 - refers to gen 2 If this replica becomes a primary: - Local recovery will replay translog gen 2 and up, causing index #1 to be re-index. - Even if recovery will start at gen 3, the translog retention policy will cause file based recovery to replay the entire translog. If it happens to start at gen 2 (but not 1), we will run into the same problem. #### Some context - out of order delivery involving deletes: On normal operations, this relies on the gc_deletes setting. We assume that the setting represents an upper bound on the time between the index and the delete operation. The index operation will be detected as stale based on the tombstone map in the LiveVersionMap. Recovery presents a challenge as it can replay an old index operation that was in the translog and override a delete operation that was done when the engine was opened (and is not part of the replayed snapshot). To deal with this situation, we disable GC deletes (i.e. retain all deletes) for the duration of recoveries. This means that the delete operation will be remembered and the index operation ignored. Both of the above scenarios (local recover + peer recovery) create a situation where the delete operation is never replayed. It this "lost" as lucene doesn't remember it happened and our LiveVersionMap is populated with it. #### Solution: Note that both local and peer recovery represent a scenario where we replay translog ops on top of an existing lucene index, potentially with ongoing indexing. Therefore we can treat them the same. The local checkpoint in Lucene represent a marker indicating that all operations below it were performed on the index. This is the only form of "memory" that we have that relates to deletes. If we can achieve the following: 1) All ops below the local checkpoint are not indexed to lucene. 2) All ops above the local checkpoint are It will mean that all variants are covered: (i# == index op seq#, d# == delete op seq#, lc == local checkpoint in commit) 1) i# < d# <= lc - document is already deleted in lucene and stays that way. 2) i# <= lc < d# - delete is replayed on index - document is deleted 3) lc < i# < d# - index is replayed and then delete - document is deleted. More formally - we want to make sure that for all ops that performed on the primary o1 and o2, if o2 is processed on a shard before o1, o1 will be dropped. We have the following scenarios 1) If both o1 or o2 are not included in the replayed snapshot and are above it (i.e., have a higher seq#), they fall under the gc deletes assumption. 2) If both o1 is part of the replayed snapshot but o2 is above it: - if o2 arrives first, o1 must arrive due to the recovery and potentially via replication as well. since gc deletes is disabled we are guaranteed to know of o2's existence. 3) If both o2 and o1 are part of the replayed snapshot: - we fall under the same scenarios as #2 - disabling GC deletes ensures we know of o2 if it arrives first. 4) If o1 falls before the snapshot and o2 is either part of the snapshot or higher: - Since the snapshot is guaranteed to contain all ops that are not part of lucene and are above the lc in the commit used, this means that o1 is part of lucene and o1 < local checkpoint. This means it won't be processed and we're not in the scenario we're discussing. 5) If o2 falls before the snapshot but o1 is part of it: - by the same reasoning above, o2 is < local checkpoint. Since o1 < o2, we also get o1 < local checkpoint and this will be dropped. #### Implementation: For local recovery, we can filter the ops we read of the translog and avoid replaying them. For peer recovery this is tricky as we do want to send the operations in order to have some history on the target shard. Filtering operations on the engine level (i.e., not indexing to lucene if op seq# <= lc) would work for both.
This commit is contained in:
Boaz Leskes
GitHub
parent
c3784326eb
commit
ab1636d547
|
|
@ -693,14 +693,23 @@ public class InternalEngine extends Engine {
|
|||
// this allows to ignore the case where a document was found in the live version maps in
|
||||
// a delete state and return false for the created flag in favor of code simplicity
|
||||
final OpVsLuceneDocStatus opVsLucene;
|
||||
if (index.seqNo() != SequenceNumbersService.UNASSIGNED_SEQ_NO) {
|
||||
opVsLucene = compareOpToLuceneDocBasedOnSeqNo(index);
|
||||
} else {
|
||||
if (index.seqNo() == SequenceNumbersService.UNASSIGNED_SEQ_NO) {
|
||||
// This can happen if the primary is still on an old node and send traffic without seq# or we recover from translog
|
||||
// created by an old version.
|
||||
assert config().getIndexSettings().getIndexVersionCreated().before(Version.V_6_0_0_alpha1) :
|
||||
"index is newly created but op has no sequence numbers. op: " + index;
|
||||
opVsLucene = compareOpToLuceneDocBasedOnVersions(index);
|
||||
} else if (index.seqNo() <= seqNoService.getLocalCheckpoint()){
|
||||
// the operation seq# is lower then the current local checkpoint and thus was already put into lucene
|
||||
// this can happen during recovery where older operations are sent from the translog that are already
|
||||
// part of the lucene commit (either from a peer recovery or a local translog)
|
||||
// or due to concurrent indexing & recovery. For the former it is important to skip lucene as the operation in
|
||||
// question may have been deleted in an out of order op that is not replayed.
|
||||
// See testRecoverFromStoreWithOutOfOrderDelete for an example of local recovery
|
||||
// See testRecoveryWithOutOfOrderDelete for an example of peer recovery
|
||||
opVsLucene = OpVsLuceneDocStatus.OP_STALE_OR_EQUAL;
|
||||
} else {
|
||||
opVsLucene = compareOpToLuceneDocBasedOnSeqNo(index);
|
||||
}
|
||||
if (opVsLucene == OpVsLuceneDocStatus.OP_STALE_OR_EQUAL) {
|
||||
plan = IndexingStrategy.processButSkipLucene(false, index.seqNo(), index.version());
|
||||
|
|
@ -979,12 +988,21 @@ public class InternalEngine extends Engine {
|
|||
// this allows to ignore the case where a document was found in the live version maps in
|
||||
// a delete state and return true for the found flag in favor of code simplicity
|
||||
final OpVsLuceneDocStatus opVsLucene;
|
||||
if (delete.seqNo() != SequenceNumbersService.UNASSIGNED_SEQ_NO) {
|
||||
opVsLucene = compareOpToLuceneDocBasedOnSeqNo(delete);
|
||||
} else {
|
||||
if (delete.seqNo() == SequenceNumbersService.UNASSIGNED_SEQ_NO) {
|
||||
assert config().getIndexSettings().getIndexVersionCreated().before(Version.V_6_0_0_alpha1) :
|
||||
"index is newly created but op has no sequence numbers. op: " + delete;
|
||||
opVsLucene = compareOpToLuceneDocBasedOnVersions(delete);
|
||||
} else if (delete.seqNo() <= seqNoService.getLocalCheckpoint()) {
|
||||
// the operation seq# is lower then the current local checkpoint and thus was already put into lucene
|
||||
// this can happen during recovery where older operations are sent from the translog that are already
|
||||
// part of the lucene commit (either from a peer recovery or a local translog)
|
||||
// or due to concurrent indexing & recovery. For the former it is important to skip lucene as the operation in
|
||||
// question may have been deleted in an out of order op that is not replayed.
|
||||
// See testRecoverFromStoreWithOutOfOrderDelete for an example of local recovery
|
||||
// See testRecoveryWithOutOfOrderDelete for an example of peer recovery
|
||||
opVsLucene = OpVsLuceneDocStatus.OP_STALE_OR_EQUAL;
|
||||
} else {
|
||||
opVsLucene = compareOpToLuceneDocBasedOnSeqNo(delete);
|
||||
}
|
||||
|
||||
final DeletionStrategy plan;
|
||||
|
|
|
|||
|
|
@ -125,12 +125,14 @@ import java.util.concurrent.atomic.AtomicInteger;
|
|||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.function.BiConsumer;
|
||||
import java.util.function.Consumer;
|
||||
import java.util.function.LongFunction;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.stream.IntStream;
|
||||
|
||||
import static java.util.Collections.emptyMap;
|
||||
import static java.util.Collections.emptySet;
|
||||
import static org.elasticsearch.cluster.routing.TestShardRouting.newShardRouting;
|
||||
import static org.elasticsearch.common.lucene.Lucene.cleanLuceneIndex;
|
||||
import static org.elasticsearch.common.xcontent.ToXContent.EMPTY_PARAMS;
|
||||
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
|
||||
|
|
@ -339,7 +341,7 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
// promote the replica
|
||||
final ShardRouting replicaRouting = indexShard.routingEntry();
|
||||
final ShardRouting primaryRouting =
|
||||
TestShardRouting.newShardRouting(
|
||||
newShardRouting(
|
||||
replicaRouting.shardId(),
|
||||
replicaRouting.currentNodeId(),
|
||||
null,
|
||||
|
|
@ -416,7 +418,7 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
// promote the replica
|
||||
final ShardRouting replicaRouting = indexShard.routingEntry();
|
||||
final ShardRouting primaryRouting =
|
||||
TestShardRouting.newShardRouting(
|
||||
newShardRouting(
|
||||
replicaRouting.shardId(),
|
||||
replicaRouting.currentNodeId(),
|
||||
null,
|
||||
|
|
@ -458,13 +460,13 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
|
||||
if (randomBoolean()) {
|
||||
// relocation target
|
||||
indexShard = newShard(TestShardRouting.newShardRouting(shardId, "local_node", "other node",
|
||||
indexShard = newShard(newShardRouting(shardId, "local_node", "other node",
|
||||
true, ShardRoutingState.INITIALIZING, AllocationId.newRelocation(AllocationId.newInitializing())));
|
||||
} else if (randomBoolean()) {
|
||||
// simulate promotion
|
||||
indexShard = newStartedShard(false);
|
||||
ShardRouting replicaRouting = indexShard.routingEntry();
|
||||
ShardRouting primaryRouting = TestShardRouting.newShardRouting(replicaRouting.shardId(), replicaRouting.currentNodeId(), null,
|
||||
ShardRouting primaryRouting = newShardRouting(replicaRouting.shardId(), replicaRouting.currentNodeId(), null,
|
||||
true, ShardRoutingState.STARTED, replicaRouting.allocationId());
|
||||
indexShard.updateShardState(primaryRouting, indexShard.getPrimaryTerm() + 1, (shard, listener) -> {}, 0L,
|
||||
Collections.singleton(indexShard.routingEntry().allocationId().getId()),
|
||||
|
|
@ -520,7 +522,7 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
case 1: {
|
||||
// initializing replica / primary
|
||||
final boolean relocating = randomBoolean();
|
||||
ShardRouting routing = TestShardRouting.newShardRouting(shardId, "local_node",
|
||||
ShardRouting routing = newShardRouting(shardId, "local_node",
|
||||
relocating ? "sourceNode" : null,
|
||||
relocating ? randomBoolean() : false,
|
||||
ShardRoutingState.INITIALIZING,
|
||||
|
|
@ -533,7 +535,7 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
// relocation source
|
||||
indexShard = newStartedShard(true);
|
||||
ShardRouting routing = indexShard.routingEntry();
|
||||
routing = TestShardRouting.newShardRouting(routing.shardId(), routing.currentNodeId(), "otherNode",
|
||||
routing = newShardRouting(routing.shardId(), routing.currentNodeId(), "otherNode",
|
||||
true, ShardRoutingState.RELOCATING, AllocationId.newRelocation(routing.allocationId()));
|
||||
IndexShardTestCase.updateRoutingEntry(indexShard, routing);
|
||||
indexShard.relocated("test", primaryContext -> {});
|
||||
|
|
@ -1377,6 +1379,47 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
closeShards(shard);
|
||||
}
|
||||
|
||||
public void testRecoverFromStoreWithOutOfOrderDelete() throws IOException {
|
||||
final IndexShard shard = newStartedShard(false);
|
||||
final Consumer<Mapping> mappingConsumer = getMappingUpdater(shard, "test");
|
||||
shard.applyDeleteOperationOnReplica(1, 1, 2, "test", "id", VersionType.EXTERNAL, mappingConsumer);
|
||||
shard.getEngine().rollTranslogGeneration(); // isolate the delete in it's own generation
|
||||
shard.applyIndexOperationOnReplica(0, 1, 1, VersionType.EXTERNAL, IndexRequest.UNSET_AUTO_GENERATED_TIMESTAMP, false,
|
||||
SourceToParse.source(shard.shardId().getIndexName(), "test", "id", new BytesArray("{}"), XContentType.JSON), mappingConsumer);
|
||||
|
||||
// index a second item into the second generation, skipping seq# 2. Local checkpoint is now 1, which will make this generation stick
|
||||
// around
|
||||
shard.applyIndexOperationOnReplica(3, 1, 1, VersionType.EXTERNAL, IndexRequest.UNSET_AUTO_GENERATED_TIMESTAMP, false,
|
||||
SourceToParse.source(shard.shardId().getIndexName(), "test", "id2", new BytesArray("{}"), XContentType.JSON), mappingConsumer);
|
||||
|
||||
final int translogOps;
|
||||
if (randomBoolean()) {
|
||||
logger.info("--> flushing shard");
|
||||
flushShard(shard);
|
||||
translogOps = 2;
|
||||
} else if (randomBoolean()) {
|
||||
shard.getEngine().rollTranslogGeneration();
|
||||
translogOps = 3;
|
||||
} else {
|
||||
translogOps = 3;
|
||||
}
|
||||
|
||||
final ShardRouting replicaRouting = shard.routingEntry();
|
||||
IndexShard newShard = reinitShard(shard,
|
||||
newShardRouting(replicaRouting.shardId(), replicaRouting.currentNodeId(), true, ShardRoutingState.INITIALIZING,
|
||||
RecoverySource.StoreRecoverySource.EXISTING_STORE_INSTANCE));
|
||||
DiscoveryNode localNode = new DiscoveryNode("foo", buildNewFakeTransportAddress(), emptyMap(), emptySet(), Version.CURRENT);
|
||||
newShard.markAsRecovering("store", new RecoveryState(newShard.routingEntry(), localNode, null));
|
||||
assertTrue(newShard.recoverFromStore());
|
||||
assertEquals(translogOps, newShard.recoveryState().getTranslog().recoveredOperations());
|
||||
assertEquals(translogOps, newShard.recoveryState().getTranslog().totalOperations());
|
||||
assertEquals(translogOps, newShard.recoveryState().getTranslog().totalOperationsOnStart());
|
||||
assertEquals(100.0f, newShard.recoveryState().getTranslog().recoveredPercent(), 0.01f);
|
||||
updateRoutingEntry(newShard, ShardRoutingHelper.moveToStarted(newShard.routingEntry()));
|
||||
assertDocCount(newShard, 1);
|
||||
closeShards(newShard);
|
||||
}
|
||||
|
||||
public void testRecoverFromStore() throws IOException {
|
||||
final IndexShard shard = newStartedShard(true);
|
||||
int totalOps = randomInt(10);
|
||||
|
|
@ -1939,7 +1982,7 @@ public class IndexShardTests extends IndexShardTestCase {
|
|||
sourceShard.refresh("test");
|
||||
|
||||
|
||||
ShardRouting targetRouting = TestShardRouting.newShardRouting(new ShardId("index_1", "index_1", 0), "n1", true,
|
||||
ShardRouting targetRouting = newShardRouting(new ShardId("index_1", "index_1", 0), "n1", true,
|
||||
ShardRoutingState.INITIALIZING, RecoverySource.LocalShardsRecoverySource.INSTANCE);
|
||||
|
||||
final IndexShard targetShard;
|
||||
|
|
|
|||
|
|
@ -19,9 +19,14 @@
|
|||
|
||||
package org.elasticsearch.indices.recovery;
|
||||
|
||||
import org.elasticsearch.action.index.IndexRequest;
|
||||
import org.elasticsearch.cluster.metadata.IndexMetaData;
|
||||
import org.elasticsearch.common.bytes.BytesArray;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.xcontent.XContentType;
|
||||
import org.elasticsearch.index.IndexSettings;
|
||||
import org.elasticsearch.index.VersionType;
|
||||
import org.elasticsearch.index.mapper.SourceToParse;
|
||||
import org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase;
|
||||
import org.elasticsearch.index.replication.RecoveryDuringReplicationTests;
|
||||
import org.elasticsearch.index.shard.IndexShard;
|
||||
|
|
@ -79,4 +84,52 @@ public class RecoveryTests extends ESIndexLevelReplicationTestCase {
|
|||
assertBusy(() -> assertThat(replica.getTranslog().totalOperations(), equalTo(0)));
|
||||
}
|
||||
}
|
||||
|
||||
public void testRecoveryWithOutOfOrderDelete() throws Exception {
|
||||
try (ReplicationGroup shards = createGroup(1)) {
|
||||
shards.startAll();
|
||||
// create out of order delete and index op on replica
|
||||
final IndexShard orgReplica = shards.getReplicas().get(0);
|
||||
orgReplica.applyDeleteOperationOnReplica(1, 1, 2, "type", "id", VersionType.EXTERNAL, u -> {});
|
||||
orgReplica.getTranslog().rollGeneration(); // isolate the delete in it's own generation
|
||||
orgReplica.applyIndexOperationOnReplica(0, 1, 1, VersionType.EXTERNAL, IndexRequest.UNSET_AUTO_GENERATED_TIMESTAMP, false,
|
||||
SourceToParse.source(orgReplica.shardId().getIndexName(), "type", "id", new BytesArray("{}"), XContentType.JSON),
|
||||
u -> {});
|
||||
|
||||
// index a second item into the second generation, skipping seq# 2. Local checkpoint is now 1, which will make this generation
|
||||
// stick around
|
||||
orgReplica.applyIndexOperationOnReplica(3, 1, 1, VersionType.EXTERNAL, IndexRequest.UNSET_AUTO_GENERATED_TIMESTAMP, false,
|
||||
SourceToParse.source(orgReplica.shardId().getIndexName(), "type", "id2", new BytesArray("{}"), XContentType.JSON), u -> {});
|
||||
|
||||
final int translogOps;
|
||||
if (randomBoolean()) {
|
||||
if (randomBoolean()) {
|
||||
logger.info("--> flushing shard (translog will be trimmed)");
|
||||
IndexMetaData.Builder builder = IndexMetaData.builder(orgReplica.indexSettings().getIndexMetaData());
|
||||
builder.settings(Settings.builder().put(orgReplica.indexSettings().getSettings())
|
||||
.put(IndexSettings.INDEX_TRANSLOG_RETENTION_AGE_SETTING.getKey(), "-1")
|
||||
.put(IndexSettings.INDEX_TRANSLOG_RETENTION_SIZE_SETTING.getKey(), "-1")
|
||||
);
|
||||
orgReplica.indexSettings().updateIndexMetaData(builder.build());
|
||||
orgReplica.onSettingsChanged();
|
||||
translogOps = 3; // 2 ops + seqno gaps
|
||||
} else {
|
||||
logger.info("--> flushing shard (translog will be retained)");
|
||||
translogOps = 4; // 3 ops + seqno gaps
|
||||
}
|
||||
flushShard(orgReplica);
|
||||
} else {
|
||||
translogOps = 4; // 3 ops + seqno gaps
|
||||
}
|
||||
|
||||
final IndexShard orgPrimary = shards.getPrimary();
|
||||
shards.promoteReplicaToPrimary(orgReplica).get(); // wait for primary/replica sync to make sure seq# gap is closed.
|
||||
|
||||
IndexShard newReplica = shards.addReplicaWithExistingPath(orgPrimary.shardPath(), orgPrimary.routingEntry().currentNodeId());
|
||||
shards.recoverReplica(newReplica);
|
||||
shards.assertAllEqual(1);
|
||||
|
||||
assertThat(newReplica.getTranslog().totalOperations(), equalTo(translogOps));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue