Commit Graph

312 Commits

Author SHA1 Message Date
Luca Cavanna c1beb95aa1 Mute LocalIndexFollowingIT#testRemoveRemoteConnection
Relates to 
2018-12-28 16:39:36 +01:00
Nhat Nguyen 7580d9d925
Make SourceToParse immutable ()
Today the routing of a SourceToParse is assigned in a separate step
after the object is created. We can easily forget to set the routing.
With this commit, the routing must be provided in the constructor of
SourceToParse.

Relates 
2018-12-24 14:06:50 -05:00
Martijn van Groningen 561b704129
[CCR] AutoFollowCoordinator and follower index already created ()
The AutoFollowCoordinator should be resilient to the fact that the follower
index has already been created and in that case it should only update
the auto follow metadata with the fact that the follower index was created.

Relates to 
2018-12-24 10:16:38 +01:00
Martijn van Groningen 44fe265d82
[CCR] Added auto_follow_exception.timestamp field to auto follow stats ()
Currently auto follow stats users are unable to see whether an auto follow
error was recent or old. The new timestamp field will help user distinguish
between old and new errors.
2018-12-24 07:53:51 +01:00
Martijn van Groningen 4fb62fcba6
Make CCR resilient against missing remote cluster connections ()
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behavior similar how CCR
deals with authorization errors.

Closes 
Closes 
2018-12-24 07:28:34 +01:00
Martijn van Groningen 4ded4717fe
[CCR] Add `ccr.auto_follow_coordinator.wait_for_timeout` setting ()
This setting controls the wait for timeout the autofollow coordinator
should use when setting cluster state requests to a remote cluster.
2018-12-21 09:36:40 +01:00
Tim Brooks d9b2ed6135
Send clear session as routable remote request ()
This commit adds a RemoteClusterAwareRequest interface that allows a
request to specify which remote node it should be routed to. The remote
cluster aware client will attempt to route the request directly to this
node. Otherwise it will send it as a proxy action to eventually end up
on the requested node.

It implements the ccr clean_session action with this client.
2018-12-20 17:43:12 -07:00
Tim Brooks 4cd570593d
Update index mappings when ccr restore complete ()
This is related to . When the shard restore process is complete,
the index mappings need to be updated to ensure that the data in the
files restores is compatible with the follower mappings. This commit
implements a mapping update as the final step in a shard restore.
2018-12-20 13:53:04 -07:00
Martijn van Groningen b42074c1cc
[CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled ()
Currently if a leader index with soft deletes disabled is auto followed then this index is silently ignored.
This commit changes this behavior to mark these indices as auto followed and report an error, which is visible in auto follow stats. Marking the index as auto follow is important, because otherwise the auto follower will continuously try to auto follow and fail.

Relates to 
2018-12-20 15:21:52 +01:00
Martijn van Groningen 7b1dfeff2e
Renamed `WHITE_LISTED_SETTINGS` to `NON_REPLICATED_SETTINGS`
because the latter better describes the purpose of this field.
2018-12-20 15:08:04 +01:00
Martijn van Groningen 18691daebe
[TEST] Renamed ccr qa module. 2018-12-19 13:57:12 +01:00
Martijn van Groningen 3cc0cf03c6
[TEST] No need to specifically check licensesMetaData on master node. 2018-12-19 13:51:24 +01:00
Martijn van Groningen a6af33ef0b
[TEST] Wait for license metadata to be installed 2018-12-19 13:03:45 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task ()
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Tim Brooks 1fa105658e
Add CcrRestoreSourceService to track sessions ()
This commit is related to . It adds a CcrRestoreSourceService to
track Engine.IndexCommitRef need for in-process file restores. When a
follower starts restoring a shard through the CcrRepository it opens a
session with the leader through the PutCcrRestoreSessionAction. The
leader responds to the request by telling the follower what files it
needs to fetch for a restore. This is not yet implemented.

Once, the restore is complete, the follower closes the session with the
DeleteCcrRestoreSessionAction action.
2018-12-18 11:23:13 -07:00
Martijn van Groningen 1afcfc97bd
[TEST] Added more logging
Relates to 
2018-12-18 16:01:02 +01:00
Boaz Leskes 5f76f39386
Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm ()
This PR renames the parameters previously introduce to the following:

### URL Parameters
```
PUT twitter/_doc/1?if_seq_no=501&if_primary_term=1
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

DELETE twitter/_doc/1?if_seq_no=501&if_primary_term=1
```

### Bulk API
```
POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1", "if_seq_no": 501, "if_primary_term": 1 } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2", "if_seq_no": 501, "if_primary_term": 1 } }
```

### Java API
```
IndexRequest.ifSeqNo(long seqNo)
IndexRequest.ifPrimaryTerm(long primaryTerm)
DeleteRequest.ifSeqNo(long seqNo)
DeleteRequest.ifPrimaryTerm(long primaryTerm)
```

Relates 
Relates 
2018-12-18 14:35:18 +01:00
Martijn van Groningen 0ff1f1fa18
Muted tests.
Relates to 
2018-12-18 13:39:01 +01:00
Martijn van Groningen 57e1a4bc9f
[TEST] Ensure shard follow tasks have really stopped.
Relates to 
2018-12-18 10:43:27 +01:00
Tim Brooks 3dd5a5a3c5
Initialize startup `CcrRepositories` ()
Currently, the CcrRepositoryManger only listens for settings updates
and installs new repositories. It does not install the repositories that
are in the initial settings. This commit, modifies the manager to
install the initial repositories. Additionally, it modifies the ccr
integration test to configure the remote leader node at startup, instead
of using a settings update.
2018-12-17 13:19:32 -07:00
Martijn van Groningen a181a25226
[CCR] Add time since last auto follow fetch to auto follow stats ()
For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to 
Originates from 
2018-12-17 14:14:56 +01:00
Martijn van Groningen f27d2c2927
[TEST] Pause index following at end of test,
so that no unexpected failures happen at test teardown.
2018-12-17 07:55:27 +01:00
Nhat Nguyen 2028c2af14 TEST: Do not assert max_seq_of_updates if promotion
If a primary promotion happens in the test testAddRemoveShardOnLeader, the
max_seq_no_of_updates_or_deletes on a  new primary might be higher than the
max_seq_no_of_updates_or_deletes on the replicas or copies of the follower.

Relates 
2018-12-16 16:48:04 -05:00
Martijn van Groningen 97107e99e8
Moved test to its rightful place. 2018-12-16 13:57:51 +01:00
Boaz Leskes 733a6d34c1
Add seq no powered optimistic locking support to the index and delete transport actions ()
This commit add support for using sequence numbers to power [optimistic concurrency control](http://en.wikipedia.org/wiki/Optimistic_concurrency_control) 
in the delete and index transport actions and requests. A follow up will come with adding sequence
numbers to the update and get results.

Relates  
Relates 
2018-12-15 17:59:57 +01:00
Albert Zaharovits a30e8c2fa3
HasPrivilegesResponse use TreeSet for fields ()
For class fields of type collection whose order is not important 
and for which duplicates are not permitted we declare them as `Set`s.
Usually the definition is a `HashSet` but in this case `TreeSet` is used
instead to aid testing.
2018-12-15 08:34:54 +02:00
Martijn van Groningen 68a674ef1f
[CCR] Fix follow stats API's follower index filtering feature ()
Currently always all follow stats for all follower indices are being
returned even if follow stats for only specific indices are requested.
2018-12-14 19:39:30 +01:00
Armin Braun c5b3ac5578
SNAPSHOTS: Allow Parallel Restore Operations ()
* Enable parallel restore operations
* Add uuid to restore in progress entries to uniquely identify them
* Adjust restore in progress entries to be a map in cluster state
* Added tests for:
   * Parallel restore from two different snapshots
   * Parallel restore from a single snapshot to different indices to test uuid identifiers are correctly used by `RestoreService` and routing allocator
   * Parallel restore with waiting for completion to test transport actions correctly use uuid identifiers
2018-12-14 11:39:23 +01:00
Nhat Nguyen a4b32f1143
Remove concurrency in testFailLeaderReplicaShard ()
testFailLeaderReplicaShard periodically fails because we concurrently
index to the leader group and close one of its replicas. If a
replication request hits a closing shard, we will fail that shard;
however, failing a shard is supported by the test framework - this makes
the test fail.
2018-12-13 19:02:13 -05:00
Boaz Leskes f6b5d7e013
Add sequence numbers based optimistic concurrency control support to Engine ()
This commit add support to engine operations for resolving and verifying the sequence number and
primary term of the last modification to a document before performing an operation. This is
infrastructure to move our (optimistic concurrency control)[http://en.wikipedia.org/wiki/Optimistic_concurrency_control] API to use sequence numbers instead of internal versioning.

Relates  
Relates 
2018-12-13 08:08:40 +01:00
Martijn van Groningen 883940ad92
[CCR] Change AutofollowCoordinator to use wait_for_metadata_version ()
Changed AutofollowCoordinator makes use of the wait_for_metadata_version
feature in cluster state API and removed hard coded poll interval.

Originates from 
Relates to 
2018-12-12 12:47:24 +01:00
Martijn van Groningen 4a825e2e86
[CCR] Clean followed leader index UUIDs in auto follow metadata ()
The auto follow coordinator keeps track of the UUIDs of indices that it has followed. The index UUID strings need to be cleaned up in the case that these indices are removed in the remote cluster.

Relates to 
2018-12-12 09:55:37 +01:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later ()
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates 

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
Nhat Nguyen f23701406b CCR/TEST: Enable soft-deletes in ShardChangesActionTests
Relates 
2018-12-11 15:00:09 -05:00
Andrey Ershov 8b821706cc
Switch more tests to zen2 ()
1. CCR tests work without any changes
2. `testDanglingIndices` require changes the source code (added TODO).
3. `testIndexDeletionWhenNodeRejoins` because it's using just two
nodes, adding the node to exclusions is needed on restart.
4. `testCorruptTranslogTruncationOfReplica` starts dedicated master
one, because otherwise, the cluster does not form, if nodes are stopped
and one node is started back.
5. `testResolvePath` needs TEST cluster, because all nodes are stopped
at the end of the test and it's not possible to perform checks needed
by SUITE cluster.
6. `SnapshotDisruptionIT`. Without changes, the test fails because Zen2
retries snapshot creation as soon as network partition heals. This
results into the race between creating snapshot and test cleanup logic
(deleting index). Zen1 on the
other hand, also schedules retry, but it takes some time after network
partition heals, so cleanup logic executes latter and test passes. The
check that snapshot is eventually created is added to
the end of the test.
2018-12-11 17:12:17 +01:00
Martijn van Groningen 633ab24017
[CCR] Restructured QA modules ()
Renamed the follow qa modules:
`multi-cluster-downgraded-to-basic-license` to `downgraded-to-basic-license`
`multi-cluster-with-non-compliant-license` to `non-compliant-license`
`multi-cluster-with-security` to `security`

Moved the `chain` module into the `multi-cluster` module and
changed the `multi-cluster` to start 3 clusters.

Followup from 
2018-12-09 19:34:48 +01:00
Nhat Nguyen 95bafb0593 TEST: Always enable soft-deletes in ShardChangesTests 2018-12-09 02:57:13 -05:00
Tim Brooks 8a53f2b464
Implement basic `CcrRepository` restore ()
This is related to . It implements a basic restore functionality
for the CcrRepository. When the restore process is kicked off, it
configures the new index as expected for a follower index. This means
that the index has a different uuid, the version is not incremented, and
the Ccr metadata is installed.

When the restore shard method is called, an empty shard is initialized.
2018-12-07 15:27:04 -07:00
Nhat Nguyen f2df0a5be4
Remove LocalCheckpointTracker#resetCheckpoint ()
In , we added a new assertion to ensure that the
LocalCheckpointTracker is always consistent with Lucene index. However,
we reset LocalCheckpoinTracker in testDedupByPrimaryTerm cause this
assertion to be violated.

This commit removes resetCheckpoint from LocalCheckpointTracker and
rewrites testDedupByPrimaryTerm without resetting the local checkpoint.

Relates 
2018-12-07 12:22:20 -05:00
Ryan Ernst 37b3fc383f
Build: Use explicit deps on test tasks for check ()
This commit moves back to use explicit dependsOn for test tasks on
check. Not all tasks extending RandomizedTestingTask should be run by
check directly.
2018-12-06 14:13:49 -08:00
Yannick Welsch a0ae1cc987 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 23:13:12 +01:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response ()
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates 
2018-12-05 19:49:06 +01:00
Yannick Welsch cc11953724 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 16:55:45 +01:00
Tim Brooks 068c856e88
Rename internal repository actions to be internal ()
This is a follow-up to . It renames the internal repository
actions to be prefixed by "internal". This allows the system user to
execute the actions.

Additionally, this PR stops casting Client to NodeClient. The client we
have is a NodeClient so executing the actions will be local.
2018-12-05 08:11:47 -07:00
Yannick Welsch b20497560c Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 14:06:38 +01:00
Martijn van Groningen a264cb6ddb
Refactor AutoFollowCoordinator to track leader indices per remote cluster ()
and replaced poll interval setting with a hardcoded poll interval.
The hard coded interval will be removed in a follow up change to make
use of cluster state API's wait_for_metatdata_version.

Before the auto following was bootstrapped from thread pool scheduler,
but now auto followers for new remote clusters are bootstrapped when
a new cluster state is published.

Originates from 
Relates to 
2018-12-05 13:39:14 +01:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 ()
Closes 

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Martijn van Groningen 11935cd480
Replace Streamable w/ Writeable in BaseTasksResponse and subclasses ()
This commit replaces usages of Streamable with Writeable for the
BaseTasksResponse / TransportTasksAction classes and subclasses of
these classes.

Note that where possible response fields were made final.

Relates to 
2018-12-05 13:14:10 +01:00
Yannick Welsch 42457b5960 Merge remote-tracking branch 'elastic/master' into zen2 2018-12-05 11:39:38 +01:00
Martijn van Groningen b9707c29a1
[CCR] Change get autofollow patterns API response format ()
The current response format is:

```
{
    "pattern1": {
        ...
    },
    "pattern2": {
        ...
    }
}
```

The new format is:

```
{
    "patterns": [
        {
            "name": "pattern1",
            "pattern": {
                ...
            }
        },
        {
            "name": "pattern2",
            "pattern": {
                ...
            }
        }
    ]
}
```

This format is more structured and more friendly for parsing and generating specs.
This is a breaking change, but it is better to do this now while ccr
is still a beta feature than later.

Follow up from 
2018-12-05 08:41:27 +01:00