Commit Graph

675 Commits

Author SHA1 Message Date
Nhat Nguyen 8fefa8a661 Update InternalEngine tests on ccr side for #30121
Relates #30121
2018-05-04 10:57:54 -04:00
Nhat Nguyen d621fc7a00
Add tombstone document into Lucene for Noop (#30226)
This commit adds a tombstone document into Lucene for every No-op. 
With this change, Lucene index is expected to have a complete history 
of operations like Translog. In fact, this guarantee is subjected to the
soft-deletes retention merge policy.

Relates #29530
2018-05-02 09:08:29 -04:00
Nhat Nguyen eb4281edef CCR side #30244
Relates #30244
2018-05-01 21:08:24 -04:00
Martijn van Groningen 8a2df6c3b9
[CCR] Only normalize -1 seqno in shard changes request. (#30238)
Prior to this change a -1 seqno would be normalized earlier, which
caused a leader shard containing a single operation to be ignored.

Closes #30227
2018-05-01 08:40:23 +02:00
Martijn van Groningen e6b88fa5a0
removed duplicated license 2018-04-25 12:18:02 +02:00
Martijn van Groningen 5a67a0f78f
Applying changes required for ccr after moving ccr code to elasticsearch 2018-04-25 08:03:29 +02:00
Martijn van Groningen 9b9d0f9057 Enabled licence header check and fixed unchecked casts. (#4408) 2018-04-20 11:15:52 +02:00
Martijn van Groningen cfd7847628 fixed issues after merging in master 2018-04-20 07:59:13 +02:00
Nhat Nguyen f97aec7b8b Sibling of enforce access to translog via engine
Since elastic/elasticsearch#29542, we no longer expose translog instance
but only provide creating translog snapshot method. This commit adapts
that change in CCR branch.

Relates elastic/elasticsearch#29542
2018-04-18 11:54:00 -04:00
Martijn van Groningen 56ca59a513 Add the ability to the follow index to follow an index in a remote cluster.
The follow index api completely reuses CCS infrastructure that was exposed via:
https://github.com/elastic/elasticsearch/pull/29495

This means that the leader index parameter support the same ccs index
to indicate that an index resides in a different cluster.

I also added a qa module that smoke tests the cross cluster nature of ccr.
The idea is that this test just verifies that ccr can read data from a
remote leader index and that is it, no crazy randomization or indirectly
testing other features.
2018-04-17 07:36:40 +02:00
Martijn van Groningen c0d42e9cd1 Fixed test 2018-04-16 10:48:46 +02:00
Martijn van Groningen a94b38b88e Fixed compile errors and test failures after merging master into ccr. 2018-04-13 16:35:09 +02:00
Martijn van Groningen d77f756f5c ccr: use indices stats api to fetch global checkpoint of the follower shards and
keep track of shard follow stats inside shard follow stats' node task instead of persistent task status.

By maintaining the shard follow stats inside its node task the stats update is quicker as
no cluster state update is required. The stats are now transient; meaning if the task
is going to run a different node then the stats are gone too. Currently only the processed
global checkpoint is being tracked and this is being restored when a shard follow node task
starts via the indices stats api (the reason of the first change of this change). Other stats
that we may add in the future (like fetch_time, see: https://gist.github.com/s1monw/dba13daf8493bf48431b72365e110717)
it is ok if we start from zero in case a shard follow task moves to another node.
2018-04-05 14:52:20 +02:00
Martijn van Groningen d976fa44e7 Removed LocalCheckpointTracker usage. 2018-03-29 07:41:23 +02:00
Martijn van Groningen a22a7d079d ccr: Added maximum translog limit that a single shard changes response can return.
This limit is based on the number of estimate bytes in each translog
operation that fall between the minimum and maximum request sequence number.

If this limit is met then the shard follow task executor will make sure
that a subsequent shard changes request will be performed to fetch the
remaining translog operations.

This limit is needed in order to protect against returning too many
translog operations in a single shard changes response.

Relates to #2436
2018-03-28 15:49:57 +02:00
Martijn van Groningen 282740610b Fixed test after merging in master branch. 2018-03-28 09:54:41 +02:00
Nhat Nguyen 51111a8106 CCR: Stop FollowExistingIndexAction after report failure (#4111)
We check for the existence of both leader and follower index, then properly 
report to the caller. However, we do not return after reporting failure. This
causes the caller receive exception twice: IllegalArgumentException then
NullPointerException. This commit makes sure to stop the action after reporting
failure.
2018-03-26 13:56:47 -04:00
Martijn van Groningen 9e4c68c389 Fixed compile and test errors after merging in master 2018-03-16 17:47:10 +01:00
Martijn van Groningen 10cfa21a68 required changes after merge master branch into ccr branch. 2018-02-22 15:03:33 +01:00
Martijn van Groningen 1a9a7ffe97 removed hack 2018-02-07 17:54:28 +01:00
Martijn van Groningen c442d14f1d Several changes that were required after merging master into the ccr branch. 2018-02-05 13:25:58 +01:00
Martijn van Groningen 4e818254ad re-enabled java integration tests 2018-01-25 14:18:34 +01:00
Martijn van Groningen 05d3d2e49c fix packages after merge 2018-01-24 09:28:42 +01:00
Jason Tedor 9b6bb2c635 Enable run task for CCR
This commit enables the run task for ccr by specifying that the ccr
project not be evaluated until after core is evaluated. This is
important since ccr is alphabetically before core and thus Gradle
evaluates it first.

Relates #3665
2018-01-22 15:07:20 -05:00
Martijn van Groningen 83a82d83d0 Moved ccr source code to its own gradle module after xpack split. 2018-01-22 11:09:04 +01:00