2019-01-30 04:49:42 -05:00
|
|
|
[testenv="platinum"]
|
Allow follower indices to override leader settings (#58103)
Today when creating a follower index via the put follow API, or via an
auto-follow pattern, it is not possible to specify settings overrides
for the follower index. Instead, we copy all of the leader index
settings to the follower. Yet, there are cases where a user would want
some different settings on the follower index such as the number of
replicas, or allocation settings. This commit addresses this by allowing
the user to specify settings overrides when creating follower index via
manual put follower calls, or via auto-follow patterns. Note that not
all settings can be overrode (e.g., index.number_of_shards) so we also
have detection that prevents attempting to override settings that must
be equal between the leader and follow index. Note that we do not even
allow specifying such settings in the overrides, even if they are
specified to be equal between the leader and the follower
index. Instead, the must be implicitly copied from the leader index, not
explicitly set by the user.
2020-06-18 10:55:17 -04:00
|
|
|
`settings`::
|
|
|
|
(object) Settings to override from the leader index. Note that certain
|
|
|
|
settings can not be overrode (e.g., `index.number_of_shards`).
|
|
|
|
|
2018-10-26 11:23:35 -04:00
|
|
|
`max_read_request_operation_count`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(integer) The maximum number of operations to pull per read from the remote
|
|
|
|
cluster.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_outstanding_read_requests`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(long) The maximum number of outstanding reads requests from the remote
|
|
|
|
cluster.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_read_request_size`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(<<byte-units,byte value>>) The maximum size in bytes of per read of a batch
|
|
|
|
of operations pulled from the remote cluster.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_write_request_operation_count`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(integer) The maximum number of operations per bulk write request executed on
|
|
|
|
the follower.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_write_request_size`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(<<byte-units,byte value>>) The maximum total bytes of operations per bulk write request
|
|
|
|
executed on the follower.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_outstanding_write_requests`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(integer) The maximum number of outstanding write requests on the follower.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_write_buffer_count`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(integer) The maximum number of operations that can be queued for writing.
|
|
|
|
When this limit is reached, reads from the remote cluster will be deferred
|
|
|
|
until the number of queued operations goes below the limit.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_write_buffer_size`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(<<byte-units,byte value>>) The maximum total bytes of operations that can be
|
|
|
|
queued for
|
|
|
|
writing. When this limit is reached, reads from the remote cluster will be
|
|
|
|
deferred until the total bytes of queued operations goes below the limit.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`max_retry_delay`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(<<time-units,time value>>) The maximum time to wait before retrying an
|
|
|
|
operation that failed exceptionally. An exponential backoff strategy is
|
|
|
|
employed when retrying.
|
2018-10-26 11:23:35 -04:00
|
|
|
|
|
|
|
`read_poll_timeout`::
|
2019-07-04 11:22:17 -04:00
|
|
|
(<<time-units,time value>>) The maximum time to wait for new operations on the
|
|
|
|
remote cluster when the follower index is synchronized with the leader index.
|
|
|
|
When the timeout has elapsed, the poll for operations will return to the
|
|
|
|
follower so that it can update some statistics. Then the follower will
|
|
|
|
immediately attempt to read from the leader again.
|
2019-01-30 04:49:42 -05:00
|
|
|
|
|
|
|
===== Default values
|
|
|
|
|
|
|
|
//////////////////////////
|
|
|
|
|
2019-09-09 13:38:14 -04:00
|
|
|
[source,console]
|
2019-01-30 04:49:42 -05:00
|
|
|
--------------------------------------------------
|
2019-01-30 05:02:16 -05:00
|
|
|
PUT /follower_index/_ccr/follow?wait_for_active_shards=1
|
2019-01-30 04:49:42 -05:00
|
|
|
{
|
|
|
|
"remote_cluster" : "remote_cluster",
|
|
|
|
"leader_index" : "leader_index"
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
// TESTSETUP
|
|
|
|
// TEST[setup:remote_cluster_and_leader_index]
|
|
|
|
|
2019-09-09 13:38:14 -04:00
|
|
|
[source,console]
|
2019-01-30 04:49:42 -05:00
|
|
|
--------------------------------------------------
|
|
|
|
POST /follower_index/_ccr/pause_follow
|
|
|
|
--------------------------------------------------
|
|
|
|
// TEARDOWN
|
|
|
|
|
2019-09-09 13:38:14 -04:00
|
|
|
[source,console]
|
2019-01-30 04:49:42 -05:00
|
|
|
--------------------------------------------------
|
|
|
|
GET /follower_index/_ccr/info?filter_path=follower_indices.parameters
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
//////////////////////////
|
|
|
|
|
|
|
|
The following output from the follow info api describes all the default
|
|
|
|
values for the above described index follow request parameters:
|
|
|
|
|
2019-09-06 09:22:08 -04:00
|
|
|
[source,console-result]
|
2019-01-30 04:49:42 -05:00
|
|
|
--------------------------------------------------
|
|
|
|
{
|
|
|
|
"follower_indices" : [
|
|
|
|
{
|
|
|
|
"parameters" : {
|
|
|
|
"max_read_request_operation_count" : 5120,
|
|
|
|
"max_read_request_size" : "32mb",
|
|
|
|
"max_outstanding_read_requests" : 12,
|
|
|
|
"max_write_request_operation_count" : 5120,
|
|
|
|
"max_write_request_size" : "9223372036854775807b",
|
|
|
|
"max_outstanding_write_requests" : 9,
|
|
|
|
"max_write_buffer_count" : 2147483647,
|
|
|
|
"max_write_buffer_size" : "512mb",
|
|
|
|
"max_retry_delay" : "500ms",
|
|
|
|
"read_poll_timeout" : "1m"
|
|
|
|
}
|
|
|
|
}
|
|
|
|
]
|
|
|
|
}
|
|
|
|
|
|
|
|
--------------------------------------------------
|