Introduce CCR overview (#35436)

This commit introduces a basic overview for cross-cluster replication to
the docs.

Co-authored-by: "lcawl <lcawley@elastic.co>"

docs/reference/ccr/auto-follow.asciidoc

@@ -0,0 +1,32 @@
+[role="xpack"]
+[testenv="platinum"]
+[[ccr-overview-auto-follow]]
+=== Automatically following indices
+
+beta[]
+
+In time series use cases where you want to follow new indices that are
+periodically created (such as daily Beats indices), manually configuring follower
+indices for each new leader index can be an operational burden. The auto-follow
+functionality in {ccr} is aimed at easing this burden. With the auto-follow
+functionality, you can specify that new indices in a remote cluster that have a 
+name that matches a pattern are automatically followed.
+
+==== Managing auto-follow patterns
+
+You can add a new auto-follow pattern configuration with the
+{ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API].  When you create
+a new auto-follow pattern configuration, you are configuring a collection of
+patterns against a single remote cluster. Any time a new index with a name that 
+matches one of the patterns in the collection is created in the remote cluster,
+a follower index is configured in the local cluster. The follower index uses the 
+new index as its leader index.
+
+You can inspect all configured auto-follow pattern collections with the
+{ref}/ccr-get-auto-follow-pattern.html[get auto-follow pattern API]. To delete a
+configured auto-follow pattern collection, use the
+{ref}ccr-delete-auto-follow-pattern.html[delete auto-follow pattern API].
+
+Since auto-follow functionality is handled automatically in the background on
+your behalf, error reporting is done through logs on the elected master node
+and through the {ref}/ccr-get-stats.html[{ccr} stats API].

docs/reference/ccr/index.asciidoc

@@ -8,10 +8,23 @@
 
 beta[]
 
-* <<ccr-overview, Overview>>
+The {ccr} (CCR) feature enables replication of indices in remote clusters to a 
-* <<ccr-getting-started,Getting Started>>
+local cluster. This functionality can be used in some common production use cases:
+
+* Disaster recovery in case a primary cluster fails. A secondary cluster can
+  serve as a hot backup
+* Geo-proximity so that reads can be served locally
+
+This guide provides an overview of {ccr}:
+
+* <<ccr-overview>>
+* <<ccr-requirements>>
+* <<ccr-overview-auto-follow>>
+* <<ccr-getting-started>>
 
 --
 
 include::overview.asciidoc[]
+include::requirements.asciidoc[]
+include::auto-follow.asciidoc[]
 include::getting-started.asciidoc[]

docs/reference/ccr/overview.asciidoc

@@ -4,4 +4,109 @@
 == Overview
 
 beta[]
-This is the overview section of the {ccr} docs.
+
+Cross-cluster replication is done on an index-by-index basis. Replication is
+configured at the index level. For each configured replication there is a
+replication source index called the _leader index_ and a replication target
+index called the _follower index_.
+
+Replication is active-passive. This means that while the leader index
+can directly be written into, the follower index can not directly receive
+writes.
+
+Replication is pull-based. This means that replication is driven by the
+follower index. This simplifies state management on the leader index and means
+that {ccr} does not interfere with indexing on the leader index.
+
+[float]
+=== Configuring replication
+
+Replication can be configured in two ways:
+
+* Manually using the
+  {ref}/ccr-put-follow.html[create follower API]
+
+* Automatically using
+  <<ccr-overview-auto-follow,auto-follow patterns>>
+
+NOTE: You must also <<ccr-requirements,configure the leader index>>.
+
+[float]
+=== The mechanics of replication
+
+While replication is managed at the index level, replication is performed at the
+shard level. When a follower index is created, it is automatically
+configured to have an identical number of shards as the leader index. A follower
+shard task in the follower index pulls from the corresponding leader shard in
+the leader index by sending read requests for new operations. These read
+requests can be served from any copy of the leader shard (primary or replicas).
+
+For each read request sent by the follower shard task, if there are new
+operations available on the leader shard, the leader shard responds with
+operations limited by the read parameters that you established when you
+configured the follower index. If there are no new operations available on the
+leader shard, the leader shard waits up to a configured timeout for new
+operations. If new operations occur within that timeout, the leader shard
+immediately responds with those new operations. Otherwise, if the timeout
+elapses, the follower shard replies that there are no new operations. The
+follower shard task updates some statistics and immediately sends another read
+request to the leader shard. This ensures that the network connections between
+the remote cluster and the local cluster are continually being used so as to
+avoid forceful termination by an external source (such as a firewall).
+
+If a read request fails, the cause of the failure is inspected. If the
+cause of the failure is deemed to be a failure that can be recovered from (for 
+example, a network failure), the follower shard task enters into a retry
+loop. Otherwise, the follower shard task is paused and requires user
+intervention before the it can be resumed with the
+{ref}/ccr-post-resume-follow.html[resume follower API].
+
+When operations are received by the follower shard task, they are placed in a
+write buffer. The follower shard task manages this write buffer and submits
+bulk write requests from this write buffer to the follower shard.  The write
+buffer and these write requests are managed by the write parameters that you 
+established when you configured the follower index.  The write buffer serves as
+back-pressure against read requests. If the write buffer exceeds its configured
+limits, no additional read requests are sent by the follower shard task. The
+follower shard task resumes sending read requests when the write buffer no
+longer exceeds its configured limits.
+
+Mapping updates applied to the leader index are automatically retrieved
+as-needed by the follower index.
+
+Settings updates applied to the leader index that are needed by the follower
+index are automatically retried as-needed by the follower index. Not all
+settings updates are needed by the follower index. For example, changing the
+number of replicas on the leader index is not replicated by the follower index.
+
+NOTE: If you apply a non-dynamic settings change to the leader index that is
+needed by the follower index, the follower index will go through a cycle of
+closing itself, applying the settings update, and then re-opening itself. The
+follower index will be unavailable for reads and not replicating writes
+during this cycle.
+
+[float]
+=== Inspecting the progress of replication
+
+You can inspect the progress of replication at the shard level with the
+{ref}/ccr-get-follow-stats.html[get follower stats API]. This API gives you
+insight into the read and writes managed by the follower shard task. It also
+reports read exceptions that can be retried and fatal exceptions that require
+user intervention.
+
+[float]
+=== Pausing and resuming replication
+
+You can pause replication with the
+{ref}/ccr-post-pause-follow.html[pause follower API] and then later resume
+replication with the {ref}/ccr-post-resume-follow.html[resume follower API].
+Using these APIs in tandem enables you to adjust the read and write parameters
+on the follower shard task if your initial configuration is not suitable for
+your use case.
+
+[float]
+=== Terminating replication
+
+You can terminate replication with the
+{ref}/ccr-post-unfollow.html[unfollow API]. This API converts a follower index
+to a regular (non-follower) index.

docs/reference/ccr/requirements.asciidoc

@@ -0,0 +1,45 @@
+[role="xpack"]
+[testenv="platinum"]
+[[ccr-requirements]]
+=== Requirements for leader indices
+
+beta[]
+
+Cross-cluster replication works by replaying the history of individual write
+operations that were performed on the shards of the leader index. This means that the
+history of these operations needs to be retained on the leader shards so that
+they can be pulled by the follower shard tasks. The underlying mechanism used to
+retain these operations is _soft deletes_. A soft delete occurs whenever an
+existing document is deleted or updated. By retaining these soft deletes up to
+configurable limits, the history of operations can be retained on the leader
+shards and made available to the follower shard tasks as it replays the history
+of operations.
+
+Soft deletes must be enabled for indices that you want to use as leader
+indices. Enabling soft deletes requires the addition of some index settings at
+index creation time. You must add these settings to your create index
+requests or to the index templates that you use to manage the creation of new
+indices.
+
+IMPORTANT: This means that {ccr} can not be used on existing indices. If you have
+existing data that you want to replicate from another cluster, you must
+{ref}/docs-reindex.html[reindex] your data into a new index with soft deletes 
+enabled.
+
+[float]
+[[ccr-overview-soft-deletes]]
+==== Soft delete settings
+
+`index.soft_deletes.enabled`::
+
+Whether or not soft deletes are enabled on the index. Soft deletes can only be
+configured at index creation and only on indices created on or after 6.5.0. The
+default value is `false`.
+
+`index.soft_deletes.retention.operations`::
+
+The number of soft deletes to retain. Soft deletes are collected during merges
+on the underlying Lucene index yet retained up to the number of operations
+configured by this setting. The default value is `0`.
+
+For more information about index settings, see {ref}/index-modules.html[Index modules].