mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-06 21:18:31 +00:00
fadbe0de08
Today we require users to prepare their indices for split operations. Yet, we can do this automatically when an index is created which would make the split feature a much more appealing option since it doesn't have any 3rd party prerequisites anymore. This change automatically sets the number of routinng shards such that an index is guaranteed to be able to split once into twice as many shards. The number of routing shards is scaled towards the default shard limit per index such that indices with a smaller amount of shards can be split more often than larger ones. For instance an index with 1 or 2 shards can be split 10x (until it approaches 1024 shards) while an index created with 128 shards can only be split 3x by a factor of 2. Please note this is just a default value and users can still prepare their indices with `index.number_of_routing_shards` for custom splitting. NOTE: this change has an impact on the document distribution since we are changing the hash space. Documents are still uniformly distributed across all shards but since we are artificually changing the number of buckets in the consistent hashign space document might be hashed into different shards compared to previous versions. This is a 7.0 only change.
174 lines
6.0 KiB
Plaintext
174 lines
6.0 KiB
Plaintext
[[indices-split-index]]
|
|
== Split Index
|
|
|
|
The split index API allows you to split an existing index into a new index,
|
|
where each original primary shard is split into two or more primary shards in
|
|
the new index.
|
|
|
|
The number of times the index can be split (and the number of shards that each
|
|
original shard can be split into) is determined by the
|
|
`index.number_of_routing_shards` setting. The number of routing shards
|
|
specifies the hashing space that is used internally to distribute documents
|
|
across shards with consistent hashing. For instance, a 5 shard index with
|
|
`number_of_routing_shards` set to `30` (`5 x 2 x 3`) could be split by a
|
|
factor of `2` or `3`. In other words, it could be split as follows:
|
|
|
|
* `5` -> `10` -> `30` (split by 2, then by 3)
|
|
* `5` -> `15` -> `30` (split by 3, then by 2)
|
|
* `5` -> `30` (split by 6)
|
|
|
|
While you can set the `index.number_of_routing_shards` setting explicitly at
|
|
index creation time, the default value depends upon the number of primary
|
|
shards in the original index. The default is designed to allow you to split
|
|
by factors of 2 up to a maximum of 1024 shards. However, the original number
|
|
of primary shards must taken into account. For instance, an index created
|
|
with 5 primary shards could be split into 10, 20, 40, 80, 160, 320, or a
|
|
maximum of 740 shards (with a single split action or multiple split actions).
|
|
|
|
If the original index contains one primary shard (or a multi-shard index has
|
|
been <<indices-shrink-index,shrunk>> down to a single primary shard), then the
|
|
index may by split into an arbitrary number of shards greater than 1. The
|
|
properties of the default number of routing shards will then apply to the
|
|
newly split index.
|
|
|
|
|
|
Splitting works as follows:
|
|
|
|
* First, it creates a new target index with the same definition as the source
|
|
index, but with a larger number of primary shards.
|
|
|
|
* Then it hard-links segments from the source index into the target index. (If
|
|
the file system doesn't support hard-linking, then all segments are copied
|
|
into the new index, which is a much more time consuming process.)
|
|
|
|
* Once the low level files are created all documents will be `hashed` again to delete
|
|
documents that belong to a different shard.
|
|
|
|
* Finally, it recovers the target index as though it were a closed index which
|
|
had just been re-opened.
|
|
|
|
[float]
|
|
=== Preparing an index for splitting
|
|
|
|
Create a new index:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT my_source_index
|
|
{
|
|
"settings": {
|
|
"index.number_of_shards" : 1
|
|
}
|
|
}
|
|
-------------------------------------------------
|
|
// CONSOLE
|
|
|
|
In order to split an index, the index must be marked as read-only,
|
|
and have <<cluster-health,health>> `green`.
|
|
|
|
This can be achieved with the following request:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /my_source_index/_settings
|
|
{
|
|
"settings": {
|
|
"index.blocks.write": true <1>
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
<1> Prevents write operations to this index while still allowing metadata
|
|
changes like deleting the index.
|
|
|
|
[float]
|
|
=== Splitting an index
|
|
|
|
To split `my_source_index` into a new index called `my_target_index`, issue
|
|
the following request:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST my_source_index/_split/my_target_index
|
|
{
|
|
"settings": {
|
|
"index.number_of_shards": 2
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
The above request returns immediately once the target index has been added to
|
|
the cluster state -- it doesn't wait for the split operation to start.
|
|
|
|
[IMPORTANT]
|
|
=====================================
|
|
|
|
Indices can only be split if they satisfy the following requirements:
|
|
|
|
* the target index must not exist
|
|
|
|
* The source index must have fewer primary shards than the target index.
|
|
|
|
* The number of primary shards in the target index must be a factor of the
|
|
number of primary shards in the source index.
|
|
|
|
* The node handling the split process must have sufficient free disk space to
|
|
accommodate a second copy of the existing index.
|
|
|
|
=====================================
|
|
|
|
The `_split` API is similar to the <<indices-create-index, `create index` API>>
|
|
and accepts `settings` and `aliases` parameters for the target index:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST my_source_index/_split/my_target_index
|
|
{
|
|
"settings": {
|
|
"index.number_of_shards": 5 <1>
|
|
},
|
|
"aliases": {
|
|
"my_search_indices": {}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/^/PUT my_source_index\n{"settings": {"index.blocks.write": true, "index.number_of_shards": "1"}}\n/]
|
|
|
|
<1> The number of shards in the target index. This must be a factor of the
|
|
number of shards in the source index.
|
|
|
|
|
|
NOTE: Mappings may not be specified in the `_split` request, and all
|
|
`index.analysis.*` and `index.similarity.*` settings will be overwritten with
|
|
the settings from the source index.
|
|
|
|
[float]
|
|
=== Monitoring the split process
|
|
|
|
The split process can be monitored with the <<cat-recovery,`_cat recovery`
|
|
API>>, or the <<cluster-health, `cluster health` API>> can be used to wait
|
|
until all primary shards have been allocated by setting the `wait_for_status`
|
|
parameter to `yellow`.
|
|
|
|
The `_split` API returns as soon as the target index has been added to the
|
|
cluster state, before any shards have been allocated. At this point, all
|
|
shards are in the state `unassigned`. If, for any reason, the target index
|
|
can't be allocated, its primary shard will remain `unassigned` until it
|
|
can be allocated on that node.
|
|
|
|
Once the primary shard is allocated, it moves to state `initializing`, and the
|
|
split process begins. When the split operation completes, the shard will
|
|
become `active`. At that point, Elasticsearch will try to allocate any
|
|
replicas and may decide to relocate the primary shard to another node.
|
|
|
|
[float]
|
|
=== Wait For Active Shards
|
|
|
|
Because the split operation creates a new index to split the shards to,
|
|
the <<create-index-wait-for-active-shards,wait for active shards>> setting
|
|
on index creation applies to the split index action as well. |