mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-23 21:38:15 +00:00
James Rodewig
GitHub
parent
e3469bb6e2
commit
d66084dcaf
@ -1,8 +1,5 @@
|
||||
[[data-streams-change-mappings-and-settings]]
|
||||
== Change mappings and settings of backing indices
|
||||
++++
|
||||
<titleabbrev>Change mappings and settings</titleabbrev>
|
||||
++++
|
||||
== Change mappings and settings for a data stream
|
||||
|
||||
Each data stream has a <<create-a-data-stream-template,matching index
|
||||
template>>. Mappings and index settings from this template are applied to new
|
||||
@ -10,26 +7,22 @@ backing indices created for the stream. This includes the stream's first
|
||||
backing index, which is auto-generated when the stream is created.
|
||||
|
||||
Before creating a data stream, we recommend you carefully consider which
|
||||
mappings and settings to include in this template. However, if you later need to
|
||||
change the mappings or settings of a data stream's backing indices, you have a
|
||||
couple options:
|
||||
mappings and settings to include in this template.
|
||||
|
||||
* To apply changes to future backing indices, simply update the index
|
||||
template used by the data stream. Mapping and setting changes will be
|
||||
automatically applied to any backing indices created after the update.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
`logs_data_stream` is an existing index template used by the
|
||||
`logs` data stream.
|
||||
If you later need to change the mappings or settings for a data stream, you have
|
||||
a few options:
|
||||
|
||||
The following <<indices-templates,put index template API>> makes several
|
||||
changes to the `logs_data_stream` template:
|
||||
* <<add-new-field-mapping-to-a-data-stream>>
|
||||
* <<change-existing-field-mapping-in-a-data-stream>>
|
||||
* <<change-dynamic-index-setting-for-a-data-stream>>
|
||||
* <<change-static-index-setting-for-a-data-stream>>
|
||||
|
||||
* It changes the `@timestamp` field mapping from the `date` field datatype to
|
||||
the `date_nanos` datatype.
|
||||
* It adds new `sort.field` and `sort.order` index settings.
|
||||
TIP: If your changes include modifications to existing field mappings or
|
||||
<<index-modules-settings,static index settings>>, a reindex is often required to
|
||||
apply the changes to a data stream's backing indices. If you are already
|
||||
performing a reindex, you can use the same process to add new field
|
||||
mappings and change <<index-modules-settings,dynamic index settings>>. See
|
||||
<<data-streams-use-reindex-to-change-mappings-settings>>.
|
||||
|
||||
////
|
||||
[source,console]
|
||||
@ -68,23 +61,61 @@ PUT /_index_template/logs_data_stream
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"index.lifecycle.name": "logs_policy"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
PUT /logs/_bulk?refresh
|
||||
{"create":{"_index" : "logs"}}
|
||||
{ "@timestamp": "2020-12-08T11:04:05.000Z" }
|
||||
{"create":{"_index" : "logs"}}
|
||||
{ "@timestamp": "2020-12-08T11:06:07.000Z" }
|
||||
{"create":{"_index" : "logs"}}
|
||||
{ "@timestamp": "2020-12-09T11:07:08.000Z" }
|
||||
PUT /_index_template/new_logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "new_logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
PUT /_data_stream/logs
|
||||
|
||||
PUT /_data_stream/new_logs
|
||||
----
|
||||
// TESTSETUP
|
||||
|
||||
[source,console]
|
||||
----
|
||||
DELETE /_data_stream/*
|
||||
|
||||
DELETE /_index_template/*
|
||||
|
||||
DELETE /_ilm/policy/logs_policy
|
||||
----
|
||||
// TEARDOWN
|
||||
////
|
||||
|
||||
[discrete]
|
||||
[[add-new-field-mapping-to-a-data-stream]]
|
||||
=== Add a new field mapping to a data stream
|
||||
|
||||
To add a mapping for a new field to a data stream, following these steps:
|
||||
|
||||
. Update the index template used by the data stream. This ensures the new
|
||||
field mapping is added to future backing indices created for the stream.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
`logs_data_stream` is an existing index template used by the `logs` data stream.
|
||||
|
||||
The following <<indices-templates,put index template>> request adds a mapping
|
||||
for a new field, `message`, to the template.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /_index_template/logs_data_stream
|
||||
@ -97,62 +128,265 @@ PUT /_index_template/logs_data_stream
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date_nanos" <1>
|
||||
"type": "date"
|
||||
},
|
||||
"message": { <1>
|
||||
"type": "text"
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"index.lifecycle.name": "logs_policy",
|
||||
"sort.field" : [ "@timestamp"], <2>
|
||||
"sort.order" : [ "desc"] <3>
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
|
||||
<1> Changes the `@timestamp` field mapping to the `date_nanos` datatype.
|
||||
<2> Adds the `sort.field` index setting.
|
||||
<3> Adds the `sort.order` index setting.
|
||||
<1> Adds a mapping for the new `message` field.
|
||||
====
|
||||
+
|
||||
If wanted, you can <<manually-roll-over-a-data-stream,roll over the data
|
||||
stream>> to immediately apply the new mappings and settings to the data stream's
|
||||
write index. This affects any new data added to the stream after the rollover.
|
||||
|
||||
. Use the <<indices-put-mapping,put mapping API>> to add the new field mapping
|
||||
to the data stream. By default, this adds the mapping to the stream's existing
|
||||
backing indices, including the write index.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
The following <<indices-rollover-index,rollover API>> request rolls over the
|
||||
`logs` data stream. This creates a new write index with mappings and index
|
||||
settings from the recently updated `logs_data_stream` template.
|
||||
The following put mapping API request adds the new `message` field mapping to
|
||||
the `logs` data stream.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
POST /logs/_rollover/
|
||||
PUT /logs/_mapping
|
||||
{
|
||||
"properties": {
|
||||
"message": {
|
||||
"type": "text"
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
====
|
||||
|
||||
[discrete]
|
||||
[[change-existing-field-mapping-in-a-data-stream]]
|
||||
=== Change an existing field mapping in a data stream
|
||||
|
||||
The documentation for each <<mapping-params,mapping parameter>> indicates
|
||||
whether you can update it for an existing field using the
|
||||
<<indices-put-mapping,put mapping API>>. To update these parameters for an
|
||||
existing field, follow these steps:
|
||||
|
||||
. Update the index template used by the data stream. This ensures the updated
|
||||
field mapping is added to future backing indices created for the stream.
|
||||
+
|
||||
IMPORTANT: You cannot use these methods to change the mapping of a data stream's
|
||||
<<create-a-data-stream-template,timestamp field>>. To change the timestamp
|
||||
field's mapping, you must reindex the data stream. See
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
`logs_data_stream` is an existing index template used by the `logs` data stream.
|
||||
|
||||
The following <<indices-templates,put index template>> request changes the
|
||||
argument for the `host.ip` field's <<ignore-malformed,`ignore_malformed`>>
|
||||
mapping parameter to `true`.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /_index_template/logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
},
|
||||
"host": {
|
||||
"properties": {
|
||||
"ip": {
|
||||
"type": "ip",
|
||||
"ignore_malformed": true <1>
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
<1> Changes the `host.ip` field's `ignore_malformed` value to `true`.
|
||||
====
|
||||
|
||||
. Use the <<indices-put-mapping,put mapping API>> to apply the mapping changes
|
||||
to the data stream. By default, this applies the changes to the stream's
|
||||
existing backing indices, including the write index.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
The following <<indices-put-mapping,put mapping API>> request targets the `logs`
|
||||
data stream. The request changes the argument for the `host.ip` field's
|
||||
`ignore_malformed` mapping parameter to `true`.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /logs/_mapping
|
||||
{
|
||||
"properties": {
|
||||
"host": {
|
||||
"properties": {
|
||||
"ip": {
|
||||
"type": "ip",
|
||||
"ignore_malformed": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
====
|
||||
|
||||
Except for supported mapping parameters, we don't recommend you change the
|
||||
mapping or field datatype of existing fields, even in a data stream's matching
|
||||
index template or its backing indices. Changing the mapping of an existing
|
||||
field could invalidate any data that’s already indexed.
|
||||
|
||||
If you need to change the mapping of an existing field, create a new
|
||||
data stream and reindex your data into it. See
|
||||
<<data-streams-use-reindex-to-change-mappings-settings>>.
|
||||
|
||||
* To apply mapping and setting changes to all existing backing indices and
|
||||
future ones, you must create a new data stream and reindex your data into it.
|
||||
See <<data-streams-use-reindex-to-change-mappings-settings>>.
|
||||
[discrete]
|
||||
[[change-dynamic-index-setting-for-a-data-stream]]
|
||||
=== Change a dynamic index setting for a data stream
|
||||
|
||||
To change a <<index-modules-settings,dynamic index setting>> for a data stream,
|
||||
follow these steps:
|
||||
|
||||
. Update the index template used by the data stream. This ensures the setting is
|
||||
applied to future backing indices created for the stream.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
`logs_data_stream` is an existing index template used by the `logs` data stream.
|
||||
|
||||
The following <<indices-templates,put index template>> request changes the
|
||||
template's `index.refresh_interval` index setting to `30s` (30 seconds).
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /_index_template/logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"index.refresh_interval": "30s" <1>
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
<1> Changes the `index.refresh_interval` setting to `30s` (30 seconds).
|
||||
====
|
||||
|
||||
. Use the <<indices-update-settings,update index settings API>> to update the
|
||||
index setting for the data stream. By default, this applies the setting to
|
||||
the stream's existing backing indices, including the write index.
|
||||
+
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
The following update index settings API request updates the
|
||||
`index.refresh_interval` setting for the `logs` data stream.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /logs/_settings
|
||||
{
|
||||
"index": {
|
||||
"refresh_interval": "30s"
|
||||
}
|
||||
}
|
||||
----
|
||||
====
|
||||
|
||||
[discrete]
|
||||
[[change-static-index-setting-for-a-data-stream]]
|
||||
=== Change a static index setting for a data stream
|
||||
|
||||
<<index-modules-settings,Static index settings>> can only be set when a backing
|
||||
index is created. You cannot update static index settings using the
|
||||
<<indices-update-settings,update index settings API>>.
|
||||
|
||||
To apply a new static setting to future backing indices, update the index
|
||||
template used by the data stream. The setting is automatically applied to any
|
||||
backing index created after the update.
|
||||
|
||||
.*Example*
|
||||
[%collapsible]
|
||||
====
|
||||
`logs_data_stream` is an existing index template used by the `logs` data stream.
|
||||
|
||||
The following <<indices-templates,put index template API>> requests adds new
|
||||
`sort.field` and `sort.order index` settings to the template.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT /_index_template/logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"sort.field": [ "@timestamp"], <1>
|
||||
"sort.order": [ "desc"] <2>
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
<1> Adds the `sort.field` index setting.
|
||||
<2> Adds the `sort.order` index setting.
|
||||
====
|
||||
|
||||
If wanted, you can <<manually-roll-over-a-data-stream,roll over the data
|
||||
stream>> to immediately apply the setting to the data stream’s write index. This
|
||||
affects any new data added to the stream after the rollover. However, it does
|
||||
not affect the data stream's existing backing indices or existing data.
|
||||
|
||||
To apply static setting changes to existing backing indices, you must create a
|
||||
new data stream and reindex your data into it. See
|
||||
<<data-streams-use-reindex-to-change-mappings-settings>>.
|
||||
|
||||
[discrete]
|
||||
[[data-streams-use-reindex-to-change-mappings-settings]]
|
||||
=== Use reindex to change mappings or settings
|
||||
|
||||
To change the mappings or settings for every backing index in a data stream, you
|
||||
must first create or update an index template so that it contains the
|
||||
changes. You can then reindex the existing data stream into a new one matching
|
||||
the template. This applies the mapping and setting changes in the template
|
||||
to each document and backing index of the data stream destination. These changes
|
||||
also affect any future backing index created by the new stream.
|
||||
You can use a reindex to change the mappings or settings of a data stream. This
|
||||
is often required to change the datatype of an existing field or update static
|
||||
index settings for backing indices.
|
||||
|
||||
To reindex a data stream, first create or update an index template so that it
|
||||
contains the wanted mapping or setting changes. You can then reindex the
|
||||
existing data stream into a new stream matching the template. This applies the
|
||||
mapping and setting changes in the template to each document and backing index
|
||||
added to the new data stream. These changes also affect any future backing
|
||||
index created by the new stream.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
@ -174,7 +408,6 @@ wildcard pattern can be used to create a new data stream.
|
||||
----
|
||||
GET /_resolve/index/new_logs*
|
||||
----
|
||||
// TEST[continued]
|
||||
|
||||
The API returns the following response, indicating no existing targets match
|
||||
this pattern.
|
||||
@ -182,11 +415,12 @@ this pattern.
|
||||
[source,console-result]
|
||||
----
|
||||
{
|
||||
"indices" : [ ],
|
||||
"aliases" : [ ],
|
||||
"data_streams" : [ ]
|
||||
"indices": [ ],
|
||||
"aliases": [ ],
|
||||
"data_streams": [ ]
|
||||
}
|
||||
----
|
||||
// TESTRESPONSE[s/"data_streams": \[ \]/"data_streams": $body.data_streams/]
|
||||
====
|
||||
|
||||
. Create or update an index template. This template should contain the
|
||||
@ -235,15 +469,12 @@ PUT /_index_template/new_logs_data_stream
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"index.lifecycle.name": "logs_policy",
|
||||
"sort.field" : [ "@timestamp"], <2>
|
||||
"sort.order" : [ "desc"] <3>
|
||||
"sort.field": [ "@timestamp"], <2>
|
||||
"sort.order": [ "desc"] <3>
|
||||
}
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
|
||||
<1> Changes the `@timestamp` field mapping to the `date_nanos` field datatype.
|
||||
<2> Adds the `sort.field` index setting.
|
||||
<3> Adds the `sort.order` index setting.
|
||||
@ -280,7 +511,7 @@ existing index or data stream uses this name, this request creates the
|
||||
----
|
||||
PUT /_data_stream/new_logs
|
||||
----
|
||||
// TEST[continued]
|
||||
// TEST[s/new_logs/new_logs_two/]
|
||||
====
|
||||
|
||||
. If you do not want to mix new and old data in your new data stream, pause the
|
||||
@ -308,23 +539,6 @@ PUT /_cluster/settings
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
|
||||
////
|
||||
[source,console]
|
||||
----
|
||||
DELETE /_data_stream/logs
|
||||
|
||||
DELETE /_data_stream/new_logs
|
||||
|
||||
DELETE /_index_template/logs_data_stream
|
||||
|
||||
DELETE /_index_template/new_logs_data_stream
|
||||
|
||||
DELETE /_ilm/policy/logs_policy
|
||||
----
|
||||
// TEST[continued]
|
||||
////
|
||||
====
|
||||
|
||||
. Reindex your data to the new data stream using an `op_type` of `create`.
|
||||
@ -383,49 +597,6 @@ The following <<docs-reindex,reindex API>> request copies documents from
|
||||
`.ds-logs-000001` to the `new_logs` data stream. Note the request's `op_type` is
|
||||
`create`.
|
||||
|
||||
////
|
||||
[source,console]
|
||||
----
|
||||
PUT /_index_template/logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
PUT /_index_template/new_logs_data_stream
|
||||
{
|
||||
"index_patterns": [ "new_logs*" ],
|
||||
"data_stream": {
|
||||
"timestamp_field": "@timestamp"
|
||||
},
|
||||
"template": {
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"@timestamp": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
PUT /_data_stream/logs
|
||||
|
||||
PUT /_data_stream/new_logs
|
||||
----
|
||||
////
|
||||
|
||||
[source,console]
|
||||
----
|
||||
POST /_reindex
|
||||
@ -439,7 +610,6 @@ POST /_reindex
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
====
|
||||
+
|
||||
You can also use a query to reindex only a subset of documents with each
|
||||
@ -474,7 +644,6 @@ POST /_reindex
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
====
|
||||
|
||||
. If you previously changed your {ilm-init} poll interval, change it back to its
|
||||
@ -496,7 +665,6 @@ PUT /_cluster/settings
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
====
|
||||
|
||||
. Resume indexing using the new data stream. Searches on this stream will now
|
||||
@ -516,17 +684,4 @@ indices and any data they contain.
|
||||
----
|
||||
DELETE /_data_stream/logs
|
||||
----
|
||||
// TEST[continued]
|
||||
====
|
||||
|
||||
////
|
||||
[source,console]
|
||||
----
|
||||
DELETE /_data_stream/new_logs
|
||||
|
||||
DELETE /_index_template/logs_data_stream
|
||||
|
||||
DELETE /_index_template/new_logs_data_stream
|
||||
----
|
||||
// TEST[continued]
|
||||
////
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user