parent
ea3c49979e
commit
747f8bfe79
|
@ -56,20 +56,19 @@ image::images/data-streams/data-streams-diagram.svg[align="center"]
|
||||||
|
|
||||||
To create backing indices, each data stream requires a matching
|
To create backing indices, each data stream requires a matching
|
||||||
<<indices-templates,index template>>. This template acts as a blueprint for the
|
<<indices-templates,index template>>. This template acts as a blueprint for the
|
||||||
stream's backing indices. It contains:
|
stream's backing indices. It specifies:
|
||||||
|
|
||||||
* The mappings and settings applied to each backing index when it's created.
|
* One or more wildcard (`*`) patterns that match the name of the stream.
|
||||||
|
|
||||||
* A name or wildcard (`*`) pattern that matches the data stream's name.
|
* The mappings and settings for the stream's backing indices.
|
||||||
|
|
||||||
* A `data_stream` object with an empty body (`{ }`). This object indicates the
|
* That the template is used exclusively for data streams.
|
||||||
template is used for data streams.
|
|
||||||
|
|
||||||
A `@timestamp` field must be included in every document indexed to the data
|
Every document indexed to a data stream must have a `@timestamp` field. This
|
||||||
stream. This field can be mapped as a <<date,`date`>> or
|
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field
|
||||||
<<date_nanos,`date_nanos`>> field data type in the stream's matching index
|
data type by the stream's index template. If the template does not specify a
|
||||||
template. If no mapping is specified in the template, the `date` field data type
|
mapping, the `@timestamp` field is mapped as a `date` field with default
|
||||||
with default options is used.
|
options.
|
||||||
|
|
||||||
The same index template can be used to create multiple data streams.
|
The same index template can be used to create multiple data streams.
|
||||||
|
|
||||||
|
|
|
@ -55,10 +55,18 @@ triggers for such actions.
|
||||||
TIP: While optional, we recommend using {ilm-init} to manage the backing indices
|
TIP: While optional, we recommend using {ilm-init} to manage the backing indices
|
||||||
associated with a data stream.
|
associated with a data stream.
|
||||||
|
|
||||||
The following <<ilm-put-lifecycle,create lifecycle policy API>> request
|
You can create the policy through the Kibana UI. In Kibana, open the menu and go
|
||||||
configures the `logs_policy` lifecycle policy.
|
to *Stack Management > Index Lifecycle Policies*. Click *Index Lifecycle
|
||||||
|
Policies*.
|
||||||
|
|
||||||
The `logs_policy` policy uses the <<ilm-rollover,`rollover` action>> to create a
|
[role="screenshot"]
|
||||||
|
image::images/ilm/create-policy.png[Index Lifecycle Policies page]
|
||||||
|
|
||||||
|
You can also create a policy using the <<ilm-put-lifecycle,create lifecycle
|
||||||
|
policy API>>.
|
||||||
|
|
||||||
|
The following request configures the `logs_policy` lifecycle policy. The
|
||||||
|
`logs_policy` policy uses the <<ilm-rollover,`rollover` action>> to create a
|
||||||
new <<data-stream-write-index,write index>> for the data stream when the current
|
new <<data-stream-write-index,write index>> for the data stream when the current
|
||||||
one reaches 25GB in size. The policy also deletes backing indices 30 days after
|
one reaches 25GB in size. The policy also deletes backing indices 30 days after
|
||||||
their rollover.
|
their rollover.
|
||||||
|
@ -95,19 +103,18 @@ PUT /_ilm/policy/logs_policy
|
||||||
A data stream uses an index template to configure its backing indices. A
|
A data stream uses an index template to configure its backing indices. A
|
||||||
template for a data stream must specify:
|
template for a data stream must specify:
|
||||||
|
|
||||||
* An index pattern that matches the name of the stream.
|
* One or more wildcard (`*`) patterns that match the name of the stream.
|
||||||
|
|
||||||
* An empty `data_stream` object that indicates the template is used for data
|
|
||||||
streams.
|
|
||||||
|
|
||||||
* The mappings and settings for the stream's backing indices.
|
* The mappings and settings for the stream's backing indices.
|
||||||
|
|
||||||
|
* That the template is used exclusively for data streams.
|
||||||
|
|
||||||
Every document indexed to a data stream must have a `@timestamp` field. This
|
Every document indexed to a data stream must have a `@timestamp` field. This
|
||||||
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field
|
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field
|
||||||
data type by the stream's index template. This mapping can include other
|
data type by the stream's index template. This mapping can include other
|
||||||
<<mapping-params,mapping parameters>>, such as <<mapping-date-format,`format`>>.
|
<<mapping-params,mapping parameters>>, such as <<mapping-date-format,`format`>>.
|
||||||
If the template does not specify a mapping is specified in the template, the
|
If the template does not specify a mapping, the `@timestamp` field is mapped as
|
||||||
`@timestamp` field is mapped as a `date` field with default options.
|
a `date` field with default options.
|
||||||
|
|
||||||
We recommend using {ilm-init} to manage a data stream's backing indices. Specify
|
We recommend using {ilm-init} to manage a data stream's backing indices. Specify
|
||||||
the name of the lifecycle policy with the `index.lifecycle.name` setting.
|
the name of the lifecycle policy with the `index.lifecycle.name` setting.
|
||||||
|
@ -117,11 +124,24 @@ in this template before creating a data stream. Later changes to the mappings or
|
||||||
settings of a stream's backing indices may require reindexing. See
|
settings of a stream's backing indices may require reindexing. See
|
||||||
<<data-streams-change-mappings-and-settings>>.
|
<<data-streams-change-mappings-and-settings>>.
|
||||||
|
|
||||||
The following <<indices-templates,put index template API>> request
|
You can create an index template through the Kibana UI:
|
||||||
configures the `logs_data_stream` template.
|
|
||||||
|
|
||||||
Because no field mapping is specified, the `@timestamp` field uses the `date`
|
. From Kibana, open the menu and go to *Stack Management > Index Management*.
|
||||||
field data type by default.
|
. In the *Index Templates* tab, click *Create template*.
|
||||||
|
. In the Create template wizard, use the *Data stream* toggle to indicate the
|
||||||
|
template is used exclusively for data streams.
|
||||||
|
|
||||||
|
[role="screenshot"]
|
||||||
|
image::images/data-streams/create-index-template.png[Create template page]
|
||||||
|
|
||||||
|
You can also create a template using the <<indices-put-template,put index
|
||||||
|
template API>>. The template must include a `data_stream` object with an empty
|
||||||
|
body (`{ }`). This object indicates the template is used exclusively for data
|
||||||
|
streams.
|
||||||
|
|
||||||
|
The following request configures the `logs_data_stream` index template. Because
|
||||||
|
no field mapping is specified, the `@timestamp` field uses the `date` field data
|
||||||
|
type by default.
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
----
|
----
|
||||||
|
@ -179,7 +199,7 @@ You can create a data stream using one of two methods:
|
||||||
[[index-documents-to-create-a-data-stream]]
|
[[index-documents-to-create-a-data-stream]]
|
||||||
==== Index documents to create a data stream
|
==== Index documents to create a data stream
|
||||||
|
|
||||||
You can automatically generate a data stream using an indexing request. Submit
|
You can automatically create a data stream using an indexing request. Submit
|
||||||
an <<add-documents-to-a-data-stream,indexing request>> to a target
|
an <<add-documents-to-a-data-stream,indexing request>> to a target
|
||||||
matching the name or wildcard pattern defined in the template's `index_patterns`
|
matching the name or wildcard pattern defined in the template's `index_patterns`
|
||||||
property.
|
property.
|
||||||
|
@ -254,10 +274,16 @@ PUT /_data_stream/logs_alt
|
||||||
[[get-info-about-a-data-stream]]
|
[[get-info-about-a-data-stream]]
|
||||||
=== Get information about a data stream
|
=== Get information about a data stream
|
||||||
|
|
||||||
You can use the <<indices-get-data-stream,get data stream API>> to get
|
To view information about a data stream in Kibana, open the menu and go to
|
||||||
information about one or more data streams, including:
|
*Stack Management > Index Management*. In the *Data Streams* tab, click a data
|
||||||
|
stream's name to view information about the stream.
|
||||||
|
|
||||||
|
[role="screenshot"]
|
||||||
|
image::images/data-streams/data-streams-list.png[Data Streams tab]
|
||||||
|
|
||||||
|
You can also use the <<indices-get-data-stream,get data stream API>> to retrieve
|
||||||
|
the following information about one or more data streams:
|
||||||
|
|
||||||
* The timestamp field
|
|
||||||
* The current backing indices, which is returned as an array. The last item in
|
* The current backing indices, which is returned as an array. The last item in
|
||||||
the array contains information about the stream's current write index.
|
the array contains information about the stream's current write index.
|
||||||
* The current generation
|
* The current generation
|
||||||
|
@ -266,8 +292,6 @@ information about one or more data streams, including:
|
||||||
* The current {ilm-init} lifecycle policy in the stream's matching index
|
* The current {ilm-init} lifecycle policy in the stream's matching index
|
||||||
template
|
template
|
||||||
|
|
||||||
This is also handy way to verify that a recently created data stream exists.
|
|
||||||
|
|
||||||
The following get data stream API request retrieves information about the
|
The following get data stream API request retrieves information about the
|
||||||
`logs` data stream.
|
`logs` data stream.
|
||||||
|
|
||||||
|
@ -334,11 +358,18 @@ data. See <<data-stream-privileges>>.
|
||||||
[[delete-a-data-stream]]
|
[[delete-a-data-stream]]
|
||||||
=== Delete a data stream
|
=== Delete a data stream
|
||||||
|
|
||||||
You can use the <<indices-delete-data-stream,delete data stream API>> to delete
|
You can use the Kibana UI to delete a data stream and its backing indices. In
|
||||||
a data stream and its backing indices.
|
Kibana, open the menu and go to *Stack Management > Index Management*. In the
|
||||||
|
*Data Streams* tab, click the trash can icon to delete a stream and its backing
|
||||||
|
indices.
|
||||||
|
|
||||||
The following delete data stream API request deletes the `logs` data stream. This
|
[role="screenshot"]
|
||||||
request also deletes the stream's backing indices and any data they contain.
|
image::images/data-streams/data-streams-list.png[Data Streams tab]
|
||||||
|
|
||||||
|
You can also use the the <<indices-delete-data-stream,delete data stream API>>
|
||||||
|
to delete a data stream. The following delete data stream API request deletes
|
||||||
|
the `logs` data stream. This request also deletes the stream's backing indices
|
||||||
|
and any data they contain.
|
||||||
|
|
||||||
[source,console]
|
[source,console]
|
||||||
----
|
----
|
||||||
|
|
|
@ -55,9 +55,10 @@ reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
|
||||||
* A `delete` phase that sets `min_age` to remove the index 90 days after rollover.
|
* A `delete` phase that sets `min_age` to remove the index 90 days after rollover.
|
||||||
Note that this value is relative to the rollover time, not the index creation time.
|
Note that this value is relative to the rollover time, not the index creation time.
|
||||||
|
|
||||||
You can create the policy through {kib} Management or with the
|
You can create the policy through {kib} or with the
|
||||||
<<ilm-put-lifecycle, put policy>> API.
|
<<ilm-put-lifecycle, put policy>> API.
|
||||||
To create the policy from {kib}, go to Management and click **Index Lifecycle Policies**.
|
To create the policy from {kib}, open the menu and go to *Stack Management >
|
||||||
|
Index Lifecycle Policies*. Click *Index Lifecycle Policies*.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image:images/ilm/create-policy.png[]
|
image:images/ilm/create-policy.png[]
|
||||||
|
@ -109,9 +110,14 @@ To enable the {ilm-init} to manage the data stream, the template configures one
|
||||||
|
|
||||||
* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to the data stream.
|
* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to the data stream.
|
||||||
|
|
||||||
You can use the {kib} Create template wizard to add the template.
|
You can use the {kib} Create template wizard to add the template. From Kibana,
|
||||||
This wizard invokes the put _index_template API to create the <<indices-templates,index template>>
|
open the menu and go to *Stack Management > Index Management*. In the *Index
|
||||||
with the options you specify.
|
Templates* tab, click *Create template*.
|
||||||
|
|
||||||
|
image::images/data-streams/create-index-template.png[Create template page]
|
||||||
|
|
||||||
|
This wizard invokes the <<indices-put-template,put index template API>> to create
|
||||||
|
the index template with the options you specify.
|
||||||
|
|
||||||
.API example
|
.API example
|
||||||
[%collapsible]
|
[%collapsible]
|
||||||
|
@ -303,9 +309,9 @@ that match the index pattern.
|
||||||
* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
|
* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
|
||||||
when the rollover action is triggered for an index.
|
when the rollover action is triggered for an index.
|
||||||
|
|
||||||
You can use the {kib} Create template wizard to add the template.
|
You can use the {kib} Create template wizard to add the template. To access the
|
||||||
To access the wizard, go to Management, click **Index Management**,
|
wizard, open the menu, go to *Stack Management > Index Management*, and click
|
||||||
and select the **Index Templates** view.
|
the *Index Templates* tab.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image:images/ilm/create-template-wizard.png[]
|
image:images/ilm/create-template-wizard.png[]
|
||||||
|
|
Binary file not shown.
After Width: | Height: | Size: 86 KiB |
Binary file not shown.
After Width: | Height: | Size: 16 KiB |
Loading…
Reference in New Issue