honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[DOCS] Tighten data streams copy (#64085) (#64111)

This commit is contained in:
James Rodewig 2020-10-24 14:39:13 -04:00 committed by GitHub
parent f19f170811
commit 361fa021fa
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 188 additions and 809 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/reference/data-streams/change-mappings-and-settings.asciidoc
View File

@ -471,7 +471,7 @@ PUT /_index_template/new-data-stream-template
create the new data stream. The name of the data stream must match the index create the new data stream. The name of the data stream must match the index
pattern defined in the new template's `index_patterns` property. pattern defined in the new template's `index_patterns` property.
+ +
We do not recommend <<index-documents-to-create-a-data-stream,indexing new data We do not recommend <<create-a-data-stream,indexing new data
to create this data stream>>. Later, you will reindex older data from an to create this data stream>>. Later, you will reindex older data from an
existing data stream into this new stream. This could result in one or more existing data stream into this new stream. This could result in one or more
backing indices that contains a mix of new and old data. backing indices that contains a mix of new and old data.

196
docs/reference/data-streams/data-streams.asciidoc
View File

@ -5,111 +5,49 @@
<titleabbrev>Data streams</titleabbrev> <titleabbrev>Data streams</titleabbrev>
++++ ++++
A _data stream_ is a convenient, scalable way to ingest, search, and manage A {ref}/data-streams.html[data stream] lets you store append-only time series
continuously generated time series data. data across multiple indices while giving you a single named resource for
requests. Data streams are well-suited for logs, events, metrics, and other
Time series data, such as logs, tends to grow over time. While storing an entire continuously generated data.
time series in a single {es} index is simpler, it is often more efficient and
cost-effective to store large volumes of data across multiple, time-based
indices. Multiple indices let you move indices containing older, less frequently
queried data to less expensive hardware and delete indices when they're no
longer needed, reducing overhead and storage costs.
A data stream is designed to give you the best of both worlds:
* The simplicity of a single named resource you can use for requests
* The storage, scalability, and cost-saving benefits of multiple indices
You can submit indexing and search requests directly to a data stream. The You can submit indexing and search requests directly to a data stream. The
stream automatically routes the requests to a collection of hidden stream automatically routes the request to backing indices that store the
_backing indices_ that store the stream's data. stream's data. You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to
automate the management of these backing indices. For example, you can use
You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to automate the {ilm-init} to automatically move older backing indices to less expensive
management of these backing indices. {ilm-init} lets you automatically spin up hardware and delete unneeded indices. {ilm-init} can help you reduce costs and
new backing indices, allocate indices to different hardware, delete old indices, overhead as your data grows.
and take other automatic actions based on age or size criteria you set. Use data
streams and {ilm-init} to seamlessly scale your data storage based on your
budget, performance, resiliency, and retention needs.
[discrete]
[[when-to-use-data-streams]]
== When to use data streams
We recommend using data streams if you:
* Use {es} to ingest, search, and manage large volumes of time series data
* Want to scale and reduce costs by using {ilm-init} to automate the management
of your indices
* Index large volumes of time series data in {es} but rarely delete or update
individual documents
[discrete] [discrete]
[[backing-indices]] [[backing-indices]]
== Backing indices == Backing indices
A data stream consists of one or more _backing indices_. Backing indices are
<<index-hidden,hidden>>, auto-generated indices used to store a stream's A data stream consists of one or more <<index-hidden,hidden>>, auto-generated
documents. backing indices.
image::images/data-streams/data-streams-diagram.svg[align="center"] image::images/data-streams/data-streams-diagram.svg[align="center"]
To create backing indices, each data stream requires a matching To configure its backing indices, each data stream requires a matching
<<indices-templates,index template>>. This template acts as a blueprint for the <<indices-templates,index template>>. This template contains:
stream's backing indices. It specifies:
* One or more wildcard (`*`) patterns that match the name of the stream. * An index pattern matching the stream's name.
* The mappings and settings for the stream's backing indices. * Mappings and settings for the stream's backing indices.
* That the template is used exclusively for data streams. Every document indexed to a data stream must contain a `@timestamp` field,
mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field type. If the
index template doesn't specify a mapping for the `@timestamp` field, {es} maps
`@timestamp` as a `date` field with default options.
Every document indexed to a data stream must have a `@timestamp` field. This The same index template can be used for multiple data streams. You cannot
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field delete an index template in use by a data stream.
data type by the stream's index template. If the template does not specify a
mapping, the `@timestamp` field is mapped as a `date` field with default
options.
The same index template can be used to create multiple data streams.
[discrete]
[[data-streams-generation]]
== Generation
Each data stream tracks its _generation_: a six-digit, zero-padded integer
that acts as a cumulative count of the data stream's backing indices. This count
includes any deleted indices for the stream. The generation is incremented
whenever a new backing index is added to the stream.
When a backing index is created, the index is named using the following
convention:
[source,text]
----
.ds-<data-stream>-<generation>
----
For example, the `web-server-logs` data stream has a generation of `34`. The
most recently created backing index for this data stream is named
`.ds-web-server-logs-000034`.
Because the generation increments with each new backing index, backing indices
with a higher generation contain more recent data. Backing indices with a lower
generation contain older data.
A backing index's name can change after its creation due to a
<<indices-shrink-index,shrink>>, <<snapshots-restore-snapshot,restore>>, or
other operations. However, renaming a backing index does not detach it from a
data stream.
[discrete] [discrete]
[[data-stream-read-requests]] [[data-stream-read-requests]]
== Read requests == Read requests
When a read request is sent to a data stream, it routes the request to all its When you submit a read request to a data stream, the stream routes the request
backing indices. For example, a search request sent to a data stream would query to all its backing indices.
all its backing indices.
image::images/data-streams/data-streams-search-request.svg[align="center"] image::images/data-streams/data-streams-search-request.svg[align="center"]
@ -117,18 +55,16 @@ image::images/data-streams/data-streams-search-request.svg[align="center"]
[[data-stream-write-index]] [[data-stream-write-index]]
== Write index == Write index
The most recently created backing index is the data stream’s only The most recently created backing index is the data stream’s write index.
_write index_. The data stream routes all indexing requests for new documents to The stream adds new documents to this index only.
this index.
image::images/data-streams/data-streams-index-request.svg[align="center"] image::images/data-streams/data-streams-index-request.svg[align="center"]
You cannot add new documents to a stream's other backing indices, even by You cannot add new documents to other backing indices, even by sending requests
sending requests directly to the index. directly to the index.
Because it's the only index capable of ingesting new documents, you cannot You also cannot perform operations on a write index that may hinder indexing,
perform operations on a write index that might hinder indexing. These such as:
prohibited operations include:
* <<indices-clone-index,Clone>> * <<indices-clone-index,Clone>>
* <<indices-close,Close>> * <<indices-close,Close>>
@ -141,45 +77,57 @@ prohibited operations include:
[[data-streams-rollover]] [[data-streams-rollover]]
== Rollover == Rollover
When a data stream is created, one backing index is automatically created. When you create a data stream, {es} automatically creates a backing index for
Because this single index is also the most recently created backing index, it the stream. This index also acts as the stream's first write index. A
acts as the stream's write index. <<indices-rollover-index,rollover>> creates a new backing index that becomes the
stream's new write index.
A <<indices-rollover-index,rollover>> creates a new backing index for a data We recommend using <<index-lifecycle-management,{ilm-init}>> to automatically
stream. This new backing index becomes the stream's write index, replacing roll over data streams when the write index reaches a specified age or size.
the current one, and increments the stream's generation. If needed, you can also <<manually-roll-over-a-data-stream,manually roll over>>
a data stream.
In most cases, we recommend using <<index-lifecycle-management,{ilm}
({ilm-init})>> to automate rollovers for data streams. This lets you
automatically roll over the current write index when it meets specified
criteria, such as a maximum age or size.
However, you can also use the <<indices-rollover-index,rollover API>> to [discrete]
manually perform a rollover. See <<manually-roll-over-a-data-stream>>. [[data-streams-generation]]
== Generation
Each data stream tracks its generation: a six-digit, zero-padded integer that
acts as a cumulative count of the stream's rollovers, starting at `000001`.
When a backing index is created, the index is named using the following
convention:
[source,text]
----
.ds-<data-stream>-<generation>
----
Backing indices with a higher generation contain more recent data. For example,
the `web-server-logs` data stream has a generation of `34`. The stream's most
recent backing index is named `.ds-web-server-logs-000034`.
Some operations, such as a <<indices-shrink-index,shrink>> or
<<snapshots-restore-snapshot,restore>>, can change a backing index's name.
These name changes do not remove a backing index from its data stream.
[discrete] [discrete]
[[data-streams-append-only]] [[data-streams-append-only]]
== Append-only == Append-only
For most time series use cases, existing data is rarely, if ever, updated. Data streams are designed for use cases where existing data is rarely,
Because of this, data streams are designed to be append-only. if ever, updated. You cannot send update or deletion requests for existing
documents directly to a data stream. Instead, use the
<<update-docs-in-a-data-stream-by-query,update by query>> and
<<delete-docs-in-a-data-stream-by-query,delete by query>> APIs.
You can send <<add-documents-to-a-data-stream,indexing requests for new If needed, you can <<update-delete-docs-in-a-backing-index,update or delete
documents>> directly to a data stream. However, you cannot send the update or documents>> by submitting requests directly to the document's backing index.
deletion requests for existing documents directly to a data stream.
Instead, you can use the <<docs-update-by-query,update by query>> and TIP: If you frequently update or delete existing documents, use an
<<docs-delete-by-query,delete by query>> APIs to update or delete existing <<indices-add-alias,index alias>> and <<indices-templates,index template>>
documents in a data stream. See <<update-docs-in-a-data-stream-by-query>> and <<delete-docs-in-a-data-stream-by-query>>. instead of a data stream. You can still use
<<index-lifecycle-management,{ilm-init}>> to manage indices for the alias.
If needed, you can update or delete a document by submitting requests to the
backing index containing the document. See
<<update-delete-docs-in-a-backing-index>>.
TIP: If you frequently update or delete existing documents,
we recommend using an <<indices-add-alias,index alias>> and
<<indices-templates,index template>> instead of a data stream. You can still
use <<index-lifecycle-management,{ilm-init}>> to manage indices for the alias.
include::set-up-a-data-stream.asciidoc[] include::set-up-a-data-stream.asciidoc[]
include::use-a-data-stream.asciidoc[] include::use-a-data-stream.asciidoc[]

325
docs/reference/data-streams/set-up-a-data-stream.asciidoc
View File

@ -4,69 +4,29 @@
To set up a data stream, follow these steps: To set up a data stream, follow these steps:
. Check the <<data-stream-prereqs, prerequisites>>.
. <<configure-a-data-stream-ilm-policy>>. . <<configure-a-data-stream-ilm-policy>>.
. <<create-a-data-stream-template>>. . <<create-a-data-stream-template>>.
. <<create-a-data-stream>>. . <<create-a-data-stream>>.
. <<get-info-about-a-data-stream>> to verify it exists.
. <<secure-a-data-stream>>. . <<secure-a-data-stream>>.
After you set up a data stream, you can <<use-a-data-stream, use the data
stream>> for indexing, searches, and other supported operations.
If you no longer need it, you can <<delete-a-data-stream,delete a data stream>>
and its backing indices.
[discrete]
[[data-stream-prereqs]]
=== Prerequisites
* {es} data streams are intended for time series data only. Each document
indexed to a data stream must contain the `@timestamp` field. This field must be
mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field data type.
* Data streams are best suited for time-based,
<<data-streams-append-only,append-only>> use cases. If you frequently need to
update or delete existing documents, we recommend using an index alias and an
index template instead.
[discrete] [discrete]
[[configure-a-data-stream-ilm-policy]] [[configure-a-data-stream-ilm-policy]]
=== Optional: Configure an {ilm-init} lifecycle policy for a data stream === Optional: Configure an {ilm-init} lifecycle policy
You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to automatically While optional, we recommend you configure an <<set-up-lifecycle-policy,{ilm}
manage a data stream's backing indices. For example, you could use {ilm-init} ({ilm-init}) policy>> to automate the management of your data stream's backing
to: indices.
* Spin up a new write index for the data stream when the current one reaches a In {kib}, open the menu and go to *Stack Management > Index Lifecycle Policies*.
certain size or age. Click *Index Lifecycle Policies*.
* Move older backing indices to slower, less expensive hardware.
* Delete stale backing indices to enforce data retention standards.
To use {ilm-init} with a data stream, you must
<<set-up-lifecycle-policy,configure a lifecycle policy>>. This lifecycle policy
should contain the automated actions to take on backing indices and the
triggers for such actions.
TIP: While optional, we recommend using {ilm-init} to manage the backing indices
associated with a data stream.
You can create the policy through the Kibana UI. In Kibana, open the menu and go
to *Stack Management > Index Lifecycle Policies*. Click *Index Lifecycle
Policies*.
[role="screenshot"] [role="screenshot"]
image::images/ilm/create-policy.png[Index Lifecycle Policies page] image::images/ilm/create-policy.png[Index Lifecycle Policies page]
You can also create a policy using the <<ilm-put-lifecycle,create lifecycle [%collapsible]
policy API>>. .API example
====
The following request configures the `my-data-stream-policy` lifecycle policy. Use the <<ilm-put-lifecycle,create lifecycle policy API>> to configure a policy:
The policy uses the <<ilm-rollover,`rollover` action>> to create a
new <<data-stream-write-index,write index>> for the data stream when the current
one reaches 25GB in size. The policy also deletes backing indices 30 days after
their rollover.
[source,console] [source,console]
---- ----
@ -91,73 +51,59 @@ PUT /_ilm/policy/my-data-stream-policy
} }
} }
---- ----
====
[discrete] [discrete]
[[create-a-data-stream-template]] [[create-a-data-stream-template]]
=== Create an index template for a data stream === Create an index template
A data stream uses an index template to configure its backing indices. A . In {kib}, open the menu and go to *Stack Management > Index Management*.
template for a data stream must specify: . In the *Index Templates* tab, click *Create template*.
. In the Create template wizard, use the *Data stream* toggle to indicate the
template is used for data streams.
. Use the wizard to finish defining your template. Specify:
* One or more index patterns that match the name of the stream. * One or more index patterns that match the data stream's name.
* The mappings and settings for the stream's backing indices. * Mappings and settings for the stream's backing indices.
* That the template is used exclusively for data streams.
* A priority for the template.
* A priority for the index template
+
[IMPORTANT] [IMPORTANT]
==== ====
{es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and `synthetics-*-*` index {es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and
patterns, each with a priority of `100`. `synthetics-*-*` index patterns, each with a priority of `100`.
{ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to {ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to
create data streams. If you use {agent}, assign your index templates a priority create data streams.
lower than `100` to avoid overriding the built-in templates.
Otherwise, to avoid accidentally applying the built-in templates, use a If you use {agent}, assign your index templates a priority lower than `100` to
non-overlapping index pattern or assign templates with an overlapping pattern a avoid overriding the built-in templates. Otherwise, use a non-overlapping index
`priority` higher than `100`. pattern or assign templates with an overlapping pattern a `priority` higher than
`100`.
For example, if you don't use {agent} and want to create a template for the For example, if you don't use {agent} and want to create a template for the
`logs-*` index pattern, assign your template a priority of `200`. This ensures `logs-*` index pattern, assign your template a priority of `200`. This ensures
your template is applied instead of the built-in template for `logs-*-*`. your template is applied instead of the built-in template for `logs-*-*`.
==== ====
Every document indexed to a data stream must have a `@timestamp` field. This If the index template doesn't specify a mapping for the `@timestamp` field, {es}
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field maps `@timestamp` as a `date` field with default options.
data type by the stream's index template. This mapping can include other
<<mapping-params,mapping parameters>>, such as <<mapping-date-format,`format`>>.
If the template does not specify a mapping, the `@timestamp` field is mapped as
a `date` field with default options.
We recommend using {ilm-init} to manage a data stream's backing indices. Specify If using {ilm-init}, specify your lifecycle policy in the `index.lifecycle.name`
the name of the lifecycle policy with the `index.lifecycle.name` setting. setting.
TIP: We recommend you carefully consider which mappings and settings to include TIP: Carefully consider your template's mappings and settings. Later changes may
in this template before creating a data stream. Later changes to the mappings or require reindexing. See <<data-streams-change-mappings-and-settings>>.
settings of a stream's backing indices may require reindexing. See
<<data-streams-change-mappings-and-settings>>.
You can create an index template through the Kibana UI:
. From Kibana, open the menu and go to *Stack Management > Index Management*.
. In the *Index Templates* tab, click *Create template*.
. In the Create template wizard, use the *Data stream* toggle to indicate the
template is used exclusively for data streams.
[role="screenshot"] [role="screenshot"]
image::images/data-streams/create-index-template.png[Create template page] image::images/data-streams/create-index-template.png[Create template page]
You can also create a template using the <<indices-put-template,put index [%collapsible]
template API>>. The template must include a `data_stream` object with an empty .API example
body (`{ }`). This object indicates the template is used exclusively for data ====
streams. Use the <<indices-put-template,put index template API>> to create an index
template. The template must include an empty `data_stream` object, indicating
The following request configures the `my-data-stream-template` index template. it's used for data streams.
Because no field mapping is specified, the `@timestamp` field uses the `date`
field data type by default.
[source,console] [source,console]
---- ----
@ -174,64 +120,15 @@ PUT /_index_template/my-data-stream-template
} }
---- ----
// TEST[continued] // TEST[continued]
====
Alternatively, the following template maps `@timestamp` as a `date_nanos` field.
[source,console]
----
PUT /_index_template/my-data-stream-template
{
"index_patterns": [ "my-data-stream*" ],
"data_stream": { },
"priority": 200,
"template": {
"mappings": {
"properties": {
"@timestamp": { "type": "date_nanos" } <1>
}
},
"settings": {
"index.lifecycle.name": "my-data-stream-policy"
}
}
}
----
// TEST[continued]
<1> Maps `@timestamp` as a `date_nanos` field. You can include other supported
mapping parameters in this field mapping.
NOTE: You cannot delete an index template that's in use by a data stream.
This would prevent the data stream from creating new backing indices.
[discrete] [discrete]
[[create-a-data-stream]] [[create-a-data-stream]]
=== Create a data stream === Create the data stream
You can create a data stream using one of two methods: To automatically create the data stream, submit an
<<add-documents-to-a-data-stream,indexing request>> to the stream. The stream's
* <<index-documents-to-create-a-data-stream>> name must match one of your template's index patterns.
* <<manually-create-a-data-stream>>
[discrete]
[[index-documents-to-create-a-data-stream]]
==== Index documents to create a data stream
You can automatically create a data stream using an indexing request. Submit
an <<add-documents-to-a-data-stream,indexing request>> to a target
matching the index pattern defined in the template's `index_patterns`
property.
If the indexing request's target doesn't exist, {es} creates the data stream and
uses the target name as the name for the stream.
NOTE: Data streams support only specific types of indexing requests. See
<<add-documents-to-a-data-stream>>.
The following <<docs-index_,index API>> request targets `my-data-stream`, which
matches the index pattern for `my-data-stream-template`. Because
no existing index or data stream uses this name, this request creates the
`my-data-stream` data stream and indexes the document to it.
[source,console] [source,console]
---- ----
@ -246,41 +143,9 @@ POST /my-data-stream/_doc/
---- ----
// TEST[continued] // TEST[continued]
The API returns the following response. Note the `_index` property contains You can also use the <<indices-create-data-stream,create data stream API>> to
`.ds-my-data-stream-000001`, indicating the document was indexed to the write manually create the data stream. The stream's name must match one of your
index of the new data stream. template's index patterns.
[source,console-result]
----
{
"_index": ".ds-my-data-stream-000001",
"_id": "qecQmXIBT4jB8tq1nG0j",
"_type": "_doc",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
----
// TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/]
[discrete]
[[manually-create-a-data-stream]]
==== Manually create a data stream
You can use the <<indices-create-data-stream,create data stream API>> to
manually create a data stream. The name of the data stream must match the index
pattern defined in the template's `index_patterns` property.
The following create data stream request targets `my-data-stream-alt`, which
matches the index pattern for `my-data-stream-template`. Because
no existing index or data stream uses this name, this request creates the
`my-data-stream-alt` data stream.
[source,console] [source,console]
---- ----
@ -288,30 +153,28 @@ PUT /_data_stream/my-data-stream-alt
---- ----
// TEST[continued] // TEST[continued]
[discrete]
[[secure-a-data-stream]]
=== Secure the data stream
To control access to the data stream and its
data, use <<data-stream-privileges,{es}'s {security-features}>>.
[discrete] [discrete]
[[get-info-about-a-data-stream]] [[get-info-about-a-data-stream]]
=== Get information about a data stream === Get information about a data stream
To view information about a data stream in Kibana, open the menu and go to In {kib}, open the menu and go to *Stack Management > Index Management*. In the
*Stack Management > Index Management*. In the *Data Streams* tab, click a data *Data Streams* tab, click the data stream's name.
stream's name to view information about the stream.
[role="screenshot"] [role="screenshot"]
image::images/data-streams/data-streams-list.png[Data Streams tab] image::images/data-streams/data-streams-list.png[Data Streams tab]
You can also use the <<indices-get-data-stream,get data stream API>> to retrieve [%collapsible]
the following information about one or more data streams: .API example
====
* The current backing indices, which is returned as an array. The last item in Use the <<indices-get-data-stream,get data stream API>> to retrieve information
the array contains information about the stream's current write index. about one or more data streams:
* The current generation
* The data stream's health status
* The index template used to create the stream's backing indices
* The current {ilm-init} lifecycle policy in the stream's matching index
template
The following get data stream API request retrieves information about
`my-data-stream`.
//// ////
[source,console] [source,console]
@ -326,75 +189,31 @@ POST /my-data-stream/_rollover/
GET /_data_stream/my-data-stream GET /_data_stream/my-data-stream
---- ----
// TEST[continued] // TEST[continued]
====
The API returns the following response. Note the `indices` property contains an
array of the stream's current backing indices. The last item in this array
contains information about the stream's write index, `.ds-my-data-stream-000002`.
[source,console-result]
----
{
"data_streams": [
{
"name": "my-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-000001",
"index_uuid": "krR78LfvTOe6gr5dj2_1xQ"
},
{
"index_name": ".ds-my-data-stream-000002", <1>
"index_uuid": "C6LWyNJHQWmA08aQGvqRkA"
}
],
"generation": 2,
"status": "GREEN",
"template": "my-data-stream-template",
"ilm_policy": "my-data-stream-policy"
}
]
}
----
// TESTRESPONSE[s/"index_uuid": "krR78LfvTOe6gr5dj2_1xQ"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_uuid": "C6LWyNJHQWmA08aQGvqRkA"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW"/]
<1> Last item in the `indices` array for `my-data-stream`. This
item contains information about the stream's current write index,
`.ds-my-data-stream-000002`.
[discrete]
[[secure-a-data-stream]]
=== Secure a data stream
You can use {es} {security-features} to control access to a data stream and its
data. See <<data-stream-privileges>>.
[discrete] [discrete]
[[delete-a-data-stream]] [[delete-a-data-stream]]
=== Delete a data stream === Delete a data stream
You can use the Kibana UI to delete a data stream and its backing indices. In To delete a data stream and its backing indices, open the {kib} menu and go to
Kibana, open the menu and go to *Stack Management > Index Management*. In the *Stack Management > Index Management*. In the *Data Streams* tab, click the
*Data Streams* tab, click the trash can icon to delete a stream and its backing trash can icon.
indices.
[role="screenshot"] [role="screenshot"]
image::images/data-streams/data-streams-list.png[Data Streams tab] image::images/data-streams/data-streams-list.png[Data Streams tab]
You can also use the the <<indices-delete-data-stream,delete data stream API>> [%collapsible]
to delete a data stream. The following delete data stream API request deletes .API example
`my-data-stream`. This request also deletes the stream's backing ====
indices and any data they contain. Use the <<indices-delete-data-stream,delete data stream API>> to delete a data
stream and its backing indices:
[source,console] [source,console]
---- ----
DELETE /_data_stream/my-data-stream DELETE /_data_stream/my-data-stream
---- ----
// TEST[continued] // TEST[continued]
====
//// ////
[source,console] [source,console]

474
docs/reference/data-streams/use-a-data-stream.asciidoc
View File

@ -56,35 +56,8 @@ DELETE /_index_template/*
[[add-documents-to-a-data-stream]] [[add-documents-to-a-data-stream]]
=== Add documents to a data stream === Add documents to a data stream
You can add documents to a data stream using two types of indexing requests: To add an individual document, use the <<docs-index_,index API>>.
<<ingest,Ingest pipelines>> are supported.
* <<data-streams-individual-indexing-requests>>
* <<data-streams-bulk-indexing-requests>>
Adding a document to a data stream adds the document to stream's current
<<data-stream-write-index,write index>>.
You cannot add new documents to a stream's other backing indices, even by
sending requests directly to the index. This means you cannot submit the
following requests directly to any backing index except the write index:
* An <<docs-index_,index API>> request with an
<<docs-index-api-op_type,`op_type`>> of `create`. The `op_type` parameter
defaults to `create` when adding new documents.
* A <<docs-bulk,bulk API>> request using a `create` action
[discrete]
[[data-streams-individual-indexing-requests]]
==== Individual indexing requests
You can use an <<docs-index_,index API>> request with an
<<docs-index-api-op_type,`op_type`>> of `create` to add individual documents
to a data stream.
NOTE: The `op_type` parameter defaults to `create` when adding new documents.
The following index API request adds a new document to `my-data-stream`.
[source,console] [source,console]
---- ----
@ -98,23 +71,13 @@ POST /my-data-stream/_doc/
} }
---- ----
IMPORTANT: You cannot add new documents to a data stream using the index API's You cannot add new documents to a data stream using the index API's `PUT
`PUT /<target>/_doc/<_id>` request format. To specify a document ID, use the /<target>/_doc/<_id>` request format. To specify a document ID, use the `PUT
`PUT /<target>/_create/<_id>` format instead. /<target>/_create/<_id>` format instead. Only an
<<docs-index-api-op_type,`op_type`>> of `create` is supported.
To add multiple documents with a single request, use the <<docs-bulk,bulk API>>.
[discrete] Only `create` actions are supported.
[[data-streams-bulk-indexing-requests]]
==== Bulk indexing requests
You can use the <<docs-bulk,bulk API>> to add multiple documents to a data
stream in a single request. Each action in the bulk request must use the
`create` action.
NOTE: Data streams do not support other bulk actions, such as `index`.
The following bulk API request adds several new documents to
`my-data-stream`. Only the `create` action is used.
[source,console] [source,console]
---- ----
@ -127,65 +90,6 @@ PUT /my-data-stream/_bulk?refresh
{ "@timestamp": "2020-12-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" } { "@timestamp": "2020-12-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }
---- ----
[discrete]
[[data-streams-index-with-an-ingest-pipeline]]
==== Index with an ingest pipeline
You can use an <<ingest,ingest pipeline>> with an indexing request to
pre-process data before it's indexed to a data stream.
The following <<put-pipeline-api,put pipeline API>> request creates the
`lowercase_message_field` ingest pipeline. The pipeline uses the
<<lowercase-processor,`lowercase` ingest processor>> to change the `message`
field value to lowercase before indexing.
[source,console]
----
PUT /_ingest/pipeline/lowercase_message_field
{
"description" : "Lowercases the message field value",
"processors" : [
{
"lowercase" : {
"field" : "message"
}
}
]
}
----
// TEST[continued]
The following index API request adds a new document to `my-data-stream`.
The request includes a `?pipeline=lowercase_message_field` query parameter.
This parameter indicates {es} should use the `lowercase_message_field` pipeline
to pre-process the document before indexing it.
During pre-processing, the pipeline changes the letter case of the document's
`message` field value from `LOGIN Successful` to `login successful`.
[source,console]
----
POST /my-data-stream/_doc?pipeline=lowercase_message_field
{
"@timestamp": "2020-12-08T11:12:01.000Z",
"user": {
"id": "I1YBEOxJ"
},
"message": "LOGIN Successful"
}
----
// TEST[continued]
////
[source,console]
----
DELETE /_ingest/pipeline/lowercase_message_field
----
// TEST[continued]
////
[discrete] [discrete]
[[search-a-data-stream]] [[search-a-data-stream]]
=== Search a data stream === Search a data stream
@ -198,157 +102,24 @@ The following search APIs support data streams:
* <<search-field-caps, Field capabilities>> * <<search-field-caps, Field capabilities>>
* <<eql-search-api, EQL search>> * <<eql-search-api, EQL search>>
The following <<search-search,search API>> request searches `my-data-stream`
for documents with a timestamp between today and yesterday that also have
`message` value of `login successful`.
[source,console]
----
GET /my-data-stream/_search
{
"query": {
"bool": {
"must": {
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lt": "now/d"
}
}
},
"should": {
"match": {
"message": "login successful"
}
}
}
}
}
----
You can use a comma-separated list to search
multiple data streams, indices, and index aliases in the same request.
The following request searches `my-data-stream` and `my-data-stream-alt`,
which are specified as a comma-separated list in the request path.
[source,console]
----
GET /my-data-stream,my-data-stream-alt/_search
{
"query": {
"match": {
"user.id": "8a4f500d"
}
}
}
----
Index patterns are also supported.
The following request uses the `my-data-stream*` index pattern to search any
data stream, index, or index alias beginning with `my-data-stream`.
[source,console]
----
GET /my-data-stream*/_search
{
"query": {
"match": {
"user.id": "vlb44hny"
}
}
}
----
The following search request omits a target in the request path. The request
searches all data streams and indices in the cluster.
[source,console]
----
GET /_search
{
"query": {
"match": {
"user.id": "l7gk7f82"
}
}
}
----
[discrete] [discrete]
[[get-stats-for-a-data-stream]] [[get-stats-for-a-data-stream]]
=== Get statistics for a data stream === Get statistics for a data stream
You can use the <<data-stream-stats-api,data stream stats API>> to retrieve Use the <<data-stream-stats-api,data stream stats API>> to get
statistics for one or more data streams. These statistics include: statistics for one or more data streams:
* A count of the stream's backing indices
* The total store size of all shards for the stream's backing indices
* The highest `@timestamp` value for the stream
.*Example*
[%collapsible]
====
The following data stream stats API request retrieves statistics for
`my-data-stream`.
[source,console] [source,console]
---- ----
GET /_data_stream/my-data-stream/_stats?human=true GET /_data_stream/my-data-stream/_stats?human=true
---- ----
The API returns the following response.
[source,console-result]
----
{
"_shards": {
"total": 6,
"successful": 3,
"failed": 0
},
"data_stream_count": 1,
"backing_indices": 3,
"total_store_size": "624b",
"total_store_size_bytes": 624,
"data_streams": [
{
"data_stream": "my-data-stream",
"backing_indices": 3,
"store_size": "624b",
"store_size_bytes": 624,
"maximum_timestamp": 1607339167000
}
]
}
----
// TESTRESPONSE[s/"total_store_size": "624b"/"total_store_size": $body.total_store_size/]
// TESTRESPONSE[s/"total_store_size_bytes": 624/"total_store_size_bytes": $body.total_store_size_bytes/]
// TESTRESPONSE[s/"store_size": "624b"/"store_size": $body.data_streams.0.store_size/]
// TESTRESPONSE[s/"store_size_bytes": 624/"store_size_bytes": $body.data_streams.0.store_size_bytes/]
====
[discrete] [discrete]
[[manually-roll-over-a-data-stream]] [[manually-roll-over-a-data-stream]]
=== Manually roll over a data stream === Manually roll over a data stream
A rollover creates a new backing index for a data stream. This new backing index Use the <<indices-rollover-index,rollover API>> to manually
becomes the stream's <<data-stream-write-index,write index>> and increments <<data-streams-rollover,roll over>> a data stream:
the stream's <<data-streams-generation,generation>>.
In most cases, we recommend using <<index-lifecycle-management,{ilm-init}>> to
automate rollovers for data streams. This lets you automatically roll over the
current write index when it meets specified criteria, such as a maximum age or
size.
However, you can also use the <<indices-rollover-index,rollover API>> to
manually perform a rollover. This can be useful if you want to
<<data-streams-change-mappings-and-settings,apply mapping or setting changes>>
to the stream's write index after updating a data stream's template.
The following <<indices-rollover-index,rollover API>> request submits a manual
rollover request for `my-data-stream`.
[source,console] [source,console]
---- ----
@ -359,112 +130,35 @@ POST /my-data-stream/_rollover/
[[open-closed-backing-indices]] [[open-closed-backing-indices]]
=== Open closed backing indices === Open closed backing indices
You may <<indices-close,close>> one or more of a data stream's backing indices You cannot search a <<indices-close,closed>> backing index, even by searching
as part of its {ilm-init} lifecycle or another workflow. A closed backing index its data stream. You also cannot <<update-docs-in-a-data-stream-by-query,update>>
cannot be searched, even for searches targeting its data stream. You also can't or <<delete-docs-in-a-data-stream-by-query,delete>> documents in a closed index.
<<update-docs-in-a-data-stream-by-query,update>> or
<<delete-docs-in-a-data-stream-by-query,delete>> documents in a closed index.
You can re-open individual backing indices by sending an To re-open a closed backing index, submit an <<indices-open-close,open
<<indices-open-close,open request>> directly to the index. index API request>> directly to the index:
You also can conveniently re-open all closed backing indices for a data stream
by sending an open request directly to the stream.
The following <<cat-indices,cat indices>> API request retrieves the status for
`my-data-stream`'s backing indices.
////
[source,console]
----
POST /.ds-my-data-stream-000001,.ds-my-data-stream-000002/_close/
----
////
[source,console] [source,console]
---- ----
GET /_cat/indices/my-data-stream?v&s=index&h=index,status POST /.ds-my-data-stream-000001/_open/
---- ----
// TEST[continued]
The API returns the following response. The response indicates To re-open all closed backing indices for a data stream, submit an open index
`my-data-stream` contains two closed backing indices: API request to the stream:
`.ds-my-data-stream-000001` and `.ds-my-data-stream-000002`.
[source,txt]
----
index status
.ds-my-data-stream-000001 close
.ds-my-data-stream-000002 close
.ds-my-data-stream-000003 open
----
// TESTRESPONSE[non_json]
The following <<indices-open-close,open API>> request re-opens any closed
backing indices for `my-data-stream`, including
`.ds-my-data-stream-000001` and `.ds-my-data-stream-000002`.
[source,console] [source,console]
---- ----
POST /my-data-stream/_open/ POST /my-data-stream/_open/
---- ----
// TEST[continued]
You can resubmit the original cat indices API request to verify
`.ds-my-data-stream-000001` and `.ds-my-data-stream-000002` were re-opened.
[source,console]
----
GET /_cat/indices/my-data-stream?v&s=index&h=index,status
----
// TEST[continued]
The API returns the following response.
[source,txt]
----
index status
.ds-my-data-stream-000001 open
.ds-my-data-stream-000002 open
.ds-my-data-stream-000003 open
----
// TESTRESPONSE[non_json]
[discrete] [discrete]
[[reindex-with-a-data-stream]] [[reindex-with-a-data-stream]]
=== Reindex with a data stream === Reindex with a data stream
You can use the <<docs-reindex,reindex API>> to copy documents to a data stream Use the <<docs-reindex,reindex API>> to copy documents from an
from an existing index, index alias, or data stream. existing index, index alias, or data stream to a data stream. Because data streams are
<<data-streams-append-only,append-only>>, a reindex into a data stream must use
A reindex copies documents from a _source_ to a _destination_. The source and an `op_type` of `create`. A reindex cannot update existing documents in a data
destination can be any pre-existing index, index alias, or data stream. However, stream.
the source and destination must be different. You cannot reindex a data stream
into itself.
Because data streams are <<data-streams-append-only,append-only>>, a reindex
request to a data stream destination must have an `op_type` of `create`. This
means a reindex can only add new documents to a data stream. It cannot update
existing documents in the data stream destination.
A reindex can be used to:
* Convert an existing index alias and collection of time-based indices into a
data stream.
* Apply a new or updated <<create-a-data-stream-template,index template>>
by reindexing an existing data stream into a new one. This applies mapping
and setting changes in the template to each document and backing index of the
data stream destination. See
<<data-streams-use-reindex-to-change-mappings-settings>>.
TIP: If you only want to update the mappings or settings of a data stream's
write index, we recommend you update the <<create-a-data-stream-template,data
stream's template>> and perform a <<manually-roll-over-a-data-stream,rollover>>.
The following reindex request copies documents from the `archive` index alias to
`my-data-stream`. Because the destination is a data
stream, the request's `op_type` is `create`.
//// ////
[source,console] [source,console]
@ -504,48 +198,12 @@ POST /_reindex
---- ----
// TEST[continued] // TEST[continued]
You can also reindex documents from a data stream to an index, index
alias, or data stream.
The following reindex request copies documents from `my-data-stream`
to the existing `archive` index alias. Because the destination is not a
data stream, the `op_type` does not need to be specified.
[source,console]
----
POST /_reindex
{
"source": {
"index": "my-data-stream"
},
"dest": {
"index": "archive"
}
}
----
// TEST[continued]
[discrete] [discrete]
[[update-docs-in-a-data-stream-by-query]] [[update-docs-in-a-data-stream-by-query]]
=== Update documents in a data stream by query === Update documents in a data stream by query
You cannot send indexing or update requests for existing documents directly to a Use the <<docs-update-by-query,update by query API>> to update documents in a
data stream. These prohibited requests include: data stream that match a provided query:
* An <<docs-index_,index API>> request with an
<<docs-index-api-op_type,`op_type`>> of `index`. The `op_type` parameter
defaults to `index` for existing documents.
* A <<docs-bulk,bulk API>> request using the `index` or `update`
action.
Instead, you can use the <<docs-update-by-query,update by query API>> to update
documents in a data stream that matches a provided query.
The following update by query request updates documents in `my-data-stream`
with a `user.id` of `l7gk7f82`. The request uses a
<<modules-scripting-using,script>> to assign matching documents a new `user.id`
value of `XgdX0NoX`.
[source,console] [source,console]
---- ----
@ -569,18 +227,8 @@ POST /my-data-stream/_update_by_query
[[delete-docs-in-a-data-stream-by-query]] [[delete-docs-in-a-data-stream-by-query]]
=== Delete documents in a data stream by query === Delete documents in a data stream by query
You cannot send document deletion requests directly to a data stream. These Use the <<docs-delete-by-query,delete by query API>> to delete documents in a
prohibited requests include: data stream that match a provided query:
* A <<docs-delete,delete API>> request
* A <<docs-bulk,bulk API>> request using the `delete` action.
Instead, you can use the <<docs-delete-by-query,delete by query API>> to delete
documents in a data stream that matches a provided query.
The following delete by query request deletes documents in `my-data-stream`
with a `user.id` of `vlb44hny`.
[source,console] [source,console]
---- ----
@ -598,26 +246,15 @@ POST /my-data-stream/_delete_by_query
[[update-delete-docs-in-a-backing-index]] [[update-delete-docs-in-a-backing-index]]
=== Update or delete documents in a backing index === Update or delete documents in a backing index
Alternatively, you can update or delete documents in a data stream by sending If needed, you can update or delete documents in a data stream by sending
the update or deletion request to the backing index containing the document. To requests to the backing index containing the document. You'll need:
do this, you first need to get:
* The <<mapping-id-field,document ID>> * The <<mapping-id-field,document ID>>
* The name of the backing index that contains the document * The name of the backing index containing the document
* If updating the document, its <<optimistic-concurrency-control,sequence number
and primary term>>
If you want to update a document, you must also get its current To get this information, use a <<search-a-data-stream,search request>>:
<<optimistic-concurrency-control,sequence number and primary term>>.
You can use a <<search-a-data-stream,search request>> to retrieve this
information.
The following search request retrieves documents in `my-data-stream`
with a `user.id` of `yWIumJd7`. By default, this search returns the
document ID and backing index for any matching documents.
The request includes a `"seq_no_primary_term": true` argument. This means the
search also returns the sequence number and primary term for any matching
documents.
[source,console] [source,console]
---- ----
@ -632,8 +269,7 @@ GET /my-data-stream/_search
} }
---- ----
The API returns the following response. The `hits.hits` property contains Response:
information for any documents matching the search.
[source,console-result] [source,console-result]
---- ----
@ -681,17 +317,8 @@ information for any documents matching the search.
<3> Current sequence number for the document <3> Current sequence number for the document
<4> Primary term for the document <4> Primary term for the document
You can use an <<docs-index_,index API>> request to update an individual To update the document, use an <<docs-index_,index API>> request with valid
document. To prevent an accidental overwrite, this request must include valid `if_seq_no` and `if_primary_term` arguments:
`if_seq_no` and `if_primary_term` arguments.
The following index API request updates an existing document in
`my-data-stream`. The request targets document ID
`bfspvnIBr7VVZlfp2lqX` in the `.ds-my-data-stream-000003` backing index.
The request also includes the current sequence number and primary term in the
respective `if_seq_no` and `if_primary_term` query parameters. The request body
contains a new JSON source for the document.
[source,console] [source,console]
---- ----
@ -705,32 +332,17 @@ PUT /.ds-my-data-stream-000003/_doc/bfspvnIBr7VVZlfp2lqX?if_seq_no=0&if_primary_
} }
---- ----
You use the <<docs-delete,delete API>> to delete individual documents. Deletion To delete the document, use the <<docs-delete,delete API>>:
requests do not require a sequence number or primary term.
The following index API request deletes an existing document in
`my-data-stream`. The request targets document ID
`bfspvnIBr7VVZlfp2lqX` in the `.ds-my-data-stream-000003` backing index.
[source,console] [source,console]
---- ----
DELETE /.ds-my-data-stream-000003/_doc/bfspvnIBr7VVZlfp2lqX DELETE /.ds-my-data-stream-000003/_doc/bfspvnIBr7VVZlfp2lqX
---- ----
You can use the <<docs-bulk,bulk API>> to delete or update multiple documents in To delete or update multiple documents with a single request, use the
one request using `delete`, `index`, or `update` actions. <<docs-bulk,bulk API>>'s `delete`, `index`, and `update` actions. For `index`
actions, include valid <<bulk-optimistic-concurrency-control,`if_seq_no` and
If the action type is `index`, the action must include valid `if_primary_term`>> arguments.
<<bulk-optimistic-concurrency-control,`if_seq_no` and `if_primary_term`>>
arguments.
The following bulk API request uses an `index` action to update an existing
document in `my-data-stream`.
The `index` action targets document ID `bfspvnIBr7VVZlfp2lqX` in the
`.ds-my-data-stream-000003` backing index. The action also includes the current
sequence number and primary term in the respective `if_seq_no` and
`if_primary_term` parameters.
[source,console] [source,console]
---- ----