[role="xpack"] [[set-up-a-data-stream]] == Set up a data stream To set up a data stream, follow these steps: . Check the <>. . <>. . <>. . <>. . <> to verify it exists. . <>. After you set up a data stream, you can <> for indexing, searches, and other supported operations. If you no longer need it, you can <> and its backing indices. [discrete] [[data-stream-prereqs]] === Prerequisites * {es} data streams are intended for time-series data only. Each document indexed to a data stream must contain a shared timestamp field. + TIP: Data streams work well with most common log formats. While no schema is required to use data streams, we recommend the {ecs-ref}[Elastic Common Schema (ECS)]. * Data streams are best suited for time-based, <> use cases. If you frequently need to update or delete existing documents, we recommend using an index alias and an index template instead. [discrete] [[configure-a-data-stream-ilm-policy]] === Optional: Configure an {ilm-init} lifecycle policy for a data stream You can use <> to automatically manage a data stream's backing indices. For example, you could use {ilm-init} to: * Spin up a new write index for the data stream when the current one reaches a certain size or age. * Move older backing indices to slower, less expensive hardware. * Delete stale backing indices to enforce data retention standards. To use {ilm-init} with a data stream, you must <>. This lifecycle policy should contain the automated actions to take on backing indices and the triggers for such actions. TIP: While optional, we recommend using {ilm-init} to manage the backing indices associated with a data stream. You can create the policy through the Kibana UI. In Kibana, open the menu and go to *Stack Management > Index Lifecycle Policies*. Click *Index Lifecycle Policies*. [role="screenshot"] image::images/ilm/create-policy.png[Index Lifecycle Policies page] You can also create a policy using the <>. The following request configures the `my-data-stream-policy` lifecycle policy. The policy uses the <> to create a new <> for the data stream when the current one reaches 25GB in size. The policy also deletes backing indices 30 days after their rollover. [source,console] ---- PUT /_ilm/policy/my-data-stream-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "25GB" } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } } ---- [discrete] [[create-a-data-stream-template]] === Create an index template for a data stream A data stream uses an index template to configure its backing indices. A template for a data stream must specify: * One or more index patterns that match the name of the stream. * The mappings and settings for the stream's backing indices. * That the template is used exclusively for data streams. Every document indexed to a data stream must have a `@timestamp` field. This field can be mapped as a <> or <> field data type by the stream's index template. This mapping can include other <>, such as <>. If the template does not specify a mapping, the `@timestamp` field is mapped as a `date` field with default options. We recommend using {ilm-init} to manage a data stream's backing indices. Specify the name of the lifecycle policy with the `index.lifecycle.name` setting. TIP: We recommend you carefully consider which mappings and settings to include in this template before creating a data stream. Later changes to the mappings or settings of a stream's backing indices may require reindexing. See <>. You can create an index template through the Kibana UI: . From Kibana, open the menu and go to *Stack Management > Index Management*. . In the *Index Templates* tab, click *Create template*. . In the Create template wizard, use the *Data stream* toggle to indicate the template is used exclusively for data streams. [role="screenshot"] image::images/data-streams/create-index-template.png[Create template page] You can also create a template using the <>. The template must include a `data_stream` object with an empty body (`{ }`). This object indicates the template is used exclusively for data streams. The following request configures the `my-data-stream-template` index template. Because no field mapping is specified, the `@timestamp` field uses the `date` field data type by default. [source,console] ---- PUT /_index_template/my-data-stream-template { "index_patterns": [ "my-data-stream*" ], "data_stream": { }, "template": { "settings": { "index.lifecycle.name": "my-data-stream-policy" } } } ---- // TEST[continued] Alternatively, the following template maps `@timestamp` as a `date_nanos` field. [source,console] ---- PUT /_index_template/my-data-stream-template { "index_patterns": [ "my-data-stream*" ], "data_stream": { }, "template": { "mappings": { "properties": { "@timestamp": { "type": "date_nanos" } <1> } }, "settings": { "index.lifecycle.name": "my-data-stream-policy" } } } ---- // TEST[continued] <1> Maps `@timestamp` as a `date_nanos` field. You can include other supported mapping parameters in this field mapping. NOTE: You cannot delete an index template that's in use by a data stream. This would prevent the data stream from creating new backing indices. [discrete] [[create-a-data-stream]] === Create a data stream You can create a data stream using one of two methods: * <> * <> [discrete] [[index-documents-to-create-a-data-stream]] ==== Index documents to create a data stream You can automatically create a data stream using an indexing request. Submit an <> to a target matching the index pattern defined in the template's `index_patterns` property. If the indexing request's target doesn't exist, {es} creates the data stream and uses the target name as the name for the stream. NOTE: Data streams support only specific types of indexing requests. See <>. The following <> request targets `my-data-stream`, which matches the index pattern for `my-data-stream-template`. Because no existing index or data stream uses this name, this request creates the `my-data-stream` data stream and indexes the document to it. [source,console] ---- POST /my-data-stream/_doc/ { "@timestamp": "2020-12-06T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" } ---- // TEST[continued] The API returns the following response. Note the `_index` property contains `.ds-my-data-stream-000001`, indicating the document was indexed to the write index of the new data stream. [source,console-result] ---- { "_index": ".ds-my-data-stream-000001", "_id": "qecQmXIBT4jB8tq1nG0j", "_type": "_doc", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 1, "failed": 0 }, "_seq_no": 0, "_primary_term": 1 } ---- // TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/] [discrete] [[manually-create-a-data-stream]] ==== Manually create a data stream You can use the <> to manually create a data stream. The name of the data stream must match the index pattern defined in the template's `index_patterns` property. The following create data stream request targets `my-data-stream-alt`, which matches the index pattern for `my-data-stream-template`. Because no existing index or data stream uses this name, this request creates the `my-data-stream-alt` data stream. [source,console] ---- PUT /_data_stream/my-data-stream-alt ---- // TEST[continued] [discrete] [[get-info-about-a-data-stream]] === Get information about a data stream To view information about a data stream in Kibana, open the menu and go to *Stack Management > Index Management*. In the *Data Streams* tab, click a data stream's name to view information about the stream. [role="screenshot"] image::images/data-streams/data-streams-list.png[Data Streams tab] You can also use the <> to retrieve the following information about one or more data streams: * The current backing indices, which is returned as an array. The last item in the array contains information about the stream's current write index. * The current generation * The data stream's health status * The index template used to create the stream's backing indices * The current {ilm-init} lifecycle policy in the stream's matching index template The following get data stream API request retrieves information about `my-data-stream`. //// [source,console] ---- POST /my-data-stream/_rollover/ ---- // TEST[continued] //// [source,console] ---- GET /_data_stream/my-data-stream ---- // TEST[continued] The API returns the following response. Note the `indices` property contains an array of the stream's current backing indices. The last item in this array contains information about the stream's write index, `.ds-my-data-stream-000002`. [source,console-result] ---- { "data_streams": [ { "name": "my-data-stream", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-my-data-stream-000001", "index_uuid": "krR78LfvTOe6gr5dj2_1xQ" }, { "index_name": ".ds-my-data-stream-000002", <1> "index_uuid": "C6LWyNJHQWmA08aQGvqRkA" } ], "generation": 2, "status": "GREEN", "template": "my-data-stream-template", "ilm_policy": "my-data-stream-policy" } ] } ---- // TESTRESPONSE[s/"index_uuid": "krR78LfvTOe6gr5dj2_1xQ"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/] // TESTRESPONSE[s/"index_uuid": "C6LWyNJHQWmA08aQGvqRkA"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/] // TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW"/] <1> Last item in the `indices` array for `my-data-stream`. This item contains information about the stream's current write index, `.ds-my-data-stream-000002`. [discrete] [[secure-a-data-stream]] === Secure a data stream You can use {es} {security-features} to control access to a data stream and its data. See <>. [discrete] [[delete-a-data-stream]] === Delete a data stream You can use the Kibana UI to delete a data stream and its backing indices. In Kibana, open the menu and go to *Stack Management > Index Management*. In the *Data Streams* tab, click the trash can icon to delete a stream and its backing indices. [role="screenshot"] image::images/data-streams/data-streams-list.png[Data Streams tab] You can also use the the <> to delete a data stream. The following delete data stream API request deletes `my-data-stream`. This request also deletes the stream's backing indices and any data they contain. [source,console] ---- DELETE /_data_stream/my-data-stream ---- // TEST[continued] //// [source,console] ---- DELETE /_data_stream/* DELETE /_index_template/* DELETE /_ilm/policy/my-data-stream-policy ---- // TEST[continued] ////