[role="xpack"] [[set-up-a-data-stream]] == Set up a data stream To set up a data stream, follow these steps: . Check the <>. . <>. . <>. . <>. . <> to verify it exists. . <>. After you set up a data stream, you can <> for indexing, searches, and other supported operations. If you no longer need it, you can <> and its backing indices. [discrete] [[data-stream-prereqs]] === Prerequisites * {es} data streams are intended for time-series data only. Each document indexed to a data stream must contain a shared timestamp field. + TIP: Data streams work well with most common log formats. While no schema is required to use data streams, we recommend the {ecs-ref}[Elastic Common Schema (ECS)]. * Data streams are best suited for time-based, <> use cases. If you frequently need to update or delete existing documents, we recommend using an index alias and an index template instead. [discrete] [[configure-a-data-stream-ilm-policy]] === Optional: Configure an {ilm-init} lifecycle policy for a data stream You can use <> to automatically manage a data stream's backing indices. For example, you could use {ilm-init} to: * Spin up a new write index for the data stream when the current one reaches a certain size or age. * Move older backing indices to slower, less expensive hardware. * Delete stale backing indices to enforce data retention standards. To use {ilm-init} with a data stream, you must <>. This lifecycle policy should contain the automated actions to take on backing indices and the triggers for such actions. TIP: While optional, we recommend using {ilm-init} to manage the backing indices associated with a data stream. The following <> request configures the `logs_policy` lifecycle policy. The `logs_policy` policy uses the <> to create a new <> for the data stream when the current one reaches 25GB in size. The policy also deletes backing indices 30 days after their rollover. [source,console] ---- PUT /_ilm/policy/logs_policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "25GB" } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } } ---- [discrete] [[create-a-data-stream-template]] === Create an index template for a data stream A data stream uses an index template to configure its backing indices. A template for a data stream must specify: * An index pattern that matches the name of the stream. * An empty `data_stream` object that indicates the template is used for data streams. * The mappings and settings for the stream's backing indices. Every document indexed to a data stream must have a `@timestamp` field. This field can be mapped as a <> or <> field data type by the stream's index template. This mapping can include other <>, such as <>. If the template does not specify a mapping is specified in the template, the `@timestamp` field is mapped as a `date` field with default options. We recommend using {ilm-init} to manage a data stream's backing indices. Specify the name of the lifecycle policy with the `index.lifecycle.name` setting. TIP: We recommend you carefully consider which mappings and settings to include in this template before creating a data stream. Later changes to the mappings or settings of a stream's backing indices may require reindexing. See <>. The following <> request configures the `logs_data_stream` template. Because no field mapping is specified, the `@timestamp` field uses the `date` field data type by default. [source,console] ---- PUT /_index_template/logs_data_stream { "index_patterns": [ "logs*" ], "data_stream": { }, "template": { "settings": { "index.lifecycle.name": "logs_policy" } } } ---- // TEST[continued] Alternatively, the following template maps `@timestamp` as a `date_nanos` field. [source,console] ---- PUT /_index_template/logs_data_stream { "index_patterns": [ "logs*" ], "data_stream": { }, "template": { "mappings": { "properties": { "@timestamp": { "type": "date_nanos" } <1> } }, "settings": { "index.lifecycle.name": "logs_policy" } } } ---- // TEST[continued] <1> Maps `@timestamp` as a `date_nanos` field. You can include other supported mapping parameters in this field mapping. NOTE: You cannot delete an index template that's in use by a data stream. This would prevent the data stream from creating new backing indices. [discrete] [[create-a-data-stream]] === Create a data stream You can create a data stream using one of two methods: * <> * <> [discrete] [[index-documents-to-create-a-data-stream]] ==== Index documents to create a data stream You can automatically generate a data stream using an indexing request. Submit an <> to a target matching the name or wildcard pattern defined in the template's `index_patterns` property. If the indexing request's target doesn't exist, {es} creates the data stream and uses the target name as the name for the stream. NOTE: Data streams support only specific types of indexing requests. See <>. The following <> request targets `logs`, which matches the wildcard pattern for the `logs_data_stream` template. Because no existing index or data stream uses this name, this request creates the `logs` data stream and indexes the document to it. [source,console] ---- POST /logs/_doc/ { "@timestamp": "2020-12-06T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" } ---- // TEST[continued] The API returns the following response. Note the `_index` property contains `.ds-logs-000001`, indicating the document was indexed to the write index of the new `logs` data stream. [source,console-result] ---- { "_index": ".ds-logs-000001", "_id": "qecQmXIBT4jB8tq1nG0j", "_type": "_doc", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 1, "failed": 0 }, "_seq_no": 0, "_primary_term": 1 } ---- // TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/] [discrete] [[manually-create-a-data-stream]] ==== Manually create a data stream You can use the <> to manually create a data stream. The name of the data stream must match the name or wildcard pattern defined in the template's `index_patterns` property. The following create data stream request targets `logs_alt`, which matches the wildcard pattern for the `logs_data_stream` template. Because no existing index or data stream uses this name, this request creates the `logs_alt` data stream. [source,console] ---- PUT /_data_stream/logs_alt ---- // TEST[continued] [discrete] [[get-info-about-a-data-stream]] === Get information about a data stream You can use the <> to get information about one or more data streams, including: * The timestamp field * The current backing indices, which is returned as an array. The last item in the array contains information about the stream's current write index. * The current generation * The data stream's health status * The index template used to create the stream's backing indices * The current {ilm-init} lifecycle policy in the stream's matching index template This is also handy way to verify that a recently created data stream exists. The following get data stream API request retrieves information about the `logs` data stream. //// [source,console] ---- POST /logs/_rollover/ ---- // TEST[continued] //// [source,console] ---- GET /_data_stream/logs ---- // TEST[continued] The API returns the following response. Note the `indices` property contains an array of the stream's current backing indices. The last item in this array contains information about the stream's write index, `.ds-logs-000002`. [source,console-result] ---- { "data_streams": [ { "name": "logs", "timestamp_field": { "name": "@timestamp" }, "indices": [ { "index_name": ".ds-logs-000001", "index_uuid": "krR78LfvTOe6gr5dj2_1xQ" }, { "index_name": ".ds-logs-000002", <1> "index_uuid": "C6LWyNJHQWmA08aQGvqRkA" } ], "generation": 2, "status": "GREEN", "template": "logs_data_stream", "ilm_policy": "logs_policy" } ] } ---- // TESTRESPONSE[s/"index_uuid": "krR78LfvTOe6gr5dj2_1xQ"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/] // TESTRESPONSE[s/"index_uuid": "C6LWyNJHQWmA08aQGvqRkA"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/] // TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW"/] <1> Last item in the `indices` array for the `logs` data stream. This item contains information about the stream's current write index, `.ds-logs-000002`. [discrete] [[secure-a-data-stream]] === Secure a data stream You can use {es} {security-features} to control access to a data stream and its data. See <>. [discrete] [[delete-a-data-stream]] === Delete a data stream You can use the <> to delete a data stream and its backing indices. The following delete data stream API request deletes the `logs` data stream. This request also deletes the stream's backing indices and any data they contain. [source,console] ---- DELETE /_data_stream/logs ---- // TEST[continued] //// [source,console] ---- DELETE /_data_stream/* DELETE /_index_template/* DELETE /_ilm/policy/logs_policy ---- // TEST[continued] ////