OpenSearch/docs/reference/data-streams/set-up-a-data-stream.asciidoc

[[set-up-a-data-stream]]
== Set up a data stream

To set up a data stream, follow these steps:

. Check the <<data-stream-prereqs, prerequisites>>.
. <<configure-a-data-stream-ilm-policy>>.
. <<create-a-data-stream-template>>.
. <<create-a-data-stream>>.

After you set up a data stream, you can <<use-a-data-stream, use the data
stream>> for indexing, searches, and other supported operations.

[discrete]
[[data-stream-prereqs]]
=== Prerequisites

* {es} data streams are intended for time-series data only. Each document
indexed to a data stream must contain a shared timestamp field.
+
TIP: Data streams work well with most common log formats. While no schema is
required to use data streams, we recommend the {ecs-ref}[Elastic Common Schema
(ECS)].

* Data streams are designed to be append-only. While you can index new documents
directly to a data stream, you cannot use a data stream to directly update or
delete individual documents. To update or delete specific documents in a data
stream, submit a <<docs-delete,delete>> or <<docs-update,update>> API request to
the backing index containing the document.


[discrete]
[[configure-a-data-stream-ilm-policy]]
=== Optional: Configure an {ilm-init} lifecycle policy for a data stream

You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to automatically
manage a data stream's backing indices. For example, you could use {ilm-init}
to:

* Spin up a new write index for the data stream when the current one reaches a
  certain size or age.
* Move older backing indices to slower, less expensive hardware.
* Delete stale backing indices to enforce data retention standards.

To use {ilm-init} with a data stream, you must
<<set-up-lifecycle-policy,configure a lifecycle policy>>. This lifecycle policy
should contain the automated actions to take on backing indices and the
triggers for such actions.

TIP: While optional, we recommend using {ilm-init} to scale data streams in
production.

.*Example*
[%collapsible]
====
The following <<ilm-put-lifecycle,create lifecycle policy API>> request
configures the `logs_policy` lifecycle policy.

The `logs_policy` policy uses the <<ilm-rollover,`rollover` action>> to create a
new write index for the data stream when the current one reaches 25GB in size.
The policy also deletes backing indices 30 days after their rollover.

[source,console]
----
PUT /_ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "25GB"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----
====


[discrete]
[[create-a-data-stream-template]]
=== Create a composable template for a data stream

Each data stream requires a <<indices-templates,composable template>>. The data
stream uses this template to create its backing indices.

Composable templates for data streams must contain:

* A name or wildcard (`*`) pattern for the data stream in the `index_patterns`
  property.

* A `data_stream` definition containing the `timestamp_field` property.
  This timestamp field must be included in every document indexed to the data
  stream.

* A <<date,`date`>> or <<date_nanos,`date_nanos`>> field mapping for the
  timestamp field specified in the `timestamp_field` property.

* If you intend to use {ilm-init}, you must specify the
  <<configure-a-data-stream-ilm-policy,lifecycle policy>> in the 
  `index.lifecycle.name` setting.

You can also specify other mappings and settings you'd like to apply to the
stream's backing indices.

.*Example*
[%collapsible]
====
The following <<indices-templates,put composable template API>> request
configures the `logs_data_stream` template.

[source,console]
----
PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp"
  },
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        }
      }
    },
    "settings": {
      "index.lifecycle.name": "logs_policy"
    }
  }
}
----
// TEST[continued]
====

[discrete]
[[create-a-data-stream]]
=== Create a data stream

With a composable template, you can create a data stream using one of two
methods:

* Submit an <<add-documents-to-a-data-stream,indexing request>> to a target
matching the name or wildcard pattern defined in the template's `index_patterns`
property.
+
--
If the indexing request's target doesn't exist, {es} creates the data stream and
uses the target name as the name for the stream.

NOTE: Data streams support only specific types of indexing requests. See
<<add-documents-to-a-data-stream>>.

.*Example: Index documents to create a data stream*
[%collapsible]
====
The following <<docs-index_,index API>> request targets `logs`, which matches
the wildcard pattern for the `logs_data_stream` template. Because no existing
index or data stream uses this name, this request creates the `logs` data stream
and indexes the document to it.

[source,console]
----
POST /logs/_doc/
{
  "@timestamp": "2020-12-06T11:04:05.000Z",
  "user": {
    "id": "vlb44hny"
  },
  "message": "Login attempt failed"
}
----
// TEST[continued]

The API returns the following response. Note the `_index` property contains
`.ds-logs-000001`, indicating the document was indexed to the write index of the
new `logs` data stream.

[source,console-result]
----
{
  "_index": ".ds-logs-000001",
  "_id": "qecQmXIBT4jB8tq1nG0j",
  "_type": "_doc",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
----
// TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/]
====
--

* Use the <<indices-create-data-stream,create data stream API>> to manually
create a data stream. The name of the data stream must match the
name or wildcard pattern defined in the template's `index_patterns` property.
+
--
.*Example: Manually create a data stream*
[%collapsible]
====
The following <<indices-create-data-stream,create data stream API>> request
targets `logs_alt`, which matches the wildcard pattern for the
`logs_data_stream` template. Because no existing index or data stream uses this
name, this request creates the `logs_alt` data stream.

[source,console]
----
PUT /_data_stream/logs_alt
----
// TEST[continued]
====
--

////
[source,console]
----
DELETE /_data_stream/logs

DELETE /_data_stream/logs_alt

DELETE /_index_template/logs_data_stream

DELETE /_ilm/policy/logs_policy
----
// TEST[continued]
////