[[data-streams-overview]] == Data streams overview ++++ Overview ++++ A data stream consists of one or more _backing indices_. Backing indices are <>, auto-generated indices used to store a stream's documents. image::images/data-streams/data-streams-diagram.svg[align="center"] The creation of a data stream requires an associated <>. This template acts as a blueprint for the stream's backing indices. It contains: * A name or wildcard (`*`) pattern for the data stream. * The data stream's _timestamp field_. This field must be mapped as a <> or <> field datatype and must be included in every document indexed to the data stream. * The mappings and settings applied to each backing index when it's created. The same composable template can be used to create multiple data streams. See <>. [discrete] [[data-streams-generation]] === Generation Each data stream tracks its _generation_: a six-digit, zero-padded integer that acts as a cumulative count of the data stream's backing indices. This count includes any deleted indices for the stream. The generation is incremented whenever a new backing index is added to the stream. When a backing index is created, the index is named using the following convention: [source,text] ---- .ds-- ---- .*Example* [%collapsible] ==== The `web_server_logs` data stream has a generation of `34`. The most recently created backing index for this data stream is named `.ds-web_server_logs-000034`. ==== Because the generation increments with each new backing index, backing indices with a higher generation contain more recent data. Backing indices with a lower generation contain older data. A backing index's name can change after its creation due to a <>, <>, or other operations. [discrete] [[data-stream-write-index]] === Write index When a read request is sent to a data stream, it routes the request to all its backing indices. For example, a search request sent to a data stream would query all its backing indices. image::images/data-streams/data-streams-search-request.svg[align="center"] However, the most recently created backing index is the data stream’s only _write index_. The data stream routes all indexing requests for new documents to this index. image::images/data-streams/data-streams-index-request.svg[align="center"] You cannot add new documents to a stream's other backing indices, even by sending requests directly to the index. This means you cannot submit the following requests directly to any backing index except the write index: * An <> request with an <> of `create`. The `op_type` parameter defaults to `create` when adding new documents. * A <> request using a `create` action Because it's the only index capable of ingesting new documents, you cannot perform operations on a write index that might hinder indexing. These prohibited operations include: * <> * <> * <> * <> * <> * <> [discrete] [[data-streams-rollover]] === Rollover When a data stream is created, one backing index is automatically created. Because this single index is also the most recently created backing index, it acts as the stream's write index. A <> creates a new backing index for a data stream. This new backing index becomes the stream's write index, replacing the current one, and increments the stream's generation. In most cases, we recommend using <> to automate rollovers for data streams. This lets you automatically roll over the current write index when it meets specified criteria, such as a maximum age or size. However, you can also use the <> to manually perform a rollover. See <>. [discrete] [[data-streams-append-only]] === Append-only For most time-series use cases, existing data is rarely, if ever, updated. Because of this, data streams are designed to be append-only. This means you can send indexing requests for new documents directly to a data stream. However, you cannot send update or deletion requests for existing documents to a data stream. To update or delete specific documents in a data stream, submit one of the following requests to the backing index containing the document: * An <> request with an <> of `index`. These requests must include valid <> arguments. * A <> request using the `delete`, `index`, or `update` action. If the action type is `index`, the action must include valid <> arguments. * A <> request See <>. TIP: If you need to frequently update or delete existing documents across multiple indices, we recommend using an <> and <> instead of a data stream. You can still use <> to manage the indices.