[role="xpack"] [[data-streams]] = Data streams ++++ Data streams ++++ A _data stream_ is a convenient, scalable way to ingest, search, and manage continuously generated time-series data. Time-series data, such as logs, tends to grow over time. While storing an entire time series in a single {es} index is simpler, it is often more efficient and cost-effective to store large volumes of data across multiple, time-based indices. Multiple indices let you move indices containing older, less frequently queried data to less expensive hardware and delete indices when they're no longer needed, reducing overhead and storage costs. A data stream is designed to give you the best of both worlds: * The simplicity of a single named resource you can use for requests * The storage, scalability, and cost-saving benefits of multiple indices You can submit indexing and search requests directly to a data stream. The stream automatically routes the requests to a collection of hidden _backing indices_ that store the stream's data. You can use <> to automate the management of these backing indices. {ilm-init} lets you automatically spin up new backing indices, allocate indices to different hardware, delete old indices, and take other automatic actions based on age or size criteria you set. Use data streams and {ilm-init} to seamlessly scale your data storage based on your budget, performance, resiliency, and retention needs. [discrete] [[when-to-use-data-streams]] == When to use data streams We recommend using data streams if you: * Use {es} to ingest, search, and manage large volumes of time-series data * Want to scale and reduce costs by using {ilm-init} to automate the management of your indices * Index large volumes of time-series data in {es} but rarely delete or update individual documents [discrete] [[backing-indices]] == Backing indices A data stream consists of one or more _backing indices_. Backing indices are <>, auto-generated indices used to store a stream's documents. image::images/data-streams/data-streams-diagram.svg[align="center"] To create backing indices, each data stream requires a matching <>. This template acts as a blueprint for the stream's backing indices. It contains: * The mappings and settings applied to each backing index when it's created. * A name or wildcard (`*`) pattern that matches the data stream's name. * A `data_stream` object with an empty body (`{ }`). This object indicates the template is used for data streams. A `@timestamp` field must be included in every document indexed to the data stream. This field can be mapped as a <> or <> field data type in the stream's matching index template. If no mapping is specified in the template, the `date` field data type with default options is used. The same index template can be used to create multiple data streams. [discrete] [[data-streams-generation]] == Generation Each data stream tracks its _generation_: a six-digit, zero-padded integer that acts as a cumulative count of the data stream's backing indices. This count includes any deleted indices for the stream. The generation is incremented whenever a new backing index is added to the stream. When a backing index is created, the index is named using the following convention: [source,text] ---- .ds-- ---- For example, the `web_server_logs` data stream has a generation of `34`. The most recently created backing index for this data stream is named `.ds-web_server_logs-000034`. Because the generation increments with each new backing index, backing indices with a higher generation contain more recent data. Backing indices with a lower generation contain older data. A backing index's name can change after its creation due to a <>, <>, or other operations. However, renaming a backing index does not detach it from a data stream. [discrete] [[data-stream-read-requests]] == Read requests When a read request is sent to a data stream, it routes the request to all its backing indices. For example, a search request sent to a data stream would query all its backing indices. image::images/data-streams/data-streams-search-request.svg[align="center"] [discrete] [[data-stream-write-index]] == Write index The most recently created backing index is the data stream’s only _write index_. The data stream routes all indexing requests for new documents to this index. image::images/data-streams/data-streams-index-request.svg[align="center"] You cannot add new documents to a stream's other backing indices, even by sending requests directly to the index. Because it's the only index capable of ingesting new documents, you cannot perform operations on a write index that might hinder indexing. These prohibited operations include: * <> * <> * <> * <> * <> * <> [discrete] [[data-streams-rollover]] == Rollover When a data stream is created, one backing index is automatically created. Because this single index is also the most recently created backing index, it acts as the stream's write index. A <> creates a new backing index for a data stream. This new backing index becomes the stream's write index, replacing the current one, and increments the stream's generation. In most cases, we recommend using <> to automate rollovers for data streams. This lets you automatically roll over the current write index when it meets specified criteria, such as a maximum age or size. However, you can also use the <> to manually perform a rollover. See <>. [discrete] [[data-streams-append-only]] == Append-only For most time-series use cases, existing data is rarely, if ever, updated. Because of this, data streams are designed to be append-only. You can send <> directly to a data stream. However, you cannot send the update or deletion requests for existing documents directly to a data stream. Instead, you can use the <> and <> APIs to update or delete existing documents in a data stream. See <> and <>. If needed, you can update or delete a document by submitting requests to the backing index containing the document. See <>. TIP: If you frequently update or delete existing documents, we recommend using an <> and <> instead of a data stream. You can still use <> to manage indices for the alias. include::set-up-a-data-stream.asciidoc[] include::use-a-data-stream.asciidoc[] include::change-mappings-and-settings.asciidoc[]