diff --git a/docs/reference/data-streams.asciidoc b/docs/reference/data-streams/data-streams.asciidoc similarity index 94% rename from docs/reference/data-streams.asciidoc rename to docs/reference/data-streams/data-streams.asciidoc index ca3139638c9..43352ce1614 100644 --- a/docs/reference/data-streams.asciidoc +++ b/docs/reference/data-streams/data-streams.asciidoc @@ -1,7 +1,8 @@ -[chapter] [[data-streams]] -= Data Streams += Data streams +[partintro] +-- You can use data streams to index time-based data that's continuously generated. A data stream groups indices from the same time-based data source. A data stream tracks its indices, known as _backing indices_, using an ordered @@ -116,3 +117,14 @@ The following APIs are available for managing data streams: * To get information about data streams, use the <>. * To delete data streams, use the <>. * To manually create a data stream, use the <>. + +[discrete] +[[data-streams-toc]] +== In this section + +* <> +* <> +-- + +include::set-up-a-data-stream.asciidoc[] +include::use-a-data-stream.asciidoc[] diff --git a/docs/reference/data-streams/set-up-a-data-stream.asciidoc b/docs/reference/data-streams/set-up-a-data-stream.asciidoc new file mode 100644 index 00000000000..dec5f55700c --- /dev/null +++ b/docs/reference/data-streams/set-up-a-data-stream.asciidoc @@ -0,0 +1,243 @@ +[[set-up-a-data-stream]] +== Set up a data stream + +To set up a data stream, follow these steps: + +. Check the <>. +. <>. +. <>. +. <>. + +After you set up a data stream, you can <> for indexing, searches, and other supported operations. + +[discrete] +[[data-stream-prereqs]] +=== Prerequisites + +* {es} data streams are intended for time-series data only. Each document +indexed to a data stream must contain a shared timestamp field. ++ +TIP: Data streams work well with most common log formats. While no schema is +required to use data streams, we recommend the {ecs-ref}[Elastic Common Schema +(ECS)]. + +* Data streams are designed to be append-only. While you can index new documents +directly to a data stream, you cannot use a data stream to directly update or +delete individual documents. To update or delete specific documents in a data +stream, submit a <> or <> API request to +the backing index containing the document. + + +[discrete] +[[configure-a-data-stream-ilm-policy]] +=== Optional: Configure an {ilm-init} lifecycle policy for a data stream + +You can use <> to automatically +manage a data stream's backing indices. For example, you could use {ilm-init} +to: + +* Spin up a new write index for the data stream when the current one reaches a + certain size or age. +* Move older backing indices to slower, less expensive hardware. +* Delete stale backing indices to enforce data retention standards. + +To use {ilm-init} with a data stream, you must +<>. This lifecycle policy +should contain the automated actions to take on backing indices and the +triggers for such actions. + +TIP: While optional, we recommend using {ilm-init} to scale data streams in +production. + +.*Example* +[%collapsible] +==== +The following <> request +configures the `logs_policy` lifecycle policy. + +The `logs_policy` policy uses the <> to create a +new write index for the data stream when the current one reaches 25GB in size. +The policy also deletes backing indices 30 days after their rollover. + +[source,console] +---- +PUT /_ilm/policy/logs_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_size": "25GB" + } + } + }, + "delete": { + "min_age": "30d", + "actions": { + "delete": {} + } + } + } + } +} +---- +==== + + +[discrete] +[[create-a-data-stream-template]] +=== Create a composable template for a data stream + +Each data stream requires a <>. The data +stream uses this template to create its backing indices. + +Composable templates for data streams must contain: + +* A name or wildcard (`*`) pattern for the data stream in the `index_patterns` + property. + +* A `data_stream` definition containing the `timestamp_field` property. + This timestamp field must be included in every document indexed to the data + stream. + +* A <> or <> field mapping for the + timestamp field specified in the `timestamp_field` property. + +* If you intend to use {ilm-init}, you must specify the + <> in the + `index.lifecycle.name` setting. + +You can also specify other mappings and settings you'd like to apply to the +stream's backing indices. + +.*Example* +[%collapsible] +==== +The following <> request +configures the `logs_data_stream` template. + +[source,console] +---- +PUT /_index_template/logs_data_stream +{ + "index_patterns": [ "logs*" ], + "data_stream": { + "timestamp_field": "@timestamp" + }, + "template": { + "mappings": { + "properties": { + "@timestamp": { + "type": "date" + } + } + }, + "settings": { + "index.lifecycle.name": "logs_policy" + } + } +} +---- +// TEST[continued] +==== + +[discrete] +[[create-a-data-stream]] +=== Create a data stream + +With a composable template, you can create a data stream using one of two +methods: + +* Submit an <> to a target +matching the name or wildcard pattern defined in the template's `index_patterns` +property. ++ +-- +If the indexing request's target doesn't exist, {es} creates the data stream and +uses the target name as the name for the stream. + +NOTE: Data streams support only specific types of indexing requests. See +<>. + +.*Example: Index documents to create a data stream* +[%collapsible] +==== +The following <> request targets `logs`, which matches +the wildcard pattern for the `logs_data_stream` template. Because no existing +index or data stream uses this name, this request creates the `logs` data stream +and indexes the document to it. + +[source,console] +---- +POST /logs/_doc/ +{ + "@timestamp": "2020-12-06T11:04:05.000Z", + "user": { + "id": "vlb44hny" + }, + "message": "Login attempt failed" +} +---- +// TEST[continued] + +The API returns the following response. Note the `_index` property contains +`.ds-logs-000001`, indicating the document was indexed to the write index of the +new `logs` data stream. + +[source,console-result] +---- +{ + "_index": ".ds-logs-000001", + "_id": "qecQmXIBT4jB8tq1nG0j", + "_type": "_doc", + "_version": 1, + "result": "created", + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 0, + "_primary_term": 1 +} +---- +// TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/] +==== +-- + +* Use the <> to manually +create a data stream. The name of the data stream must match the +name or wildcard pattern defined in the template's `index_patterns` property. ++ +-- +.*Example: Manually create a data stream* +[%collapsible] +==== +The following <> request +targets `logs_alt`, which matches the wildcard pattern for the +`logs_data_stream` template. Because no existing index or data stream uses this +name, this request creates the `logs_alt` data stream. + +[source,console] +---- +PUT /_data_stream/logs_alt +---- +// TEST[continued] +==== +-- + +//// +[source,console] +---- +DELETE /_data_stream/logs + +DELETE /_data_stream/logs_alt + +DELETE /_index_template/logs_data_stream + +DELETE /_ilm/policy/logs_policy +---- +// TEST[continued] +//// \ No newline at end of file diff --git a/docs/reference/data-streams/use-a-data-stream.asciidoc b/docs/reference/data-streams/use-a-data-stream.asciidoc new file mode 100644 index 00000000000..cdca8b0ff1b --- /dev/null +++ b/docs/reference/data-streams/use-a-data-stream.asciidoc @@ -0,0 +1,185 @@ +[[use-a-data-stream]] +== Use a data stream + +After you <>, you can do +the following: + +* <> +* <> +* <> + +//// +[source,console] +---- +PUT /_index_template/logs_data_stream +{ + "index_patterns": [ "logs*" ], + "data_stream": { + "timestamp_field": "@timestamp" + }, + "template": { + "mappings": { + "properties": { + "@timestamp": { + "type": "date" + } + } + } + } +} + +PUT /_data_stream/logs +---- +//// + +[discrete] +[[add-documents-to-a-data-stream]] +=== Add documents to a data stream + +You can add documents to a data stream using the following requests: + +* An <> request with an +<> set to `create`. Specify the data +stream's name in place of an index name. ++ +-- +NOTE: The `op_type` parameter defaults to `create` when adding new documents. + +.*Example: Index API request* +[%collapsible] +==== +The following <> adds a new document to the `logs` data +stream. + +[source,console] +---- +POST /logs/_doc/ +{ + "@timestamp": "2020-12-07T11:06:07.000Z", + "user": { + "id": "8a4f500d" + }, + "message": "Login successful" +} +---- +// TEST[continued] +==== +-- + +* A <> request using the `create` action. Specify the data +stream's name in place of an index name. ++ +-- +NOTE: Data streams do not support other bulk actions, such as `index`. + +.*Example: Bulk API request* +[%collapsible] +==== +The following <> index request adds several new documents to +the `logs` data stream. Note that only the `create` action is used. + +[source,console] +---- +PUT /logs/_bulk?refresh +{"create":{"_index" : "logs"}} +{ "@timestamp": "2020-12-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" } +{"create":{"_index" : "logs"}} +{ "@timestamp": "2020-12-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" } +{"create":{"_index" : "logs"}} +{ "@timestamp": "2020-12-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" } +---- +// TEST[continued] +==== +-- + +[discrete] +[[search-a-data-stream]] +=== Search a data stream + +The following search APIs support data streams: + +* <> +* <> +* <> +* <> +//// +* <> +//// + +.*Example* +[%collapsible] +==== +The following <> request searches the `logs` data +stream for documents with a timestamp between today and yesterday that also have +`message` value of `login successful`. + +[source,console] +---- +GET /logs/_search +{ + "query": { + "bool": { + "must": { + "range": { + "@timestamp": { + "gte": "now-1d/d", + "lt": "now/d" + } + } + }, + "should": { + "match": { + "message": "login successful" + } + } + } + } +} +---- +// TEST[continued] +==== + +[discrete] +[[manually-roll-over-a-data-stream]] +=== Manually roll over a data stream + +A rollover creates a new backing index for a data stream. This new backing index +becomes the stream's new write index and increments the stream's generation. + +In most cases, we recommend using <> to +automate rollovers for data streams. This lets you automatically roll over the +current write index when it meets specified criteria, such as a maximum age or +size. + +However, you can also use the <> to +manually perform a rollover. This can be useful if you want to apply mapping or +setting changes to the stream's write index after updating a data stream's +template. + +.*Example* +[%collapsible] +==== +The following <> request submits a manual +rollover request for the `logs` data stream. + +[source,console] +---- +POST /logs/_rollover/ +{ + "conditions": { + "max_docs": "1" + } +} +---- +// TEST[continued] +==== + +//// +[source,console] +---- +DELETE /_data_stream/logs + +DELETE /_index_template/logs_data_stream +---- +// TEST[continued] +//// \ No newline at end of file diff --git a/docs/reference/docs/index_.asciidoc b/docs/reference/docs/index_.asciidoc index 89130ba4894..59aa3426feb 100644 --- a/docs/reference/docs/index_.asciidoc +++ b/docs/reference/docs/index_.asciidoc @@ -41,6 +41,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=if_seq_no] include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=if_primary_term] +[[docs-index-api-op_type]] `op_type`:: (Optional, enum) Set to `create` to only index the document if it does not already exist (_put if absent_). If a document with the specified diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 31aac2de8ec..f4b97d2efb2 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -18,6 +18,8 @@ include::setup.asciidoc[] include::upgrade.asciidoc[] +include::data-streams/data-streams.asciidoc[] + include::search/index.asciidoc[] include::query-dsl.asciidoc[] @@ -58,8 +60,6 @@ include::high-availability.asciidoc[] include::snapshot-restore/index.asciidoc[] -include::data-streams.asciidoc[] - include::{xes-repo-dir}/security/index.asciidoc[] include::{xes-repo-dir}/watcher/index.asciidoc[]