291 lines
7.1 KiB
Plaintext
291 lines
7.1 KiB
Plaintext
[[use-a-data-stream]]
|
|
== Use a data stream
|
|
|
|
After you <<set-up-a-data-stream,set up a data stream>>, you can do
|
|
the following:
|
|
|
|
* <<add-documents-to-a-data-stream>>
|
|
* <<search-a-data-stream>>
|
|
* <<manually-roll-over-a-data-stream>>
|
|
* <<reindex-with-a-data-stream>>
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
PUT /_index_template/logs_data_stream
|
|
{
|
|
"index_patterns": [ "logs*" ],
|
|
"data_stream": {
|
|
"timestamp_field": "@timestamp"
|
|
},
|
|
"template": {
|
|
"mappings": {
|
|
"properties": {
|
|
"@timestamp": {
|
|
"type": "date"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
PUT /_data_stream/logs
|
|
----
|
|
////
|
|
|
|
[discrete]
|
|
[[add-documents-to-a-data-stream]]
|
|
=== Add documents to a data stream
|
|
|
|
You can add documents to a data stream using the following requests:
|
|
|
|
* An <<docs-index_,index API>> request with an
|
|
<<docs-index-api-op_type,`op_type`>> set to `create`. Specify the data
|
|
stream's name in place of an index name.
|
|
+
|
|
--
|
|
NOTE: The `op_type` parameter defaults to `create` when adding new documents.
|
|
|
|
.*Example: Index API request*
|
|
[%collapsible]
|
|
====
|
|
The following <<docs-index_,index API>> adds a new document to the `logs` data
|
|
stream.
|
|
|
|
[source,console]
|
|
----
|
|
POST /logs/_doc/
|
|
{
|
|
"@timestamp": "2020-12-07T11:06:07.000Z",
|
|
"user": {
|
|
"id": "8a4f500d"
|
|
},
|
|
"message": "Login successful"
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
--
|
|
|
|
* A <<docs-bulk,bulk API>> request using the `create` action. Specify the data
|
|
stream's name in place of an index name.
|
|
+
|
|
--
|
|
NOTE: Data streams do not support other bulk actions, such as `index`.
|
|
|
|
.*Example: Bulk API request*
|
|
[%collapsible]
|
|
====
|
|
The following <<docs-bulk,bulk API>> index request adds several new documents to
|
|
the `logs` data stream. Note that only the `create` action is used.
|
|
|
|
[source,console]
|
|
----
|
|
PUT /logs/_bulk?refresh
|
|
{"create":{"_index" : "logs"}}
|
|
{ "@timestamp": "2020-12-08T11:04:05.000Z", "user": { "id": "vlb44hny" }, "message": "Login attempt failed" }
|
|
{"create":{"_index" : "logs"}}
|
|
{ "@timestamp": "2020-12-08T11:06:07.000Z", "user": { "id": "8a4f500d" }, "message": "Login successful" }
|
|
{"create":{"_index" : "logs"}}
|
|
{ "@timestamp": "2020-12-09T11:07:08.000Z", "user": { "id": "l7gk7f82" }, "message": "Logout successful" }
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
--
|
|
|
|
[discrete]
|
|
[[search-a-data-stream]]
|
|
=== Search a data stream
|
|
|
|
The following search APIs support data streams:
|
|
|
|
* <<search-search, Search>>
|
|
* <<async-search, Async search>>
|
|
* <<search-multi-search, Multi search>>
|
|
* <<search-field-caps, Field capabilities>>
|
|
////
|
|
* <<eql-search-api, EQL search>>
|
|
////
|
|
|
|
.*Example*
|
|
[%collapsible]
|
|
====
|
|
The following <<search-search,search API>> request searches the `logs` data
|
|
stream for documents with a timestamp between today and yesterday that also have
|
|
`message` value of `login successful`.
|
|
|
|
[source,console]
|
|
----
|
|
GET /logs/_search
|
|
{
|
|
"query": {
|
|
"bool": {
|
|
"must": {
|
|
"range": {
|
|
"@timestamp": {
|
|
"gte": "now-1d/d",
|
|
"lt": "now/d"
|
|
}
|
|
}
|
|
},
|
|
"should": {
|
|
"match": {
|
|
"message": "login successful"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
|
|
[discrete]
|
|
[[manually-roll-over-a-data-stream]]
|
|
=== Manually roll over a data stream
|
|
|
|
A rollover creates a new backing index for a data stream. This new backing index
|
|
becomes the stream's <<data-stream-write-index,write index>> and increments
|
|
the stream's <<data-streams-generation,generation>>.
|
|
|
|
In most cases, we recommend using <<index-lifecycle-management,{ilm-init}>> to
|
|
automate rollovers for data streams. This lets you automatically roll over the
|
|
current write index when it meets specified criteria, such as a maximum age or
|
|
size.
|
|
|
|
However, you can also use the <<indices-rollover-index,rollover API>> to
|
|
manually perform a rollover. This can be useful if you want to
|
|
<<data-streams-change-mappings-and-settings,apply mapping or setting changes>>
|
|
to the stream's write index after updating a data stream's template.
|
|
|
|
.*Example*
|
|
[%collapsible]
|
|
====
|
|
The following <<indices-rollover-index,rollover API>> request submits a manual
|
|
rollover request for the `logs` data stream.
|
|
|
|
[source,console]
|
|
----
|
|
POST /logs/_rollover/
|
|
{
|
|
"conditions": {
|
|
"max_docs": "1"
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
|
|
[discrete]
|
|
[[reindex-with-a-data-stream]]
|
|
=== Reindex with a data stream
|
|
|
|
You can use the <<docs-reindex,reindex API>> to copy documents to a data stream
|
|
from an existing index, index alias, or data stream.
|
|
|
|
A reindex copies documents from a _source_ to a _destination_. The source and
|
|
destination can be any pre-existing index, index alias, or data stream. However,
|
|
the source and destination must be different. You cannot reindex a data stream
|
|
into itself.
|
|
|
|
Because data streams are <<data-streams-append-only,append-only>>, a reindex
|
|
request to a data stream destination must have an `op_type` of `create`. This
|
|
means a reindex can only add new documents to a data stream. It cannot update
|
|
existing documents in the data stream destination.
|
|
|
|
A reindex can be used to:
|
|
|
|
* Convert an existing index alias and collection of time-based indices into a
|
|
data stream.
|
|
|
|
* Apply a new or updated <<create-a-data-stream-template,composable template>>
|
|
by reindexing an existing data stream into a new one. This applies mapping
|
|
and setting changes in the template to each document and backing index of the
|
|
data stream destination. See
|
|
<<data-streams-use-reindex-to-change-mappings-settings>>.
|
|
|
|
TIP: If you only want to update the mappings or settings of a data stream's
|
|
write index, we recommend you update the <<create-a-data-stream-template,data
|
|
stream's template>> and perform a <<manually-roll-over-a-data-stream,rollover>>.
|
|
|
|
.*Example*
|
|
[%collapsible]
|
|
====
|
|
The following reindex request copies documents from the `archive` index alias to
|
|
the existing `logs` data stream. Because the destination is a data stream, the
|
|
request's `op_type` is `create`.
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
PUT /_bulk?refresh=wait_for
|
|
{"create":{"_index" : "archive_1"}}
|
|
{ "@timestamp": "2020-12-08T11:04:05.000Z" }
|
|
{"create":{"_index" : "archive_2"}}
|
|
{ "@timestamp": "2020-12-08T11:06:07.000Z" }
|
|
{"create":{"_index" : "archive_2"}}
|
|
{ "@timestamp": "2020-12-09T11:07:08.000Z" }
|
|
{"create":{"_index" : "archive_2"}}
|
|
{ "@timestamp": "2020-12-09T11:07:08.000Z" }
|
|
|
|
POST /_aliases
|
|
{
|
|
"actions" : [
|
|
{ "add" : { "index" : "archive_1", "alias" : "archive" } },
|
|
{ "add" : { "index" : "archive_2", "alias" : "archive", "is_write_index" : true} }
|
|
]
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
////
|
|
|
|
[source,console]
|
|
----
|
|
POST /_reindex
|
|
{
|
|
"source": {
|
|
"index": "archive"
|
|
},
|
|
"dest": {
|
|
"index": "logs",
|
|
"op_type": "create"
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
|
|
You can also reindex documents from a data stream to an index, index
|
|
alias, or data stream.
|
|
|
|
.*Example*
|
|
[%collapsible]
|
|
====
|
|
The following reindex request copies documents from the `logs` data stream
|
|
to the existing `archive` index alias. Because the destination is not a data
|
|
stream, the `op_type` does not need to be specified.
|
|
|
|
[source,console]
|
|
----
|
|
POST /_reindex
|
|
{
|
|
"source": {
|
|
"index": "logs"
|
|
},
|
|
"dest": {
|
|
"index": "archive"
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
====
|
|
|
|
////
|
|
[source,console]
|
|
----
|
|
DELETE /_data_stream/logs
|
|
|
|
DELETE /_index_template/logs_data_stream
|
|
----
|
|
// TEST[continued]
|
|
//// |