119 lines
3.7 KiB
Plaintext
119 lines
3.7 KiB
Plaintext
[chapter]
|
|
[[data-streams]]
|
|
= Data Streams
|
|
|
|
You can use data streams to index time-based data that's continuously generated.
|
|
A data stream groups indices from the same time-based data source.
|
|
A data stream tracks its indices, known as _backing indices_, using an ordered
|
|
list.
|
|
|
|
A data stream's backing indices are <<index-hidden,hidden>>.
|
|
While all backing indices handle read requests, the most recently created
|
|
backing index is the data stream's only write index. A data stream only
|
|
accepts <<docs-index_,index requests>> with `op_type` set to `create`. To update
|
|
or delete specific documents in a data stream, submit a <<docs-delete,delete>>
|
|
or <<docs-update,update>> API request to the backing index containing the
|
|
document.
|
|
|
|
To create a data stream, set up a <<indices-templates,composable index
|
|
template>> containing:
|
|
|
|
* A name or wildcard pattern for the data stream in the `index_patterns` property.
|
|
* A `data_stream` definition that contains the `timestamp_field` property.
|
|
The `timestamp_field` must be the primary timestamp field
|
|
for the data source. This field must be included in every
|
|
document indexed to the data stream.
|
|
|
|
When you index one or more documents to a not-yet-existent target matching
|
|
the template's name or pattern, {es} automatically creates the corresponding
|
|
data stream. You can also manually create a data stream using the
|
|
<<indices-create-data-stream,create data stream API>>. However, a composable
|
|
template for the stream is still required.
|
|
|
|
You can use the <<indices-rollover-index,rollover API>> to roll a data stream
|
|
over to a new index when the current write index meets specified criteria, such
|
|
as a maximum age or size. A rollover creates a new backing index and updates the
|
|
data stream's list of backing indices. This new index then becomes the stream's
|
|
new write index. See <<rollover-data-stream-ex>>.
|
|
|
|
[discrete]
|
|
[[create-data-stream]]
|
|
== Create a data stream
|
|
|
|
Create a composable template with a `data_stream` definition:
|
|
|
|
[source,console]
|
|
-----------------------------------
|
|
PUT /_index_template/logs_template
|
|
{
|
|
"index_patterns": ["logs-*"],
|
|
"data_stream": {
|
|
"timestamp_field": "@timestamp"
|
|
}
|
|
}
|
|
-----------------------------------
|
|
|
|
Start indexing data to a target matching the composable template's wildcard
|
|
pattern:
|
|
|
|
[source,console]
|
|
----
|
|
POST /logs-foobar/_doc
|
|
{
|
|
"@timestamp": "2050-11-15T14:12:12",
|
|
...
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
// TEST[s/,//]
|
|
// TEST[s/\.\.\.//]
|
|
|
|
Response:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"_shards" : {
|
|
"total" : 2,
|
|
"failed" : 0,
|
|
"successful" : 1
|
|
},
|
|
"_index" : ".ds-logs-foobar-000001",
|
|
"_type" : "_doc",
|
|
"_id" : "W0tpsmIBdwcYyG50zbta",
|
|
"_version" : 1,
|
|
"_seq_no" : 0,
|
|
"_primary_term" : 1,
|
|
"result": "created"
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/W0tpsmIBdwcYyG50zbta/$body._id/]
|
|
|
|
Or create a data stream using the create data stream API:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_data_stream/logs-barbaz
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
////
|
|
[source,console]
|
|
-----------------------------------
|
|
DELETE /_data_stream/logs-foobar
|
|
DELETE /_data_stream/logs-barbaz
|
|
DELETE /_index_template/logs_template
|
|
-----------------------------------
|
|
// TEST[continued]
|
|
////
|
|
|
|
[discrete]
|
|
[[data-streams-apis]]
|
|
== Data stream APIs
|
|
|
|
The following APIs are available for managing data streams:
|
|
|
|
* To get information about data streams, use the <<indices-get-data-stream, get data stream API>>.
|
|
* To delete data streams, use the <<indices-delete-data-stream, delete data stream API>>.
|
|
* To manually create a data stream, use the <<indices-create-data-stream, create data stream API>>.
|