honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ingest: document fields that support templating (#34536) 

This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
This commit is contained in:
Jake Landis 2018-10-23 13:28:44 -05:00 committed by GitHub
parent e242fd2e42
commit a8e1ee34ca
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 32 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
docs/reference/ingest/ingest-node.asciidoc
View File

@ -776,16 +776,16 @@ Accepts a single value or an array of values.
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be appended to
| `value` | yes | - | The value to be appended
| `field` | yes | - | The field to be appended to. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be appended. Supports <<accessing-template-fields,template snippets>>.
|======
[source,js]
--------------------------------------------------
{
"append": {
"field": "field1",
"value": ["item2", "item3", "item4"]
"field": "tags",
"value": ["production", "{{app}}", "{{owner}}"]
}
}
--------------------------------------------------
@ -812,7 +812,7 @@ the field is not a supported format or resultant value exceeds 2^63.
--------------------------------------------------
{
"bytes": {
"field": "foo"
"field": "file.size"
}
}
--------------------------------------------------
@ -850,7 +850,7 @@ still be updated with the unconverted field value.
--------------------------------------------------
{
"convert": {
"field" : "foo",
"field" : "url.port",
"type": "integer"
}
}
@ -874,8 +874,8 @@ in the same order they were defined as part of the processor definition.
| `field` | yes | - | The field to get the date from.
| `target_field` | no | @timestamp | The field that will hold the parsed date.
| `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days.
| `timezone` | no | UTC | The timezone to use when parsing the date. Supports <<accessing-template-fields,template snippets>>.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <<accessing-template-fields,template snippets>>.
|======
Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
@ -913,8 +913,8 @@ the timezone and locale values.
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["ISO8601"],
"timezone" : "{{ my_timezone }}",
"locale" : "{{ my_locale }}"
"timezone" : "{{my_timezone}}",
"locale" : "{{my_locale}}"
}
}
]
@ -1059,12 +1059,12 @@ understands this to mean `2016-04-01` as is explained in the <<date-math-index-n
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to get the date or timestamp from.
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. Supports <<accessing-template-fields,template snippets>>.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports <<accessing-template-fields,template snippets>>.
| `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here. Supports <<accessing-template-fields,template snippets>>.
|======
[[dissect-processor]]
@ -1405,14 +1405,15 @@ to the requester.
[options="header"]
|======
| Name | Required | Default | Description
| `message` | yes | - | The error message of the `FailException` thrown by the processor
| `message` | yes | - | The error message thrown by the processor. Supports <<accessing-template-fields,template snippets>>.
|======
[source,js]
--------------------------------------------------
{
"fail": {
"message": "an error message"
"if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
}
}
--------------------------------------------------
@ -2117,7 +2118,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | Fields to be removed
| `field` | yes | - | Fields to be removed. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document
|======
@ -2127,7 +2128,7 @@ Here is an example to remove a single field:
--------------------------------------------------
{
"remove": {
"field": "foo"
"field": "user_agent"
}
}
--------------------------------------------------
@ -2139,7 +2140,7 @@ To remove multiple fields, you can use the following query:
--------------------------------------------------
{
"remove": {
"field": ["foo", "bar"]
"field": ["user_agent", "url"]
}
}
--------------------------------------------------
@ -2153,18 +2154,18 @@ Renames an existing field. If the field doesn't exist or the new name is already
.Rename Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed
| `target_field` | yes | - | The new name of the field
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed. Supports <<accessing-template-fields,template snippets>>.
| `target_field` | yes | - | The new name of the field. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|======
[source,js]
--------------------------------------------------
{
"rename": {
"field": "foo",
"target_field": "foobar"
"field": "provider",
"target_field": "cloud.provider"
}
}
--------------------------------------------------
@ -2282,18 +2283,18 @@ its value will be replaced with the provided one.
.Set Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update
| `value` | yes | - | The value to be set for the field
| `override`| no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be set for the field. Supports <<accessing-template-fields,template snippets>>.
| `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
|======
[source,js]
--------------------------------------------------
{
"set": {
"field": "field1",
"value": 582.1
"field": "host.os.name",
"value": "{{os}}"
}
}
--------------------------------------------------
@ -2346,7 +2347,7 @@ Throws an error when the field is not an array.
--------------------------------------------------
{
"sort": {
"field": "field_to_sort",
"field": "array_field_to_sort",
"order": "desc"
}
}