honeymoose/OpenSearch
1
0
Fork 0
mirror of https://github.com/honeymoose/OpenSearch.git synced 2025-02-17 10:25:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

ingest: document fields that support templating (#34536)

Browse Source 
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
This commit is contained in:
Jake Landis 2018-10-23 13:28:44 -05:00 committed by GitHub
parent e242fd2e42
commit a8e1ee34ca
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 32 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
docs/reference/ingest/ingest-node.asciidoc
View File

@ -776,16 +776,16 @@ Accepts a single value or an array of values.
[options="header"] [options="header"]
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `field` | yes | - | The field to be appended to | `field` | yes | - | The field to be appended to. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be appended | `value` | yes | - | The value to be appended. Supports <<accessing-template-fields,template snippets>>.
|====== |======
[source,js] [source,js]
-------------------------------------------------- --------------------------------------------------
{ {
"append": { "append": {
"field": "field1", "field": "tags",
"value": ["item2", "item3", "item4"] "value": ["production", "{{app}}", "{{owner}}"]
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -812,7 +812,7 @@ the field is not a supported format or resultant value exceeds 2^63.
-------------------------------------------------- --------------------------------------------------
{ {
"bytes": { "bytes": {
"field": "foo" "field": "file.size"
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -850,7 +850,7 @@ still be updated with the unconverted field value.
-------------------------------------------------- --------------------------------------------------
{ {
"convert": { "convert": {
"field" : "foo", "field" : "url.port",
"type": "integer" "type": "integer"
} }
} }
@ -874,8 +874,8 @@ in the same order they were defined as part of the processor definition.
| `field` | yes | - | The field to get the date from. | `field` | yes | - | The field to get the date from.
| `target_field` | no | @timestamp | The field that will hold the parsed date. | `target_field` | no | @timestamp | The field that will hold the parsed date.
| `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. | `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date. | `timezone` | no | UTC | The timezone to use when parsing the date. Supports <<accessing-template-fields,template snippets>>.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. | `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <<accessing-template-fields,template snippets>>.
|====== |======
Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field: Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
@ -1059,12 +1059,12 @@ understands this to mean `2016-04-01` as is explained in the <<date-math-index-n
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `field` | yes | - | The field to get the date or timestamp from. | `field` | yes | - | The field to get the date or timestamp from.
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. | `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. Supports <<accessing-template-fields,template snippets>>.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). | `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports <<accessing-template-fields,template snippets>>.
| `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. | `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names. | `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days. | `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here. | `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here. Supports <<accessing-template-fields,template snippets>>.
|====== |======
[[dissect-processor]] [[dissect-processor]]
@ -1405,14 +1405,15 @@ to the requester.
[options="header"] [options="header"]
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `message` | yes | - | The error message of the `FailException` thrown by the processor | `message` | yes | - | The error message thrown by the processor. Supports <<accessing-template-fields,template snippets>>.
|====== |======
[source,js] [source,js]
-------------------------------------------------- --------------------------------------------------
{ {
"fail": { "fail": {
"message": "an error message" "if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -2117,7 +2118,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown
[options="header"] [options="header"]
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `field` | yes | - | Fields to be removed | `field` | yes | - | Fields to be removed. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document | `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document
|====== |======
@ -2127,7 +2128,7 @@ Here is an example to remove a single field:
-------------------------------------------------- --------------------------------------------------
{ {
"remove": { "remove": {
"field": "foo" "field": "user_agent"
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -2139,7 +2140,7 @@ To remove multiple fields, you can use the following query:
-------------------------------------------------- --------------------------------------------------
{ {
"remove": { "remove": {
"field": ["foo", "bar"] "field": ["user_agent", "url"]
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -2154,8 +2155,8 @@ Renames an existing field. If the field doesn't exist or the new name is already
[options="header"] [options="header"]
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `field` | yes | - | The field to be renamed | `field` | yes | - | The field to be renamed. Supports <<accessing-template-fields,template snippets>>.
| `target_field` | yes | - | The new name of the field | `target_field` | yes | - | The new name of the field. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document | `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|====== |======
@ -2163,8 +2164,8 @@ Renames an existing field. If the field doesn't exist or the new name is already
-------------------------------------------------- --------------------------------------------------
{ {
"rename": { "rename": {
"field": "foo", "field": "provider",
"target_field": "foobar" "target_field": "cloud.provider"
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -2283,8 +2284,8 @@ its value will be replaced with the provided one.
[options="header"] [options="header"]
|====== |======
| Name | Required | Default | Description | Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update | `field` | yes | - | The field to insert, upsert, or update. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be set for the field | `value` | yes | - | The value to be set for the field. Supports <<accessing-template-fields,template snippets>>.
| `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched. | `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
|====== |======
@ -2292,8 +2293,8 @@ its value will be replaced with the provided one.
-------------------------------------------------- --------------------------------------------------
{ {
"set": { "set": {
"field": "field1", "field": "host.os.name",
"value": 582.1 "value": "{{os}}"
} }
} }
-------------------------------------------------- --------------------------------------------------
@ -2346,7 +2347,7 @@ Throws an error when the field is not an array.
-------------------------------------------------- --------------------------------------------------
{ {
"sort": { "sort": {
"field": "field_to_sort", "field": "array_field_to_sort",
"order": "desc" "order": "desc"
} }
} }