update ingest docs
- move ingest plugin docs to core reference docs - move geoip processor docs to plugins/ingest-geoip.asciidoc - add missing options tables for some processors - add description of pipeline definition - add description of processor definitions including common parameters like "tag" and "on_failure"
This commit is contained in:
Tal Levy
parent
a5a62932b9
commit
894efa3fb6
|
|
@ -0,0 +1,64 @@
|
||||||
|
[[ingest-geoip]]
|
||||||
|
== Ingest Geoip Processor Plugin
|
||||||
|
|
||||||
|
The GeoIP processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases.
|
||||||
|
This processor adds this information by default under the `geoip` field.
|
||||||
|
|
||||||
|
The ingest plugin ships by default with the GeoLite2 City and GeoLite2 Country geoip2 databases from Maxmind made available
|
||||||
|
under the CCA-ShareAlike 3.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/
|
||||||
|
|
||||||
|
The GeoIP processor can run with other geoip2 databases from Maxmind. The files must be copied into the geoip config directory
|
||||||
|
and the `database_file` option should be used to specify the filename of the custom database. The geoip config directory
|
||||||
|
is located at `$ES_HOME/config/ingest/geoip` and holds the shipped databases too.
|
||||||
|
|
||||||
|
[[geoip-options]]
|
||||||
|
.Geoip options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `source_field` | yes | - | The field to get the ip address or hostname from for the geographical lookup.
|
||||||
|
| `target_field` | no | geoip | The field that will hold the geographical information looked up from the Maxmind database.
|
||||||
|
| `database_file` | no | GeoLite2-City.mmdb | The database filename in the geoip config directory. The ingest plugin ships with the GeoLite2-City.mmdb and GeoLite2-Country.mmdb files.
|
||||||
|
| `fields` | no | [`continent_name`, `country_iso_code`, `region_name`, `city_name`, `location`] <1> | Controls what properties are added to the `target_field` based on the geoip lookup.
|
||||||
|
|======
|
||||||
|
|
||||||
|
<1> Depends on what is available in `database_field`:
|
||||||
|
* If the GeoLite2 City database is used then the following fields may be added under the `target_field`: `ip`,
|
||||||
|
`country_iso_code`, `country_name`, `continent_name`, `region_name`, `city_name`, `timezone`, `latitude`, `longitude`
|
||||||
|
and `location`. The fields actually added depend on what has been found and which fields were configured in `fields`.
|
||||||
|
* If the GeoLite2 Country database is used then the following fields may be added under the `target_field`: `ip`,
|
||||||
|
`country_iso_code`, `country_name` and `continent_name`.The fields actually added depend on what has been found and which fields were configured in `fields`.
|
||||||
|
|
||||||
|
An example that uses the default city database and adds the geographical information to the `geoip` field based on the `ip` field:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"description" : "...",
|
||||||
|
"processors" : [
|
||||||
|
{
|
||||||
|
"geoip" : {
|
||||||
|
"source_field" : "ip"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
An example that uses the default country database and add the geographical information to the `geo` field based on the `ip` field`:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"description" : "...",
|
||||||
|
"processors" : [
|
||||||
|
{
|
||||||
|
"geoip" : {
|
||||||
|
"source_field" : "ip",
|
||||||
|
"target_field" : "geo",
|
||||||
|
"database_file" : "GeoLite2-Country.mmdb"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
|
@ -28,12 +28,59 @@ PUT /my-index/my-type/my-id?pipeline=my_pipeline_id
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
// AUTOSENSE
|
// AUTOSENSE
|
||||||
|
|
||||||
|
=== Pipeline Definition
|
||||||
|
|
||||||
|
A pipeline is a definition of a series of processors that are to be
|
||||||
|
executed in the same sequential order as they are declared.
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"description" : "...",
|
||||||
|
"processors" : [ ... ]
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
The `description` is a special field to store a helpful description of
|
||||||
|
what the pipeline attempts to achieve.
|
||||||
|
|
||||||
|
The `processors` parameter defines a list of processors to be executed in
|
||||||
|
order.
|
||||||
|
|
||||||
=== Processors
|
=== Processors
|
||||||
|
|
||||||
|
All processors are defined in the following way within a pipeline definition:
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
{
|
||||||
|
"PROCESSOR_NAME" : {
|
||||||
|
... processor configuration options ...
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
Each processor defines its own configuration parameters, but all processors have
|
||||||
|
the ability to declare `tag` and `on_failure` fields. These fields are optional.
|
||||||
|
|
||||||
|
A `tag` is simply a string identifier of the specific instatiation of a certain
|
||||||
|
processor in a pipeline. The `tag` field does not affect any processor's behavior,
|
||||||
|
but is very useful for bookkeeping and tracing errors to specific processors.
|
||||||
|
|
||||||
|
See <<handling-failure-in-pipelines>> to learn more about the `on_failure` field and error handling in pipelines.
|
||||||
|
|
||||||
==== Set processor
|
==== Set processor
|
||||||
Sets one field and associates it with the specified value. If the field already exists,
|
Sets one field and associates it with the specified value. If the field already exists,
|
||||||
its value will be replaced with the provided one.
|
its value will be replaced with the provided one.
|
||||||
|
|
||||||
|
[[set-options]]
|
||||||
|
.Set Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to insert, upsert, or update
|
||||||
|
| `value` | yes | - | The value to be set for the field
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -50,6 +97,15 @@ Converts a scalar to an array and appends one or more values to it if the field
|
||||||
Creates an array containing the provided values if the fields doesn't exist.
|
Creates an array containing the provided values if the fields doesn't exist.
|
||||||
Accepts a single value or an array of values.
|
Accepts a single value or an array of values.
|
||||||
|
|
||||||
|
[[append-options]]
|
||||||
|
.Append Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to be appended to
|
||||||
|
| `value` | yes | - | The value to be appended
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -63,6 +119,14 @@ Accepts a single value or an array of values.
|
||||||
==== Remove processor
|
==== Remove processor
|
||||||
Removes an existing field. If the field doesn't exist, an exception will be thrown
|
Removes an existing field. If the field doesn't exist, an exception will be thrown
|
||||||
|
|
||||||
|
[[remove-options]]
|
||||||
|
.Remove Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to be removed
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -76,6 +140,15 @@ Removes an existing field. If the field doesn't exist, an exception will be thro
|
||||||
Renames an existing field. If the field doesn't exist, an exception will be thrown. Also, the new field
|
Renames an existing field. If the field doesn't exist, an exception will be thrown. Also, the new field
|
||||||
name must not exist.
|
name must not exist.
|
||||||
|
|
||||||
|
[[rename-options]]
|
||||||
|
.Rename Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to be renamed
|
||||||
|
| `to` | yes | - | The new name of the field
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -96,6 +169,15 @@ The supported types include: `integer`, `float`, `string`, and `boolean`.
|
||||||
`boolean` will set the field to true if its string value is equal to `true` (ignore case), to
|
`boolean` will set the field to true if its string value is equal to `true` (ignore case), to
|
||||||
false if its string value is equal to `false` (ignore case) and it will throw exception otherwise.
|
false if its string value is equal to `false` (ignore case) and it will throw exception otherwise.
|
||||||
|
|
||||||
|
[[convert-options]]
|
||||||
|
.Convert Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field whose value is to be converted
|
||||||
|
| `type` | yes | - | The type to convert the existing value to
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -110,9 +192,15 @@ false if its string value is equal to `false` (ignore case) and it will throw ex
|
||||||
Converts a string field by applying a regular expression and a replacement.
|
Converts a string field by applying a regular expression and a replacement.
|
||||||
If the field is not a string, the processor will throw an exception.
|
If the field is not a string, the processor will throw an exception.
|
||||||
|
|
||||||
This configuration takes a `field` for the field name, `pattern` for the
|
[[gsub-options]]
|
||||||
pattern to be replaced, and `replacement` for the string to replace the matching patterns with.
|
.Gsub Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field apply the replacement for
|
||||||
|
| `pattern` | yes | - | The pattern to be replaced
|
||||||
|
| `replacement` | yes | - | The string to replace the matching patterns with.
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
@ -129,6 +217,15 @@ pattern to be replaced, and `replacement` for the string to replace the matching
|
||||||
Joins each element of an array into a single string using a separator character between each element.
|
Joins each element of an array into a single string using a separator character between each element.
|
||||||
Throws error when the field is not an array.
|
Throws error when the field is not an array.
|
||||||
|
|
||||||
|
[[join-options]]
|
||||||
|
.Join Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to be separated
|
||||||
|
| `separator` | yes | - | The separator character
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -142,6 +239,14 @@ Throws error when the field is not an array.
|
||||||
==== Split processor
|
==== Split processor
|
||||||
Split a field to an array using a separator character. Only works on string fields.
|
Split a field to an array using a separator character. Only works on string fields.
|
||||||
|
|
||||||
|
[[split-options]]
|
||||||
|
.Split Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to split
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -154,6 +259,14 @@ Split a field to an array using a separator character. Only works on string fiel
|
||||||
==== Lowercase processor
|
==== Lowercase processor
|
||||||
Converts a string to its lowercase equivalent.
|
Converts a string to its lowercase equivalent.
|
||||||
|
|
||||||
|
[[lowercase-options]]
|
||||||
|
.Lowercase Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to lowercase
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -166,6 +279,14 @@ Converts a string to its lowercase equivalent.
|
||||||
==== Uppercase processor
|
==== Uppercase processor
|
||||||
Converts a string to its uppercase equivalent.
|
Converts a string to its uppercase equivalent.
|
||||||
|
|
||||||
|
[[uppercase-options]]
|
||||||
|
.Uppercase Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The field to uppercase
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -178,6 +299,14 @@ Converts a string to its uppercase equivalent.
|
||||||
==== Trim processor
|
==== Trim processor
|
||||||
Trims whitespace from field. NOTE: this only works on leading and trailing whitespaces.
|
Trims whitespace from field. NOTE: this only works on leading and trailing whitespaces.
|
||||||
|
|
||||||
|
[[trim-options]]
|
||||||
|
.Trim Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `field` | yes | - | The string-valued field to trim whitespace from
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -346,71 +475,6 @@ An example of a pipeline specifying custom pattern definitions:
|
||||||
}
|
}
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
==== Geoip processor
|
|
||||||
|
|
||||||
The GeoIP processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases.
|
|
||||||
This processor adds this information by default under the `geoip` field.
|
|
||||||
|
|
||||||
The ingest plugin ships by default with the GeoLite2 City and GeoLite2 Country geoip2 databases from Maxmind made available
|
|
||||||
under the CCA-ShareAlike 3.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/
|
|
||||||
|
|
||||||
The GeoIP processor can run with other geoip2 databases from Maxmind. The files must be copied into the geoip config directory
|
|
||||||
and the `database_file` option should be used to specify the filename of the custom database. The geoip config directory
|
|
||||||
is located at `$ES_HOME/config/ingest/geoip` and holds the shipped databases too.
|
|
||||||
|
|
||||||
[[geoip-options]]
|
|
||||||
.Geoip options
|
|
||||||
[options="header"]
|
|
||||||
|======
|
|
||||||
| Name | Required | Default | Description
|
|
||||||
| `source_field` | yes | - | The field to get the ip address or hostname from for the geographical lookup.
|
|
||||||
| `target_field` | no | geoip | The field that will hold the geographical information looked up from the Maxmind database.
|
|
||||||
| `database_file` | no | GeoLite2-City.mmdb | The database filename in the geoip config directory. The ingest plugin ships with the GeoLite2-City.mmdb and GeoLite2-Country.mmdb files.
|
|
||||||
| `fields` | no | [`continent_name`, `country_iso_code`, `region_name`, `city_name`, `location`] <1> | Controls what properties are added to the `target_field` based on the geoip lookup.
|
|
||||||
|======
|
|
||||||
|
|
||||||
<1> Depends on what is available in `database_field`:
|
|
||||||
* If the GeoLite2 City database is used then the following fields may be added under the `target_field`: `ip`,
|
|
||||||
`country_iso_code`, `country_name`, `continent_name`, `region_name`, `city_name`, `timezone`, `latitude`, `longitude`
|
|
||||||
and `location`. The fields actually added depend on what has been found and which fields were configured in `fields`.
|
|
||||||
* If the GeoLite2 Country database is used then the following fields may be added under the `target_field`: `ip`,
|
|
||||||
`country_iso_code`, `country_name` and `continent_name`.The fields actually added depend on what has been found and which fields were configured in `fields`.
|
|
||||||
|
|
||||||
An example that uses the default city database and adds the geographical information to the `geoip` field based on the `ip` field:
|
|
||||||
|
|
||||||
[source,js]
|
|
||||||
--------------------------------------------------
|
|
||||||
{
|
|
||||||
"description" : "...",
|
|
||||||
"processors" : [
|
|
||||||
{
|
|
||||||
"geoip" : {
|
|
||||||
"source_field" : "ip"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
An example that uses the default country database and add the geographical information to the `geo` field based on the `ip` field`:
|
|
||||||
|
|
||||||
[source,js]
|
|
||||||
--------------------------------------------------
|
|
||||||
{
|
|
||||||
"description" : "...",
|
|
||||||
"processors" : [
|
|
||||||
{
|
|
||||||
"geoip" : {
|
|
||||||
"source_field" : "ip",
|
|
||||||
"target_field" : "geo",
|
|
||||||
"database_file" : "GeoLite2-Country.mmdb"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
==== Date processor
|
==== Date processor
|
||||||
|
|
||||||
The date processor is used for parsing dates from fields, and then using that date or timestamp as the timestamp for that document.
|
The date processor is used for parsing dates from fields, and then using that date or timestamp as the timestamp for that document.
|
||||||
|
|
@ -454,6 +518,14 @@ The Fail Processor is used to raise an exception. This is useful for when
|
||||||
a user expects a pipeline to fail and wishes to relay a specific message
|
a user expects a pipeline to fail and wishes to relay a specific message
|
||||||
to the requester.
|
to the requester.
|
||||||
|
|
||||||
|
[[fail-options]]
|
||||||
|
.Fail Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `message` | yes | - | The error message of the `FailException` thrown by the processor
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
|
|
@ -467,6 +539,14 @@ to the requester.
|
||||||
The DeDot Processor is used to remove dots (".") from field names and
|
The DeDot Processor is used to remove dots (".") from field names and
|
||||||
replace them with a specific `separator` string.
|
replace them with a specific `separator` string.
|
||||||
|
|
||||||
|
[[dedot-options]]
|
||||||
|
.DeDot Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `separator` | yes | "_" | The string to replace dots with in all field names
|
||||||
|
|======
|
||||||
|
|
||||||
[source,js]
|
[source,js]
|
||||||
--------------------------------------------------
|
--------------------------------------------------
|
||||||
{
|
{
|
||||||
Loading…
Reference in New Issue