Commit Graph

75 Commits

Author SHA1 Message Date
Adrien Grand db9af54ec0 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-22 08:35:54 +02:00
David Pilato cb8073e990 Wrong name for values field
We wrote that the document is:

```json
{
  "value" : ["foo", "bar", "baz"]
}
```

But the processor is using a `values` field:

```json
{
  "foreach" : {
    "field" : "values",
    "processors" : [
      // ...
    ]
  }
}
```

It should be `values`.
2016-06-20 18:58:41 +02:00
Tal Levy a26260fb72 new ScriptProcessor for Ingest (#18193)
add new ScriptProcessor for executing ES Scripts within pipelines
2016-06-15 14:57:18 -07:00
Martijn van Groningen a2ad5c0282 docs: fix typo
Closes #18877
2016-06-15 10:56:46 +02:00
Christoph Wurm d71894a226 Update ingest-node.asciidoc
Fixed forgotten rename from `match_formats` to `formats` in documentation (changed in dd2184ab25)
2016-06-07 15:47:38 +02:00
Martijn van Groningen 766789b0f0 ingest: added `ignore_failure` option to all processors
If this option is enabled on a processor it silently catches any processor related failure and continues executing the rest of the pipeline.

 Closes #18493
2016-06-01 10:29:12 +02:00
Tal Levy edfbdf2748 add ability to specify multiple grok patterns (#18074)
- now you can specify a list of grok patterns to match your field with
and the first one to successfully match wins.
- only non-null captures will be inserted into your matched document.

Fixes #17903.
2016-05-25 12:20:39 -07:00
Zachary Tong 7c46b57ff2 Add a Sort ingest processor
Sorts an array of values in ascending or descending order. If all elements are numerics, they will be sorted numerically. If values are strings, or mixtures of strings/numbers, the elements will be sorted lexicographically.
2016-05-17 12:06:48 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE (#18210) 2016-05-09 15:42:23 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes #12583. That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Martijn van Groningen 7aca1389e2 ingest: Add `date_index_name` processor.
Closes #17814
2016-04-29 17:20:48 +02:00
Tal Levy 6302fb65a3 add ability to disable ability to override values of existing fields in set processor 2016-04-28 13:50:19 -07:00
Martijn van Groningen dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Clinton Gormley 102a398d9f Fixed split processor example 2016-04-19 14:11:45 +02:00
Clinton Gormley a2ab13ddd1 Update ingest-node.asciidoc
Documented `separator` in the `split processor

Closes https://github.com/elastic/elasticsearch/issues/17831
2016-04-19 11:11:58 +02:00
Martijn van Groningen 16fa3e546e docs: remove mention of file based grok pattern 2016-04-13 22:51:12 +02:00
Martijn van Groningen ca5bd89581 docs: adjust grok processor docs to not mention pattern files as these no longer exist
Closes #17692
2016-04-13 12:37:50 +02:00
Tal Levy 2064fe3985 add type conversion support to ConvertProcessor 2016-03-29 07:56:53 -07:00
Clinton Gormley 432f0cc193 Docs: Added the ingest node to the modules/nodes page
Closes #17113
2016-03-15 19:03:18 +01:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
DeDe Morton 6b52b0bdc3 Ingest node edits 2016-03-03 22:29:27 -08:00
Martijn van Groningen ddc2b60c4b docs: fix incorrect grok pattern parameter names
Closes #16819
2016-02-26 06:38:25 -08:00
DeDe Morton d3e3f19a1f Change topic order 2016-02-12 15:00:07 -08:00
DeDe Morton 2734737d55 Add ingest docs to the build 2016-02-11 14:16:56 -08:00