Commit Graph

99 Commits

Author SHA1 Message Date
Pascal Borreli fcb01deb34 Fixed typos (#20843) 2016-10-10 14:51:47 -06:00
Boaz Leskes 27eab74510 merge from master 2016-09-30 17:19:30 +02:00
Martijn van Groningen 55dce523c2 docs: marked `foreach` processor as experimental
Closes #19602
2016-09-30 12:23:42 +02:00
Jason Tedor 8879360f66 Fix failing doc tests in feature/seq_no
This commit fixes failing doc tests in feature/seq_no after merging
master into this branch.
2016-09-29 03:58:02 +02:00
Tal Levy f3a5ee671b [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-29 03:04:43 +02:00
Tal Levy 550a0449bc [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-28 21:50:32 +02:00
Tal Levy 9f1f5fdedc introduce the JSON Processor (#20128)
introduce the JSON Processor
2016-09-09 14:34:32 -07:00
Tal Levy dda32545bb add ignore_missing option to relevant processors (#20194) 2016-09-09 12:20:18 -07:00
Martijn van Groningen 6f6d17dc9c ingest: Add `dot_expander` processor that can turn fields with dots in the field name into object fields. 2016-09-05 07:28:38 +02:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00
Tal Levy bf046f8f93 update ingest date index name processor with runnable CONSOLE examples (#19935) 2016-08-11 11:36:14 -07:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes #19585
2016-07-29 17:58:30 +02:00
Martijn van Groningen 24d7fa6d54 ingest: Change the `foreach` processor to use the `_ingest._value` ingest metadata attribute to store the current array element being processed.
Closes #19592
2016-07-27 09:35:09 +02:00
Martijn van Groningen 1bc12f5214 docs: fix broken link
Closes #19430
2016-07-14 11:12:47 +02:00
Tal Levy 8fd01554bc update foreach processor to only support one applied processor. (#19402)
Closes #19345.
2016-07-13 13:13:00 -07:00
javanna 62462f5d9b [TEST] replace ResponseBodyAssertion with existing MatchAssertion
We introduced a special response_body assertion to test our docs snippets. The match assertion does the same job though and can be reused and adapted where needed. ResponseBodyAssertion contains provides much better and accurate errors though, which can be now utilized in MatchAssertion so that many more REST tests can benefit from readable error messages.

 Each response body gets always stashed and can be retrieved for later evaluations already. Instead of providing the response body as strings that get parsed to json objects separately, then converted to maps as ResponseBodyAssertion did, we parse everything once, the json is part of the yaml test, which is supported. The only downside is that json comments cannot be used, rather yaml comments should be used (// C style vs # ). There were only two docs tests that were using comments in ingest-node.asciidoc where I went ahead and remove the comments which didn't seem that useful anyways.
2016-07-01 11:13:10 +02:00
David Pilato 157645fe9e Merge pull request #18981 from elastic/doc/ingest-foreach
Wrong name for values field
2016-06-22 23:14:02 +02:00
Adrien Grand db9af54ec0 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-22 08:35:54 +02:00
David Pilato cb8073e990 Wrong name for values field
We wrote that the document is:

```json
{
  "value" : ["foo", "bar", "baz"]
}
```

But the processor is using a `values` field:

```json
{
  "foreach" : {
    "field" : "values",
    "processors" : [
      // ...
    ]
  }
}
```

It should be `values`.
2016-06-20 18:58:41 +02:00
Tal Levy a26260fb72 new ScriptProcessor for Ingest (#18193)
add new ScriptProcessor for executing ES Scripts within pipelines
2016-06-15 14:57:18 -07:00
Martijn van Groningen a2ad5c0282 docs: fix typo
Closes #18877
2016-06-15 10:56:46 +02:00
Christoph Wurm d71894a226 Update ingest-node.asciidoc
Fixed forgotten rename from `match_formats` to `formats` in documentation (changed in dd2184ab25)
2016-06-07 15:47:38 +02:00
Martijn van Groningen 766789b0f0 ingest: added `ignore_failure` option to all processors
If this option is enabled on a processor it silently catches any processor related failure and continues executing the rest of the pipeline.

 Closes #18493
2016-06-01 10:29:12 +02:00
Tal Levy edfbdf2748 add ability to specify multiple grok patterns (#18074)
- now you can specify a list of grok patterns to match your field with
and the first one to successfully match wins.
- only non-null captures will be inserted into your matched document.

Fixes #17903.
2016-05-25 12:20:39 -07:00
Zachary Tong 7c46b57ff2 Add a Sort ingest processor
Sorts an array of values in ascending or descending order. If all elements are numerics, they will be sorted numerically. If values are strings, or mixtures of strings/numbers, the elements will be sorted lexicographically.
2016-05-17 12:06:48 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE (#18210) 2016-05-09 15:42:23 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes #12583. That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Martijn van Groningen 7aca1389e2 ingest: Add `date_index_name` processor.
Closes #17814
2016-04-29 17:20:48 +02:00
Tal Levy 6302fb65a3 add ability to disable ability to override values of existing fields in set processor 2016-04-28 13:50:19 -07:00
Martijn van Groningen dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Clinton Gormley 102a398d9f Fixed split processor example 2016-04-19 14:11:45 +02:00
Clinton Gormley a2ab13ddd1 Update ingest-node.asciidoc
Documented `separator` in the `split processor

Closes https://github.com/elastic/elasticsearch/issues/17831
2016-04-19 11:11:58 +02:00
Martijn van Groningen 16fa3e546e docs: remove mention of file based grok pattern 2016-04-13 22:51:12 +02:00
Martijn van Groningen ca5bd89581 docs: adjust grok processor docs to not mention pattern files as these no longer exist
Closes #17692
2016-04-13 12:37:50 +02:00
Tal Levy 2064fe3985 add type conversion support to ConvertProcessor 2016-03-29 07:56:53 -07:00
Clinton Gormley 432f0cc193 Docs: Added the ingest node to the modules/nodes page
Closes #17113
2016-03-15 19:03:18 +01:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
DeDe Morton 6b52b0bdc3 Ingest node edits 2016-03-03 22:29:27 -08:00
Martijn van Groningen ddc2b60c4b docs: fix incorrect grok pattern parameter names
Closes #16819
2016-02-26 06:38:25 -08:00
DeDe Morton d3e3f19a1f Change topic order 2016-02-12 15:00:07 -08:00
DeDe Morton 2734737d55 Add ingest docs to the build 2016-02-11 14:16:56 -08:00
Dongjoon Hyun 21ea552070 Fix typos in docs. 2016-02-09 02:07:32 -08:00
Martijn van Groningen 7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Tal Levy 9e7e2ab10b remove DeDotProcessor from Ingest 2016-02-02 14:16:01 -08:00
Tal Levy 61e4283a16 Add processor tags to on_failure metadata in ingest pipeline
closes #16202
2016-01-29 14:36:11 -08:00
Martijn van Groningen 63bc108e7e docs: s/processor_id/tag 2016-01-27 10:29:49 +01:00
Martijn van Groningen b784b81665 docs: Remove the fact that ingest was a plugin from the docs. 2016-01-26 15:49:47 +01:00
Tal Levy 894efa3fb6 update ingest docs
- move ingest plugin docs to core reference docs
- move geoip processor docs to plugins/ingest-geoip.asciidoc
- add missing options tables for some processors
- add description of pipeline definition
- add description of processor definitions including common parameters
  like "tag" and "on_failure"
2016-01-25 12:08:17 -08:00