Commit Graph

23 Commits

Author SHA1 Message Date
Tal Levy edfbdf2748 add ability to specify multiple grok patterns (#18074)
- now you can specify a list of grok patterns to match your field with
and the first one to successfully match wins.
- only non-null captures will be inserted into your matched document.

Fixes #17903.
2016-05-25 12:20:39 -07:00
polyfractal 978c1e3e36 [TEST] Add missing sort processor to test
Also fixes the naming of the sort REST test
to follow the numbering convention
2016-05-17 14:52:17 -04:00
Martijn van Groningen 7aca1389e2 ingest: Add `date_index_name` processor.
Closes #17814
2016-04-29 17:20:48 +02:00
Tal Levy 534caa8927 Handle regex parsing errors in Gsub and Grok Processors
Currently, both Gsub and Grok parse regex strings during
Pipeline creation. Thrown parsing exceptions were leaking out, this
commit wraps those exceptions in ElasticsearchParseExceptions.
2016-03-22 15:06:29 -07:00
Simon Willnauer 121e7c8ca4 Add infrastructure to run REST tests on a multi-version cluster
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Relates to #14406
Closes #17072
2016-03-13 10:52:39 +01:00
Martijn van Groningen 7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Tal Levy 3191fc7347 Merge pull request #16355 from talevy/fix_ingest_exception
revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation
2016-02-02 14:11:24 -08:00
Tal Levy 0a1580eefa revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation 2016-02-02 14:08:22 -08:00
Martijn van Groningen 64ac037a82 Added an ingest qa that tests processor real world like configurations.
Renamed `ingest-with-mustache` to `smoke-test-ingest-with-all-dependencies`
Also renamed `ingest-disabled` to `smoke-test-ingest-disabled` so that the name is more inline with other qa smoke test modules.
2016-02-02 22:37:24 +01:00
Tal Levy fca442f4d1 Introduce Pipeline Factory Error Responses in Node Ingest
When there is an exception thrown during pipeline creation within
Rest calls (in put pipeline, and simulate) We now return a structured
error response to the user with details around which processor's
configuration is the cause of the issue, or which configuration property
is misconfigured, etc.
2016-01-29 13:37:27 -08:00
javanna 9bb4caa6f1 adapt to upstream changes, jvm flag doesn't get returned anymore 2016-01-25 18:01:19 +01:00
Tal Levy 3a6c2d008e rename processor_tag to tag 2016-01-21 09:05:42 -08:00
Tal Levy f1204cb1ab add an AbstractProcessor to help hold the re-used processorTag variable 2016-01-20 14:34:26 -08:00
Martijn van Groningen e2e207687d Fixes due to changes in master branch. 2016-01-19 22:03:26 +01:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
javanna 050585e89f remove BiFunction<Environment, TemplateService, Processor.Factory> in favour of Function<TemplateService, Processor.Factory>
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
2016-01-15 15:12:53 +01:00
Martijn van Groningen 21cc0b2316 Cleanup ingest initialization code.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
2016-01-15 13:35:06 +01:00
javanna 169b3c75c5 move grok patterns loading to IngestGrokPlugin class out of the GrokProcessor.Factory 2016-01-14 16:24:07 +01:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Simon Willnauer ce42ae4cf3 Merge pull request #15922 from s1monw/feature/ingest
Review feedback and several cleanups
2016-01-13 12:04:04 +01:00
Tal Levy b39a64e49f accept grok matches with no captures instead of NPE 2016-01-12 12:01:55 -08:00
Simon Willnauer 4d38a47eb5 Review feedback and several cleanups 2016-01-12 12:06:14 +01:00
Martijn van Groningen 1637fe9e0b Moved the grok processor to its own module, so that it will available out-of-the-box, while its dependencies are isolated 2016-01-08 21:59:23 +01:00