28 Commits

Author SHA1 Message Date
polyfractal
72094feb12 [TEST] Add missing sort processor to tests, continued 2016-05-17 16:39:53 -04:00
Martijn van Groningen
7aca1389e2 ingest: Add date_index_name processor.
Closes #17814
2016-04-29 17:20:48 +02:00
Tal Levy
07c2fbf83a Validate properties values according to database type (#17940)
Fixes #17683.
2016-04-29 07:58:27 -07:00
Martijn van Groningen
dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Chris Earle
d97d5ebb8b Remove hostname from NetworkAddress.format
This removes the inconsistent output of IP addresses. The format was parsing-unfriendly and it makes it hard
to reason about API responses, such as to _nodes.

With this change in place, it will never print the hostname as part of the default format, which has the
added benefit that it can be used consistently for URIs, which was not the case when the hostname might
appear at the front with "hostname/ip:port".
2016-04-07 17:27:59 -04:00
Simon Willnauer
554bf2c282 [TEST] Test that all processors are available 2016-03-14 22:35:25 +01:00
Martijn van Groningen
82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
Martijn van Groningen
c7b626c615 updated SHAs 2016-02-28 13:20:39 +01:00
Sylwester Lachiewicz
3af735c69e Update MaxMind geoip2 version to 2.6
Update to align with #16801 jackson 2.7.1
2016-02-27 12:57:24 +01:00
Martijn van Groningen
7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Tal Levy
3191fc7347 Merge pull request #16355 from talevy/fix_ingest_exception
revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation
2016-02-02 14:11:24 -08:00
Tal Levy
0a1580eefa revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation 2016-02-02 14:08:22 -08:00
Martijn van Groningen
64ac037a82 Added an ingest qa that tests processor real world like configurations.
Renamed `ingest-with-mustache` to `smoke-test-ingest-with-all-dependencies`
Also renamed `ingest-disabled` to `smoke-test-ingest-disabled` so that the name is more inline with other qa smoke test modules.
2016-02-02 22:37:24 +01:00
Tal Levy
fca442f4d1 Introduce Pipeline Factory Error Responses in Node Ingest
When there is an exception thrown during pipeline creation within
Rest calls (in put pipeline, and simulate) We now return a structured
error response to the user with details around which processor's
configuration is the cause of the issue, or which configuration property
is misconfigured, etc.
2016-01-29 13:37:27 -08:00
Martijn van Groningen
2b2552c301 test: cleanup static test resources 2016-01-26 21:17:05 +01:00
javanna
720c531364 [TEST] remove check for jvm true, site plugins have been removed 2016-01-22 16:51:51 +01:00
javanna
4ad5e67433 Geoip processor: remove redundant latitude and longitude fields and make location an object with lat and lon subfields 2016-01-22 12:26:39 +01:00
Tal Levy
3a6c2d008e rename processor_tag to tag 2016-01-21 09:05:42 -08:00
Tal Levy
f1204cb1ab add an AbstractProcessor to help hold the re-used processorTag variable 2016-01-20 14:34:26 -08:00
Martijn van Groningen
e2e207687d Fixes due to changes in master branch. 2016-01-19 22:03:26 +01:00
Tal Levy
9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
javanna
af36e5f01e updated ingest-geoip plugin description 2016-01-15 15:16:09 +01:00
javanna
050585e89f remove BiFunction<Environment, TemplateService, Processor.Factory> in favour of Function<TemplateService, Processor.Factory>
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
2016-01-15 15:12:53 +01:00
javanna
dd7cae7c19 move DatabaseReaders initialization to IngestGeoIpPlugin#onModule 2016-01-15 15:02:35 +01:00
Martijn van Groningen
21cc0b2316 Cleanup ingest initialization code.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
2016-01-15 13:35:06 +01:00
Martijn van Groningen
f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Simon Willnauer
4d38a47eb5 Review feedback and several cleanups 2016-01-12 12:06:14 +01:00
Martijn van Groningen
cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00