Commit Graph

38 Commits

Author SHA1 Message Date
javanna f657a7dbf3 Merge branch 'feature/ingest' into ingest/date 2015-11-09 13:42:07 +01:00
Martijn van Groningen 302621f16b geoip: don't store geoinfo if nothing can be resolved 2015-11-09 18:24:05 +07:00
javanna 1dfe6f6dcf make sure headers etc. are passed over to inner index and delete requests in put/delete pipeline 2015-11-06 19:21:30 +01:00
javanna d318990339 added REST test 2015-11-06 16:10:22 +01:00
javanna 682876f7d7 added factory tests 2015-11-06 15:44:20 +01:00
javanna dbf5c96876 Merge branch 'feature/ingest' into ingest/date 2015-11-06 13:04:51 +01:00
javanna 588de6ccff Add generic type to Processor.Factory and rename Geo*Tests to GeoIp*Tests 2015-11-06 12:53:20 +01:00
javanna 1d2e244bac Merge branch 'feature/ingest' into ingest/date 2015-11-06 12:08:14 +01:00
javanna 7798c6cd49 unified date parser tests in a single test class, added more tests for date processor 2015-11-06 12:05:30 +01:00
Martijn van Groningen e7f0f0ed4e Enforce strict pipeline configuration
Closes #14552
2015-11-05 23:38:29 +07:00
Martijn van Groningen 92452ff99a rename the `ingest` parameter to `pipeline_id` param, because it is more descriptive what the parameter should hold. 2015-11-05 14:35:49 +07:00
javanna b45815da36 adapt to changes upstream 2015-11-04 14:56:03 +01:00
javanna 363454ce4a Merge branch 'feature/ingest' into ingest/date 2015-11-04 14:48:24 +01:00
Martijn van Groningen 1a4b5bba2b Simplify processor creation from map of maps by folding the build and builder factory in one interface called Factory.
In tests processors can be created from the their constructors instead of builders.
In the IngestModule, register instances instead of class instances.
2015-11-03 13:25:58 +07:00
Tal Levy d935a3ab81 Make Grok-specific classes package-protected. 2015-11-02 21:30:09 -08:00
Martijn van Groningen 95e6b99d2b renamed test file 2015-11-03 12:17:10 +07:00
Martijn van Groningen 03387266ca ingest: Added new `geoip` processor, that adds geographical information to documents based on an ip address.
The information is fetched from the Maxmind geolite2 database, that is embedded in the ingest plugin.
2015-11-03 12:06:29 +07:00
Tal Levy 464b46437f Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a
ConfigException class to be a catch-all for things that can go wrong
when constructing new processors with configurations that possibly throw
exceptions. The GrokProcessor loads patterns from the resources
directory.

fix resource path issue, and add rest-api-spec test for grok

fix rest-spec tests

changes: license, remove configexception, throw IOException

add more tests and fix iso8601-hour pattern

move grok patterns from resources to config

fix tests with pom changes, updated IngestClientIT with grok processor

update gradle build script for grok deps and test configuration

move config files to src/main/packaging

move Env out of Processor, fix test for src/main/packaging change

add docs

clean up test resources task

update Grok to be immutable

- Updated the Grok class to be immutable. This means that all the
  pattern bank loading is handled by an external utility class called
  PatternUtils.
- fixed tabs in the nagios patterns file's comments
2015-11-02 20:27:49 -08:00
Tal Levy a8d76f0117 remove pom.xml from ingest plugin 2015-11-02 10:09:36 -08:00
javanna 14e09ad20c Add the ingest param to the context only if present 2015-11-02 17:39:30 +01:00
Martijn van Groningen 376c739fe6 added TODO 2015-11-02 23:37:09 +07:00
javanna 7b3c034caf [DOCS] fixed javadocs minor typos 2015-11-02 15:54:46 +01:00
javanna 8425d26b70 use existing constant to check whether client is a transport client 2015-11-02 15:27:56 +01:00
Tal Levy c6f4cd68da add gradle build script to plugin 2015-10-29 13:40:23 -07:00
Martijn van Groningen 2b26381c08 fix build errors 2015-10-29 12:48:15 +07:00
Tal Levy a2ce7b1995 add tests / remove TODO 2015-10-27 16:20:20 -07:00
Tal Levy cf52ab5df0 add ability to add nested field values to Data document 2015-10-27 15:21:45 -07:00
Martijn van Groningen 9706b47240 field rename 2015-10-27 17:03:02 +07:00
Martijn van Groningen 8df9fc82d9 added custom rest specs for index and bulk that support the ingest param and added rest tests that verify that the ingest param is working 2015-10-27 15:34:54 +07:00
Tal Levy e204671aa9 fix forbiddenapis 2015-10-26 22:50:58 -07:00
Tal Levy 80849bbd6b add date processor 2015-10-20 12:17:01 +03:00
Tal Levy 17cf06ccc2 restructure processors
This Commit does the following:

 - moves processors into their own sub-packages
 - adds ability to add any typed field value into Data
2015-10-20 11:08:04 +03:00
Martijn van Groningen 5a3c75ebac added put, get and delete pipeline APIs. 2015-10-09 18:28:55 +02:00
Martijn van Groningen 11f17c0d7d rename constant name and removed the todo 2015-10-09 18:16:13 +02:00
Martijn van Groningen b3ad3f35fa prevent IndexRequest from being processed multipel times 2015-10-09 18:16:13 +02:00
Martijn van Groningen 82a9ba355d Added pipeline execution service that deals with updating data as it comes in using a dedicated thread pool.
Also changed how bulk requests are handled, because before it just didn't work, but added a todo there because it can potentially be handled differently.
2015-10-09 18:16:13 +02:00
Martijn van Groningen 2071db688c only update pipeline if the content has been changed
split the actual fetching of pipeline docs from the pipeline store to make unit testing easier
intoduced factory for builders
replaced hardcoded processor lookups with simple factory based registry
2015-10-06 14:56:10 +02:00
Martijn van Groningen 18c1cc678a wip 2015-10-05 10:03:25 +02:00