javanna
90743d8db0
add REST test for bulk api integration with ingest
2016-01-11 19:04:34 +01:00
javanna
b4baa6c7ab
remove use of already processed header in favour of resetting the pipeline id to null
2016-01-11 18:11:16 +01:00
javanna
362deb4579
[TEST] move testBulkRequestModifier to existing BulkRequestModifierTests class
2016-01-11 18:11:16 +01:00
javanna
fad2571ba5
add test for bulk without any request holiding a pipeline id
2016-01-11 18:11:16 +01:00
javanna
21c89e6c5b
make sure we don't go ahead if no index request holds a pipeline id
2016-01-11 18:11:16 +01:00
javanna
bac1206161
remove use of request headers/context for pipeline id in favour of instance members added to IndexRequest and BulkRequest
...
Now that the ingest infra is part of es core we can remove some code that was required by the plugin and have a better integration with es core. We allow to specify the pipeline id in bulk and index as a request parameter, we have a REST filter that parses it and adds it to the relevant action request. That is not required anymore, as we can add this logic to RestIndexAction and RestBulkAction directly, no need for a filter. Also, we can allow to specify a pipeline id for each index requests in a bulk request. The small downside of this is that the ingest filter has to go over each item of a bulk request, all the time, to figure out whether they have a pipeline id.
2016-01-11 18:11:15 +01:00
Martijn van Groningen
cd2155311f
renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor
2016-01-08 22:50:54 +01:00
Martijn van Groningen
1637fe9e0b
Moved the grok processor to its own module, so that it will available out-of-the-box, while its dependencies are isolated
2016-01-08 21:59:23 +01:00
Robert Muir
46a8a48d23
Merge pull request #15851 from rmuir/geoip-nodns
...
[ingest] Don't do DNS lookups from GeoIpProcessor
2016-01-08 13:17:20 -05:00
javanna
3e08c24991
clarify comment in TemplateService, explain why we have this class
2016-01-08 14:31:57 +01:00
javanna
de2eac4c49
removed leftover jodatime dependency
2016-01-08 14:31:35 +01:00
Martijn van Groningen
b9dd99636d
dynamic option is a string, not a boolean
2016-01-08 11:36:52 +01:00
Luca Cavanna
c8aeb11fd6
Merge pull request #15837 from javanna/enhancement/move_processors_to_core
...
Move ingest rest spec and tests to core with basic processors
2016-01-08 11:28:55 +01:00
javanna
8675784e3f
adapt error message when node.ingest is set to false and we fail executing a pipeline
2016-01-08 10:39:39 +01:00
javanna
ae69d46f92
move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default
2016-01-08 10:39:39 +01:00
Martijn van Groningen
e35e9bd736
Removed index template and instead create .ingest with settings and mapping when we find out that the index is missing during the put pipeline call.
2016-01-08 10:35:52 +01:00
Robert Muir
9c3ebb83a7
Don't do DNS lookups from GeoIpProcessor
...
There is no need to involve DNS in this!
2016-01-07 23:12:44 -05:00
javanna
adac314328
revert move of IngestPlugin class
...
This was moved accidentally as part of a previous refactoring.
2016-01-07 18:32:46 +01:00
javanna
79c1377661
[TEST] set node.ingest to true otherwise we may try to modify values for a threadpool that hasn't been started
2016-01-07 16:16:14 +01:00
Luca Cavanna
e149704ba2
Merge pull request #15769 from javanna/enhancement/move_to_core
...
Move ingest api to core
2016-01-07 16:04:16 +01:00
javanna
03fe38681e
renamed qa package o.e.plugin.ingest to o.e.ingest
...
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna
694eeffe92
catch the right exception...
2016-01-07 15:36:34 +01:00
Martijn van Groningen
0d70d92aa0
Also ignore rejected execution exception when installing index template
2016-01-07 15:25:35 +01:00
Martijn van Groningen
c44b83dc29
Also ignore rejected execution exception when retying restart
2016-01-07 15:21:21 +01:00
javanna
1ea690e814
Merge branch 'feature/ingest' into enhancement/move_to_core
2016-01-07 15:13:31 +01:00
javanna
7d7e0db91d
Merge branch 'master' into feature/ingest
2016-01-07 15:09:13 +01:00
javanna
234371811d
awaited fix a couple of index template tests till we have removed the ingest index template
2016-01-07 15:06:05 +01:00
Martijn van Groningen
fa9aab91eb
Ignore rejected execution exception
2016-01-07 14:52:15 +01:00
Martijn van Groningen
ba97b7f7bd
Don't activate ingest on tribe nodes
2016-01-07 14:36:23 +01:00
Nik Everett
52f28888d5
Merge pull request #15813 from nik9000/xlint1
...
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
Nik Everett
439832d0b3
Merge pull request #15815 from nik9000/xlint2
...
Remove a few more Xlint skips
2016-01-07 08:34:28 -05:00
javanna
eca1594969
start ingest thread pool only when node.ingest is set to true
2016-01-07 14:33:56 +01:00
Boaz Leskes
d5e6eb58a8
Log uncaught exceptions from scheduled once tasks
...
`ScheduledThreadPoolExecutor` allows you to schedule tasks to run once or periodically at the future. If such a task throws an exception, that exception is caught and reported in the future that `ScheduledThreadPoolExecutor#schedule` returns. However, we typically do not capture the future / do not test it for errors. This results in exception being swallowed and not reported. To mitigate this we now wrap any command in a LoggingRunnable (already used for periodic tasks). Also, RunnableCommand is changed not to swallow exception but percolate them further for reporting by the future.
Closes #15824
2016-01-07 14:04:35 +01:00
Martijn van Groningen
0b06648205
template installation is async, so use assertBusy
2016-01-07 14:00:50 +01:00
javanna
0bfe6de75c
move all transport actions under same package org.elasticsearch.action.ingest
2016-01-07 13:52:42 +01:00
Martijn van Groningen
e6cc79a976
test: added ingest script context
2016-01-07 13:48:36 +01:00
javanna
95bc0ed7a2
move constants to IngestActionFilter
2016-01-07 13:40:28 +01:00
javanna
2803ae09dc
addProcessor -> registerProcessor
2016-01-07 13:25:25 +01:00
javanna
18aabd67c8
adapt qa tests for when ingest.node is set to false
...
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
javanna
52c2a273f9
amended error message
2016-01-07 13:21:06 +01:00
Martijn van Groningen
7e56f65939
since ingest is part of core the ingest template always gets installed, so the assumptions this test had had to be updated
2016-01-07 12:58:13 +01:00
Martijn van Groningen
4dca3cb38f
fix thread pools, use management the do the operation and response handeling
2016-01-07 12:14:13 +01:00
javanna
c185c1339a
remove grok and join usage from IngestClientIT
2016-01-07 11:30:02 +01:00
javanna
e036b5896d
move registerFilter up
2016-01-07 11:29:43 +01:00
Martijn van Groningen
9ec2e140b8
Merge branch 'master' into feature/ingest
2016-01-07 10:44:21 +01:00
Simon Willnauer
e7f9d685f1
[TEST] Test that translog can recover after random IOException
...
This commit adds a new test that can throw an IOException at any point in time
and ensures that all previously synced documents can be successfully recovered after hitting
an excepiton.
Relates to #15788
2016-01-07 10:17:31 +01:00
Adrien Grand
132df10342
Merge pull request #15614 from jpountz/remove/warmers
...
Remove query warmers and the warmer API.
2016-01-07 09:59:14 +01:00
Adrien Grand
67d233cecd
Remove warmers and the warmer API.
...
Warmers are now barely useful and will be removed in 3.0. Note that this only
removes the warmer API and query-based warmers. We still have warmers internally
for eg. global ordinals.
Close #15607
2016-01-07 09:57:07 +01:00
Martijn van Groningen
604d59a95e
muted test
2016-01-07 09:54:59 +01:00
Nik Everett
244120a065
Remove more Xlint skips
2016-01-06 23:53:05 -05:00
