f1ee29f22e
Currently the msearch api is used to execute buffered search requests; however the msearch api doesn't deal with search requests in an intelligent way. It basically executes each search separately in a concurrent manner. This api reuses the msearch request and response classes and executes the searches as one request in the node holding the enrich index shard. Things like engine.searcher and query shard context are only created once. Also there are less layers than executing a regular msearch request. This results in an interesting speedup. Without this change, in a single node cluster, enriching documents with a bulk size of 5000 items, the ingest time in each bulk response varied from 174ms to 822ms. With this change the ingest time in each bulk response varied from 54ms to 109ms. I think we should add a change like this based on this improvement in ingest time. However I do wonder if instead of doing this change, we should improve the msearch api to execute more efficiently. That would be more complicated then this change, because in this change the custom api can only search enrich index shards and these are special because they always have a single primary shard. If msearch api is to be improved then that should work for any search request to any indices. Making the same optimization for indices with more than 1 primary shard requires much more work. The current change is isolated in the enrich plugin and LOC / complexity is small. So this good enough for now. |
||
|---|---|---|
| .. | ||
| licenses | ||
| src | ||
| build.gradle | ||