OpenSearch/x-pack/plugin/ml
Dimitris Athanasiou 3ed65eb418
[7.x][ML] Recover data frame extraction search from latest sort key (#61544) (#61572)
If a search failure occurs during data frame extraction we catch
the error and retry once. However, we retry another search that is
identical to the first one. This means we will re-fetch any docs
that were already processed. This may result either to training
a model using duplicate data or in the case of outlier detection to
an error message that the process received more records than it
expected.

This commit fixes this issue by tracking the latest doc's sort key
and then using that in a range query in case we restart the search
due to a failure.

Backport of #61544

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-08-26 17:54:00 +03:00
..
licenses [ML] Rename log structure to file structure (#33421) 2018-09-06 09:13:08 +01:00
qa [7.x] [ML] write warning if configured memory limit is too low for analytics job (#61505) (#61528) 2020-08-26 10:35:38 -04:00
src [7.x][ML] Recover data frame extraction search from latest sort key (#61544) (#61572) 2020-08-26 17:54:00 +03:00
build.gradle [7.x] Cleanup xpack build.gradle (#60554) (#60603) 2020-08-03 13:11:43 -05:00