OpenSearch/x-pack/plugin/ml/qa
Benjamin Trent afd90647c9
[ML] Adds feature importance to option to inference processor (#52218) (#52666)
This adds machine learning model feature importance calculations to the inference processor.

The new flag in the configuration matches the analytics parameter name: `num_top_feature_importance_values`
Example:
```
"inference": {
   "field_mappings": {},
   "model_id": "my_model",
   "inference_config": {
      "regression": {
         "num_top_feature_importance_values": 3
      }
   }
}
```

This will write to the document as follows:
```
"inference" : {
   "feature_importance" : {
      "FlightTimeMin" : -76.90955548511226,
      "FlightDelayType" : 114.13514762158526,
      "DistanceMiles" : 13.731580450792187
   },
   "predicted_value" : 108.33165831875137,
   "model_id" : "my_model"
}
```

This is done through calculating the [SHAP values](https://arxiv.org/abs/1802.03888).

It requires that models have populated `number_samples` for each tree node. This is not available to models that were created before 7.7.

Additionally, if the inference config is requesting feature_importance, and not all nodes have been upgraded yet, it will not allow the pipeline to be created. This is to safe-guard in a mixed-version environment where only some ingest nodes have been upgraded.

NOTE: the algorithm is a Java port of the one laid out in ml-cpp: https://github.com/elastic/ml-cpp/blob/master/lib/maths/CTreeShapFeatureImportance.cc

usability blocked by: https://github.com/elastic/ml-cpp/pull/991
2020-02-21 18:42:31 -05:00
..
basic-multi-node [ML] Make ml internal indices hidden (#52423) (#52509) 2020-02-19 14:02:32 +01:00
disabled Apply 2-space indent to all gradle scripts (#49071) 2019-11-14 11:01:23 +00:00
ml-with-security [Ml] Validate tree feature index is within range (#52514) 2020-02-19 14:41:43 +00:00
native-multi-node-tests [ML] Adds feature importance to option to inference processor (#52218) (#52666) 2020-02-21 18:42:31 -05:00
no-bootstrap-tests Apply 2-space indent to all gradle scripts (#49071) 2019-11-14 11:01:23 +00:00
single-node-tests Convert most awaitBusy calls to assertBusy (#45794) (#47112) 2019-09-29 12:21:46 +01:00
build.gradle Apply 2-space indent to all gradle scripts (#49071) 2019-11-14 11:01:23 +00:00