601 lines
18 KiB
Plaintext
601 lines
18 KiB
Plaintext
[role="xpack"]
|
||
[testenv="platinum"]
|
||
[[put-dfanalytics]]
|
||
=== Create {dfanalytics-jobs} API
|
||
[subs="attributes"]
|
||
++++
|
||
<titleabbrev>Create {dfanalytics-jobs}</titleabbrev>
|
||
++++
|
||
|
||
Instantiates a {dfanalytics-job}.
|
||
|
||
experimental[]
|
||
|
||
[[ml-put-dfanalytics-request]]
|
||
==== {api-request-title}
|
||
|
||
`PUT _ml/data_frame/analytics/<data_frame_analytics_id>`
|
||
|
||
|
||
[[ml-put-dfanalytics-prereq]]
|
||
==== {api-prereq-title}
|
||
|
||
If the {es} {security-features} are enabled, you must have the following built-in roles and privileges:
|
||
|
||
* `machine_learning_admin`
|
||
* `kibana_admin` (UI only)
|
||
|
||
|
||
* source indices: `read`, `view_index_metadata`
|
||
* destination index: `read`, `create_index`, `manage` and `index`
|
||
* cluster: `monitor` (UI only)
|
||
|
||
For more information, see <<security-privileges>> and <<built-in-roles>>.
|
||
|
||
NOTE: It is possible that secondary authorization headers are supplied in the
|
||
request. If this is the case, the secondary authorization headers are used
|
||
instead of the primary headers.
|
||
|
||
[[ml-put-dfanalytics-desc]]
|
||
==== {api-description-title}
|
||
|
||
This API creates a {dfanalytics-job} that performs an analysis on the source
|
||
indices and stores the outcome in a destination index.
|
||
|
||
If the destination index does not exist, it is created automatically when you
|
||
start the job. See <<start-dfanalytics>>.
|
||
|
||
[[ml-hyperparam-optimization]]
|
||
If you supply only a subset of the {regression} or {classification} parameters,
|
||
_hyperparameter optimization_ occurs. It determines a value for each of the
|
||
undefined parameters.
|
||
|
||
////
|
||
The starting point is calculated for data dependent parameters by examining the loss
|
||
on the training data. Subject to the size constraint, this operation provides an
|
||
upper bound on the improvement in validation loss.
|
||
|
||
The optimization starts with random search, then
|
||
Bayesian optimization is performed that is targeting maximum expected
|
||
improvement. If you override any parameters by explicitely setting it, the
|
||
optimization calculates the value of the remaining parameters accordingly and
|
||
uses the value you provided for the overridden parameter. The number of rounds
|
||
are reduced respectively. The validation error is estimated in each round by
|
||
using 4-fold cross validation.
|
||
////
|
||
|
||
[[ml-put-dfanalytics-path-params]]
|
||
==== {api-path-parms-title}
|
||
|
||
`<data_frame_analytics_id>`::
|
||
(Required, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=job-id-data-frame-analytics-define]
|
||
|
||
[role="child_attributes"]
|
||
[[ml-put-dfanalytics-request-body]]
|
||
==== {api-request-body-title}
|
||
|
||
`allow_lazy_start`::
|
||
(Optional, boolean)
|
||
Specifies whether this job can start when there is insufficient {ml} node
|
||
capacity for it to be immediately assigned to a node. The default is `false`; if
|
||
a {ml} node with capacity to run the job cannot immediately be found, the
|
||
<<start-dfanalytics>> API returns an error. However, this is also subject to the
|
||
cluster-wide `xpack.ml.max_lazy_ml_nodes` setting. See <<advanced-ml-settings>>.
|
||
If this option is set to `true`, the API does not return an error and the job
|
||
waits in the `starting` state until sufficient {ml} node capacity is available.
|
||
|
||
//Begin analysis
|
||
`analysis`::
|
||
(Required, object)
|
||
The analysis configuration, which contains the information necessary to perform
|
||
one of the following types of analysis: {classification}, {oldetection}, or
|
||
{regression}.
|
||
+
|
||
.Properties of `analysis`
|
||
[%collapsible%open]
|
||
====
|
||
//Begin classification
|
||
`classification`:::
|
||
(Required^*^, object)
|
||
The configuration information necessary to perform
|
||
{ml-docs}/dfa-classification.html[{classification}].
|
||
+
|
||
TIP: Advanced parameters are for fine-tuning {classanalysis}. They are set
|
||
automatically by hyperparameter optimization to give the minimum validation
|
||
error. It is highly recommended to use the default values unless you fully
|
||
understand the function of these parameters.
|
||
+
|
||
.Properties of `classification`
|
||
[%collapsible%open]
|
||
=====
|
||
`class_assignment_objective`::::
|
||
(Optional, string)
|
||
Defines the objective to optimize when assigning class labels:
|
||
`maximize_accuracy` or `maximize_minimum_recall`. When maximizing accuracy,
|
||
class labels are chosen to maximize the number of correct predictions. When
|
||
maximizing minimum recall, labels are chosen to maximize the minimum recall
|
||
for any class. Defaults to `maximize_minimum_recall`.
|
||
|
||
`dependent_variable`::::
|
||
(Required, string)
|
||
+
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=dependent-variable]
|
||
+
|
||
The data type of the field must be numeric (`integer`, `short`, `long`, `byte`),
|
||
categorical (`ip`, `keyword`, `text`), or boolean. There must be no more than 30
|
||
different values in this field.
|
||
|
||
`eta`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=eta]
|
||
|
||
`feature_bag_fraction`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=feature-bag-fraction]
|
||
|
||
`gamma`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=gamma]
|
||
|
||
`lambda`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=lambda]
|
||
|
||
`max_trees`::::
|
||
(Optional, integer)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=max-trees]
|
||
|
||
`num_top_classes`::::
|
||
(Optional, integer)
|
||
Defines the number of categories for which the predicted probabilities are
|
||
reported. It must be non-negative. If it is greater than the total number of
|
||
categories, the API reports all category probabilities. Defaults to 2.
|
||
|
||
`num_top_feature_importance_values`::::
|
||
(Optional, integer)
|
||
Advanced configuration option. Specifies the maximum number of
|
||
{ml-docs}/dfa-classification.html#dfa-classification-feature-importance[feature
|
||
importance] values per document to return. By default, it is zero and no feature importance
|
||
calculation occurs.
|
||
|
||
`prediction_field_name`::::
|
||
(Optional, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=prediction-field-name]
|
||
|
||
`randomize_seed`::::
|
||
(Optional, long)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=randomize-seed]
|
||
|
||
`training_percent`::::
|
||
(Optional, integer)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=training-percent]
|
||
//End classification
|
||
=====
|
||
//Begin outlier_detection
|
||
`outlier_detection`:::
|
||
(Required^*^, object)
|
||
The configuration information necessary to perform
|
||
{ml-docs}/dfa-outlier-detection.html[{oldetection}]:
|
||
+
|
||
.Properties of `outlier_detection`
|
||
[%collapsible%open]
|
||
=====
|
||
`compute_feature_influence`::::
|
||
(Optional, boolean)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=compute-feature-influence]
|
||
|
||
`feature_influence_threshold`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=feature-influence-threshold]
|
||
|
||
`method`::::
|
||
(Optional, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=method]
|
||
|
||
`n_neighbors`::::
|
||
(Optional, integer)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=n-neighbors]
|
||
|
||
`outlier_fraction`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=outlier-fraction]
|
||
|
||
`standardization_enabled`::::
|
||
(Optional, boolean)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=standardization-enabled]
|
||
//End outlier_detection
|
||
=====
|
||
//Begin regression
|
||
`regression`:::
|
||
(Required^*^, object)
|
||
The configuration information necessary to perform
|
||
{ml-docs}/dfa-regression.html[{regression}].
|
||
+
|
||
TIP: Advanced parameters are for fine-tuning {reganalysis}. They are set
|
||
automatically by hyperparameter optimization to give minimum validation error.
|
||
It is highly recommended to use the default values unless you fully understand
|
||
the function of these parameters.
|
||
+
|
||
.Properties of `regression`
|
||
[%collapsible%open]
|
||
=====
|
||
`dependent_variable`::::
|
||
(Required, string)
|
||
+
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=dependent-variable]
|
||
+
|
||
The data type of the field must be numeric.
|
||
|
||
`eta`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=eta]
|
||
|
||
`feature_bag_fraction`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=feature-bag-fraction]
|
||
|
||
`gamma`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=gamma]
|
||
|
||
`lambda`::::
|
||
(Optional, double)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=lambda]
|
||
|
||
`max_trees`::::
|
||
(Optional, integer)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=max-trees]
|
||
|
||
`num_top_feature_importance_values`::::
|
||
(Optional, integer)
|
||
Advanced configuration option. Specifies the maximum number of
|
||
{ml-docs}/dfa-regression.html#dfa-regression-feature-importance[feature importance]
|
||
values per document to return. By default, it is zero and no feature importance
|
||
calculation occurs.
|
||
|
||
`prediction_field_name`::::
|
||
(Optional, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=prediction-field-name]
|
||
|
||
`randomize_seed`::::
|
||
(Optional, long)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=randomize-seed]
|
||
|
||
`training_percent`::::
|
||
(Optional, integer)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=training-percent]
|
||
=====
|
||
//End regression
|
||
====
|
||
//End analysis
|
||
|
||
//Begin analyzed_fields
|
||
`analyzed_fields`::
|
||
(Optional, object)
|
||
Specify `includes` and/or `excludes` patterns to select which fields will be
|
||
included in the analysis. The patterns specified in `excludes` are applied last,
|
||
therefore `excludes` takes precedence. In other words, if the same field is
|
||
specified in both `includes` and `excludes`, then the field will not be included
|
||
in the analysis.
|
||
+
|
||
--
|
||
[[dfa-supported-fields]]
|
||
The supported fields for each type of analysis are as follows:
|
||
|
||
* {oldetection-cap} requires numeric or boolean data to analyze. The algorithms
|
||
don't support missing values therefore fields that have data types other than
|
||
numeric or boolean are ignored. Documents where included fields contain missing
|
||
values, null values, or an array are also ignored. Therefore the `dest` index
|
||
may contain documents that don't have an {olscore}.
|
||
* {regression-cap} supports fields that are numeric, `boolean`, `text`,
|
||
`keyword`, and `ip`. It is also tolerant of missing values. Fields that are
|
||
supported are included in the analysis, other fields are ignored. Documents
|
||
where included fields contain an array with two or more values are also
|
||
ignored. Documents in the `dest` index that don’t contain a results field are
|
||
not included in the {reganalysis}.
|
||
* {classification-cap} supports fields that are numeric, `boolean`, `text`,
|
||
`keyword`, and `ip`. It is also tolerant of missing values. Fields that are
|
||
supported are included in the analysis, other fields are ignored. Documents
|
||
where included fields contain an array with two or more values are also ignored.
|
||
Documents in the `dest` index that don’t contain a results field are not
|
||
included in the {classanalysis}. {classanalysis-cap} can be improved by mapping
|
||
ordinal variable values to a single number. For example, in case of age ranges,
|
||
you can model the values as "0-14" = 0, "15-24" = 1, "25-34" = 2, and so on.
|
||
|
||
If `analyzed_fields` is not set, only the relevant fields will be included. For
|
||
example, all the numeric fields for {oldetection}. For more information about
|
||
field selection, see <<explain-dfanalytics>>.
|
||
--
|
||
+
|
||
.Properties of `analyzed_fields`
|
||
[%collapsible%open]
|
||
====
|
||
`excludes`:::
|
||
(Optional, array)
|
||
An array of strings that defines the fields that will be excluded from the
|
||
analysis. You do not need to add fields with unsupported data types to
|
||
`excludes`, these fields are excluded from the analysis automatically.
|
||
|
||
`includes`:::
|
||
(Optional, array)
|
||
An array of strings that defines the fields that will be included in the
|
||
analysis.
|
||
//End analyzed_fields
|
||
====
|
||
|
||
`description`::
|
||
(Optional, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=description-dfa]
|
||
|
||
`dest`::
|
||
(Required, object)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=dest]
|
||
|
||
`model_memory_limit`::
|
||
(Optional, string)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=model-memory-limit-dfa]
|
||
|
||
`source`::
|
||
(object)
|
||
include::{docdir}/ml/ml-shared.asciidoc[tag=source-put-dfa]
|
||
|
||
|
||
[[ml-put-dfanalytics-example]]
|
||
==== {api-examples-title}
|
||
|
||
|
||
[[ml-put-dfanalytics-example-preprocess]]
|
||
===== Preprocessing actions example
|
||
|
||
The following example shows how to limit the scope of the analysis to certain
|
||
fields, specify excluded fields in the destination index, and use a query to
|
||
filter your data before analysis.
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT _ml/data_frame/analytics/model-flight-delays-pre
|
||
{
|
||
"source": {
|
||
"index": [
|
||
"kibana_sample_data_flights" <1>
|
||
],
|
||
"query": { <2>
|
||
"range": {
|
||
"DistanceKilometers": {
|
||
"gt": 0
|
||
}
|
||
}
|
||
},
|
||
"_source": { <3>
|
||
"includes": [],
|
||
"excludes": [
|
||
"FlightDelay",
|
||
"FlightDelayType"
|
||
]
|
||
}
|
||
},
|
||
"dest": { <4>
|
||
"index": "df-flight-delays",
|
||
"results_field": "ml-results"
|
||
},
|
||
"analysis": {
|
||
"regression": {
|
||
"dependent_variable": "FlightDelayMin",
|
||
"training_percent": 90
|
||
}
|
||
},
|
||
"analyzed_fields": { <5>
|
||
"includes": [],
|
||
"excludes": [
|
||
"FlightNum"
|
||
]
|
||
},
|
||
"model_memory_limit": "100mb"
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[skip:setup kibana sample data]
|
||
|
||
<1> The source index to analyze.
|
||
<2> This query filters out entire documents that will not be present in the
|
||
destination index.
|
||
<3> The `_source` object defines fields in the dataset that will be included or
|
||
excluded in the destination index. In this case, `includes` does not specify any
|
||
fields, so the default behavior takes place: all the fields of the source index
|
||
will included except the ones that are explicitly specified in `excludes`.
|
||
<4> Defines the destination index that contains the results of the analysis and
|
||
the fields of the source index specified in the `_source` object. Also defines
|
||
the name of the `results_field`.
|
||
<5> Specifies fields to be included in or excluded from the analysis. This does
|
||
not affect whether the fields will be present in the destination index, only
|
||
affects whether they are used in the analysis.
|
||
|
||
In this example, we can see that all the fields of the source index are included
|
||
in the destination index except `FlightDelay` and `FlightDelayType` because
|
||
these are defined as excluded fields by the `excludes` parameter of the
|
||
`_source` object. The `FlightNum` field is included in the destination index,
|
||
however it is not included in the analysis because it is explicitly specified as
|
||
excluded field by the `excludes` parameter of the `analyzed_fields` object.
|
||
|
||
|
||
[[ml-put-dfanalytics-example-od]]
|
||
===== {oldetection-cap} example
|
||
|
||
The following example creates the `loganalytics` {dfanalytics-job}, the analysis
|
||
type is `outlier_detection`:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT _ml/data_frame/analytics/loganalytics
|
||
{
|
||
"description": "Outlier detection on log data",
|
||
"source": {
|
||
"index": "logdata"
|
||
},
|
||
"dest": {
|
||
"index": "logdata_out"
|
||
},
|
||
"analysis": {
|
||
"outlier_detection": {
|
||
"compute_feature_influence": true,
|
||
"outlier_fraction": 0.05,
|
||
"standardization_enabled": true
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[setup:setup_logdata]
|
||
|
||
|
||
The API returns the following result:
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"id": "loganalytics",
|
||
"description": "Outlier detection on log data",
|
||
"source": {
|
||
"index": ["logdata"],
|
||
"query": {
|
||
"match_all": {}
|
||
}
|
||
},
|
||
"dest": {
|
||
"index": "logdata_out",
|
||
"results_field": "ml"
|
||
},
|
||
"analysis": {
|
||
"outlier_detection": {
|
||
"compute_feature_influence": true,
|
||
"outlier_fraction": 0.05,
|
||
"standardization_enabled": true
|
||
}
|
||
},
|
||
"model_memory_limit": "1gb",
|
||
"create_time" : 1562265491319,
|
||
"version" : "7.6.0",
|
||
"allow_lazy_start" : false
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/1562265491319/$body.$_path/]
|
||
// TESTRESPONSE[s/"version" : "7.6.0"/"version" : $body.version/]
|
||
|
||
|
||
[[ml-put-dfanalytics-example-r]]
|
||
===== {regression-cap} examples
|
||
|
||
The following example creates the `house_price_regression_analysis`
|
||
{dfanalytics-job}, the analysis type is `regression`:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT _ml/data_frame/analytics/house_price_regression_analysis
|
||
{
|
||
"source": {
|
||
"index": "houses_sold_last_10_yrs"
|
||
},
|
||
"dest": {
|
||
"index": "house_price_predictions"
|
||
},
|
||
"analysis":
|
||
{
|
||
"regression": {
|
||
"dependent_variable": "price"
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[skip:TBD]
|
||
|
||
|
||
The API returns the following result:
|
||
|
||
[source,console-result]
|
||
----
|
||
{
|
||
"id" : "house_price_regression_analysis",
|
||
"source" : {
|
||
"index" : [
|
||
"houses_sold_last_10_yrs"
|
||
],
|
||
"query" : {
|
||
"match_all" : { }
|
||
}
|
||
},
|
||
"dest" : {
|
||
"index" : "house_price_predictions",
|
||
"results_field" : "ml"
|
||
},
|
||
"analysis" : {
|
||
"regression" : {
|
||
"dependent_variable" : "price",
|
||
"training_percent" : 100
|
||
}
|
||
},
|
||
"model_memory_limit" : "1gb",
|
||
"create_time" : 1567168659127,
|
||
"version" : "8.0.0",
|
||
"allow_lazy_start" : false
|
||
}
|
||
----
|
||
// TESTRESPONSE[s/1567168659127/$body.$_path/]
|
||
// TESTRESPONSE[s/"version": "8.0.0"/"version": $body.version/]
|
||
|
||
|
||
The following example creates a job and specifies a training percent:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT _ml/data_frame/analytics/student_performance_mathematics_0.3
|
||
{
|
||
"source": {
|
||
"index": "student_performance_mathematics"
|
||
},
|
||
"dest": {
|
||
"index":"student_performance_mathematics_reg"
|
||
},
|
||
"analysis":
|
||
{
|
||
"regression": {
|
||
"dependent_variable": "G3",
|
||
"training_percent": 70, <1>
|
||
"randomize_seed": 19673948271 <2>
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[skip:TBD]
|
||
|
||
<1> The `training_percent` defines the percentage of the data set that will be
|
||
used for training the model.
|
||
<2> The `randomize_seed` is the seed used to randomly pick which data is used
|
||
for training.
|
||
|
||
|
||
[[ml-put-dfanalytics-example-c]]
|
||
===== {classification-cap} example
|
||
|
||
The following example creates the `loan_classification` {dfanalytics-job}, the
|
||
analysis type is `classification`:
|
||
|
||
[source,console]
|
||
--------------------------------------------------
|
||
PUT _ml/data_frame/analytics/loan_classification
|
||
{
|
||
"source" : {
|
||
"index": "loan-applicants"
|
||
},
|
||
"dest" : {
|
||
"index": "loan-applicants-classified"
|
||
},
|
||
"analysis" : {
|
||
"classification": {
|
||
"dependent_variable": "label",
|
||
"training_percent": 75,
|
||
"num_top_classes": 2
|
||
}
|
||
}
|
||
}
|
||
--------------------------------------------------
|
||
// TEST[skip:TBD]
|