From 33c50f12015e2a4edcccab97af15cd0b454d32b8 Mon Sep 17 00:00:00 2001 From: Lisa Cawley Date: Tue, 2 May 2017 15:34:30 -0700 Subject: [PATCH] [DOCS] Add property table for ML Update Jobs API (elastic/x-pack-elasticsearch#1268) * [DOCS] Add property table for ML Update Jobs API * [DOCS] Updates based on feedback for ML Update Jobs API * [DOCS] Removed detector properties from ML Update Jobs API * [DOCS] Fixes typos Original commit: elastic/x-pack-elasticsearch@68d1b5598c1f1dd94fe39de41eb407eecfb64cb9 --- docs/en/ml/getting-started.asciidoc | 6 +- docs/en/rest-api/ml/jobresource.asciidoc | 7 +- docs/en/rest-api/ml/snapshotresource.asciidoc | 23 +-- docs/en/rest-api/ml/update-job.asciidoc | 133 ++++++++++++++---- 4 files changed, 123 insertions(+), 46 deletions(-) diff --git a/docs/en/ml/getting-started.asciidoc b/docs/en/ml/getting-started.asciidoc index da508076b09..abb3575c134 100644 --- a/docs/en/ml/getting-started.asciidoc +++ b/docs/en/ml/getting-started.asciidoc @@ -489,9 +489,9 @@ The number of records that have been processed by the job. Memory status:: The status of the mathematical models. When you create jobs by using the APIs or by using the advanced options in {kib}, you can specify a `model_memory_limit`. -That value is the maximum amount of memory, in MiB, that the mathematical models -can use. Once that limit is approached, data pruning becomes more aggressive. -Upon exceeding that limit, new entities are not modeled. +That value is the maximum amount of memory resources, in MiB, that the +mathematical models can use. Once that limit is approached, data pruning becomes +more aggressive. Upon exceeding that limit, new entities are not modeled. The default value is `4096`. The memory status field reflects whether you have reached or exceeded the model memory limit. It can have one of the following values: + diff --git a/docs/en/rest-api/ml/jobresource.asciidoc b/docs/en/rest-api/ml/jobresource.asciidoc index 592d0b1a2e1..38ee2ec0cd5 100644 --- a/docs/en/rest-api/ml/jobresource.asciidoc +++ b/docs/en/rest-api/ml/jobresource.asciidoc @@ -260,9 +260,10 @@ The `analysis_limits` object has the following properties: NOTE: The `categorization_examples_limit` only applies to analysis that uses categorization. `model_memory_limit`:: - (long) The maximum amount of memory, in MiB, that the mathematical models can use. - Once this limit is approached, data pruning becomes more aggressive. - Upon exceeding this limit, new entities are not modeled. The default value is 4096. + (long) The approximate maximum amount of memory resources that are required + for analytical processing, in MiB. Once this limit is approached, data pruning + becomes more aggressive. Upon exceeding this limit, new entities are not + modeled. The default value is 4096. [float] [[ml-apimodelplotconfig]] diff --git a/docs/en/rest-api/ml/snapshotresource.asciidoc b/docs/en/rest-api/ml/snapshotresource.asciidoc index f14fd25c069..b30c0b0c64b 100644 --- a/docs/en/rest-api/ml/snapshotresource.asciidoc +++ b/docs/en/rest-api/ml/snapshotresource.asciidoc @@ -21,7 +21,7 @@ A model snapshot resource has the following properties: (string) An optional description of the job. `job_id`:: - (string) A numerical character string that uniquely identifing the job that the snapshot was created for. + (string) A numerical character string that uniquely identifies the job that the snapshot was created for. `latest_record_time_stamp`:: (date) The timestamp of the latest processed record. @@ -30,11 +30,12 @@ A model snapshot resource has the following properties: (date) The timestamp of the latest bucket result. `model_size_stats`:: - (object) Summary information describing the model. See <>. + (object) Summary information describing the model. + See <>. `retain`:: - (boolean) If true, this snapshot will not be deleted during automatic cleanup of snapshots - older than `model_snapshot_retention_days`. + (boolean) If true, this snapshot will not be deleted during automatic cleanup + of snapshots older than `model_snapshot_retention_days`. However, this snapshot will be deleted when the job is deleted. The default value is false. @@ -55,22 +56,25 @@ A model snapshot resource has the following properties: The `model_size_stats` object has the following properties: `bucket_allocation_failures_count`:: - (long) The number of buckets for which entites were not processed due to memory limit constraints. + (long) The number of buckets for which entities were not processed due to + memory limit constraints. `job_id`:: (string) A numerical character string that uniquely identifies the job. `log_time`:: - (date) The timestamp that the `model_size_stats` were recorded, according to server-time. + (date) The timestamp that the `model_size_stats` were recorded, according to + server-time. `memory_status`:: (string) The status of the memory in relation to its `model_memory_limit`. Contains one of the following values. `ok`::: The internal models stayed below the configured value. - `soft_limit`::: The internal models require more than 60% of the configured memory limit and more aggressive pruning will + `soft_limit`::: The internal models require more than 60% of the configured + memory limit and more aggressive pruning will be performed in order to try to reclaim space. - `hard_limit`::: The internal models require more space that the configured memory limit. - Some incoming data could not be processed. + `hard_limit`::: The internal models require more space that the configured + memory limit. Some incoming data could not be processed. `model_bytes`:: (long) An approximation of the memory resources required for this analysis. @@ -89,4 +93,3 @@ The `model_size_stats` object has the following properties: `total_partition_field_count`:: (long) The number of _partition_ field values analyzed. - diff --git a/docs/en/rest-api/ml/update-job.asciidoc b/docs/en/rest-api/ml/update-job.asciidoc index 73aff07fabd..44f6008a034 100644 --- a/docs/en/rest-api/ml/update-job.asciidoc +++ b/docs/en/rest-api/ml/update-job.asciidoc @@ -9,11 +9,6 @@ The update job API enables you to update certain properties of a job. `POST _xpack/ml/anomaly_detectors//_update` -//// -===== Description - -//TBD: Important:: Updates do not take effect until after then job is closed and re-opened. -//// ===== Path Parameters `job_id` (required):: @@ -23,16 +18,48 @@ The update job API enables you to update certain properties of a job. The following properties can be updated after the job is created: -`analysis_config`:: - (object) The analysis configuration, which specifies how to analyze the data. - See <>. In particular, - the following properties can be updated: `categorization_filters`, - `detector_description`. -//TBD: Full list of properties that can be updated? +[cols="<,<,<",options="header",] +|======================================================================= +|Name |Description |Requires Restart -`analysis_limits`:: - (object) Specifies runtime limits for the job. - See <>. +|`analysis_limits`: `model_memory_limit` |The approximate maximum amount of +memory resources required for analytical processing, in MiB. +See <>. | Yes + +|`background_persist_interval` |Advanced configuration option. The time between +each periodic persistence of the model. See <>. | Yes + +|`custom_settings` |Contains custom meta data about the job. | No + +|`description` |An optional description of the job. +See <>. | No + +|`model_plot_config`: `enabled` |If true, enables calculation and storage of the +model bounds for each entity that is being analyzed. +See <>. | No + +|`model_snapshot_retention_days` |The time in days that model snapshots are +retained for the job. See <>. | Yes + +|`renormalization_window_days` |Advanced configuration option. The period over +which adjustments to the score are applied, as new data is seen. +See <>. | Yes + +|`results_retention_days` |Advanced configuration option. The number of days +for which job results are retained. See <>. | Yes + +|======================================================================= + +For those properties that have `Requires Restart` set to `Yes` in this table, +if the job is open when you make the update, you must stop the data feed, close +the job, then restart the data feed and open the job for the changes to take +effect. + +//|`analysis_config`: `detectors`: `index` | A unique identifier of the +//detector. Matches the order of detectors returned by +//<>, starting from 0. | No +//|`analysis_config`: `detectors`: `detector_description` |A description of the +//detector. See <>. | No [NOTE] -- @@ -43,9 +70,6 @@ of `hard_limit`, this means that it was unable to process some data. You might want to re-run this job with an increased `model_memory_limit`. -- -`description`:: - (string) An optional description of the job. - ===== Authorization @@ -55,32 +79,81 @@ For more information, see <>. ===== Examples -The following example updates the `it-ops-kpi` job: +The following example updates the `it_ops_new_logs` job: [source,js] -------------------------------------------------- -POST _xpack/ml/anomaly_detectors/it-ops-kpi/_update +POST _xpack/ml/anomaly_detectors/it_ops_new_logs/_update { - "description":"New description", - "analysis_limits":{ - "model_memory_limit": 8192 - } + "description":"An updated job", + "model_plot_config": { + "enabled": true + }, + "analysis_limits": { + "model_memory_limit": 1024 + }, + "renormalization_window_days": 30, + "background_persist_interval": "2h", + "model_snapshot_retention_days": 7, + "results_retention_days": 60, + "custom_settings": { + "custom_urls" : [{ + "url_name" : "Lookup IP", + "url_value" : "http://geoiplookup.net/ip/$clientip$" + }] + } } -------------------------------------------------- // CONSOLE // TEST[skip:todo] -When the job is updated, you receive the following results: +When the job is updated, you receive a summary of the job configuration +information, including the updated property values. For example: + [source,js] ---- { - "job_id": "it-ops-kpi", + "job_id": "it_ops_new_logs", "job_type": "anomaly_detector", - "description": "New description", - ... - "analysis_limits": { - "model_memory_limit": 8192 + "description": "An updated job", + "create_time": 1493678314204, + "finished_time": 1493678315850, + "analysis_config": { + "bucket_span": "1800s", + "categorization_field_name": "message", + "detectors": [ + { + "detector_description": "Unusual message counts", + "function": "count", + "by_field_name": "mlcategory", + "detector_rules": [] + } + ], + "influencers": [] }, - ... + "analysis_limits": { + "model_memory_limit": 1024 + }, + "data_description": { + "time_field": "time", + "time_format": "epoch_ms" + }, + "model_plot_config": { + "enabled": true + }, + "renormalization_window_days": 30, + "background_persist_interval": "2h", + "model_snapshot_retention_days": 7, + "results_retention_days": 60, + "custom_settings": { + "custom_urls": [ + { + "url_name": "Lookup IP", + "url_value": "http://geoiplookup.net/ip/$clientip$" + } + ] + }, + "model_snapshot_id": "1493678315", + "results_index_name": "shared" } ----