258 lines
11 KiB
Plaintext
258 lines
11 KiB
Plaintext
//lcawley Verified example output 2017-04-11
|
|
[[ml-job-resource]]
|
|
==== Job Resources
|
|
|
|
A job resource has the following properties:
|
|
|
|
`analysis_config`::
|
|
(object) The analysis configuration, which specifies how to analyze the data.
|
|
See <<ml-analysisconfig, analysis configuration objects>>.
|
|
|
|
`analysis_limits`::
|
|
(object) Defines approximate limits on the memory resource requirements for the job.
|
|
See <<ml-apilimits,analysis limits>>.
|
|
|
|
`create_time`::
|
|
(string) The time the job was created, in ISO 8601 format. For example, `1491007356077`.
|
|
|
|
`data_description`::
|
|
(object) Describes the data format and how APIs parse timestamp fields. See <<ml-datadescription,data description objects>>.
|
|
|
|
`description`::
|
|
(string) An optional description of the job.
|
|
|
|
`finished_time`::
|
|
(string) If the job closed or failed, this is the time the job finished, otherwise it is `null`.
|
|
|
|
`job_id`::
|
|
(string) The unique identifier for the job.
|
|
|
|
`job_type`::
|
|
(string) Reserved for future use, currently set to `anomaly_detector`.
|
|
|
|
`model_plot_config`::
|
|
(object) Configuration properties for storing additional model information.
|
|
See <<ml-apimodelplotconfig, model plot configuration>>.
|
|
|
|
`model_snapshot_id`::
|
|
(string) A numerical character string that uniquely identifies the model
|
|
snapshot. For example, `1491007364`.
|
|
|
|
`model_snapshot_retention_days`::
|
|
(long) The time in days that model snapshots are retained for the job.
|
|
Older snapshots are deleted. The default value is 1 day.
|
|
|
|
`results_index_name`::
|
|
(string) The name of the index in which to store the results generated by {ml} results.
|
|
The default value is `shared` which corresponds to the index name `.ml-anomalies-shared`
|
|
|
|
[[ml-analysisconfig]]
|
|
===== Analysis Configuration Objects
|
|
|
|
An analysis configuration object has the following properties:
|
|
|
|
`bucket_span` (required)::
|
|
(time units) The size of the interval that the analysis is aggregated into, typically between `5m` and `1h`.
|
|
The default value is `5m`.
|
|
|
|
`categorization_field_name`::
|
|
(string) If not null, the values of the specified field will be categorized.
|
|
The resulting categories can be used in a detector by setting `by_field_name`,
|
|
`over_field_name`, or `partition_field_name` to the keyword `prelertcategory`.
|
|
|
|
`categorization_filters`::
|
|
(array of strings) If `categorization_field_name` is specified, you can also define optional filters.
|
|
This property expects an array of regular expressions.
|
|
The expressions are used to filter out matching sequences off the categorization field values.
|
|
This functionality is useful to fine tune categorization by excluding sequences
|
|
that should not be taken into consideration for defining categories.
|
|
For example, you can exclude SQL statements that appear in your log files.
|
|
|
|
`detectors` (required)::
|
|
(array) An array of detector configuration objects,
|
|
which describe the anomaly detectors that are used in the job.
|
|
See <<ml-detectorconfig,detector configuration objects>>. +
|
|
+
|
|
--
|
|
NOTE: If the `detectors` array does not contain at least one detector, no analysis can occur
|
|
and an error is returned.
|
|
|
|
--
|
|
`influencers`::
|
|
(array of strings) A comma separated list of influencer field names.
|
|
Typically these can be the by, over, or partition fields that are used in the detector configuration.
|
|
You might also want to use a field name that is not specifically named in a detector,
|
|
but is available as part of the input data. When you use multiple detectors,
|
|
the use of influencers is recommended as it aggregates results for each influencer entity.
|
|
|
|
`latency`::
|
|
(unsigned integer) The size of the window, in seconds, in which to expect data that is out of time order.
|
|
The default value is 0 (no latency).
|
|
|
|
NOTE: Latency is only applicable when you send data by using the <<ml-post-data, Post Data to Jobs>> API.
|
|
|
|
--
|
|
`multivariate_by_fields`::
|
|
(boolean) If set to `true`, the analysis will automatically find correlations
|
|
between metrics for a given `by` field value and report anomalies when those
|
|
correlations cease to hold. For example, suppose CPU and memory usage on host A
|
|
is usually highly correlated with the same metrics on host B. Perhaps this
|
|
correlation occurs because they are running a load-balanced application.
|
|
If you enable this property, then anomalies will be reported when, for example,
|
|
CPU usage on host A is high and the value of CPU usage on host B is low.
|
|
That is to say, you'll see an anomaly when the CPU of host A is unusual given the CPU of host B. +
|
|
+
|
|
--
|
|
NOTE: To use the `multivariate_by_fields` property, you must also specify `by_field_name` in your detector.
|
|
|
|
// LEAVE UNDOCUMENTED
|
|
// `overlapping_buckets`::
|
|
// (boolean) If set to `true`, an additional analysis occurs that runs out of phase by half a bucket length.
|
|
// This requires more system resources and enhances detection of anomalies that span bucket boundaries.
|
|
|
|
`summary_count_field_name`::
|
|
(string) If not null, the data fed to the job is expected to be pre-summarized.
|
|
This property value is the name of the field that contains the count of raw data points that have been summarized.
|
|
The same `summary_count_field_name` applies to all detectors in the job. +
|
|
+
|
|
--
|
|
NOTE: The `summary_count_field_name` property cannot be used with the `metric` function.
|
|
--
|
|
|
|
// LEAVE UNDOCUMENTED
|
|
// `use_per_partition_normalization`::
|
|
// () TBD
|
|
|
|
[[ml-detectorconfig]]
|
|
===== Detector Configuration Objects
|
|
|
|
Detector configuration objects specify which data fields a job analyzes.
|
|
They also specify which analytical functions are used.
|
|
You can specify multiple detectors for a job.
|
|
Each detector has the following properties:
|
|
|
|
`by_field_name`::
|
|
(string) The field used to split the data.
|
|
In particular, this property is used for analyzing the splits with respect to their own history.
|
|
It is used for finding unusual values in the context of the split.
|
|
|
|
`detector_description`::
|
|
(string) A description of the detector. For example, `Low event rate`.
|
|
|
|
// LEAVE UNDOCUMENTED
|
|
// `detector_rules`::
|
|
// (array) TBD
|
|
|
|
`exclude_frequent`::
|
|
(string) Contains one of the following values: `all`, `none`, `by`, or `over`.
|
|
If set, frequent entities are excluded from influencing the anomaly results.
|
|
Entities can be considered frequent over time or frequent in a population.
|
|
If you are working with both over and by fields, then you can set `exclude_frequent`
|
|
to `all` for both fields, or to `by` or `over` for those specific fields.
|
|
|
|
`field_name`::
|
|
(string) The field that the detector uses in the function. If you use an event rate
|
|
function such as `count` or `rare`, do not specify this field. +
|
|
+
|
|
--
|
|
NOTE: The `field_name` cannot contain double quotes or backslashes.
|
|
--
|
|
|
|
`function` (required)::
|
|
(string) The analysis function that is used.
|
|
For example, `count`, `rare`, `mean`, `min`, `max`, and `sum`.
|
|
|
|
`over_field_name`::
|
|
(string) The field used to split the data.
|
|
In particular, this property is used for analyzing the splits with respect to the history of all splits.
|
|
It is used for finding unusual values in the population of all splits.
|
|
|
|
`partition_field_name`::
|
|
(string) The field used to segment the analysis.
|
|
When you use this property, you have completely independent baselines for each value of this field.
|
|
|
|
`use_null`::
|
|
(boolean) Defines whether a new series is used as the null series
|
|
when there is no value for the by or partition fields. The default value is `false`. +
|
|
+
|
|
--
|
|
IMPORTANT: Field names are case sensitive, for example a field named 'Bytes' is different to one named 'bytes'.
|
|
--
|
|
|
|
[[ml-datadescription]]
|
|
===== Data Description Objects
|
|
|
|
The data description define the format of the input data when posting time-ordered data to the `_data` endpoint.
|
|
Please note that when configuring a datafeed, these are automatically set.
|
|
|
|
When data is received via the <<ml-post-data, Post Data to Jobs>> API,
|
|
the data posted is not stored in Elasticsearch. Only the results for anomaly detection are retained.
|
|
|
|
A data description object has the following properties:
|
|
|
|
`format`::
|
|
(string) Only `JSON` format is supported at this time.
|
|
|
|
`time_field`::
|
|
(string) The name of the field that contains the timestamp.
|
|
The default value is `time`.
|
|
|
|
`time_format`::
|
|
(string) The time format, which can be `epoch`, `epoch_ms`, or a custom pattern.
|
|
The default value is `epoch`, which refers to UNIX or Epoch time (the number of seconds
|
|
since 1 Jan 1970).
|
|
The value `epoch_ms` indicates that time is measured in milliseconds since the epoch.
|
|
The `epoch` and `epoch_ms` time formats accept either integer or real values. +
|
|
+
|
|
--
|
|
NOTE: Custom patterns must conform to the Java `DateTimeFormatter` class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: `yyyy-MM-dd'T'HH:mm:ssX`. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.
|
|
--
|
|
|
|
|
|
[[ml-apilimits]]
|
|
===== Analysis Limits
|
|
|
|
Limits can be applied for the resources required to hold the mathematical models in memory.
|
|
These limits are approximate and can be set per job.
|
|
They do not control the memory used by other processes, for example the elasticsearch Java processes.
|
|
If necessary, the limits can be increased after the job is created.
|
|
|
|
The `analysis_limits` object has the following properties:
|
|
|
|
`categorization_examples_limit`::
|
|
(long) The maximum number of examples stored per category in memory and
|
|
in the results data store. The default value is 4. If you increase this value,
|
|
more examples are available, however it requires that you have more storage available.
|
|
If you set this value to `0`, no examples are stored.
|
|
|
|
NOTE: The `categorization_examples_limit` only applies to analysis that uses categorization.
|
|
|
|
`model_memory_limit`::
|
|
(long) The maximum amount of memory, in MiB, that the mathematical models can use.
|
|
Once this limit is approached, data pruning becomes more aggressive.
|
|
Upon exceeding this limit, new entities are not modeled. The default value is 4096.
|
|
|
|
[[ml-apimodelplotconfig]]
|
|
===== Model Plot Config
|
|
|
|
This advanced configuration option will store model information along with results allowing a more detailed view into anomaly detection.
|
|
Enabling this can add considerable overhead to the performance of the system and is not feasible for jobs with many entities.
|
|
|
|
Model plot provides a simplified and indicative view of the model and its bounds.
|
|
It does not display complex features such as multivariate correlations or multimodal data.
|
|
As such, anomalies may occassionally be reported which cannot be seen in the model plot.
|
|
|
|
Model plot config can be configured when the job is created or updated later. It must be disabled if performance issues are experienced.
|
|
|
|
The `model_plot_config` object has the following properties:
|
|
|
|
`enabled`::
|
|
(boolean) If true, will enable calculation and storage of the model bounds for each entity being analyzed.
|
|
By default, this is not enabled.
|
|
|
|
`terms`::
|
|
(string) Limits data collection to this comma separated list of _partition_ or _by_ field names.
|
|
If terms are not specified or is an empty string, no filtering is applied.
|
|
E.g. `"CPU,NetworkIn,DiskWrites"`
|
|
|