291 lines
11 KiB
Plaintext
291 lines
11 KiB
Plaintext
//lcawley Verified example output 2017-04-11
|
|
[[ml-job-resource]]
|
|
==== Job Resources
|
|
|
|
A job resource has the following properties:
|
|
|
|
`analysis_config`::
|
|
(object) The analysis configuration, which specifies how to analyze the data.
|
|
See <<ml-analysisconfig, analysis configuration objects>>.
|
|
|
|
`analysis_limits`::
|
|
(object) Defines approximate limits on the memory resource requirements for the job.
|
|
See <<ml-apilimits,analysis limits>>.
|
|
|
|
`create_time`::
|
|
(string) The time the job was created, in ISO 8601 format.
|
|
For example, `1491007356077`.
|
|
|
|
`data_description`::
|
|
(object) Describes the data format and how APIs parse timestamp fields.
|
|
See <<ml-datadescription,data description objects>>.
|
|
|
|
`description`::
|
|
(string) An optional description of the job.
|
|
|
|
`finished_time`::
|
|
(string) If the job closed or failed, this is the time the job finished,
|
|
otherwise it is `null`.
|
|
|
|
`job_id`::
|
|
(string) The unique identifier for the job.
|
|
|
|
`job_type`::
|
|
(string) Reserved for future use, currently set to `anomaly_detector`.
|
|
|
|
`model_plot_config`::
|
|
(object) Configuration properties for storing additional model information.
|
|
See <<ml-apimodelplotconfig, model plot configuration>>.
|
|
|
|
`model_snapshot_id`::
|
|
(string) A numerical character string that uniquely identifies the model
|
|
snapshot. For example, `1491007364`.
|
|
|
|
`model_snapshot_retention_days`::
|
|
(long) The time in days that model snapshots are retained for the job.
|
|
Older snapshots are deleted. The default value is 1 day.
|
|
|
|
`results_index_name`::
|
|
(string) The name of the index in which to store the {ml} results.
|
|
The default value is `shared`,
|
|
which corresponds to the index name `.ml-anomalies-shared`
|
|
|
|
[[ml-analysisconfig]]
|
|
===== Analysis Configuration Objects
|
|
|
|
An analysis configuration object has the following properties:
|
|
|
|
`bucket_span` (required)::
|
|
(time units) The size of the interval that the analysis is aggregated into,
|
|
typically between `5m` and `1h`. The default value is `5m`.
|
|
|
|
`categorization_field_name`::
|
|
(string) If not null, the values of the specified field will be categorized.
|
|
The resulting categories can be used in a detector by setting `by_field_name`,
|
|
`over_field_name`, or `partition_field_name` to the keyword `prelertcategory`.
|
|
|
|
`categorization_filters`::
|
|
(array of strings) If `categorization_field_name` is specified,
|
|
you can also define optional filters. This property expects an array of
|
|
regular expressions. The expressions are used to filter out matching sequences
|
|
off the categorization field values. This functionality is useful to fine tune
|
|
categorization by excluding sequences that should not be taken into
|
|
consideration for defining categories. For example, you can exclude SQL
|
|
statements that appear in your log files.
|
|
|
|
`detectors` (required)::
|
|
(array) An array of detector configuration objects,
|
|
which describe the anomaly detectors that are used in the job.
|
|
See <<ml-detectorconfig,detector configuration objects>>. +
|
|
+
|
|
--
|
|
NOTE: If the `detectors` array does not contain at least one detector, no analysis can occur
|
|
and an error is returned.
|
|
|
|
--
|
|
`influencers`::
|
|
(array of strings) A comma separated list of influencer field names.
|
|
Typically these can be the by, over, or partition fields that are used in the
|
|
detector configuration. You might also want to use a field name that is not
|
|
specifically named in a detector, but is available as part of the input data.
|
|
When you use multiple detectors, the use of influencers is recommended as it
|
|
aggregates results for each influencer entity.
|
|
|
|
`latency`::
|
|
(unsigned integer) The size of the window, in seconds, in which to expect data
|
|
that is out of time order. The default value is 0 (no latency). +
|
|
+
|
|
--
|
|
NOTE: Latency is only applicable when you send data by using
|
|
the <<ml-post-data,post data>> API.
|
|
|
|
--
|
|
`multivariate_by_fields`::
|
|
(boolean) If set to `true`, the analysis will automatically find correlations
|
|
between metrics for a given `by` field value and report anomalies when those
|
|
correlations cease to hold. For example, suppose CPU and memory usage on host A
|
|
is usually highly correlated with the same metrics on host B. Perhaps this
|
|
correlation occurs because they are running a load-balanced application.
|
|
If you enable this property, then anomalies will be reported when, for example,
|
|
CPU usage on host A is high and the value of CPU usage on host B is low.
|
|
That is to say, you'll see an anomaly when the CPU of host A is unusual given the CPU of host B. +
|
|
+
|
|
--
|
|
NOTE: To use the `multivariate_by_fields` property, you must also specify
|
|
`by_field_name` in your detector.
|
|
|
|
--
|
|
|
|
`summary_count_field_name`::
|
|
(string) If not null, the data fed to the job is expected to be pre-summarized.
|
|
This property value is the name of the field that contains the count of raw
|
|
data points that have been summarized. The same `summary_count_field_name`
|
|
applies to all detectors in the job. +
|
|
+
|
|
--
|
|
NOTE: The `summary_count_field_name` property cannot be used with the `metric` function.
|
|
|
|
--
|
|
|
|
////
|
|
LEAVE UNDOCUMENTED
|
|
`overlapping_buckets`::
|
|
(boolean) If set to `true`, an additional analysis occurs that runs out of phase by half a bucket length.
|
|
This requires more system resources and enhances detection of anomalies that span bucket boundaries.
|
|
`use_per_partition_normalization`::
|
|
() TBD
|
|
////
|
|
|
|
[float]
|
|
[[ml-detectorconfig]]
|
|
===== Detector Configuration Objects
|
|
|
|
Detector configuration objects specify which data fields a job analyzes.
|
|
They also specify which analytical functions are used.
|
|
You can specify multiple detectors for a job.
|
|
Each detector has the following properties:
|
|
|
|
`by_field_name`::
|
|
(string) The field used to split the data.
|
|
In particular, this property is used for analyzing the splits with respect to their own history.
|
|
It is used for finding unusual values in the context of the split.
|
|
|
|
`detector_description`::
|
|
(string) A description of the detector. For example, `Low event rate`.
|
|
|
|
`exclude_frequent`::
|
|
(string) Contains one of the following values: `all`, `none`, `by`, or `over`.
|
|
If set, frequent entities are excluded from influencing the anomaly results.
|
|
Entities can be considered frequent over time or frequent in a population.
|
|
If you are working with both over and by fields, then you can set `exclude_frequent`
|
|
to `all` for both fields, or to `by` or `over` for those specific fields.
|
|
|
|
`field_name`::
|
|
(string) The field that the detector uses in the function. If you use an event rate
|
|
function such as `count` or `rare`, do not specify this field. +
|
|
+
|
|
--
|
|
NOTE: The `field_name` cannot contain double quotes or backslashes.
|
|
|
|
--
|
|
|
|
`function` (required)::
|
|
(string) The analysis function that is used.
|
|
For example, `count`, `rare`, `mean`, `min`, `max`, and `sum`.
|
|
|
|
`over_field_name`::
|
|
(string) The field used to split the data.
|
|
In particular, this property is used for analyzing the splits with respect to the history of all splits.
|
|
It is used for finding unusual values in the population of all splits.
|
|
|
|
`partition_field_name`::
|
|
(string) The field used to segment the analysis.
|
|
When you use this property, you have completely independent baselines for each value of this field.
|
|
|
|
`use_null`::
|
|
(boolean) Defines whether a new series is used as the null series
|
|
when there is no value for the by or partition fields. The default value is `false`. +
|
|
+
|
|
--
|
|
IMPORTANT: Field names are case sensitive, for example a field named 'Bytes'
|
|
is different from one named 'bytes'.
|
|
|
|
--
|
|
|
|
////
|
|
LEAVE UNDOCUMENTED
|
|
`detector_rules`::
|
|
(array) TBD
|
|
////
|
|
|
|
[float]
|
|
[[ml-datadescription]]
|
|
===== Data Description Objects
|
|
|
|
The data description defines the format of the input data when you send data to
|
|
the job by using the <<ml-post-data,post data>> API. Note that when configure
|
|
a data feed, these properties are automatically set.
|
|
|
|
When data is received via the <<ml-post-data,post data>> API, it is not stored
|
|
in Elasticsearch. Only the results for anomaly detection are retained.
|
|
|
|
A data description object has the following properties:
|
|
|
|
`format`::
|
|
(string) Only `JSON` format is supported at this time.
|
|
|
|
`time_field`::
|
|
(string) The name of the field that contains the timestamp.
|
|
The default value is `time`.
|
|
|
|
`time_format`::
|
|
(string) The time format, which can be `epoch`, `epoch_ms`, or a custom pattern.
|
|
The default value is `epoch`, which refers to UNIX or Epoch time (the number of seconds
|
|
since 1 Jan 1970).
|
|
The value `epoch_ms` indicates that time is measured in milliseconds since the epoch.
|
|
The `epoch` and `epoch_ms` time formats accept either integer or real values. +
|
|
+
|
|
--
|
|
NOTE: Custom patterns must conform to the Java `DateTimeFormatter` class.
|
|
When you use date-time formatting patterns, it is recommended that you provide
|
|
the full date, time and time zone. For example: `yyyy-MM-dd'T'HH:mm:ssX`.
|
|
If the pattern that you specify is not sufficient to produce a complete timestamp,
|
|
job creation fails.
|
|
|
|
--
|
|
|
|
|
|
[float]
|
|
[[ml-apilimits]]
|
|
===== Analysis Limits
|
|
|
|
Limits can be applied for the resources required to hold the mathematical models in memory.
|
|
These limits are approximate and can be set per job. They do not control the
|
|
memory used by other processes, for example the Elasticsearch Java processes.
|
|
If necessary, you can increase the limits after the job is created.
|
|
|
|
The `analysis_limits` object has the following properties:
|
|
|
|
`categorization_examples_limit`::
|
|
(long) The maximum number of examples stored per category in memory and
|
|
in the results data store. The default value is 4. If you increase this value,
|
|
more examples are available, however it requires that you have more storage available.
|
|
If you set this value to `0`, no examples are stored. +
|
|
+
|
|
--
|
|
NOTE: The `categorization_examples_limit` only applies to analysis that uses categorization.
|
|
|
|
--
|
|
|
|
`model_memory_limit`::
|
|
(long) The maximum amount of memory, in MiB, that the mathematical models can use.
|
|
Once this limit is approached, data pruning becomes more aggressive.
|
|
Upon exceeding this limit, new entities are not modeled. The default value is 4096.
|
|
|
|
[float]
|
|
[[ml-apimodelplotconfig]]
|
|
===== Model Plot Config
|
|
|
|
This advanced configuration option stores model information along with the
|
|
results. It provides a more detailed view into anomaly detection. If you enable
|
|
this option, it can add considerable overhead to the performance of the system;
|
|
it is not feasible for jobs with many entities.
|
|
|
|
Model plot provides a simplified and indicative view of the model and its bounds.
|
|
It does not display complex features such as multivariate correlations or multimodal data.
|
|
As such, anomalies may occasionally be reported which cannot be seen in the model plot.
|
|
|
|
Model plot config can be configured when the job is created or updated later.
|
|
It must be disabled if performance issues are experienced.
|
|
|
|
The `model_plot_config` object has the following properties:
|
|
|
|
`enabled`::
|
|
(boolean) If true, enables calculation and storage of the model bounds for
|
|
each entity that is being analyzed. By default, this is not enabled.
|
|
|
|
`terms`::
|
|
(string) Limits data collection to this comma separated list of _partition_
|
|
or _by_ field names. If terms are not specified or it is an empty string,
|
|
no filtering is applied. For example, `"CPU,NetworkIn,DiskWrites"`
|