Dimitris Athanasiou 126c2fd2d5
[7.x][ML] Machine learning data frame analytics () ()
This merges the initial work that adds a framework for performing
machine learning analytics on data frames. The feature is currently experimental
and requires a platinum license. Note that the original commits can be
found in the `feature-ml-data-frame-analytics` branch.

A new set of APIs is added which allows the creation of data frame analytics
jobs. Configuration allows specifying different types of analysis to be performed
on a data frame. At first there is support for outlier detection.

The APIs are:

- PUT _ml/data_frame/analysis/{id}
- GET _ml/data_frame/analysis/{id}
- GET _ml/data_frame/analysis/{id}/_stats
- POST _ml/data_frame/analysis/{id}/_start
- POST _ml/data_frame/analysis/{id}/_stop
- DELETE _ml/data_frame/analysis/{id}

When a data frame analytics job is started a persistent task is created and started.
The main steps of the task are:

1. reindex the source index into the dest index
2. analyze the data through the data_frame_analyzer c++ process
3. merge the results of the process back into the destination index

In addition, an evaluation API is added which packages commonly used metrics
that provide evaluation of various analysis:

- POST _ml/data_frame/_evaluate
2019-06-25 20:29:11 +03:00

585 lines
18 KiB
Plaintext

[[java-rest-high-supported-apis]]
== Document APIs
:upid: {mainid}-document
:doc-tests-file: {doc-tests}/CRUDDocumentationIT.java
The Java High Level REST Client supports the following Document APIs:
[[single-doc]]
Single document APIs::
* <<{upid}-index>>
* <<{upid}-get>>
* <<{upid}-exists>>
* <<{upid}-delete>>
* <<{upid}-update>>
* <<{upid}-term-vectors>>
[[multi-doc]]
Multi-document APIs::
* <<{upid}-bulk>>
* <<{upid}-multi-get>>
* <<{upid}-reindex>>
* <<{upid}-update-by-query>>
* <<{upid}-delete-by-query>>
* <<{upid}-rethrottle>>
* <<{upid}-multi-term-vectors>>
include::document/index.asciidoc[]
include::document/get.asciidoc[]
include::document/exists.asciidoc[]
include::document/delete.asciidoc[]
include::document/update.asciidoc[]
include::document/term-vectors.asciidoc[]
include::document/bulk.asciidoc[]
include::document/multi-get.asciidoc[]
include::document/reindex.asciidoc[]
include::document/update-by-query.asciidoc[]
include::document/delete-by-query.asciidoc[]
include::document/rethrottle.asciidoc[]
include::document/multi-term-vectors.asciidoc[]
== Search APIs
:upid: {mainid}
:doc-tests-file: {doc-tests}/SearchDocumentationIT.java
The Java High Level REST Client supports the following Search APIs:
* <<{upid}-search>>
* <<{upid}-search-scroll>>
* <<{upid}-clear-scroll>>
* <<{upid}-search-template>>
* <<{upid}-multi-search-template>>
* <<{upid}-multi-search>>
* <<{upid}-field-caps>>
* <<{upid}-rank-eval>>
* <<{upid}-explain>>
* <<{upid}-count>>
include::search/search.asciidoc[]
include::search/scroll.asciidoc[]
include::search/multi-search.asciidoc[]
include::search/search-template.asciidoc[]
include::search/multi-search-template.asciidoc[]
include::search/field-caps.asciidoc[]
include::search/rank-eval.asciidoc[]
include::search/explain.asciidoc[]
include::search/count.asciidoc[]
== Miscellaneous APIs
The Java High Level REST Client supports the following Miscellaneous APIs:
* <<java-rest-high-main>>
* <<java-rest-high-ping>>
* <<java-rest-high-x-pack-info>>
* <<java-rest-high-x-pack-usage>>
include::miscellaneous/main.asciidoc[]
include::miscellaneous/ping.asciidoc[]
include::miscellaneous/x-pack-info.asciidoc[]
include::miscellaneous/x-pack-usage.asciidoc[]
== Indices APIs
:upid: {mainid}
:doc-tests-file: {doc-tests}/IndicesClientDocumentationIT.java
The Java High Level REST Client supports the following Indices APIs:
Index Management::
* <<{upid}-analyze>>
* <<{upid}-create-index>>
* <<{upid}-delete-index>>
* <<{upid}-indices-exists>>
* <<{upid}-open-index>>
* <<{upid}-close-index>>
* <<{upid}-shrink-index>>
* <<{upid}-split-index>>
* <<{upid}-refresh>>
* <<{upid}-flush>>
* <<{upid}-flush-synced>>
* <<{upid}-clear-cache>>
* <<{upid}-force-merge>>
* <<{upid}-rollover-index>>
* <<{upid}-indices-put-settings>>
* <<{upid}-get-settings>>
* <<{upid}-indices-validate-query>>
* <<{upid}-get-index>>
Mapping Management::
* <<{upid}-put-mapping>>
* <<{upid}-get-mappings>>
* <<{upid}-get-field-mappings>>
Alias Management::
* <<{upid}-update-aliases>>
* <<{upid}-exists-alias>>
* <<{upid}-get-alias>>
Template Management::
* <<{upid}-get-templates>>
* <<{upid}-templates-exist>>
* <<{upid}-put-template>>
include::indices/analyze.asciidoc[]
include::indices/create_index.asciidoc[]
include::indices/delete_index.asciidoc[]
include::indices/indices_exists.asciidoc[]
include::indices/open_index.asciidoc[]
include::indices/close_index.asciidoc[]
include::indices/shrink_index.asciidoc[]
include::indices/split_index.asciidoc[]
include::indices/refresh.asciidoc[]
include::indices/flush.asciidoc[]
include::indices/flush_synced.asciidoc[]
include::indices/clear_cache.asciidoc[]
include::indices/force_merge.asciidoc[]
include::indices/rollover.asciidoc[]
include::indices/put_mapping.asciidoc[]
include::indices/get_mappings.asciidoc[]
include::indices/get_field_mappings.asciidoc[]
include::indices/update_aliases.asciidoc[]
include::indices/exists_alias.asciidoc[]
include::indices/get_alias.asciidoc[]
include::indices/put_settings.asciidoc[]
include::indices/get_settings.asciidoc[]
include::indices/put_template.asciidoc[]
include::indices/validate_query.asciidoc[]
include::indices/get_templates.asciidoc[]
include::indices/templates_exist.asciidoc[]
include::indices/get_index.asciidoc[]
include::indices/freeze_index.asciidoc[]
include::indices/unfreeze_index.asciidoc[]
include::indices/delete_template.asciidoc[]
== Cluster APIs
The Java High Level REST Client supports the following Cluster APIs:
* <<java-rest-high-cluster-put-settings>>
* <<java-rest-high-cluster-get-settings>>
* <<java-rest-high-cluster-health>>
:upid: {mainid}-cluster
:doc-tests-file: {doc-tests}/ClusterClientDocumentationIT.java
include::cluster/put_settings.asciidoc[]
include::cluster/get_settings.asciidoc[]
include::cluster/health.asciidoc[]
== Ingest APIs
The Java High Level REST Client supports the following Ingest APIs:
* <<java-rest-high-ingest-put-pipeline>>
* <<java-rest-high-ingest-get-pipeline>>
* <<java-rest-high-ingest-delete-pipeline>>
* <<java-rest-high-ingest-simulate-pipeline>>
include::ingest/put_pipeline.asciidoc[]
include::ingest/get_pipeline.asciidoc[]
include::ingest/delete_pipeline.asciidoc[]
include::ingest/simulate_pipeline.asciidoc[]
== Snapshot APIs
The Java High Level REST Client supports the following Snapshot APIs:
* <<java-rest-high-snapshot-get-repository>>
* <<java-rest-high-snapshot-create-repository>>
* <<java-rest-high-snapshot-delete-repository>>
* <<java-rest-high-snapshot-verify-repository>>
* <<java-rest-high-snapshot-create-snapshot>>
* <<java-rest-high-snapshot-get-snapshots>>
* <<java-rest-high-snapshot-snapshots-status>>
* <<java-rest-high-snapshot-delete-snapshot>>
include::snapshot/get_repository.asciidoc[]
include::snapshot/create_repository.asciidoc[]
include::snapshot/delete_repository.asciidoc[]
include::snapshot/verify_repository.asciidoc[]
include::snapshot/create_snapshot.asciidoc[]
include::snapshot/get_snapshots.asciidoc[]
include::snapshot/snapshots_status.asciidoc[]
include::snapshot/delete_snapshot.asciidoc[]
== Tasks APIs
The Java High Level REST Client supports the following Tasks APIs:
* <<java-rest-high-tasks-list>>
* <<java-rest-high-cluster-cancel-tasks>>
include::tasks/list_tasks.asciidoc[]
include::tasks/cancel_tasks.asciidoc[]
== Script APIs
The Java High Level REST Client supports the following Scripts APIs:
* <<java-rest-high-get-stored-script>>
* <<java-rest-high-put-stored-script>>
* <<java-rest-high-delete-stored-script>>
include::script/get_script.asciidoc[]
include::script/put_script.asciidoc[]
include::script/delete_script.asciidoc[]
== Licensing APIs
The Java High Level REST Client supports the following Licensing APIs:
* <<java-rest-high-put-license>>
* <<java-rest-high-get-license>>
* <<java-rest-high-delete-license>>
* <<java-rest-high-start-trial>>
* <<java-rest-high-start-basic>>
* <<java-rest-high-get-trial-status>>
* <<java-rest-high-get-basic-status>>
include::licensing/put-license.asciidoc[]
include::licensing/get-license.asciidoc[]
include::licensing/delete-license.asciidoc[]
include::licensing/start-trial.asciidoc[]
include::licensing/start-basic.asciidoc[]
include::licensing/get-trial-status.asciidoc[]
include::licensing/get-basic-status.asciidoc[]
== Machine Learning APIs
:upid: {mainid}-x-pack-ml
:doc-tests-file: {doc-tests}/MlClientDocumentationIT.java
The Java High Level REST Client supports the following Machine Learning APIs:
* <<{upid}-put-job>>
* <<{upid}-get-job>>
* <<{upid}-delete-job>>
* <<{upid}-open-job>>
* <<{upid}-close-job>>
* <<{upid}-flush-job>>
* <<{upid}-update-job>>
* <<{upid}-get-job-stats>>
* <<{upid}-put-datafeed>>
* <<{upid}-update-datafeed>>
* <<{upid}-get-datafeed>>
* <<{upid}-delete-datafeed>>
* <<{upid}-preview-datafeed>>
* <<{upid}-start-datafeed>>
* <<{upid}-stop-datafeed>>
* <<{upid}-get-datafeed-stats>>
* <<{upid}-forecast-job>>
* <<{upid}-delete-forecast>>
* <<{upid}-get-buckets>>
* <<{upid}-get-overall-buckets>>
* <<{upid}-get-records>>
* <<{upid}-post-data>>
* <<{upid}-get-influencers>>
* <<{upid}-get-categories>>
* <<{upid}-get-calendars>>
* <<{upid}-put-calendar>>
* <<{upid}-get-calendar-events>>
* <<{upid}-post-calendar-event>>
* <<{upid}-delete-calendar-event>>
* <<{upid}-put-calendar-job>>
* <<{upid}-delete-calendar-job>>
* <<{upid}-delete-calendar>>
* <<{upid}-get-data-frame-analytics>>
* <<{upid}-get-data-frame-analytics-stats>>
* <<{upid}-put-data-frame-analytics>>
* <<{upid}-delete-data-frame-analytics>>
* <<{upid}-start-data-frame-analytics>>
* <<{upid}-stop-data-frame-analytics>>
* <<{upid}-evaluate-data-frame>>
* <<{upid}-put-filter>>
* <<{upid}-get-filters>>
* <<{upid}-update-filter>>
* <<{upid}-delete-filter>>
* <<{upid}-get-model-snapshots>>
* <<{upid}-delete-model-snapshot>>
* <<{upid}-revert-model-snapshot>>
* <<{upid}-update-model-snapshot>>
* <<{upid}-get-ml-info>>
* <<{upid}-delete-expired-data>>
* <<{upid}-set-upgrade-mode>>
include::ml/put-job.asciidoc[]
include::ml/get-job.asciidoc[]
include::ml/delete-job.asciidoc[]
include::ml/open-job.asciidoc[]
include::ml/close-job.asciidoc[]
include::ml/update-job.asciidoc[]
include::ml/flush-job.asciidoc[]
include::ml/put-datafeed.asciidoc[]
include::ml/update-datafeed.asciidoc[]
include::ml/get-datafeed.asciidoc[]
include::ml/delete-datafeed.asciidoc[]
include::ml/preview-datafeed.asciidoc[]
include::ml/start-datafeed.asciidoc[]
include::ml/stop-datafeed.asciidoc[]
include::ml/get-datafeed-stats.asciidoc[]
include::ml/get-job-stats.asciidoc[]
include::ml/forecast-job.asciidoc[]
include::ml/delete-forecast.asciidoc[]
include::ml/get-buckets.asciidoc[]
include::ml/get-overall-buckets.asciidoc[]
include::ml/get-records.asciidoc[]
include::ml/post-data.asciidoc[]
include::ml/get-influencers.asciidoc[]
include::ml/get-categories.asciidoc[]
include::ml/get-calendars.asciidoc[]
include::ml/put-calendar.asciidoc[]
include::ml/get-calendar-events.asciidoc[]
include::ml/post-calendar-event.asciidoc[]
include::ml/delete-calendar-event.asciidoc[]
include::ml/put-calendar-job.asciidoc[]
include::ml/delete-calendar-job.asciidoc[]
include::ml/delete-calendar.asciidoc[]
include::ml/get-data-frame-analytics.asciidoc[]
include::ml/get-data-frame-analytics-stats.asciidoc[]
include::ml/put-data-frame-analytics.asciidoc[]
include::ml/delete-data-frame-analytics.asciidoc[]
include::ml/start-data-frame-analytics.asciidoc[]
include::ml/stop-data-frame-analytics.asciidoc[]
include::ml/evaluate-data-frame.asciidoc[]
include::ml/put-filter.asciidoc[]
include::ml/get-filters.asciidoc[]
include::ml/update-filter.asciidoc[]
include::ml/delete-filter.asciidoc[]
include::ml/get-model-snapshots.asciidoc[]
include::ml/delete-model-snapshot.asciidoc[]
include::ml/revert-model-snapshot.asciidoc[]
include::ml/update-model-snapshot.asciidoc[]
include::ml/get-info.asciidoc[]
include::ml/delete-expired-data.asciidoc[]
include::ml/set-upgrade-mode.asciidoc[]
== Migration APIs
:upid: {mainid}-migration
:doc-tests-file: {doc-tests}/MigrationClientDocumentationIT.java
The Java High Level REST Client supports the following Migration APIs:
* <<{upid}-get-deprecation-info>>
include::migration/get-deprecation-info.asciidoc[]
== Rollup APIs
:upid: {mainid}-rollup
:doc-tests-file: {doc-tests}/RollupDocumentationIT.java
The Java High Level REST Client supports the following Rollup APIs:
* <<java-rest-high-x-pack-rollup-put-job>>
* <<{upid}-rollup-start-job>>
* <<{upid}-rollup-stop-job>>
* <<{upid}-rollup-delete-job>>
* <<java-rest-high-x-pack-rollup-get-job>>
* <<{upid}-search>>
* <<{upid}-x-pack-rollup-get-rollup-caps>>
* <<{upid}-x-pack-rollup-get-rollup-index-caps>>
include::rollup/put_job.asciidoc[]
include::rollup/start_job.asciidoc[]
include::rollup/stop_job.asciidoc[]
include::rollup/delete_job.asciidoc[]
include::rollup/get_job.asciidoc[]
include::rollup/search.asciidoc[]
include::rollup/get_rollup_caps.asciidoc[]
include::rollup/get_rollup_index_caps.asciidoc[]
== Security APIs
:upid: {mainid}-security
:doc-tests-file: {doc-tests}/SecurityDocumentationIT.java
The Java High Level REST Client supports the following Security APIs:
* <<java-rest-high-security-put-user>>
* <<{upid}-get-users>>
* <<{upid}-delete-user>>
* <<java-rest-high-security-enable-user>>
* <<java-rest-high-security-disable-user>>
* <<java-rest-high-security-change-password>>
* <<{upid}-put-role>>
* <<{upid}-get-roles>>
* <<java-rest-high-security-delete-role>>
* <<{upid}-clear-roles-cache>>
* <<{upid}-clear-realm-cache>>
* <<{upid}-authenticate>>
* <<{upid}-has-privileges>>
* <<{upid}-get-user-privileges>>
* <<java-rest-high-security-get-certificates>>
* <<java-rest-high-security-put-role-mapping>>
* <<java-rest-high-security-get-role-mappings>>
* <<java-rest-high-security-delete-role-mapping>>
* <<java-rest-high-security-create-token>>
* <<{upid}-invalidate-token>>
* <<{upid}-get-privileges>>
* <<{upid}-put-privileges>>
* <<{upid}-delete-privileges>>
* <<{upid}-create-api-key>>
* <<{upid}-get-api-key>>
* <<{upid}-invalidate-api-key>>
include::security/put-user.asciidoc[]
include::security/get-users.asciidoc[]
include::security/delete-user.asciidoc[]
include::security/enable-user.asciidoc[]
include::security/disable-user.asciidoc[]
include::security/change-password.asciidoc[]
include::security/put-role.asciidoc[]
include::security/get-roles.asciidoc[]
include::security/delete-role.asciidoc[]
include::security/delete-privileges.asciidoc[]
include::security/get-privileges.asciidoc[]
include::security/clear-roles-cache.asciidoc[]
include::security/clear-realm-cache.asciidoc[]
include::security/authenticate.asciidoc[]
include::security/has-privileges.asciidoc[]
include::security/get-user-privileges.asciidoc[]
include::security/get-certificates.asciidoc[]
include::security/put-role-mapping.asciidoc[]
include::security/get-role-mappings.asciidoc[]
include::security/delete-role-mapping.asciidoc[]
include::security/create-token.asciidoc[]
include::security/invalidate-token.asciidoc[]
include::security/put-privileges.asciidoc[]
include::security/create-api-key.asciidoc[]
include::security/get-api-key.asciidoc[]
include::security/invalidate-api-key.asciidoc[]
== Watcher APIs
:upid: {mainid}-watcher
:doc-tests-file: {doc-tests}/WatcherDocumentationIT.java
The Java High Level REST Client supports the following Watcher APIs:
* <<{upid}-start-watch-service>>
* <<{upid}-stop-watch-service>>
* <<java-rest-high-x-pack-watcher-put-watch>>
* <<{upid}-get-watch>>
* <<java-rest-high-x-pack-watcher-delete-watch>>
* <<java-rest-high-watcher-deactivate-watch>>
* <<{upid}-ack-watch>>
* <<{upid}-activate-watch>>
* <<{upid}-execute-watch>>
* <<{upid}-watcher-stats>>
include::watcher/start-watch-service.asciidoc[]
include::watcher/stop-watch-service.asciidoc[]
include::watcher/put-watch.asciidoc[]
include::watcher/get-watch.asciidoc[]
include::watcher/delete-watch.asciidoc[]
include::watcher/ack-watch.asciidoc[]
include::watcher/deactivate-watch.asciidoc[]
include::watcher/activate-watch.asciidoc[]
include::watcher/execute-watch.asciidoc[]
include::watcher/watcher-stats.asciidoc[]
== Graph APIs
The Java High Level REST Client supports the following Graph APIs:
* <<java-rest-high-x-pack-graph-explore>>
include::graph/explore.asciidoc[]
////
Clear attributes that we use to document that APIs included above so they
don't leak into the rest of the documentation.
////
--
:api!:
:request!:
:response!:
:doc-tests-file!:
:upid!:
--
== CCR APIs
:upid: {mainid}-ccr
:doc-tests-file: {doc-tests}/CCRDocumentationIT.java
The Java High Level REST Client supports the following CCR APIs:
* <<{upid}-ccr-put-follow>>
* <<{upid}-ccr-pause-follow>>
* <<{upid}-ccr-resume-follow>>
* <<{upid}-ccr-unfollow>>
* <<{upid}-ccr-forget-follower>>
* <<{upid}-ccr-put-auto-follow-pattern>>
* <<{upid}-ccr-delete-auto-follow-pattern>>
* <<{upid}-ccr-get-auto-follow-pattern>>
* <<{upid}-ccr-get-stats>>
* <<{upid}-ccr-get-follow-stats>>
* <<{upid}-ccr-get-follow-info>>
include::ccr/put_follow.asciidoc[]
include::ccr/pause_follow.asciidoc[]
include::ccr/resume_follow.asciidoc[]
include::ccr/unfollow.asciidoc[]
include::ccr/forget_follower.asciidoc[]
include::ccr/put_auto_follow_pattern.asciidoc[]
include::ccr/delete_auto_follow_pattern.asciidoc[]
include::ccr/get_auto_follow_pattern.asciidoc[]
include::ccr/get_stats.asciidoc[]
include::ccr/get_follow_stats.asciidoc[]
include::ccr/get_follow_info.asciidoc[]
== Index Lifecycle Management APIs
:upid: {mainid}-ilm
:doc-tests-file: {doc-tests}/ILMDocumentationIT.java
The Java High Level REST Client supports the following Index Lifecycle
Management APIs:
* <<{upid}-ilm-put-lifecycle-policy>>
* <<{upid}-ilm-delete-lifecycle-policy>>
* <<{upid}-ilm-get-lifecycle-policy>>
* <<{upid}-ilm-explain-lifecycle>>
* <<{upid}-ilm-start-ilm>>
* <<{upid}-ilm-stop-ilm>>
* <<{upid}-ilm-status>>
* <<{upid}-ilm-retry-lifecycle-policy>>
* <<{upid}-ilm-remove-lifecycle-policy-from-index>>
include::ilm/put_lifecycle_policy.asciidoc[]
include::ilm/delete_lifecycle_policy.asciidoc[]
include::ilm/get_lifecycle_policy.asciidoc[]
include::ilm/explain_lifecycle.asciidoc[]
include::ilm/start_lifecycle_management.asciidoc[]
include::ilm/stop_lifecycle_management.asciidoc[]
include::ilm/lifecycle_management_status.asciidoc[]
include::ilm/retry_lifecycle_policy.asciidoc[]
include::ilm/remove_lifecycle_policy_from_index.asciidoc[]
== Data Frame APIs
:upid: {mainid}-dataframe
:doc-tests-file: {doc-tests}/DataFrameTransformDocumentationIT.java
The Java High Level REST Client supports the following Data Frame APIs:
* <<{upid}-get-data-frame-transform>>
* <<{upid}-get-data-frame-transform-stats>>
* <<{upid}-put-data-frame-transform>>
* <<{upid}-delete-data-frame-transform>>
* <<{upid}-preview-data-frame-transform>>
* <<{upid}-start-data-frame-transform>>
* <<{upid}-stop-data-frame-transform>>
include::dataframe/get_data_frame.asciidoc[]
include::dataframe/get_data_frame_stats.asciidoc[]
include::dataframe/put_data_frame.asciidoc[]
include::dataframe/delete_data_frame.asciidoc[]
include::dataframe/preview_data_frame.asciidoc[]
include::dataframe/start_data_frame.asciidoc[]
include::dataframe/stop_data_frame.asciidoc[]