[DOCS] Separate Enrich API Docs (#46286)

* Add enrich policy common parameter

* Add enrich APIs to REST APIs index

* Add put enrich policy API docs

* Add get enrich policy API docs

* Add delete enrich policy API docs

* Add execute enrich policy API docs
This commit is contained in:
James Rodewig 2019-09-04 14:11:52 -04:00 committed by Martijn van Groningen
parent c79a8e448d
commit d74d995382
No known key found for this signature in database
GPG Key ID: AB236F4FCF2AF12A
8 changed files with 503 additions and 199 deletions

View File

@ -0,0 +1,69 @@
[role="xpack"]
[testenv="basic"]
[[delete-enrich-policy-api]]
=== Delete enrich policy API
++++
<titleabbrev>Delete enrich policy</titleabbrev>
++++
Deletes an existing enrich policy and its enrich index.
////
[source,js]
----
PUT /users
PUT /_enrich/policy/my-policy
{
"match": {
"indices": "users",
"match_field": "email",
"enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
}
}
----
// CONSOLE
// TESTSETUP
////
[source,js]
--------------------------------------------------
DELETE /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
[[delete-enrich-policy-api-request]]
==== {api-request-title}
`DELETE /_enrich/policy/<enrich-policy>`
[[delete-enrich-policy-api-prereqs]]
==== {api-prereq-title}
include::put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]
[[delete-enrich-policy-api-desc]]
==== {api-description-title}
Use the delete enrich policy API
to delete an existing enrich policy
and its enrich index.
[IMPORTANT]
====
You must remove an enrich policy
from any in-use ingest pipelines
before deletion.
You cannot remove in-use enrich policies.
====
[[delete-enrich-policy-api-path-params]]
==== {api-path-parms-title}
`<enrich-policy>`::
(Required, string)
Enrich policy to delete.

View File

@ -0,0 +1,107 @@
[role="xpack"]
[testenv="basic"]
[[execute-enrich-policy-api]]
=== Execute enrich policy API
++++
<titleabbrev>Execute enrich policy</titleabbrev>
++++
Executes an existing enrich policy.
////
[source,js]
----
PUT /users/_doc/1?refresh
{
"email": "mardy.brown@asciidocsmith.com",
"first_name": "Mardy",
"last_name": "Brown",
"city": "New Orleans",
"county": "Orleans",
"state": "LA",
"zip": 70116,
"web": "mardy.asciidocsmith.com"
}
PUT /_enrich/policy/my-policy
{
"match": {
"indices": "users",
"match_field": "email",
"enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
}
}
----
// CONSOLE
// TESTSETUP
////
[source,js]
--------------------------------------------------
PUT /_enrich/policy/my-policy/_execute
--------------------------------------------------
// CONSOLE
////
[source,js]
--------------------------------------------------
DELETE /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
////
[[execute-enrich-policy-api-request]]
==== {api-request-title}
`PUT /_enrich/policy/<enrich-policy>/_execute`
`POST /_enrich/policy/<enrich-policy>/_execute`
[[execute-enrich-policy-api-prereqs]]
==== {api-prereq-title}
include::put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]
[[execute-enrich-policy-api-desc]]
==== {api-description-title}
Use the execute enrich policy API
to create the enrich index for an existing enrich policy.
// tag::execute-enrich-policy-desc[]
The *enrich index* contains documents from the policy's source indices.
Enrich indices always begin with `.enrich-*`,
are read-only,
and are <<indices-forcemerge,force merged>>.
[WARNING]
====
Enrich indices should be used by the <<enrich-processor,enrich processor>> only.
Avoid using enrich indices for other purposes.
====
Once created, you cannot update
or index documents to an enrich index.
Instead, update your source indices
and execute the enrich policy again.
This creates a new enrich index from your updated source indices
and deletes the previous enrich index.
Because this API request performs several operations,
it may take a while to return a response.
// end::execute-enrich-policy-desc[]
[[sample-api-path-params]]
==== {api-path-parms-title}
`<enrich-policy>`::
(Required, string)
Enrich policy to execute.

View File

@ -0,0 +1,140 @@
[role="xpack"]
[testenv="basic"]
[[get-enrich-policy-api]]
=== Get enrich policy API
++++
<titleabbrev>Get enrich policy</titleabbrev>
++++
Returns information about an enrich policy.
////
[source,js]
----
PUT /users
PUT /_enrich/policy/my-policy
{
"match": {
"indices": "users",
"match_field": "email",
"enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
}
}
----
// CONSOLE
////
[source,js]
--------------------------------------------------
GET /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
[[get-enrich-policy-api-request]]
==== {api-request-title}
`GET /_enrich/policy/<enrich-policy>`
`GET /_enrich/policy`
[[get-enrich-policy-api-prereqs]]
==== {api-prereq-title}
include::put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]
[[get-enrich-policy-api-path-params]]
==== {api-path-parms-title}
`<enrich-policy>`::
(Optional, string)
include::{docdir}/rest-api/common-parms.asciidoc[tag=enrich-policy]
[[get-enrich-policy-api-example]]
==== {api-examples-title}
[[get-enrich-policy-api-single-ex]]
===== Get a single policy
[source,js]
--------------------------------------------------
GET /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
The API returns the following response:
[source,js]
--------------------------------------------------
{
"policies": [
{
"match": {
"name" : "my-policy",
"indices" : ["users"],
"match_field" : "email",
"enrich_fields" : [
"first_name",
"last_name",
"city",
"zip",
"state"
]
}
}
]
}
--------------------------------------------------
// TESTRESPONSE
[[get-enrich-policy-api-all-ex]]
===== Get all policies
[source,js]
--------------------------------------------------
GET /_enrich/policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
The API returns the following response:
[source,js]
--------------------------------------------------
{
"policies": [
{
"match": {
"name" : "my-policy",
"indices" : ["users"],
"match_field" : "email",
"enrich_fields" : [
"first_name",
"last_name",
"city",
"zip",
"state"
]
}
}
]
}
--------------------------------------------------
// TESTRESPONSE
////
[source,js]
--------------------------------------------------
DELETE /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
////

View File

@ -0,0 +1,18 @@
[[enrich-apis]]
== Enrich APIs
The following enrich APIs are available for managing enrich policies:
* <<put-enrich-policy-api>> to add or update an enrich policy
* <<delete-enrich-policy-api>> to delete an enrich policy
* <<get-enrich-policy-api>> to return information about an enrich policy
* <<execute-enrich-policy-api>> to execute an enrich policy
include::put-enrich-policy.asciidoc[]
include::delete-enrich-policy.asciidoc[]
include::get-enrich-policy.asciidoc[]
include::execute-enrich-policy.asciidoc[]

View File

@ -0,0 +1,149 @@
[role="xpack"]
[testenv="basic"]
[[put-enrich-policy-api]]
=== Put enrich policy API
++++
<titleabbrev>Put enrich policy</titleabbrev>
++++
Creates an enrich policy.
////
[source,js]
----
PUT /users
----
// CONSOLE
////
[source,js]
----
PUT /_enrich/policy/my-policy
{
"match": {
"indices": "users",
"match_field": "email",
"enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
}
}
----
// CONSOLE
// TEST[continued]
////
[source,js]
--------------------------------------------------
DELETE /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
////
[[put-enrich-policy-api-request]]
==== {api-request-title}
`PUT /_enrich/policy/<enrich-policy>`
[[put-enrich-policy-api-prereqs]]
==== {api-prereq-title}
// tag::enrich-policy-api-prereqs[]
If you use {es} {security-features}, you must have:
* `read` index privileges for any indices used
* The `enrich_user` {stack-ov}/built-in-roles.html[built-in role]
// end::enrich-policy-api-prereqs[]
[[put-enrich-policy-api-desc]]
==== {api-description-title}
Use the put enrich policy API
to create a new enrich policy.
// tag::enrich-policy-def
An *enrich policy* is a set of rules the enrich processor uses
to append the appropriate data to incoming documents.
An enrich policy contains:
* The *policy type*,
which determines how the processor enriches incoming documents
* A list of source indices
* The *match field*, a field used to match incoming documents
* *Enrich fields*, fields appended to incoming documents
from matching documents
// end::enrich-policy-def
===== Update an enrich policy
// tag::update-enrich-policy
You cannot update an existing enrich policy.
Instead, you can:
. Create and execute a new enrich policy.
. Replace the previous enrich policy
with the new enrich policy
in any in-use enrich processors.
. Use the <<delete-enrich-policy-api, delete enrich policy API>>
to delete the previous enrich policy.
// end::update-enrich-policy
[[put-enrich-policy-api-path-params]]
==== {api-path-parms-title}
`<enrich-policy>`::
(Required, string)
include::{docdir}/rest-api/common-parms.asciidoc[tag=enrich-policy]
[[put-enrich-policy-api-request-body]]
==== {api-request-body-title}
`<policy-type>`::
+
--
(Required, enrich policy object)
The parameter key is the enrich policy type.
The enrich policy type indicates
how the enrich processor matches incoming documents
to documents in the enrich index.
Valid key values are:
`match`::
Match documents in the enrich index
using a <<query-dsl-term-query, term query>> for the `match_field`.
The parameter value is the enrich policy.
The enrich policy is a set of rules
used to create an <<execute-enrich-policy,enrich index>>.
The enrich processor also uses these rules
to append field data to incoming documents.
Parameters include:
`indices`::
(Required, array of strings)
Source indices used to create the enrich index.
`query`::
(Optional, string)
Query type used to find and select documents in the enrich index.
Valid value is <<query-dsl-match-all-query,`match_all`>> (default).
`match_field`::
(Required, string)
Field used to match incoming documents
to documents in the enrich index.
`enrich_fields`::
(Required, Array of string)
Fields appended to incoming documents
from matching documents in the enrich index.
--

View File

@ -755,7 +755,9 @@ metadata field to provide the error message.
[role="xpack"]
[testenv="basic"]
[[ingest-enriching-data]]
== Enriching data with ingest node
== Enrich your data using the ingest node
The <<enrich-processor,enrich processor>> allows documents to be enriched with data from
an enrich index that is managed by an enrich policy prior to indexing.
@ -845,6 +847,7 @@ Which returns:
--------------------------------------------------
// TESTRESPONSE
[[execute-enrich-policy]]
Execute that enrich policy:
[source,js]
@ -939,204 +942,15 @@ DELETE /_enrich/policy/users-policy
Also there are several APIs in order to manage and execute enrich policies:
* <<put-policy-api,Put policy api>>.
* <<get-policy-api,Get policy api>>.
* <<delete-policy-api,Delete policy api>>.
* <<execute-policy-api,Execute policy api>>.
* <<put-enrich-policy-api,Put policy api>>.
* <<get-enrich-policy-api,Get enrich policy api>>.
* <<delete-enrich-policy-api,Delete policy api>>.
* <<execute-enrich-policy-api,Execute policy api>>.
If security is enabled then the user managing enrich policies will need to have
the `enrich_user` builtin role. Also the user will need to have read privileges
for the indices the enrich policy is referring to.
[[put-policy-api]]
==== Put Policy API
The put policy api allows a policy to be stored by an user specified id in the url and
the enrich policy definition as body.
Request:
[source,js]
--------------------------------------------------
PUT /_enrich/policy/my-policy
{
"match": {
"indices": "users",
"match_field": "email",
"enrich_fields": ["first_name", "last_name", "address", "city", "zip", "state"]
}
}
--------------------------------------------------
// CONSOLE
Response:
[source,js]
--------------------------------------------------
{
"acknowledged": true
}
--------------------------------------------------
// TESTRESPONSE
[[get-policy-api]]
==== Get Policy API
The get policy api allows a policy to be retrieved by id.
Request:
[source,js]
--------------------------------------------------
GET /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
Response:
[source,js]
--------------------------------------------------
{
"policies": [
{
"match": {
"name" : "my-policy",
"indices" : ["users"],
"match_field" : "email",
"enrich_fields" : [
"first_name",
"last_name",
"address",
"city",
"zip",
"state"
]
}
}
]
}
--------------------------------------------------
// TESTRESPONSE
The get policy api allows all policies to be returned.
Request:
[source,js]
--------------------------------------------------
GET /_enrich/policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
Response:
[source,js]
--------------------------------------------------
{
"policies": [
{
"match": {
"name" : "my-policy",
"indices" : ["users"],
"match_field" : "email",
"enrich_fields" : [
"first_name",
"last_name",
"address",
"city",
"zip",
"state"
]
}
}
]
}
--------------------------------------------------
// TESTRESPONSE
[[execute-policy-api]]
==== Execute Policy API
The execute policy api executes a policy based on the provided id.
It may take some time before this API returns a response.
Executing a policy involves creating a new enrich index, indexing the documents from
the indices specified in policy into the enrich index and some other operations.
Note that this api needs to be re-executed in order to incorporate new changes
in the index the policy is pointing to after the policy has been executed.
This API creates an index with the `.enrich-*` prefix in the name. This index purpose
is the be used by the enrich processor only and should not be used by anything else.
Internally old `.enrich-*` are removed by an internal cleanup mechanism.
//////////////////////////
[source,js]
--------------------------------------------------
PUT /users/_doc/1?refresh
{
"email": "mardy.brown@email.me",
"first_name": "Mardy",
"last_name": "Brown",
"address": "6649 N Blue Gum St",
"city": "New Orleans",
"county": "Orleans",
"state": "LA",
"zip": 70116,
"phone1":"504-621-8927",
"phone2": "504-845-1427",
"web": "mardy-brown.me"
}
--------------------------------------------------
// CONSOLE
// TEST[continued]
//////////////////////////
Request:
[source,js]
--------------------------------------------------
POST /_enrich/policy/my-policy/_execute
--------------------------------------------------
// CONSOLE
// TEST[continued]
Response:
[source,js]
--------------------------------------------------
{
"acknowledged": true
}
--------------------------------------------------
// TESTRESPONSE
[[delete-policy-api]]
===== Delete Policy API
The delete policy api allows a policy to be removed by id.
Request:
[source,js]
--------------------------------------------------
DELETE /_enrich/policy/my-policy
--------------------------------------------------
// CONSOLE
// TEST[continued]
Response:
[source,js]
--------------------------------------------------
{
"acknowledged": true
}
--------------------------------------------------
// TESTRESPONSE
[[ingest-processors]]
== Processors

View File

@ -49,7 +49,7 @@ tag::committed[]
If `true`,
the segments is synced to disk. Segments that are synced can survive a hard reboot.
+
If `false`,
If `false`,
the data from uncommitted segments is also stored in
the transaction log so that Elasticsearch is able to replay
changes on the next start.
@ -86,6 +86,11 @@ is based on Lucene documents. {es} reclaims the disk space of deleted Lucene
documents when a segment is merged.
end::docs-deleted[]
tag::enrich-policy[]
Enrich policy name
used to limit the request.
end::enrich-policy[]
tag::expand-wildcards[]
`expand_wildcards`::
+
@ -212,8 +217,8 @@ end::include-defaults[]
tag::include-segment-file-sizes[]
`include_segment_file_sizes`::
(Optional, boolean)
If `true`, the call reports the aggregated disk usage of
each one of the Lucene index files (only applies if segment stats are
If `true`, the call reports the aggregated disk usage of
each one of the Lucene index files (only applies if segment stats are
requested). Defaults to `false`.
end::include-segment-file-sizes[]
@ -390,7 +395,7 @@ end::pipeline[]
tag::preference[]
`preference`::
(Optional, string) Specifies the node or shard the operation should be
(Optional, string) Specifies the node or shard the operation should be
performed on. Random by default.
end::preference[]
@ -518,7 +523,7 @@ end::source_includes[]
tag::stats[]
`stats`::
(Optional, string) Specific `tag` of the request for logging and statistical
(Optional, string) Specific `tag` of the request for logging and statistical
purposes.
end::stats[]

View File

@ -16,6 +16,7 @@ not be included yet.
* <<ccr-apis,{ccr-cap} APIs>>
* <<data-frame-apis,{dataframe-transform-cap} APIs>>
* <<docs, Document APIs>>
* <<enrich-apis,Enrich APIs>>
* <<graph-explore-api,Graph Explore API>>
* <<indices, Index APIs>>
* <<index-lifecycle-management-api,Index lifecycle management APIs>>
@ -39,6 +40,7 @@ include::{es-repo-dir}/cluster.asciidoc[]
include::{es-repo-dir}/ccr/apis/ccr-apis.asciidoc[]
include::{es-repo-dir}/data-frames/apis/index.asciidoc[]
include::{es-repo-dir}/docs.asciidoc[]
include::{es-repo-dir}/ingest/apis/enrich/index.asciidoc[]
include::{es-repo-dir}/graph/explore.asciidoc[]
include::{es-repo-dir}/indices.asciidoc[]
include::{es-repo-dir}/ilm/apis/ilm-api.asciidoc[]