opensearch-docs-cn/_search-plugins/search-pipelines/index.md

81 lines
4.3 KiB
Markdown
Raw Normal View History

Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
---
layout: default
title: Search pipelines
nav_order: 100
has_children: true
has_toc: false
---
# Search pipelines
You can use _search pipelines_ to build new or reuse existing result rerankers, query rewriters, and other components that operate on queries or results. Search pipelines make it easier for you to process search queries and search results within OpenSearch. Moving some of your application functionality into an OpenSearch search pipeline reduces the overall complexity of your application. As part of a search pipeline, you specify a list of processors that perform modular tasks. You can then easily add or reorder these processors to customize search results for your application.
## Terminology
The following is a list of search pipeline terminology:
Add score normalization and combination documentation (#4985) * Add search phase results processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add hybrid query Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Normalization processor additions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add more details Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Continue writing Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add more query then fetch details and diagram Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Small rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Leaner left nav headers Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Tech review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add semantic search tutorial Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded prerequisites Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Removed comma Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Rewording advanced prerequisites Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Changed searching for ML model to shorter request Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update task type in register model response Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Changing example Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added huggingface prefix to model names Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change example responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added note about huggingface prefix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update _ml-commons-plugin/semantic-search.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented doc review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * List weights under parameters Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove one-shard warning for normalization processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change links Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * More editorial feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change model-serving framework to ML framework Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Use get model API to check model status Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added neural search description and diagram Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * More editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add link to profile API Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Addressed more tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented editorial comments on changes Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-09-22 17:29:58 -04:00
* [_Search request processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-request-processors): A component that intercepts a search request (the query and the metadata passed in the request), performs an operation with or on the search request, and returns the search request.
* [_Search response processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-response-processors): A component that intercepts a search response and search request (the query, results, and metadata passed in the request), performs an operation with or on the search response, and returns the search response.
* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor intercepts the results retrieved from one search phase and transforms them before passing them to the next search phase.
* [_Processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors/): Either a search request processor or a search response processor.
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
* _Search pipeline_: An ordered list of processors that is integrated into OpenSearch. The pipeline intercepts a query, performs processing on the query, sends it to OpenSearch, intercepts the results, performs processing on the results, and returns them to the calling application, as shown in the following diagram.
![Search processor diagram]({{site.url}}{{site.baseurl}}/images/search-pipelines.png)
Both request and response processing for the pipeline are performed on the coordinator node, so there is no shard-level processing.
{: .note}
## Processors
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
To learn more about available search processors, see [Search processors]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors/).
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
## Example
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
To create a search pipeline, send a request to the search pipeline endpoint specifying an ordered list of processors, which will be applied sequentially:
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
```json
PUT /_search/pipeline/my_pipeline
{
"request_processors": [
{
"filter_query" : {
"tag" : "tag1",
"description" : "This processor is going to restrict to publicly visible documents",
"query" : {
"term": {
"visibility": "public"
}
}
}
}
],
"response_processors": [
{
"rename_field": {
"field": "message",
"target_field": "notification"
}
}
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
]
}
```
{% include copy-curl.html %}
For more information about creating and updating a search pipeline, see [Creating a search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/creating-search-pipeline/).
To use a pipeline with a query, specify the pipeline name in the `search_pipeline` query parameter:
```json
GET /my_index/_search?search_pipeline=my_pipeline
```
{% include copy-curl.html %}
Alternatively, you can use a temporary pipeline with a request or set a default pipeline for an index. To learn more, see [Using a search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/using-search-pipeline/).
To learn about retrieving details for an existing search pipeline, see [Retrieving search pipelines]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/retrieving-search-pipeline/).
Add search pipeline documentation (#4204) * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add search pipeline documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add experimental label Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Update script processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add wildcard and pipeline path parameter Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reworded get pipeline responses Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add we do not recommend to use the feature in prod Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Add header to table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented tech review and editorial comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-06-06 11:11:52 -04:00
## Search pipeline metrics
For information about retrieving search pipeline statistics, see [Search pipeline metrics]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-pipeline-metrics/).