opensearch-docs-cn/_query-dsl/specialized/neural-sparse.md

53 lines
2.1 KiB
Markdown
Raw Normal View History

Add multimodal search/sparse search/pre- and post-processing function documentation (#5168) * Add multimodal search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Text image embedding processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add prerequisite Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change query text Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added bedrock connector tutorial and renamed ML TOC Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Name changes and rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change connector link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Link fix and field name fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add default text embedding preprocessing and post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add sparse search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix links Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Pre/post processing function tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Sparse search tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented doc review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add actual test sparse pipeline response Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added tested examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added model choice for sparse search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove Bedrock connector Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add that the model must be deployed to neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Link fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add session token to sagemaker blueprint Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Formatted bullet points the same way Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Specified both model types in neural sparse query Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added more explanation for default pre/post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove framework and extensibility references Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Minor rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-16 10:45:35 -04:00
---
layout: default
title: Neural sparse
parent: Specialized queries
grand_parent: Query DSL
nav_order: 55
---
# Neural sparse query
Introduced 2.11
{: .label .label-purple }
Use the `neural_sparse` query for vector field search in [neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/).
Add multimodal search/sparse search/pre- and post-processing function documentation (#5168) * Add multimodal search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Text image embedding processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add prerequisite Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change query text Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added bedrock connector tutorial and renamed ML TOC Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Name changes and rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change connector link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Link fix and field name fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add default text embedding preprocessing and post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add sparse search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix links Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Pre/post processing function tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Sparse search tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented doc review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add actual test sparse pipeline response Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added tested examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added model choice for sparse search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove Bedrock connector Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add that the model must be deployed to neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Link fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add session token to sagemaker blueprint Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Formatted bullet points the same way Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Specified both model types in neural sparse query Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added more explanation for default pre/post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove framework and extensibility references Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Minor rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-16 10:45:35 -04:00
## Request fields
Include the following request fields in the `neural_sparse` query:
```json
"neural_sparse": {
"<vector_field>": {
"query_text": "<query_text>",
"model_id": "<model_id>",
"max_token_score": "<max_token_score>"
}
}
```
The top-level `vector_field` specifies the vector field against which to run a search query. The following table lists the other `neural_sparse` query fields.
Field | Data type | Required/Optional | Description
:--- | :--- | :---
`query_text` | String | Required | The query text from which to generate vector embeddings.
`model_id` | String | Required | The ID of the sparse encoding model or tokenizer model that will be used to generate vector embeddings from the query text. The model must be deployed in OpenSearch before it can be used in sparse neural search. For more information, see [Using custom models within OpenSearch]({{site.url}}{{site.baseurl}}/ml-commons-plugin/using-ml-models/) and [Neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/).
add pretrained model description and saving disk description (#5383) * add description and saving disk Signed-off-by: xinyual <xinyual@amazon.com> * update Signed-off-by: xinyual <xinyual@amazon.com> * update Signed-off-by: xinyual <xinyual@amazon.com> * Update _ml-commons-plugin/pretrained-models.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> * Update _search-plugins/neural-sparse-search.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> * Moved recommended model choice options outside the table Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add a link for more info Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> Signed-off-by: xinyual <74362153+xinyual@users.noreply.github.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-25 11:59:49 -04:00
`max_token_score` | Float | Optional | The theoretical upper bound of the score for all tokens in the vocabulary (required for performance optimization). For OpenSearch-provided [pretrained sparse embedding models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sparse-encoding-models), we recommend setting `max_token_score` to 2 for `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` and to 3.5 for `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1`.
Add multimodal search/sparse search/pre- and post-processing function documentation (#5168) * Add multimodal search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Text image embedding processor Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add prerequisite Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change query text Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added bedrock connector tutorial and renamed ML TOC Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Name changes and rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change connector link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Change link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Link fix and field name fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add default text embedding preprocessing and post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add sparse search documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix links Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Pre/post processing function tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix link Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Sparse search tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Implemented doc review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add actual test sparse pipeline response Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added tested examples Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added model choice for sparse search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove Bedrock connector Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add that the model must be deployed to neural search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Link fix Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Add session token to sagemaker blueprint Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Formatted bullet points the same way Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Specified both model types in neural sparse query Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Added more explanation for default pre/post-processing functions Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Remove framework and extensibility references Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Minor rewording Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
2023-10-16 10:45:35 -04:00
#### Example request
```json
GET my-nlp-index/_search
{
"query": {
"neural_sparse": {
"passage_embedding": {
"query_text": "Hi world",
"model_id": "aP2Q8ooBpBj3wT4HVS8a",
"max_token_score": 2
}
}
}
}
```
{% include copy-curl.html %}