208 lines
6.7 KiB
Markdown
208 lines
6.7 KiB
Markdown
---
|
|
layout: default
|
|
title: Sparse search
|
|
nav_order: 30
|
|
has_children: false
|
|
parent: Neural search
|
|
---
|
|
|
|
# Sparse search
|
|
Introduced 2.11
|
|
{: .label .label-purple }
|
|
|
|
[Neural text search]({{site.url}}{{site.baseurl}}/search-plugins/neural-text-search/) relies on dense retrieval that is based on text embedding models. However, dense methods use k-NN search, which consumes a large amount of memory and CPU resources. An alternative to neural text search, sparse neural search is implemented using an inverted index and thus is as efficient as BM25. Sparse search is facilitated by sparse embedding models. When you perform a sparse search, it creates a sparse vector (a list of `token: weight` key-value pairs representing an entry and its weight) and ingests data into a rank features index.
|
|
|
|
When selecting a model, choose one of the following options:
|
|
|
|
- Use a sparse encoding model at both ingestion time and search time (high performance, relatively high latency).
|
|
- Use a sparse encoding model at ingestion time and a tokenizer model at search time (low performance, relatively low latency).
|
|
|
|
**PREREQUISITE**<br>
|
|
Before using sparse search, you must set up a sparse embedding model. For more information, see [Using custom models within OpenSearch]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).
|
|
{: .note}
|
|
|
|
## Using sparse search
|
|
|
|
To use sparse search, follow these steps:
|
|
|
|
1. [Create an ingest pipeline](#step-1-create-an-ingest-pipeline).
|
|
1. [Create an index for ingestion](#step-2-create-an-index-for-ingestion).
|
|
1. [Ingest documents into the index](#step-3-ingest-documents-into-the-index).
|
|
1. [Search the index using neural search](#step-4-search-the-index-using-neural-search).
|
|
|
|
## Step 1: Create an ingest pipeline
|
|
|
|
To generate vector embeddings, you need to create an [ingest pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/index/) that contains a [`sparse_encoding` processor]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/processors/sparse-encoding/), which will convert the text in a document field to vector embeddings. The processor's `field_map` determines the input fields from which to generate vector embeddings and the output fields in which to store the embeddings.
|
|
|
|
The following example request creates an ingest pipeline where the text from `passage_text` will be converted into text embeddings and the embeddings will be stored in `passage_embedding`:
|
|
|
|
```json
|
|
PUT /_ingest/pipeline/nlp-ingest-pipeline-sparse
|
|
{
|
|
"description": "An sparse encoding ingest pipeline",
|
|
"processors": [
|
|
{
|
|
"sparse_encoding": {
|
|
"model_id": "aP2Q8ooBpBj3wT4HVS8a",
|
|
"field_map": {
|
|
"passage_text": "passage_embedding"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
{% include copy-curl.html %}
|
|
|
|
## Step 2: Create an index for ingestion
|
|
|
|
In order to use the text embedding processor defined in your pipeline, create a rank features index, adding the pipeline created in the previous step as the default pipeline. Ensure that the fields defined in the `field_map` are mapped as correct types. Continuing with the example, the `passage_embedding` field must be mapped as [`rank_features`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/rank/#rank-features). Similarly, the `passage_text` field should be mapped as `text`.
|
|
|
|
The following example request creates a rank features index that is set up with a default ingest pipeline:
|
|
|
|
```json
|
|
PUT /my-nlp-index
|
|
{
|
|
"settings": {
|
|
"default_pipeline": "nlp-ingest-pipeline-sparse"
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"id": {
|
|
"type": "text"
|
|
},
|
|
"passage_embedding": {
|
|
"type": "rank_features"
|
|
},
|
|
"passage_text": {
|
|
"type": "text"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
{% include copy-curl.html %}
|
|
|
|
|
|
## Step 3: Ingest documents into the index
|
|
|
|
To ingest documents into the index created in the previous step, send the following requests:
|
|
|
|
```json
|
|
PUT /my-nlp-index/_doc/1
|
|
{
|
|
"passage_text": "Hello world",
|
|
"id": "s1"
|
|
}
|
|
```
|
|
{% include copy-curl.html %}
|
|
|
|
```json
|
|
PUT /my-nlp-index/_doc/2
|
|
{
|
|
"passage_text": "Hi planet",
|
|
"id": "s2"
|
|
}
|
|
```
|
|
{% include copy-curl.html %}
|
|
|
|
Before the document is ingested into the index, the ingest pipeline runs the `sparse_encoding` processor on the document, generating vector embeddings for the `passage_text` field. The indexed document includes the `passage_text` field, which contains the original text, and the `passage_embedding` field, which contains the vector embeddings.
|
|
|
|
## Step 4: Search the index using neural search
|
|
|
|
To perform a sparse vector search on your index, use the `neural_sparse` query clause in [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index/) queries.
|
|
|
|
The following example request uses a `neural_sparse` query to search for relevant documents:
|
|
|
|
```json
|
|
GET my-nlp-index/_search
|
|
{
|
|
"query": {
|
|
"neural_sparse": {
|
|
"passage_embedding": {
|
|
"query_text": "Hi world",
|
|
"model_id": "aP2Q8ooBpBj3wT4HVS8a",
|
|
"max_token_score": 2
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
{% include copy-curl.html %}
|
|
|
|
The response contains the matching documents:
|
|
|
|
```json
|
|
{
|
|
"took" : 688,
|
|
"timed_out" : false,
|
|
"_shards" : {
|
|
"total" : 1,
|
|
"successful" : 1,
|
|
"skipped" : 0,
|
|
"failed" : 0
|
|
},
|
|
"hits" : {
|
|
"total" : {
|
|
"value" : 2,
|
|
"relation" : "eq"
|
|
},
|
|
"max_score" : 30.0029,
|
|
"hits" : [
|
|
{
|
|
"_index" : "my-nlp-index",
|
|
"_id" : "1",
|
|
"_score" : 30.0029,
|
|
"_source" : {
|
|
"passage_text" : "Hello world",
|
|
"passage_embedding" : {
|
|
"!" : 0.8708904,
|
|
"door" : 0.8587369,
|
|
"hi" : 2.3929274,
|
|
"worlds" : 2.7839446,
|
|
"yes" : 0.75845814,
|
|
"##world" : 2.5432441,
|
|
"born" : 0.2682308,
|
|
"nothing" : 0.8625516,
|
|
"goodbye" : 0.17146169,
|
|
"greeting" : 0.96817183,
|
|
"birth" : 1.2788506,
|
|
"come" : 0.1623208,
|
|
"global" : 0.4371151,
|
|
"it" : 0.42951578,
|
|
"life" : 1.5750692,
|
|
"thanks" : 0.26481047,
|
|
"world" : 4.7300377,
|
|
"tiny" : 0.5462298,
|
|
"earth" : 2.6555297,
|
|
"universe" : 2.0308156,
|
|
"worldwide" : 1.3903781,
|
|
"hello" : 6.696973,
|
|
"so" : 0.20279501,
|
|
"?" : 0.67785245
|
|
},
|
|
"id" : "s1"
|
|
}
|
|
},
|
|
{
|
|
"_index" : "my-nlp-index",
|
|
"_id" : "2",
|
|
"_score" : 16.480486,
|
|
"_source" : {
|
|
"passage_text" : "Hi planet",
|
|
"passage_embedding" : {
|
|
"hi" : 4.338913,
|
|
"planets" : 2.7755864,
|
|
"planet" : 5.0969057,
|
|
"mars" : 1.7405145,
|
|
"earth" : 2.6087382,
|
|
"hello" : 3.3210192
|
|
},
|
|
"id" : "s2"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|