mirror of
https://github.com/discourse/discourse-ai.git
synced 2025-08-05 20:53:28 +00:00
2c0f535bab
* FEATURE: HyDE-powered semantic search. It relies on the new outlet added on discourse/discourse#23390 to display semantic search results in an unobtrusive way. We'll use a HyDE-backed approach for semantic search, which consists on generating an hypothetical document from a given keywords, which gets transformed into a vector and used in a asymmetric similarity topic search. This PR also reorganizes the internals to have less moving parts, maintaining one hierarchy of DAOish classes for vector-related operations like transformations and querying. Completions and vectors created by HyDE will remain cached on Redis for now, but we could later use Postgres instead. * Missing translation and rate limiting --------- Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
51 lines
971 B
Ruby
51 lines
971 B
Ruby
# frozen_string_literal: true
|
|
|
|
module DiscourseAi
|
|
module Embeddings
|
|
module VectorRepresentations
|
|
class MultilingualE5Large < Base
|
|
def vector_from(text)
|
|
DiscourseAi::Inference::DiscourseClassifier.perform!(
|
|
"#{SiteSetting.ai_embeddings_discourse_service_api_endpoint}/api/v1/classify",
|
|
name,
|
|
"query: #{text}",
|
|
SiteSetting.ai_embeddings_discourse_service_api_key,
|
|
)
|
|
end
|
|
|
|
def id
|
|
3
|
|
end
|
|
|
|
def version
|
|
1
|
|
end
|
|
|
|
def name
|
|
"multilingual-e5-large"
|
|
end
|
|
|
|
def dimensions
|
|
1024
|
|
end
|
|
|
|
def max_sequence_length
|
|
512
|
|
end
|
|
|
|
def pg_function
|
|
"<=>"
|
|
end
|
|
|
|
def pg_index_type
|
|
"vector_cosine_ops"
|
|
end
|
|
|
|
def tokenizer
|
|
DiscourseAi::Tokenizer::MultilingualE5LargeTokenizer
|
|
end
|
|
end
|
|
end
|
|
end
|
|
end
|