discourse-ai/lib/tokenizer/bge_m3_tokenizer.rb

# frozen_string_literal: true

module DiscourseAi
  module Tokenizer
    class BgeM3Tokenizer < BasicTokenizer
      def self.tokenizer
        @@tokenizer ||= Tokenizers.from_file("./plugins/discourse-ai/tokenizers/bge-m3.json")
      end
    end
  end
end
FEATURE: Add BGE-M3 embeddings support (#569) BAAI/bge-m3 is an interesting model, that is multilingual and with a context size of 8192. Even with a 16x larger context, it's only 4x slower to compute it's embeddings on the worst case scenario. Also includes a minor refactor of the rake task, including setting model and concurrency levels when running the backfill task. 2024-04-10 17:24:01 -03:00			`# frozen_string_literal: true`

			`module DiscourseAi`
			`module Tokenizer`
			`class BgeM3Tokenizer < BasicTokenizer`
			`def self.tokenizer`
			`@@tokenizer \|\|= Tokenizers.from_file("./plugins/discourse-ai/tokenizers/bge-m3.json")`
			`end`
			`end`
			`end`
			`end`