discourse/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-07-12 17:13:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

FEATURE: Add old OpenAI tokenizer to embeddings (#1487)

Browse Source
This commit is contained in:
Rafael dos Santos Silva 2025-07-07 15:07:27 -03:00 committed by GitHub
parent 6f8960e549
commit 7357280e88
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
app/models/embedding_definition.rb
View File

@ -23,6 +23,7 @@ class EmbeddingDefinition < ActiveRecord::Base
DiscourseAi::Tokenizer::GeminiTokenizer,
DiscourseAi::Tokenizer::MultilingualE5LargeTokenizer,
DiscourseAi::Tokenizer::OpenAiTokenizer,
DiscourseAi::Tokenizer::OpenAiCl100kTokenizer,
DiscourseAi::Tokenizer::MistralTokenizer,
DiscourseAi::Tokenizer::QwenTokenizer,
].map(&:name)

2
plugin.rb
View File

@ -10,7 +10,7 @@
gem "tokenizers", "0.5.4"
gem "tiktoken_ruby", "0.0.11.1"
gem "discourse_ai-tokenizers", "0.2.0", require_name: "discourse_ai/tokenizers"
gem "discourse_ai-tokenizers", "0.3.1", require_name: "discourse_ai/tokenizers"
gem "ed25519", "1.2.4" #TODO remove this as existing ssl gem should handle this
# we probably want to move all dependencies directly in to the Discourse Gemfile, this