Discource-C/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-02-18 01:14:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
discourse-ai/lib/tokenizer
History
Rafael dos Santos Silva 5db7bf6e68
Mixtral (#376) 
Add both Mistral and Mixtral support. Also includes vLLM-openAI inference support.

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2023-12-26 14:49:55 -03:00
..
all_mpnet_base_v2_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
anthropic_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
basic_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
bert_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
bge_large_en_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
llama2_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
mixtral_tokenizer.rb
Mixtral (#376)
2023-12-26 14:49:55 -03:00
multilingual_e5_large_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00
open_ai_tokenizer.rb
DEV: port directory structure to Zeitwerk (#319)
2023-11-29 15:17:46 +11:00