eb93b21769
BAAI/bge-m3 is an interesting model, that is multilingual and with a context size of 8192. Even with a 16x larger context, it's only 4x slower to compute it's embeddings on the worst case scenario. Also includes a minor refactor of the rake task, including setting model and concurrency levels when running the backfill task. |
||
|---|---|---|
| .. | ||
| all_mpnet_base_v2_tokenizer.rb | ||
| anthropic_tokenizer.rb | ||
| basic_tokenizer.rb | ||
| bert_tokenizer.rb | ||
| bge_large_en_tokenizer.rb | ||
| bge_m3_tokenizer.rb | ||
| llama2_tokenizer.rb | ||
| mixtral_tokenizer.rb | ||
| multilingual_e5_large_tokenizer.rb | ||
| open_ai_tokenizer.rb | ||