Rafael dos Santos Silva eb93b21769

FEATURE: Add BGE-M3 embeddings support (#569) ...

BAAI/bge-m3 is an interesting model, that is multilingual and with a
context size of 8192. Even with a 16x larger context, it's only 4x slower
to compute it's embeddings on the worst case scenario.

Also includes a minor refactor of the rake task, including setting model
and concurrency levels when running the backfill task.

2024-04-10 17:24:01 -03:00

16 MiB

Raw Blame History

The file is too large to be shown. View Raw