Discource-C
/
discourse-ai
mirror of https://github.com/discourse/discourse-ai.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
548 Commits 26 Branches 0 Tags 23 MiB
Commit Graph

5 Commits

Author SHA1 Message Date
Rafael dos Santos Silva 3b8f900486
FIX: Handle unicode on tokenizer (#515) 
* FIX: Handle unicode on tokenizer

Our fast track code broke when strings had characters who are longer in tokens than
in UTF-8.

Admins can set `DISCOURSE_AI_STRICT_TOKEN_COUNTING: true` in app.yml to ensure token counting is strict, even if slower.


Co-authored-by: wozulong <sidle.pax_0e@icloud.com>
 2024-03-14 17:33:30 -03:00
Rafael dos Santos Silva 84cc369552
FEATURE: Bge-large-en embeddings via Cloudflare Workers AI API (#241) 
* FEATURE: Bge-large-en embeddings via Cloudflare Workers AI API

* forgot a file

* lint
 2023-10-04 13:47:51 -03:00
Rafael dos Santos Silva 3e7c99de89
FEATURE: Support for locally infered embeddings in 100 languages (#115) 
* FEATURE: Support for locally infered embeddings in 100 languages

* add table
 2023-07-27 15:50:03 -03:00
Rafael dos Santos Silva b25daed60b
FEATURE: Llama2 for summarization (#116) 2023-07-27 13:55:32 -03:00
Rafael dos Santos Silva b82074850e
DEV: Add tests to allmpnet tokenizer (#107) 
* DEV: Add tests to allmpnet tokenizer

* lint
 2023-07-14 11:37:21 -03:00