discourse/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-07-02 20:42:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,426 Commits 41 Branches 0 Tags
Commit Graph

2 Commits

Author SHA1 Message Date
Rafael dos Santos Silva
d792919ddf
DEV: Move tokenizers to a gem (#1481) 
Also renames the Mixtral tokenizer to Mistral.

See gem at github.com/discourse/discourse_ai-tokenizers


Co-authored-by: Roman Rizzi <roman@discourse.org>
 2025-07-02 14:43:03 -03:00
Rafael dos Santos Silva
3c9513e754
Refinements to embeddings and tokenizers (#61) 
* Refinements to embeddings and tokenizers

* lint

* Truncate with tokenizers for summary

* fix
 2023-05-15 15:10:42 -03:00