Rafael dos Santos Silva d792919ddf
DEV: Move tokenizers to a gem (#1481)
Also renames the Mixtral tokenizer to Mistral.

See gem at github.com/discourse/discourse_ai-tokenizers


Co-authored-by: Roman Rizzi <roman@discourse.org>
2025-07-02 14:43:03 -03:00
..