discourse/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-03-07 01:39:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
discourse-ai/lib/shared/tokenizer
History
Rafael dos Santos Silva e457c687ca
FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91) 
* FIX: OpenAI Tokenizer was failing to truncate mid emojis

* Update spec/shared/tokenizer.rb

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>

---------

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2023-06-16 15:15:36 -03:00
..
tokenizer.rb
FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91)
2023-06-16 15:15:36 -03:00