Rafael dos Santos Silva 3b8f900486
FIX: Handle unicode on tokenizer (#515)
* FIX: Handle unicode on tokenizer

Our fast track code broke when strings had characters who are longer in tokens than
in UTF-8.

Admins can set `DISCOURSE_AI_STRICT_TOKEN_COUNTING: true` in app.yml to ensure token counting is strict, even if slower.


Co-authored-by: wozulong <sidle.pax_0e@icloud.com>
2024-03-14 17:33:30 -03:00
2023-02-17 11:33:47 -03:00
2024-03-13 13:16:07 -04:00
2023-12-26 14:49:55 -03:00
2023-02-17 11:33:47 -03:00
2023-11-03 11:30:09 +00:00
2023-11-03 11:30:09 +00:00
2024-03-06 15:23:29 +01:00
2023-02-17 11:33:47 -03:00
2024-01-13 00:28:06 +01:00
2023-09-04 15:46:35 -03:00
2023-07-15 00:56:15 +02:00
2024-01-13 00:28:06 +01:00

Discourse AI Plugin

Plugin Summary

For more information, please see: https://meta.discourse.org/t/discourse-ai/259214?u=falco

Languages
Ruby 81.3%
JavaScript 15.5%
SCSS 2.2%
CSS 0.4%
HTML 0.4%
Other 0.2%