discourse/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-06-25 17:12:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

FIX: improve token counting (#234)

Browse Source 
We were running out of tokens under certain conditions (really long
chains)

Add more buffer.
This commit is contained in:
Sam 2023-09-28 15:32:22 +10:00 committed by GitHub
parent 237e9478df
commit ed7d1f06d1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
config/settings.yml
View File

@ -243,6 +243,7 @@ discourse_ai:
- sql_helper - sql_helper
- settings_explorer - settings_explorer
- researcher - researcher
- creative
ai_bot_add_to_header: ai_bot_add_to_header:
default: true default: true
client: true client: true

6
lib/modules/ai_bot/open_ai_bot.rb
View File

@ -16,9 +16,9 @@ module DiscourseAi
# note this is about 100 tokens over, OpenAI have a more optimal representation # note this is about 100 tokens over, OpenAI have a more optimal representation
@function_size ||= tokenize(available_functions.to_json).length @function_size ||= tokenize(available_functions.to_json).length
# provide a buffer of 80 tokens - our function counting is not # provide a buffer of 120 tokens - our function counting is not
# 100% accurate so this is a trial and error number # 100% accurate and getting numbers to align exactly is very hard
buffer = @function_size + reply_params[:max_tokens] + 80 buffer = @function_size + reply_params[:max_tokens] + 120
if bot_user.id == DiscourseAi::AiBot::EntryPoint::GPT4_ID if bot_user.id == DiscourseAi::AiBot::EntryPoint::GPT4_ID
8192 - buffer 8192 - buffer