Discource-C
/
discourse-ai
mirror of https://github.com/discourse/discourse-ai.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

FIX: adjust token limits to account for functions (#96) 

Reduce maximum replies to 2500 tokens and make them even for both GPT-3.5
and 4

Account for 400+ tokens in function definitions (this was unaccounted for)
This commit is contained in:
Sam 2023-06-23 10:02:04 +10:00 committed by GitHub
parent f8cabfad6b
commit 9390fba768
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 10 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
lib/modules/ai_bot/open_ai_bot.rb
View File

@ -15,22 +15,21 @@ module DiscourseAi
def prompt_limit def prompt_limit
# note GPT counts both reply and request tokens in limits... # note GPT counts both reply and request tokens in limits...
# also allow for an extra 500 or so spare tokens # also allow for an extra 500 or so spare tokens
#
# 2500 are the max reply tokens
# Then we have 400 or so for the full function suite
# 100 additional for growth around function calls
if bot_user.id == DiscourseAi::AiBot::EntryPoint::GPT4_ID if bot_user.id == DiscourseAi::AiBot::EntryPoint::GPT4_ID
8192 - 3500 8192 - 3000
else else
16_384 - 2000 16_384 - 3000
end end
end end
def reply_params def reply_params
max_tokens = # technically we could allow GPT-3.5 16k more tokens
if bot_user.id == DiscourseAi::AiBot::EntryPoint::GPT4_ID # but lets just keep it here for now
3000 { temperature: 0.4, top_p: 0.9, max_tokens: 2500 }
else
1500
end
{ temperature: 0.4, top_p: 0.9, max_tokens: max_tokens }
end end
def submit_prompt( def submit_prompt(

2
spec/lib/modules/ai_bot/jobs/regular/create_ai_reply_spec.rb
View File

@ -36,7 +36,7 @@ RSpec.describe Jobs::CreateAiReply do
req_opts: { req_opts: {
temperature: 0.4, temperature: 0.4,
top_p: 0.9, top_p: 0.9,
max_tokens: 1500, max_tokens: 2500,
functions: bot.available_functions, functions: bot.available_functions,
stream: true, stream: true,
}, },