e76fc77189
* Minor... use username suggester in case username already exists * FIX: ensure we truncate long prompts Previously we 1. Used raw length instead of token counts for counting length 2. We totally dropped a prompt if it was too long New implementation will truncate "raw" if it gets too long maintaining meaning. |
||
|---|---|---|
| .. | ||
| tokenizer.rb | ||