FIX: Disable truncation and padding in all-mpnet-base-v2 tokenizer (#105)

The tokenizer was truncating and padding to 128 tokens, and we try append
new post content until we hit 384 tokens. This was causing the tokenizer
to accept all posts in a topic, wasting CPU and memory.
This commit is contained in:
Rafael dos Santos Silva 2023-07-13 21:09:46 -03:00 committed by GitHub
parent 703762a7a9
commit d692ecc7de
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions

File diff suppressed because one or more lines are too long