FIX: Disable truncation and padding in all-mpnet-base-v2 tokenizer (#105)

Browse Source

The tokenizer was truncating and padding to 128 tokens, and we try append
new post content until we hit 384 tokens. This was causing the tokenizer
to accept all posts in a topic, wasting CPU and memory.

...

This commit is contained in:

Rafael dos Santos Silva 2023-07-13 21:09:46 -03:00 committed by GitHub

parent 703762a7a9

commit d692ecc7de

No known key found for this signature in database

GPG Key ID: 4AEE18F83AFDEB23

1 changed files with 1 additions and 1 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2

tokenizers/all-mpnet-base-v2.json

View File

File diff suppressed because one or more lines are too long