8 Commits

Author SHA1 Message Date
Rafael dos Santos Silva
9dccc1eb93
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440) 2025-06-17 10:25:03 -03:00
Roman Rizzi
df63e36ad8
FEATURE: Make Mixtral tokenizer available for embeddings (#1258) 2025-04-11 12:01:38 -03:00
Kris
99e73f09ff
UX: improve embeddings config styles (#1085)
* WIP: improve embeddings config styles

* switch to textarea, fix back button

* remove log, update button, fix tests

* stree

* fix spec

* spec fix

* remove comment
2025-01-24 16:24:59 +11:00
Rafael dos Santos Silva
67a1257b89
FEATURE: Gemini Tokenizer (#1088) 2025-01-23 18:20:35 -03:00
Roman Rizzi
e2e753d73c
FEATURE: Formalize support for matryoshka dimensions. (#1083)
We have a flag to signal we are shortening the embeddings of a model.
Only used in Open AI's text-embedding-3-*, but we plan to use it for other services.
2025-01-22 11:26:46 -03:00
Roman Rizzi
a5e5ae72a8
FIX: Open AI embedding shortening is only available for some models (#1080) 2025-01-21 17:50:40 -03:00
Roman Rizzi
3b66fb3e87
FIX: Restore the accidentally deleted query prefix. (#1079)
Additionally, we add a prefix for embedding generation.
Both are stored in the definitions table.
2025-01-21 14:10:31 -03:00
Roman Rizzi
f5cf1019fb
FEATURE: configurable embeddings (#1049)
* Use AR model for embeddings features

* endpoints

* Embeddings CRUD UI

* Add presets. Hide a couple more settings

* system specs

* Seed embedding definition from old settings

* Generate search bit index on the fly. cleanup orphaned data

* support for seeded models

* Fix run test for new embedding

* fix selected model not set correctly
2025-01-21 12:23:19 -03:00