2 Commits

Author	SHA1	Message	Date
Rafael dos Santos Silva	3c9513e754	Refinements to embeddings and tokenizers (#61) ... * Refinements to embeddings and tokenizers * lint * Truncate with tokenizers for summary * fix	2023-05-15 15:10:42 -03:00
Rafael dos Santos Silva	9783e3b025	FEATURE: Add a basic tokenizer API (#37) ... * FEATURE: Add a basic tokenizer API * Add tests * lint	2023-04-19 11:55:59 -03:00