3 Commits

Author	SHA1	Message	Date
Rafael dos Santos Silva	e457c687ca	FIX: OpenAI Tokenizer was failing to truncate mid emojis (#91) ... * FIX: OpenAI Tokenizer was failing to truncate mid emojis * Update spec/shared/tokenizer.rb Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com> --------- Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>	2023-06-16 15:15:36 -03:00
Rafael dos Santos Silva	3c9513e754	Refinements to embeddings and tokenizers (#61) ... * Refinements to embeddings and tokenizers * lint * Truncate with tokenizers for summary * fix	2023-05-15 15:10:42 -03:00
Rafael dos Santos Silva	9783e3b025	FEATURE: Add a basic tokenizer API (#37) ... * FEATURE: Add a basic tokenizer API * Add tests * lint	2023-04-19 11:55:59 -03:00