discourse/discourse-ai
1
0
Fork 0
mirror of https://github.com/discourse/discourse-ai.git synced 2025-06-29 02:52:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
discourse-ai/tokenizers
History
Rafael dos Santos Silva 9dccc1eb93
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)
2025-06-17 10:25:03 -03:00
..
all-mpnet-base-v2.json
FIX: Disable truncation and padding in all-mpnet-base-v2 tokenizer (#105)
2023-07-13 21:09:46 -03:00
Apache License
Refinements to embeddings and tokenizers (#61)
2023-05-15 15:10:42 -03:00
bert-base-uncased.json
FEATURE: Add a basic tokenizer API (#37)
2023-04-19 11:55:59 -03:00
bge-large-en.json
FEATURE: Bge-large-en embeddings via Cloudflare Workers AI API (#241)
2023-10-04 13:47:51 -03:00
bge-m3.json
FEATURE: Add BGE-M3 embeddings support (#569)
2024-04-10 17:24:01 -03:00
claude-v1-tokenization.json
Refinements to embeddings and tokenizers (#61)
2023-05-15 15:10:42 -03:00
gemma3.json
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)
2025-06-17 10:25:03 -03:00
META_LLAMA_3_COMMUNITY_LICENSE.md
FEATURE: Llama 3 tokenizer (#615)
2024-05-13 12:45:52 -03:00
Meta-Llama-3-70B-Instruct.json
FEATURE: Llama 3 tokenizer (#615)
2024-05-13 12:45:52 -03:00
MIT License
Refinements to embeddings and tokenizers (#61)
2023-05-15 15:10:42 -03:00
mixtral.json
Mixtral (#376)
2023-12-26 14:49:55 -03:00
multilingual-e5-large.json
FEATURE: Support for locally infered embeddings in 100 languages (#115)
2023-07-27 15:50:03 -03:00
qwen3.json
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)
2025-06-17 10:25:03 -03:00
README.md
FEATURE: Add Qwen3 tokenizer and update Gemma to version 3 (#1440)
2025-06-17 10:25:03 -03:00

README.md

bert-base-uncased.json

Licensed under Apache License

claude-v1-tokenization.json

Licensed under MIT License

all-mpnet-base-v2.json

Licensed under Apache License

llama-2-70b-chat-hf

Licensed under LLAMA 2 COMMUNITY LICENSE AGREEMENT

multilingual-e5-large

Licensed under MIT License

bge-large-en

Licensed under MIT License

mixtral

Licensed under Apache 2.0 License

bge-m3

Licensed under MIT License

Meta-Llama-3-70B-Instruct

Licensed under META LLAMA 3 COMMUNITY LICENSE

Gemma 3

Licensed under the Gemma Terms of Use

Qwen 3

Licensed under the Apache 2.0 License