Commit History
fix(tokenizer): Sets fim and think tokens to non-special, and removes unk_token. 3670d21 verified
fix(tokenizer_config): Sets fim and think tokens to non-special, and removes unk_token. f1527a3 verified
fix(special_tokens_map): Removes unk_token. f580375 verified
chore(root): Adds top_k information even if 50 is already the default. 0f18025
Gustavo de Rosa commited on