Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
isaacus
/
kanon-2-tokenizer
like
2
Follow
Isaacus
37
English
legal
License:
apache-2.0
Model card
Files
Files and versions
xet
Community
main
kanon-2-tokenizer
6.16 MB
1 contributor
History:
7 commits
umarbutler
fix: drop `special_tokens_map.json` to fix transformers v5
c53b842
verified
9 days ago
.gitattributes
Safe
1.52 kB
initial commit
8 months ago
README.md
Safe
190 Bytes
docs: rewording
4 months ago
merges.txt
Safe
495 kB
Added Kanon 2's tokenizer.
8 months ago
tiktoken_tokenizer.json
Safe
980 kB
Added Kanon 2's tokenizer.
8 months ago
tokenizer.json
Safe
3.64 MB
feat: increase context window
5 months ago
tokenizer_config.json
Safe
1.34 kB
feat: increase context window
4 months ago
vocab.json
Safe
1.04 MB
Added Kanon 2's tokenizer.
8 months ago