README: add Gemma-4, Qwen3.6, Kimi-K2.6 benchmarks + head-to-head vs flagships

README.md

@@ -20,18 +20,21 @@ A bilingual Arabic/English tokenizer at 65k vocab built on HuggingFace's Unigram
 
 ## The headline
 
-**v0.2 is the first bilingual tokenizer tested to beat GPT-5's ,GPT-4o's `o200k_base` on both Arabic AND English chars-per-token, while running at 32.5% of its vocabulary size.**
+**v0.2 is the first bilingual tokenizer tested to beat GPT-5 (o200k_base), GPT-4o, Gemma-4, Qwen 3.6, and Kimi K2.6 on both Arabic AND English chars-per-token — at 25–32.5% of their vocabulary sizes.**
 
 | Tokenizer | Vocab | AR CpT | EN CpT | Parity |
 |---|--:|--:|--:|--:|
 | **SARFTokenizer v0.2** | **65,000** | **3.683** | **3.522** | **1.046** |
+| Qwen/Qwen3.6-35B-A3B | 248,077 | 3.129 | 2.985 | 1.048 |
 | tiktoken/o200k_base (GPT-4o, GPT-5) | 200,019 | 3.087 | 3.409 | 0.906 |
 | ALLaM-7B | 64,000 | 2.854 | 2.518 | 1.133 |
-| gemma-3-1b-pt | 262,145 | 2.833 | 3.069 | 0.923 |
+| google/gemma-4-31B-it | 262,144 | 2.833 | 3.069 | 0.923 |
+| google/gemma-3-1b-pt | 262,145 | 2.833 | 3.069 | 0.923 |
 | Qwen2.5-0.5B | 151,665 | 2.583 | 2.923 | 0.884 |
 | SARFTokenizer v0.1 (BPE) | 62,912 | 3.262 | 2.729 | 1.196 |
+| moonshotai/Kimi-K2.6 | 163,840 | 2.074 | 3.239 | 0.640 |
 
-Full comparison: [`BENCHMARK.md`](./BENCHMARK.md). Deep-dive writeup: see `docs/V6_REPORT.md` in the [GitHub repo](https://github.com/almaghrabima/smctm).
+Full comparison: [`BENCHMARK.md`](./BENCHMARK.md). Deep-dive writeup: `docs/V6_REPORT.md` in the [GitHub repo](https://github.com/almaghrabima/smctm).
 
 ---
 
@@ -122,7 +125,7 @@ batch = tok(
 
 ## Benchmark
 
-### Headline numbers (sorted by Arabic CpT)
+### Full table (sorted by Arabic CpT)
 
 Protocol: 300 AR + 300 EN documents from `deeplatent-hq-bilingual` validation shards, 2000 chars each. `add_special_tokens=False`. Each tokenizer's own normalizer/pre-tokenizer runs (no external preprocessing). Metrics:
 - **AR CpT** = Arabic chars / tokens (higher = more compression)
@@ -134,15 +137,30 @@ Protocol: 300 AR + 300 EN documents from `deeplatent-hq-bilingual` validation sh
 |---:|---|--:|--:|--:|--:|--:|--:|
 | 1 | **SARFTokenizer v0.2** (Unigram) | **65,000** | **3.683** | **3.522** | **1.046** | **1.57** | **2.01** |
 | 2 | SARFTokenizer v0.1 (MYTE + BPE) | 62,912 | 3.262 | 2.729 | 1.196 | 1.77 | 2.59 |
-| 3 | tiktoken/o200k_base (GPT-4o, GPT-5) | 200,019 | 3.087 | 3.409 | 0.906 | 1.87 | 2.07 |
-| 4 | ALLaM-AI/ALLaM-7B-Instruct-preview | 64,000 | 2.854 | 2.518 | 1.133 | 2.03 | 2.81 |
-| 5 | google/gemma-3-1b-pt | 262,145 | 2.833 | 3.069 | 0.923 | 2.04 | 2.30 |
-| 6 | google/gemma-2-2b | 256,000 | 2.779 | 3.117 | 0.892 | 2.08 | 2.27 |
-| 7 | QCRI/Fanar-1-9B-Instruct | 128,256 | 2.778 | 3.047 | 0.911 | 2.08 | 2.32 |
-| 8 | Qwen/Qwen2.5-0.5B | 151,665 | 2.583 | 2.923 | 0.884 | 2.24 | 2.42 |
-| 9 | hammh0a/Hala-350M | 64,400 | 2.219 | 3.220 | 0.689 | 2.61 | 2.20 |
-| 10 | tiktoken/cl100k_base (GPT-4) | 100,277 | 1.429 | 3.066 | 0.466 | 4.05 | 2.31 |
-| 11 | tiiuae/falcon-7b | 65,024 | 0.991 | 2.720 | 0.364 | 5.84 | 2.60 |
+| 3 | Qwen/Qwen3.6-35B-A3B | 248,077 | 3.129 | 2.985 | 1.048 | 1.85 | 2.38 |
+| 4 | tiktoken/o200k_base (GPT-4o, GPT-5) | 200,019 | 3.087 | 3.409 | 0.906 | 1.87 | 2.07 |
+| 5 | ALLaM-AI/ALLaM-7B-Instruct-preview | 64,000 | 2.854 | 2.518 | 1.133 | 2.03 | 2.81 |
+| 6 | google/gemma-3-1b-pt | 262,145 | 2.833 | 3.069 | 0.923 | 2.04 | 2.30 |
+| 6t | google/gemma-4-31B-it | 262,144 | 2.833 | 3.069 | 0.923 | 2.04 | 2.30 |
+| 8 | google/gemma-2-2b | 256,000 | 2.779 | 3.117 | 0.892 | 2.08 | 2.27 |
+| 9 | QCRI/Fanar-1-9B-Instruct | 128,256 | 2.778 | 3.047 | 0.911 | 2.08 | 2.32 |
+| 10 | Qwen/Qwen2.5-0.5B | 151,665 | 2.583 | 2.923 | 0.884 | 2.24 | 2.42 |
+| 11 | hammh0a/Hala-350M | 64,400 | 2.219 | 3.220 | 0.689 | 2.61 | 2.20 |
+| 12 | moonshotai/Kimi-K2.6 | 163,840 | 2.074 | 3.239 | 0.640 | 2.79 | 2.20 |
+| 13 | tiktoken/cl100k_base (GPT-4) | 100,277 | 1.429 | 3.066 | 0.466 | 4.05 | 2.31 |
+| 14 | tiiuae/falcon-7b | 65,024 | 0.991 | 2.720 | 0.364 | 5.84 | 2.60 |
+
+### Head-to-head: v0.2 vs flagship 2025/2026 tokenizers
+
+| Competitor | Vocab | Their AR CpT | **Our AR CpT** | Δ AR | Their EN CpT | **Our EN CpT** | Δ EN |
+|---|--:|--:|--:|--:|--:|--:|--:|
+| **GPT-5 / GPT-4o** (o200k_base) | 200,019 | 3.087 | **3.683** | **+19.3%** | 3.409 | **3.522** | +3.3% |
+| **Gemma-4-31B-it** | 262,144 | 2.833 | **3.683** | **+30.0%** | 3.069 | **3.522** | +14.8% |
+| **Qwen3.6-35B-A3B** | 248,077 | 3.129 | **3.683** | **+17.7%** | 2.985 | **3.522** | +18.0% |
+| **Kimi-K2.6** | 163,840 | 2.074 | **3.683** | **+77.6%** | 3.239 | **3.522** | +8.7% |
+| **ALLaM-7B** | 64,000 | 2.854 | **3.683** | **+29.0%** | 2.518 | **3.522** | +39.9% |
+
+At 65k vocab — **25–32.5%** the vocabulary size of these flagships — SARFTokenizer v0.2 wins on both Arabic and English, simultaneously.
 
 ### v0.2 vs v0.1 (same vocab tier)
 
@@ -171,10 +189,10 @@ Clean-room test, fresh venv, 1M samples streamed from each of:
 - `HuggingFaceFW/fineweb-edu` config `sample-10BT` (English)
 
 | Stream | Samples | Chars | Tokens | CpT | T/W |
-|---|--:|--:|--:|--:|--:|--:|
+|---|--:|--:|--:|--:|--:|
 | AR (FineWeb-2) | 1,000,000 | 3,045,017,883 | 829,621,672 | **3.670** | 1.60 |
 | EN (FineWeb-Edu) | 1,000,000 | 4,685,926,211 | 1,041,078,482 | **4.501** | 1.37 |
-| **Total** | **2,000,000** | **7.73 GB** | **1.87 B** | | | |
+| **Total** | **2,000,000** | **7.73 GB** | **1.87 B** | | |
 
 **AR matches the internal benchmark within 0.4%** (3.683 → 3.670) — the tokenizer generalizes across corpora.
 **EN is 27.8% higher on FineWeb-Edu** (3.522 → 4.501) because FineWeb-Edu is a quality-filtered educational corpus (Wikipedia/textbook-heavy), structurally easier than mixed web text.
@@ -184,10 +202,10 @@ Raw per-stream results: `benchmark_results.json`.
 ### Run the benchmark yourself
 
 ```python
-import os
-os.environ["HF_TOKEN"] = userdata.get("HF-Token")
+from google.colab import userdata
+from huggingface_hub import login
+login(token=userdata.get("HF-Token"), add_to_git_credential=False)
 
-# Quick sanity check: count tokens on some text you own
 from transformers import AutoTokenizer
 tok = AutoTokenizer.from_pretrained("almaghrabima/SARFTokenizer")
 
@@ -197,7 +215,24 @@ print(f"AR: {len(ar)} chars -> {len(tok.encode(ar))} tokens = {len(ar)/len(tok.e
 print(f"EN: {len(en)} chars -> {len(tok.encode(en))} tokens = {len(en)/len(tok.encode(en)):.3f} CpT")
 ```
 
-Full reproducibility (all 11 tokenizer peers + E2E FineWeb run) via `scripts/bench_tokenizers.py` and `scripts/e2e_test_fineweb.py` on [GitHub](https://github.com/almaghrabima/smctm).
+### Head-to-head your own corpus vs any flagship tokenizer
+
+```python
+from transformers import AutoTokenizer
+
+sarf = AutoTokenizer.from_pretrained("almaghrabima/SARFTokenizer")
+gemma = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")
+qwen = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-35B-A3B")
+kimi = AutoTokenizer.from_pretrained("moonshotai/Kimi-K2.6", trust_remote_code=True)
+
+text = "your Arabic or English text here"
+for name, t in [("SARF v0.2", sarf), ("Gemma-4", gemma),
+                ("Qwen3.6", qwen), ("Kimi-K2.6", kimi)]:
+    ids = t.encode(text, add_special_tokens=False)
+    print(f"{name:<14} vocab={len(t):>7,} tokens={len(ids):>5} CpT={len(text)/len(ids):.3f}")
+```
+
+Full reproducibility (all 14 tokenizer peers + E2E FineWeb run) via `scripts/bench_tokenizers.py` and `scripts/e2e_test_fineweb.py` on [GitHub](https://github.com/almaghrabima/smctm).
 
 ---
 
@@ -279,7 +314,7 @@ Your token is valid but `AutoConfig` is not forwarding it. Use `login()` to set
 
 ## Version history
 
-- **v0.2** — Unigram LM at 65k vocab with AraToken normalizer. First sub-100k-vocab tokenizer to beat GPT-5 on both AR and EN simultaneously. E2E-validated on 2M FineWeb samples.
+- **v0.2** — Unigram LM at 65k vocab with AraToken normalizer. First sub-100k-vocab tokenizer to beat GPT-5, Gemma-4, Qwen3.6, and Kimi-K2.6 on both AR and EN simultaneously. E2E-validated on 2M FineWeb samples.
 - **v0.1** — MYTE + parity-aware BPE at 62,912 vocab with 14k Arabic morpheme atomic tokens.
 
 ---