Upload all models and assets for ee (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: ee
-language_name: EE
+language_name: Ewe
 language_family: atlantic_kwa
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-atlantic_kwa
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.014
+    value: 4.309
   - name: best_isotropy
     type: isotropy
-    value: 0.6434
+    value: 0.7155
   - name: vocabulary_size
     type: vocab
-    value: 12330
-generated: 2025-12-30
+    value: 0
+generated: 2026-01-04
 ---
 
-# EE - Wikilangs Models
+# Ewe - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **EE** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Ewe** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -44,12 +54,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +70,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,55 +80,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.451x | 3.42 | 0.1964% | 189,386 |
-| **16k** | 3.696x | 3.66 | 0.2104% | 176,825 |
-| **32k** | 3.905x | 3.87 | 0.2223% | 167,343 |
-| **64k** | 4.014x 🏆 | 3.97 | 0.2285% | 162,807 |
+| **8k** | 3.822x | 3.83 | 0.5658% | 181,329 |
+| **16k** | 4.082x | 4.09 | 0.6044% | 169,762 |
+| **32k** | 4.309x 🏆 | 4.31 | 0.6380% | 160,824 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Ũ, ũ — nye Kikuyutɔwo ƒe nya alfabet.`
+**Sample 1:** `Ata Messan Ajavon Zeus nye Togo dunyahela, eye wònye Save Togo Collective ƒe zim...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ ũ , ▁ ũ ▁— ▁nye ▁ki ku yu ... (+5 more)` | 15 |
-| 16k | `▁ũ , ▁ũ ▁— ▁nye ▁ki kuyu tɔwo ▁ƒe ▁nya ... (+2 more)` | 12 |
-| 32k | `▁ũ , ▁ũ ▁— ▁nye ▁kikuyutɔwo ▁ƒe ▁nya ▁alfabet .` | 10 |
-| 64k | `▁ũ , ▁ũ ▁— ▁nye ▁kikuyutɔwo ▁ƒe ▁nya ▁alfabet .` | 10 |
-
-**Sample 2:** `Hong Kong nye Asia dukɔwo dometɔ ɖeka.
+| 8k | `▁ata ▁me ssan ▁aja von ▁ze us ▁nye ▁togo ▁dunyahela ... (+18 more)` | 28 |
+| 16k | `▁ata ▁messan ▁ajavon ▁ze us ▁nye ▁togo ▁dunyahela , ▁eye ... (+15 more)` | 25 |
+| 32k | `▁ata ▁messan ▁ajavon ▁zeus ▁nye ▁togo ▁dunyahela , ▁eye ▁wònye ... (+13 more)` | 23 |
 
-Category:China`
+**Sample 2:** `South Carolina nye dukɔ aɖe le United States. States`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁h ong ▁k ong ▁nye ▁asia ▁dukɔwo ▁dometɔ ▁ɖeka . ... (+3 more)` | 13 |
-| 16k | `▁hong ▁kong ▁nye ▁asia ▁dukɔwo ▁dometɔ ▁ɖeka . ▁category : ... (+1 more)` | 11 |
-| 32k | `▁hong ▁kong ▁nye ▁asia ▁dukɔwo ▁dometɔ ▁ɖeka . ▁category : ... (+1 more)` | 11 |
-| 64k | `▁hong ▁kong ▁nye ▁asia ▁dukɔwo ▁dometɔ ▁ɖeka . ▁category : ... (+1 more)` | 11 |
+| 8k | `▁south ▁caro lina ▁nye ▁dukɔ ▁aɖe ▁le ▁united ▁states . ... (+1 more)` | 11 |
+| 16k | `▁south ▁carolina ▁nye ▁dukɔ ▁aɖe ▁le ▁united ▁states . ▁states` | 10 |
+| 32k | `▁south ▁carolina ▁nye ▁dukɔ ▁aɖe ▁le ▁united ▁states . ▁states` | 10 |
 
-**Sample 3:** `Oslo nye Norway dugã enye. Eƒe dukɔmenɔlawo ƒe xexlẽme ɖo 658,390 lɔƒo (2016).
-
-...`
+**Sample 3:** `GbɔeviAziaku, Vincent Erskine. A Linguistic Analysis of Ewe Animal Names among t...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁os lo ▁nye ▁norway ▁dugã ▁enye . ▁eƒe ▁dukɔmenɔlawo ▁ƒe ... (+30 more)` | 40 |
-| 16k | `▁oslo ▁nye ▁norway ▁dugã ▁enye . ▁eƒe ▁dukɔmenɔlawo ▁ƒe ▁xexlẽme ... (+28 more)` | 38 |
-| 32k | `▁oslo ▁nye ▁norway ▁dugã ▁enye . ▁eƒe ▁dukɔmenɔlawo ▁ƒe ▁xexlẽme ... (+26 more)` | 36 |
-| 64k | `▁oslo ▁nye ▁norway ▁dugã ▁enye . ▁eƒe ▁dukɔmenɔlawo ▁ƒe ▁xexlẽme ... (+26 more)` | 36 |
+| 8k | `▁gbɔe viaziaku , ▁vincent ▁erskine . ▁a ▁linguistic ▁analysis ▁of ... (+19 more)` | 29 |
+| 16k | `▁gbɔe viaziaku , ▁vincent ▁erskine . ▁a ▁linguistic ▁analysis ▁of ... (+19 more)` | 29 |
+| 32k | `▁gbɔeviaziaku , ▁vincent ▁erskine . ▁a ▁linguistic ▁analysis ▁of ▁ewe ... (+18 more)` | 28 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.014x compression
-- **Lowest UNK Rate:** 8k with 0.1964% unknown tokens
+- **Best Compression:** 32k achieves 4.309x compression
+- **Lowest UNK Rate:** 8k with 0.5658% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,57 +135,111 @@ Category:China`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 3,597 🏆 | 11.81 | 9,575 | 23.2% | 55.1% |
-| **2-gram** | 316 🏆 | 8.30 | 2,357 | 61.9% | 98.5% |
-| **3-gram** | 6,180 | 12.59 | 13,821 | 19.8% | 43.8% |
-| **3-gram** | 2,220 | 11.12 | 16,421 | 29.6% | 70.6% |
-| **4-gram** | 10,276 | 13.33 | 22,472 | 18.3% | 35.3% |
-| **4-gram** | 9,097 | 13.15 | 62,387 | 16.3% | 45.5% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 3,050 | 11.57 | 7,157 | 23.6% | 56.9% |
+| **2-gram** | Subword | 259 🏆 | 8.02 | 1,996 | 66.4% | 99.2% |
+| **3-gram** | Word | 4,032 | 11.98 | 8,747 | 22.9% | 48.1% |
+| **3-gram** | Subword | 1,781 | 10.80 | 12,826 | 32.5% | 74.7% |
+| **4-gram** | Word | 6,737 | 12.72 | 13,766 | 19.8% | 37.5% |
+| **4-gram** | Subword | 7,506 | 12.87 | 51,628 | 17.9% | 48.5% |
+| **5-gram** | Word | 4,126 | 12.01 | 8,899 | 24.0% | 42.0% |
+| **5-gram** | Subword | 18,211 | 14.15 | 94,077 | 11.1% | 34.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `le ƒe` | 2,279 |
+| 2 | `ƒe me` | 1,784 |
+| 3 | `me la` | 1,442 |
+| 4 | `me le` | 1,115 |
+| 5 | `si nye` | 1,012 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `la ,` | 3,065 |
-| 2 | `le ƒe` | 2,270 |
-| 3 | `me .` | 1,712 |
-| 4 | `category :` | 1,483 |
-| 5 | `me la` | 1,440 |
+| 1 | `le ƒe me` | 1,460 |
+| 2 | `ƒe me la` | 652 |
+| 3 | `va ɖo ƒe` | 327 |
+| 4 | `ƒe va ɖo` | 319 |
+| 5 | `tso ƒe va` | 311 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `me la ,` | 1,287 |
-| 2 | `: / /` | 514 |
-| 3 | `. le ƒe` | 491 |
-| 4 | `, si nye` | 458 |
-| 5 | `va ɖo ƒe` | 326 |
+| 1 | `le ƒe me la` | 540 |
+| 2 | `ƒe va ɖo ƒe` | 316 |
+| 3 | `tso ƒe va ɖo` | 302 |
+| 4 | `vincent erskine a linguistic` | 256 |
+| 5 | `erskine a linguistic analysis` | 256 |
 
-**4-grams:**
+**5-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `https : / /` | 320 |
-| 2 | `ƒe tsoƒe category :` | 278 |
-| 3 | `: / / www` | 272 |
-| 4 | `/ / www .` | 272 |
-| 5 | `nyasiawo ƒe tsoƒe category` | 267 |
+| 1 | `tso ƒe va ɖo ƒe` | 300 |
+| 2 | `linguistic analysis of ewe animal` | 256 |
+| 3 | `analysis of ewe animal names` | 256 |
+| 4 | `of ewe animal names among` | 256 |
+| 5 | `ewe animal names among the` | 256 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `e _` | 93,022 |
+| 2 | `a _` | 32,972 |
+| 3 | `o _` | 26,746 |
+| 4 | `w o` | 25,054 |
+| 5 | `_ a` | 23,819 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ƒ e _` | 21,210 |
+| 2 | `l e _` | 20,474 |
+| 3 | `_ ƒ e` | 16,656 |
+| 4 | `w o _` | 15,423 |
+| 5 | `_ l e` | 14,771 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ ƒ e _` | 16,518 |
+| 2 | `_ l e _` | 14,241 |
+| 3 | `n y e _` | 6,181 |
+| 4 | `_ s i _` | 6,094 |
+| 5 | `_ m e _` | 5,720 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `k p l e _` | 4,986 |
+| 2 | `_ k p l e` | 4,841 |
+| 3 | `o _ ƒ e _` | 4,832 |
+| 4 | `e _ ƒ e _` | 4,358 |
+| 5 | `_ n y e _` | 3,640 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 316
+- **Best Perplexity:** 2-gram (subword) with 259
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~46% of corpus
+- **Coverage:** Top-1000 patterns cover ~35% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -183,55 +247,86 @@ Category:China`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7181 | 1.645 | 4.75 | 27,803 | 28.2% |
-| **1** | 1.5928 | 3.016 | 13.09 | 400 | 0.0% |
-| **2** | 0.3176 | 1.246 | 1.78 | 131,912 | 68.2% |
-| **2** | 1.0913 | 2.131 | 6.11 | 5,232 | 0.0% |
-| **3** | 0.1273 | 1.092 | 1.22 | 234,623 | 87.3% |
-| **3** | 0.8236 | 1.770 | 3.55 | 31,943 | 17.6% |
-| **4** | 0.0520 🏆 | 1.037 | 1.08 | 286,556 | 94.8% |
-| **4** | 0.5125 🏆 | 1.426 | 2.16 | 113,293 | 48.8% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7631 | 1.697 | 4.67 | 25,800 | 23.7% |
+| **1** | Subword | 1.5369 | 2.902 | 11.32 | 389 | 0.0% |
+| **2** | Word | 0.2897 | 1.222 | 1.68 | 120,194 | 71.0% |
+| **2** | Subword | 1.0150 | 2.021 | 5.66 | 4,399 | 0.0% |
+| **3** | Word | 0.1029 | 1.074 | 1.17 | 201,432 | 89.7% |
+| **3** | Subword | 0.7954 | 1.736 | 3.60 | 24,892 | 20.5% |
+| **4** | Word | 0.0390 🏆 | 1.027 | 1.06 | 235,375 | 96.1% |
+| **4** | Subword | 0.5399 | 1.454 | 2.27 | 89,556 | 46.0% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `ƒe sewɔtakpekpea me le berlin takpekpea me manya alesi wòhiã be yeƒe dukɔa ƒe dunyahehewo ƒe`
+2. `le ho ʋlim le dukplɔla ƒe ɖoɖo aɖe ƒe bisiɔp gbãtɔ kple dzoɖagbe kple nubablawo gɔmee`
+3. `me nuzazãwo kple la ŋkoe nye eƒe sukudede dzɔdzɔmeŋutinunya ƒe nuwɔna me be wòanye nutala afia`
+
+**Context Size 2:**
+
+1. `le ƒe me eye wòtso bole le savanna nutome wodzi mahama le november 28 dzi le guadeloupe`
+2. `ƒe me emegbe exɔ ɖɔkta ƒe dzeside adre kple afã tso dukɔ yome me le south africa`
+3. `me la gold coast le tedoxe 26 dzi kple agbalẽtamɔ̃ gãwo siaa me wotsɔ nya ɖe ame`
+
+**Context Size 3:**
 
-### Generated Text Samples
+1. `le ƒe me eye archdeacon le ƒe enye sinima gbãtɔ si woɖe le ƒe me eye wòka atam`
+2. `ƒe me la eɖe eme be mefia be wò agbe mele vevie o 11 koe gblɔ be ameyibɔwo`
+3. `va ɖo ƒe dome defontaine ku le hénin sur cojeul ƒe dumegã le ƒe va ɖo ƒe le`
+
+**Context Size 4:**
 
-Below are text samples generated from each Markov chain model:
+1. `le ƒe me la enye europa dukɔwo ƒe habɔbɔ me eƒe zimenɔla si woti le ƒe me lae nye`
+2. `ƒe va ɖo ƒe tso ƒe va ɖo ƒe dɔmedzoedonamea xɔ ƒe eve agbalẽa me tɔ vevitɔe nye dɔwɔhawo`
+3. `tso ƒe va ɖo ƒe enɔ pyrénées atlantiques dɔwɔƒea teƒe grenet nye radical party me tɔ enye orléans ƒe`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. eɖo " the cursed ones - 3 . to subɔsubɔha me tɔ . m .`
-2. `ƒe nukpɔsusuwo ta si be / mps / / / / browsecollections / 20190427151234 / 9780812291445`
-3. `, dukɔ aɖe tso aʋawɔwɔwo me le united nations decade for the national democratic congress me`
+1. `_aƒe_aɖena_(_na_`
+2. `e_dzu_alaxa_etsi`
+3. `ameɖonye_si_d_ye`
 
 **Context Size 2:**
 
-1. `la , islam kpɔ ŋusẽ ɖe amewo ŋu , eye wòdidi tso tsidzɔƒe gã la dzi .`
-2. `le ƒe 2007 fifa u - 20 muhammad hidayat ullah ( nuwɔna ) ƒe habɔbɔ si le`
-3. `me . enye senyala gilbert strauss - kahn nye kɔmiunist sukuviwo ƒe habɔbɔ me tɔ le september`
+1. `e_la_nyations_me_`
+2. `a_culymmakple_du_`
+3. `o_frafia_ƒe_a._me`
 
 **Context Size 3:**
 
-1. `me la , edze be woa de dzesi eɖokui na ɖɔwɔƒe si gbalɛ nyawo gbɔ enumake alo ana`
-2. `: / / en . wikipedia . org / fellows / 979 / " . 2023 - 01`
-3. `. le ƒe geɖe ƒe nazãbubu , vovototodedeameme , kple hadome vovototodedeameme , si ɖe gabontɔ kung fu`
+1. `ƒe_3,_dzɔ_dome_ŋgɔ`
+2. `le_ta_12._don_le_a`
+3. `_ƒe_nu_dze_la,_wod`
 
 **Context Size 4:**
 
-1. `https : / / en . wikipedia . org / wiki / s2cid_ ( identifier ) 194922873 dedieu ,`
-2. `ƒe tsoƒe category : tɔmelãwo`
-3. `: / / www . fao . org / docrep / 009 / a0154e / a0154e07 . htm nɔnɔmetatawo`
+1. `_ƒe_me_da_asitsi_et`
+2. `_le_du_be_la,_eye_w`
+3. `nye_to_february_raw`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 94.8% predictability
+- **Best Predictability:** Context-4 (word) with 96.1% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (113,293 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (89,556 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -247,35 +342,35 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 12,330 |
-| Total Tokens | 280,434 |
-| Mean Frequency | 22.74 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 251.45 |
+| Vocabulary Size | 11,578 |
+| Total Tokens | 260,556 |
+| Mean Frequency | 22.50 |
+| Median Frequency | 3 |
+| Frequency Std Dev | 257.37 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ƒe | 17,053 |
-| 2 | le | 14,563 |
+| 1 | ƒe | 16,951 |
+| 2 | le | 14,512 |
 | 3 | me | 8,468 |
-| 4 | si | 6,286 |
-| 5 | la | 4,916 |
-| 6 | kple | 4,853 |
-| 7 | nye | 3,835 |
-| 8 | be | 3,754 |
-| 9 | ɖe | 3,265 |
+| 4 | si | 6,279 |
+| 5 | la | 4,866 |
+| 6 | kple | 4,852 |
+| 7 | be | 3,745 |
+| 8 | nye | 3,709 |
+| 9 | ɖe | 3,263 |
 | 10 | siwo | 2,545 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | loi | 2 |
-| 2 | 12932 | 2 |
-| 3 | crunchy | 2 |
-| 4 | kakl | 2 |
+| 1 | woɖunɛ | 2 |
+| 2 | couscous | 2 |
+| 3 | fufú | 2 |
+| 4 | loi | 2 |
 | 5 | klottey | 2 |
 | 6 | korle | 2 |
 | 7 | domelovo | 2 |
@@ -287,24 +382,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1796 |
-| R² (Goodness of Fit) | 0.990771 |
+| Zipf Coefficient | 1.1638 |
+| R² (Goodness of Fit) | 0.992157 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 47.4% |
-| Top 1,000 | 77.4% |
-| Top 5,000 | 93.3% |
-| Top 10,000 | 98.3% |
+| Top 100 | 49.7% |
+| Top 1,000 | 78.5% |
+| Top 5,000 | 93.7% |
+| Top 10,000 | 98.8% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9908 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 47.4% of corpus
-- **Long Tail:** 2,330 words needed for remaining 1.7% coverage
+- **Zipf Compliance:** R²=0.9922 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 49.7% of corpus
+- **Long Tail:** 1,578 words needed for remaining 1.2% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -317,24 +412,114 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 5,162 | 32 | 2.985 | 0.736 | 0.6434 🏆 |
-| **mono_64d** | 5,162 | 64 | 3.060 | 0.703 | 0.2784 |
-| **mono_128d** | 5,162 | 128 | 3.081 | 0.698 | 0.0599 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.7155 🏆 | 0.3892 | N/A | N/A |
+| **mono_64d** | 64 | 0.2811 | 0.3672 | N/A | N/A |
+| **mono_128d** | 128 | 0.0660 | 0.3770 | N/A | N/A |
+| **aligned_32d** | 32 | 0.7155 | 0.4123 | 0.0180 | 0.1660 |
+| **aligned_64d** | 64 | 0.2811 | 0.3853 | 0.0500 | 0.2600 |
+| **aligned_128d** | 128 | 0.0660 | 0.3736 | 0.0840 | 0.2920 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.6434 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 5,162 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.7155 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3841. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 8.4% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.210** | High formulaic/idiomatic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-e` | okeke, dzoe, exɔe |
+| `-wo` | yeyeawo, kadodowo, eɖewo |
+| `-awo` | yeyeawo, franseawo, kɔwlɔawo |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `gbal` | 1.65x | 17 contexts | gbalɛ, gbale, gbalé |
+| `lawo` | 1.59x | 14 contexts | xɔlawo, dolawo, nɔlawo |
+| `pekp` | 1.82x | 9 contexts | kpekpe, kpekpea, kpekpeme |
+| `dɔwɔ` | 1.66x | 11 contexts | dɔwɔm, dɔwɔla, dɔwɔƒe |
+| `balẽ` | 1.72x | 9 contexts | agbalẽ, gbalẽa, lãgbalẽ |
+| `omet` | 1.44x | 14 contexts | wometa, tometi, ƒometɔ |
+| `dziɖ` | 1.82x | 7 contexts | dziɖum, dziɖuɖu, dziɖula |
+| `ziɖu` | 1.89x | 6 contexts | dziɖum, dziɖuɖu, dziɖula |
+| `takp` | 1.74x | 7 contexts | takpɔha, takpɔƒe, takpɔƒea |
+| `nyat` | 1.68x | 7 contexts | nyati, nyatia, nyatiwo |
+| `iɖuɖ` | 1.91x | 5 contexts | dziɖuɖu, dziɖuɖua, dziɖuɖuha |
+| `iawo` | 1.64x | 7 contexts | siawo, fiawo, viawo |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| gbegbɔgblɔwo | **`gbegbɔgblɔ-wo`** | 4.5 | `gbegbɔgblɔ` |
+| aƒemelãwo | **`aƒemelã-wo`** | 4.5 | `aƒemelã` |
+| gbebiamewo | **`gbebiame-wo`** | 4.5 | `gbebiame` |
+| srɔ̃tɔawo | **`srɔ̃tɔ-awo`** | 4.5 | `srɔ̃tɔ` |
+| lebanontɔwo | **`lebanontɔ-wo`** | 4.5 | `lebanontɔ` |
+| wuietɔ̃awo | **`wuietɔ̃-awo`** | 4.5 | `wuietɔ̃` |
+| domenyiŋkɔwo | **`domenyiŋkɔ-wo`** | 4.5 | `domenyiŋkɔ` |
+| ŋkuɖodzikpewo | **`ŋkuɖodzikpe-wo`** | 4.5 | `ŋkuɖodzikpe` |
+| nukpɔsusuwo | **`nukpɔsusu-wo`** | 4.5 | `nukpɔsusu` |
+| swedentɔwo | **`swedentɔ-wo`** | 4.5 | `swedentɔ` |
+| asanteawo | **`asante-awo`** | 4.5 | `asante` |
+| sɔlemexɔwo | **`sɔlemexɔ-wo`** | 4.5 | `sɔlemexɔ` |
+| akpɔkplɔwo | **`akpɔkplɔ-wo`** | 4.5 | `akpɔkplɔ` |
+| amegãxiwo | **`amegãxi-wo`** | 4.5 | `amegãxi` |
+| ukrainetɔwo | **`ukrainetɔ-wo`** | 4.5 | `ukrainetɔ` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Ewe shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -342,11 +527,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.01x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (316) |
-| Markov | **Context-4** | Highest predictability (94.8%) |
+| Tokenizer | **32k BPE** | Best compression (4.31x) |
+| N-gram | **2-gram** | Lowest perplexity (259) |
+| Markov | **Context-4** | Highest predictability (96.1%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -536,7 +722,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -552,7 +739,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-30 08:47:35*
+*Report Date: 2026-01-04 03:05:37*

models/embeddings/aligned/ee_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:138d5016cf51bd2c072d7ddd0b75f3388c58c087eb47a93eafccb332d9bc7a84
+size 1029067633

models/embeddings/aligned/ee_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "ee", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ee_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28cd1530e006fb29b6f526a46114b179fcf43946ef08586e7aec8913c54ac155
+size 65664

models/embeddings/aligned/ee_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ee",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1671,
+  "vocab_size": 4869
+}

models/embeddings/aligned/ee_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9d638ed553748b884a0e65a55719c49053e00c20b2215c890b5f4692a7a0e3e
+size 257328241

models/embeddings/aligned/ee_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "ee", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ee_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa478442fcfd6038cca5ccf9c94068b8c55816705adab108a59d5ebc31a4f609
+size 4224

models/embeddings/aligned/ee_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ee",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1671,
+  "vocab_size": 4869
+}

models/embeddings/aligned/ee_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0bc8bbef8d8e32f40c1ace722ed395d7b6f5837fcf53bfe52d2a5a9a8bcdb103
+size 514574705

models/embeddings/aligned/ee_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "ee", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ee_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61a45886575d77b99f15aaca6754fe29daeee42d1c92737a611eae6b21692943
+size 16512

models/embeddings/aligned/ee_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ee",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 1671,
+  "vocab_size": 4869
+}

models/embeddings/monolingual/ee_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:79be53d480ecbdd303f06a5412089c7ee17cef7bc0cbad323afb542fe1ef7020
-size 1029372518
+oid sha256:138d5016cf51bd2c072d7ddd0b75f3388c58c087eb47a93eafccb332d9bc7a84
+size 1029067633

models/embeddings/monolingual/ee_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 5162
+  "vocab_size": 4869
 }

models/embeddings/monolingual/ee_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:467311d74f32aec3e9431003219d29fcdd8d706e1841bf53543523add2ee3bd5
-size 257408102
+oid sha256:d9d638ed553748b884a0e65a55719c49053e00c20b2215c890b5f4692a7a0e3e
+size 257328241

models/embeddings/monolingual/ee_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 5162
+  "vocab_size": 4869
 }

models/embeddings/monolingual/ee_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ac93c0c2f5f34f2d81ba1209744027f1f6951c2f1f19c923a27097fb092a0885
-size 514729574
+oid sha256:0bc8bbef8d8e32f40c1ace722ed395d7b6f5837fcf53bfe52d2a5a9a8bcdb103
+size 514574705

models/embeddings/monolingual/ee_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 5162
+  "vocab_size": 4869
 }

models/subword_markov/ee_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d34403e1217933de312c120c770452550c9b7dd57c6d112babe8c698bea9188c
-size 42190
+oid sha256:51536daa3bb9781fc6466f29b0e073c5a52e9d1659058630978baedb6b8e90d1
+size 36773

models/subword_markov/ee_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ee",
-  "unique_contexts": 400,
-  "total_transitions": 1666590
+  "unique_contexts": 389,
+  "total_transitions": 1527359
 }

models/subword_markov/ee_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:10170b919464c2ae97fc7b8be35a350080069cbdb80d19e00fb0ba06a516eae0
-size 245537
+oid sha256:1be33f68703919123c5a502f46979ea62f68c0e271d16c4c63435d040f2fd841
+size 204528

models/subword_markov/ee_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ee",
-  "unique_contexts": 5232,
-  "total_transitions": 1665084
+  "unique_contexts": 4399,
+  "total_transitions": 1526002
 }

models/subword_markov/ee_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:61926d86ee45e0b75d960d813180a497501614b05f711e3f2bfd75554da858f7
-size 826380
+oid sha256:a980c465972864bf604b7d22d020aaf3bd03523b3f16fd05ebc1a73c8728f617
+size 670860

models/subword_markov/ee_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ee",
-  "unique_contexts": 31943,
-  "total_transitions": 1663578
+  "unique_contexts": 24892,
+  "total_transitions": 1524645
 }

models/subword_markov/ee_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7e347eca6a78c86e75d24018056d32483bab6d1ed7551c155b5e8f3ccde9f170
-size 2000536
+oid sha256:8aea990056a79d55b9ce0c8e6f64dbd4a7eb692f3afce2aa8adc84e8c2221a8f
+size 1606353

models/subword_markov/ee_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ee",
-  "unique_contexts": 113293,
-  "total_transitions": 1662072
+  "unique_contexts": 89556,
+  "total_transitions": 1523288
 }

models/subword_ngram/ee_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:45fa8ae10010771f812efb4671644e49c70062d15a295251d89585d9b5c2e125
-size 31332
+oid sha256:0267731e70fe4438e6223c6ad54399ae04b3f3aaf1d78a459a72777de37bd5e3
+size 26749

models/subword_ngram/ee_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ee",
-  "unique_ngrams": 2357,
-  "total_ngrams": 1666590
+  "unique_ngrams": 1996,
+  "total_ngrams": 1527359
 }

models/subword_ngram/ee_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7af1287e4f2d30f328be5779f838d360c32ffdb71a94d4b6e34ec4de38a35e74
-size 192548
+oid sha256:f811c91a9acd7abd901fdd218900d201d946bb43abc7c89a4122f00007b347eb
+size 150482

models/subword_ngram/ee_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ee",
-  "unique_ngrams": 16421,
-  "total_ngrams": 1665084
+  "unique_ngrams": 12826,
+  "total_ngrams": 1526002
 }

models/subword_ngram/ee_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6a22344260645ccbbbed16eede3babebb91a5eee71793a2367061dca0fdb629d
-size 742988
+oid sha256:111de5625fd9154e495e06d0da73e18820eee30d44602b8a2594310118de7801
+size 622000

models/subword_ngram/ee_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ee",
-  "unique_ngrams": 62387,
-  "total_ngrams": 1663578
+  "unique_ngrams": 51628,
+  "total_ngrams": 1524645
 }

models/subword_ngram/ee_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88839dbf14600598089dc96731844545a675ef788e19d324450ff7d37f642cde
+size 1114046

models/subword_ngram/ee_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "ee",
+  "unique_ngrams": 94077,
+  "total_ngrams": 1523288
+}

models/tokenizer/ee_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:160dd67bb52f4f3475999ebd6db4427ac84c61afb6f8b043e14ade19da491390
-size 509044
+oid sha256:ea64f3995183acdab8d8fc6d8b56accc12add1e1835e8c8b3c694434019de1bc
+size 514447

models/tokenizer/ee_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e943ac9a4bc9d3a50012c53b05f27af5a9819814687ea30cdb229998edc8d1af
-size 792030
+oid sha256:a706fb494ef21b855f3968e14427ac6f55804b8b1c3d359f34002419293ba9c1
+size 792059

models/tokenizer/ee_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3f42a721cbeee5b5e939657bb0c9fdd5e93d15caaec11c43a31a2784244d344
-size 375724
+oid sha256:2316a005005e904981c9e7cfe95dde39d03e222cab215cf2d0b164d90f845191
+size 375410

models/vocabulary/ee_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:024443db6994abc4c65b068dcf563cbc5f35b4f869cd7b0cf2e901e1c2f28fe3
-size 202981
+oid sha256:771edb67abb5b274a8abd8db9b39eab47626e5b862bd77a5227c6b5c72f7829f
+size 191748

models/vocabulary/ee_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ee",
-  "vocabulary_size": 12330,
+  "vocabulary_size": 11578,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.09374725356445081,
+    "type_token_ratio": 0.0939830061316206,
     "coverage": {
-      "top_100": 0.4492695326496258,
-      "top_1000": 0.733702228922586,
-      "top_5000": 0.8841426726789662,
-      "top_10000": 0.9321790980198622
+      "top_100": 0.4716180564400211,
+      "top_1000": 0.7447135241352959,
+      "top_5000": 0.8886046469314605,
+      "top_10000": 0.936664180054948
     },
-    "hapax_count": 15404,
-    "hapax_ratio": 0.5554193408812288,
-    "total_documents": 1506
+    "hapax_count": 14249,
+    "hapax_ratio": 0.5517094513493631,
+    "total_documents": 1357
   }
 }

models/word_markov/ee_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00082b0c440aacaad2a7872bab73d561f7c3dbffcf004cdf89030cbb6cbe9a4a
-size 1001594
+oid sha256:f1c6b29e1993b44121a5d77ed0b1bf8f10088a799dee7e338f7bfa31e73aea2b
+size 921457

models/word_markov/ee_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ee",
-  "unique_contexts": 27803,
-  "total_transitions": 354172
+  "unique_contexts": 25800,
+  "total_transitions": 273448
 }

models/word_markov/ee_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d015ba78652316001814062373668dff7c07ff3cf253806b9bcc4ded1ebc5cd3
-size 2481177
+oid sha256:2cef6e9b3ab6ee2cddb2dc893bb2607758a1cdc958438d70a9ef9fcd0ef55761
+size 2274725

models/word_markov/ee_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ee",
-  "unique_contexts": 131912,
-  "total_transitions": 352666
+  "unique_contexts": 120194,
+  "total_transitions": 272091
 }

models/word_markov/ee_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f1561aa9f41051b145da5e5a559fd901b08440c07daca3aa0af224817e729c09
-size 3921939
+oid sha256:8cc008b3ad1e5a09ffebbb30a5534ee9b25d67a1b2da653a5f4de623f2f378dd
+size 3436521

models/word_markov/ee_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ee",
-  "unique_contexts": 234623,
-  "total_transitions": 351160
+  "unique_contexts": 201432,
+  "total_transitions": 270734
 }