TCAndon-Router: Adaptive Reasoning Router for Multi-Agent Collaboration
Paper
β’
2601.04544
β’
Published
β’
6
AI & ML interests
None defined yet.
Recent Activity
View all activity
nielsrΒ
submitted
a
paper
to Daily Papers
13 days ago
nielsrΒ
submitted
a
paper
to Daily Papers
about 1 month ago
Post
2602
Muon vs MuonClip vs Muon+Adamw
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fineβtuning? We ran headβtoβhead tests on Qwen3β4B (10k+ highβquality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradientβnorm spikes made training unstable. MuonClip (Kimi K2βs clipping) stabilizes long pretraining runs, yet in our smallβscale fineβtune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muonβs spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fineβtuning? We ran headβtoβhead tests on Qwen3β4B (10k+ highβquality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradientβnorm spikes made training unstable. MuonClip (Kimi K2βs clipping) stabilizes long pretraining runs, yet in our smallβscale fineβtune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muonβs spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Post
2562
I tested Muon vs MuonClip vs Muon+AdamW for fine-tuning LLMs
Just published a blog on that, Read here π https://huggingface.co/blog/KingNish/optimizer-part1
Just published a blog on that, Read here π https://huggingface.co/blog/KingNish/optimizer-part1
- 1 reply
reach-vbΒ
authored
a
paper
about 2 months ago
Post
963
The #1 trending AI/ML dataset today π
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Massive scale, diversity and end-to-end potential from nvidia !
nvidia/PhysicalAI-Autonomous-Vehicles
Post
687
The new King πhas arrived!
Moonshot AI now the top model on Hugging Face π₯
moonshotai/Kimi-K2-Thinking
Moonshot AI now the top model on Hugging Face π₯
moonshotai/Kimi-K2-Thinking
Post
2791
πΈπ€You donβt need 100 GPUs to train something amazing!
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on π€ :
HuggingFaceTB/smol-training-playbook
Our Smol Training Playbook teaches you a better path to world-class LLMs, for free!
Check out the #1 trending space on π€ :
HuggingFaceTB/smol-training-playbook
DmitryRyuminΒ
posted
an
update
3 months ago
Post
1284
πποΈπ New Research Alert - ICCV 2025 (Poster)! πποΈπ
π Title: Is Less More? Exploring Token Condensation as Training-Free Test-Time Adaptation π
π Description: Token Condensation as Adaptation (TCA) improves the performance and efficiency of Vision Language Models in zero-shot inference by introducing domain anchor tokens.
π₯ Authors: Zixin Wang, Dong Gong, Sen Wang, Zi Huang, Yadan Luo
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation (2410.14729)
π Repository: https://github.com/Jo-wang/TCA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Session 1: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/session-1.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TestTimeAdaptation #TokenCondensation #VisionLanguageModels #TrainingFreeAdaptation #ZeroShotLearning #EfficientAI #AI #ICCV2025 #ResearchHighlight
π Title: Is Less More? Exploring Token Condensation as Training-Free Test-Time Adaptation π
π Description: Token Condensation as Adaptation (TCA) improves the performance and efficiency of Vision Language Models in zero-shot inference by introducing domain anchor tokens.
π₯ Authors: Zixin Wang, Dong Gong, Sen Wang, Zi Huang, Yadan Luo
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation (2410.14729)
π Repository: https://github.com/Jo-wang/TCA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Session 1: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/session-1.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TestTimeAdaptation #TokenCondensation #VisionLanguageModels #TrainingFreeAdaptation #ZeroShotLearning #EfficientAI #AI #ICCV2025 #ResearchHighlight
xianbaoΒ
authored
a
paper
3 months ago
DmitryRyuminΒ
posted
an
update
3 months ago
Post
2417
πποΈπ New Research Alert - ICCV 2025 (Oral)! πποΈπ
π Title: Diving into the Fusion of Monocular Priors for Generalized Stereo Matching π
π Description: The proposed method enhances stereo matching by efficiently combining unbiased monocular priors from vision foundation models. This method addresses misalignment and local optima issues using a binary local ordering map and pixel-wise linear regression.
π₯ Authors: Chengtang Yao, Lidong Yu, Zhidan Liu, Jiaxi Zeng, Yuwei Wu, and Yunde Jia
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Diving into the Fusion of Monocular Priors for Generalized Stereo Matching (2505.14414)
π Repository: https://github.com/YaoChengTang/Diving-into-the-Fusion-of-Monocular-Priors-for-Generalized-Stereo-Matching
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the 3D Pose Understanding Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/3d-pose-understanding.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #StereoMatching #MonocularDepth #VisionFoundationModels #3DReconstruction #Generalization #AI #ICCV2025 #ResearchHighlight
π Title: Diving into the Fusion of Monocular Priors for Generalized Stereo Matching π
π Description: The proposed method enhances stereo matching by efficiently combining unbiased monocular priors from vision foundation models. This method addresses misalignment and local optima issues using a binary local ordering map and pixel-wise linear regression.
π₯ Authors: Chengtang Yao, Lidong Yu, Zhidan Liu, Jiaxi Zeng, Yuwei Wu, and Yunde Jia
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Diving into the Fusion of Monocular Priors for Generalized Stereo Matching (2505.14414)
π Repository: https://github.com/YaoChengTang/Diving-into-the-Fusion-of-Monocular-Priors-for-Generalized-Stereo-Matching
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the 3D Pose Understanding Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/3d-pose-understanding.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #StereoMatching #MonocularDepth #VisionFoundationModels #3DReconstruction #Generalization #AI #ICCV2025 #ResearchHighlight
Build Failure (apt-get permission denied) in Dev Mode on Gradio + ZeroGpu Space
4
#26 opened 3 months ago
by
Archime
DmitryRyuminΒ
posted
an
update
3 months ago
Post
2837
πππ New Research Alert - ICCV 2025 (Oral)! ππ€π
π Title: Understanding Co-speech Gestures in-the-wild π
π Description: JEGAL is a tri-modal model that learns from gestures, speech and text simultaneously, enabling devices to interpret co-speech gestures in the wild.
π₯ Authors: @sindhuhegde , K R Prajwal, Taein Kwon, and Andrew Zisserman
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Understanding Co-speech Gestures in-the-wild (2503.22668)
π Web Page: https://www.robots.ox.ac.uk/~vgg/research/jegal
π Repository: https://github.com/Sindhu-Hegde/jegal
πΊ Video: https://www.youtube.com/watch?v=TYFOLKfM-rM
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Human Modeling Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/human-modeling.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #CoSpeechGestures #GestureUnderstanding #TriModalRepresentation #MultimodalLearning #AI #ICCV2025 #ResearchHighlight
π Title: Understanding Co-speech Gestures in-the-wild π
π Description: JEGAL is a tri-modal model that learns from gestures, speech and text simultaneously, enabling devices to interpret co-speech gestures in the wild.
π₯ Authors: @sindhuhegde , K R Prajwal, Taein Kwon, and Andrew Zisserman
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Understanding Co-speech Gestures in-the-wild (2503.22668)
π Web Page: https://www.robots.ox.ac.uk/~vgg/research/jegal
π Repository: https://github.com/Sindhu-Hegde/jegal
πΊ Video: https://www.youtube.com/watch?v=TYFOLKfM-rM
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Human Modeling Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/human-modeling.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #CoSpeechGestures #GestureUnderstanding #TriModalRepresentation #MultimodalLearning #AI #ICCV2025 #ResearchHighlight
DmitryRyuminΒ
posted
an
update
3 months ago
Post
3965
ππ‘π New Research Alert - ICCV 2025 (Oral)! ππͺπ
π Title: LoftUp: Learning a Coordinate-based Feature Upsampler for Vision Foundation Models π
π Description: LoftUp is a coordinate-based transformer that upscales the low-resolution features of VFMs (e.g. DINOv2 and CLIP) using cross-attention and self-distilled pseudo-ground truth (pseudo-GT) from SAM.
π₯ Authors: Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, and Dan Zhang
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models (2504.14032)
π Github Page: https://andrehuang.github.io/loftup-site
π Repository: https://github.com/andrehuang/loftup
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Foundation Models and Representation Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/foundation-models-and-representation-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #LoftUp #VisionFoundationModels #FeatureUpsampling #Cross-AttentionTransformer #CoordinateBasedLearning #SelfDistillation #PseudoGroundTruth #RepresentationLearning #AI #ICCV2025 #ResearchHighlight
π Title: LoftUp: Learning a Coordinate-based Feature Upsampler for Vision Foundation Models π
π Description: LoftUp is a coordinate-based transformer that upscales the low-resolution features of VFMs (e.g. DINOv2 and CLIP) using cross-attention and self-distilled pseudo-ground truth (pseudo-GT) from SAM.
π₯ Authors: Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, and Dan Zhang
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models (2504.14032)
π Github Page: https://andrehuang.github.io/loftup-site
π Repository: https://github.com/andrehuang/loftup
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Foundation Models and Representation Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/foundation-models-and-representation-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #LoftUp #VisionFoundationModels #FeatureUpsampling #Cross-AttentionTransformer #CoordinateBasedLearning #SelfDistillation #PseudoGroundTruth #RepresentationLearning #AI #ICCV2025 #ResearchHighlight
DmitryRyuminΒ
posted
an
update
3 months ago
Post
1954
ππ·οΈπ New Research Alert - ICCV 2025 (Oral)! ππ§©π
π Title: Heavy Labels Out! Dataset Distillation with Label Space Lightening π
π Description: The HeLlO framework is a new corpus distillation method that removes the need for large soft labels. It uses a lightweight, online image-to-label projector based on CLIP. This projector has been adapted using LoRA-style, parameter-efficient tuning. It has also been initialized with text embeddings.
π₯ Authors: @roseannelexie , @Huage001 , Zigeng Chen, Jingwen Ye, and Xinchao Wang
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Heavy Labels Out! Dataset Distillation with Label Space Lightening (2408.08201)
πΊ Video: https://www.youtube.com/watch?v=kAyK_3wskgA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Efficient Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/efficient-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #DatasetDistillation #LabelCompression #CLIP #LoRA #EfficientAI #FoundationModels #AI #ICCV2025 #ResearchHighlight
π Title: Heavy Labels Out! Dataset Distillation with Label Space Lightening π
π Description: The HeLlO framework is a new corpus distillation method that removes the need for large soft labels. It uses a lightweight, online image-to-label projector based on CLIP. This projector has been adapted using LoRA-style, parameter-efficient tuning. It has also been initialized with text embeddings.
π₯ Authors: @roseannelexie , @Huage001 , Zigeng Chen, Jingwen Ye, and Xinchao Wang
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Heavy Labels Out! Dataset Distillation with Label Space Lightening (2408.08201)
πΊ Video: https://www.youtube.com/watch?v=kAyK_3wskgA
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Efficient Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/efficient-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #DatasetDistillation #LabelCompression #CLIP #LoRA #EfficientAI #FoundationModels #AI #ICCV2025 #ResearchHighlight
- 2 replies
DmitryRyuminΒ
authored
a
paper
3 months ago
DmitryRyuminΒ
posted
an
update
3 months ago
Post
4813
ππ€π New Research Alert - ICCV 2025 (Oral)! ππ€π
π Title: Variance-based Pruning for Accelerating and Compressing Trained Networks π
π Description: The one-shot pruning method efficiently compresses networks, reducing computation and memory usage while retaining almost full performance and requiring minimal fine-tuning.
π₯ Authors: Uranik Berisha, Jens Mehnert, and Alexandru Paul Condurache
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Variance-Based Pruning for Accelerating and Compressing Trained Networks (2507.12988)
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Efficient Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/efficient-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #VarianceBasedPruning #NetworkCompression #ModelAcceleration #EfficientDeepLearning #VisionTransformers #AI #ICCV2025 #ResearchHighlight
π Title: Variance-based Pruning for Accelerating and Compressing Trained Networks π
π Description: The one-shot pruning method efficiently compresses networks, reducing computation and memory usage while retaining almost full performance and requiring minimal fine-tuning.
π₯ Authors: Uranik Berisha, Jens Mehnert, and Alexandru Paul Condurache
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Variance-Based Pruning for Accelerating and Compressing Trained Networks (2507.12988)
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Efficient Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/efficient-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #VarianceBasedPruning #NetworkCompression #ModelAcceleration #EfficientDeepLearning #VisionTransformers #AI #ICCV2025 #ResearchHighlight
DmitryRyuminΒ
posted
an
update
3 months ago
Post
3014
πποΈπ New Research Alert - ICCV 2025 (Oral)! πποΈπ
π Title: Token Activation Map to Visually Explain Multimodal LLMs π
π Description: The Token Activation Map (TAM) is an advanced explainability method for multimodal LLMs. Using causal inference and a Rank Gaussian Filter, TAM reveals token-level interactions and eliminates redundant activations. The result is clearer, high-quality visualizations that enhance understanding of object localization, reasoning and multimodal alignment across models.
π₯ Authors: Yi Li, Hualiang Wang, Xinpeng Ding, Haonan Wang, and Xiaomeng Li
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Token Activation Map to Visually Explain Multimodal LLMs (2506.23270)
π Repository: https://github.com/xmed-lab/TAM
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Multi-Modal Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/multi-modal-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TokenActivationMap #TAM #CausalInference #VisualReasoning #Multimodal #Explainability #VisionLanguage #LLM #XAI #AI #ICCV2025 #ResearchHighlight
π Title: Token Activation Map to Visually Explain Multimodal LLMs π
π Description: The Token Activation Map (TAM) is an advanced explainability method for multimodal LLMs. Using causal inference and a Rank Gaussian Filter, TAM reveals token-level interactions and eliminates redundant activations. The result is clearer, high-quality visualizations that enhance understanding of object localization, reasoning and multimodal alignment across models.
π₯ Authors: Yi Li, Hualiang Wang, Xinpeng Ding, Haonan Wang, and Xiaomeng Li
π Conference: ICCV, 19 β 23 Oct, 2025 | Honolulu, Hawai'i, USA πΊπΈ
π Paper: Token Activation Map to Visually Explain Multimodal LLMs (2506.23270)
π Repository: https://github.com/xmed-lab/TAM
π ICCV-2023-25-Papers: https://github.com/DmitryRyumin/ICCV-2023-25-Papers
π Added to the Multi-Modal Learning Section: https://github.com/DmitryRyumin/ICCV-2023-25-Papers/blob/main/sections/2025/main/multi-modal-learning.md
π More Papers: more cutting-edge research presented at other conferences in the DmitryRyumin/NewEraAI-Papers curated by @DmitryRyumin
π Keywords: #TokenActivationMap #TAM #CausalInference #VisualReasoning #Multimodal #Explainability #VisionLanguage #LLM #XAI #AI #ICCV2025 #ResearchHighlight
-
- 2 replies
Post
8214
deepseek-ai/DeepSeek-OCR is out! π₯ my take ‡οΈ
> pretty insane it can parse and re-render charts in HTML
> it uses CLIP and SAM features concatenated, so better grounding
> very efficient per vision tokens/performance ratio
> covers 100 languages
> pretty insane it can parse and re-render charts in HTML
> it uses CLIP and SAM features concatenated, so better grounding
> very efficient per vision tokens/performance ratio
> covers 100 languages
- 4 replies
Post
1003
Tokenization is one of the most important processes in AI - yet many would like to kill it π
What's tokenization? The neural networks inside LLMs actually only process numbers, not text: tokenization is the process that makes text readable for them, by converting sentences into lists of numbers.
β‘οΈ For instance, "This is tokenization" would be split into "This | is | token | ization", then each of the parts (tokens) are converted to IDs according to a predefined mapping: for instance "ization" could map to id 2438.
Thus "This is tokenization" can become 1335 | 135 | 2980 | 2438 => now the model can process the sentence!
Most tokenizers today use pre-specified mappings called "vocabularies", generally built about the compression algorithme Byte-Pair Encoding (BPE) that learns from a big corpuses of texts an optimized split to efficiently encode any text from the same distribution into a list token IDs.
π€¨ Now, these current tokenizers have flaws.
For instance, the rigidity of their mapping creates losses ; the prime example being that a tokenizer designed for English (thus optimized for tokens like "has", "been", "clock", etc) will not have the right tokens to approach Burmese, thus being terribly inefficient at it.
Many alternative approaches have emerged as a result: for instance "tokenizer-free tokenizers". One that I really liked was "entropy-based": it monitors the stream of text, and trigger a split whenever the entropy increases too much, i.e. when something "surprising" happens.
But this great article argues that tokenizers are a lesser evil. Read and decide for yourself!
https://huggingface.co/blog/catherinearnett/in-defense-of-tokenizers
What's tokenization? The neural networks inside LLMs actually only process numbers, not text: tokenization is the process that makes text readable for them, by converting sentences into lists of numbers.
β‘οΈ For instance, "This is tokenization" would be split into "This | is | token | ization", then each of the parts (tokens) are converted to IDs according to a predefined mapping: for instance "ization" could map to id 2438.
Thus "This is tokenization" can become 1335 | 135 | 2980 | 2438 => now the model can process the sentence!
Most tokenizers today use pre-specified mappings called "vocabularies", generally built about the compression algorithme Byte-Pair Encoding (BPE) that learns from a big corpuses of texts an optimized split to efficiently encode any text from the same distribution into a list token IDs.
π€¨ Now, these current tokenizers have flaws.
For instance, the rigidity of their mapping creates losses ; the prime example being that a tokenizer designed for English (thus optimized for tokens like "has", "been", "clock", etc) will not have the right tokens to approach Burmese, thus being terribly inefficient at it.
Many alternative approaches have emerged as a result: for instance "tokenizer-free tokenizers". One that I really liked was "entropy-based": it monitors the stream of text, and trigger a split whenever the entropy increases too much, i.e. when something "surprising" happens.
But this great article argues that tokenizers are a lesser evil. Read and decide for yourself!
https://huggingface.co/blog/catherinearnett/in-defense-of-tokenizers