From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
Abstract
We present ELMOD - Efficient Language Model for On-Device Deployment - a compact (2.7B) German language model designed for efficient inference on resource-constrained hardware. ELMOD was trained on a limited computational budget (55k H100 GPU hours) using exclusively publicly available data. We developed a suite of German-specific data pre-processing, which differ from English-oriented counterparts in their handling of morphological variation, compounding, and orthographic conventions. Furthermore, we introduced a quality filtering and rephrasing step, which increased the instructional quality of the data, improved performance during the annealing phase, and reduced overall compute requirements. Thanks to our architectural model and data choices, including prefiltering, our educational-quality filtering and rephrasal to raise the educational-quality, ELMOD is the strongest performer in its size class (<3B), matching the performance of 7B-parameter models in German.
Community
The released checkpoint is GPTNeoXForCausalLM, but apart from the auto-generated HF tag this is mentioned nowhere in the model card text or the paper, and the only architecture source you cite is your own prior study. Why?
Hey guys, one question about the paper:
We furthermore normalize the text based on the Normalization Form Compatibility Composed (NFKC)12
form and strip away all accents (for details see Appendix B).
I don't see accent stripping:
In [4]: tokenizer = AutoTokenizer.from_pretrained("fraunhofer-iis/elmod-2.7b-base")
In [5]: tokenizer.tokenize("öäüß ÖÄÜ ß")
Out[5]: ['ö', 'ä', 'üß', '▁Ö', 'Ä', 'Ü', '▁', 'ß']
In [6]: tokenizer.backend_tokenizer.normalizer
Out[6]: Sequence(normalizers=[NFKC(), Replace(pattern=String("▁"), content="_")])
In [7]: tokenizer.backend_tokenizer.normalizer.normalize_str("o\u0308 a\u0308 u\u0308")
Out[7]: 'ö ä ü'
And accent stripping would also be a very bad idea for German.
Hi Stefan,
Viktor here, I trained the tokenizer. You are right, the accents are actually not stripped. Thank you for catching that. Maybe to elaborate a bit further, the original idea was to strip all accents except umlauts, as we mainly focused on German and English.
It's a bit difficult to track down what exactly happened here, as the training itself was finished about half a year ago and I left Fraunhofer shortly afterwards, so I don't have access to the original data any longer. But I believe I simply misremembered when I wrote this section. AFAIK, there is no easy way to remove accents without touching the umlauts in the tokenizers library. I assume that's why the original idea was discarded.
Thanks again for pointing this out. I'll ask my colleagues to upload an amended version of the paper.
The architecture section has the same kind of mismatch, the paper describes grouped-query attention and no bias terms, but the released config (GPTNeoXForCausalLM) has attention_bias: true and no num_key_value_heads, i.e. plain multi-head attention with biases.
Models citing this paper 2
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper