Languages supported and context length?
#2
by abpani1994 - opened
Is GLiNER2’s context length of 2,048 based on the number of tokens processed by gliner.data_processor.words_splitter(), or does it use a normal split(' ') word count?
I can see that the vocab size is 250k does that mean it can support 100+ languages like BGE M3?
abpani1994 changed discussion title from Languages supported to Languages supported?
abpani1994 changed discussion title from Languages supported? to Languages supported and context length?
@abpani1994 , have you found an answer for the first question? (The second is answered here: https://huggingface.co/fastino/gliner2-multi-v1/discussions/1#693ac86818cb030278ed8a4c).