RedHatAI

/

Qwen3-Next-80B-A3B-Instruct-quantized.w4a16

like 0

Follow

Red Hat AI 1.77k

2 * L40s

vllm == 0.13.0rc2.dev6+g434ac76a7

params

python3 -m vllm.entrypoints.openai.api_server \
    --served-model-name qwen3-next-80b-a3b-instruct \
    --model /data/model-cache/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

Tokenizer error

(EngineCore_DP0 pid=325) The tokenizer you are loading from '/data/model-cache/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.