[SOLVED] No instruction following, Model just outputs vaguely relevant text, or goes into loops
Both the V3.2 and V3.2-Speciale 4-bit quants do not follow instructions and just behave like text "autocomplete". Or it goes into a loop. Prompt of "Hello" just keeps repeating the word "Hello". My own quants do exactly the same.
EDIT: Summarized
Example:
mlx_lm.generate --model mlx-community_DeepSeek-V3.2-4bit --prompt "Translate from English into French: Hi there"
, I’m a student from the UK. I’m studying French at university. I’m going to France next year to study. I’m going to live in Paris for six months. I’m going to study at the Sorbonne. I’m going to live in a student residence. I’m going to have a room with a view of the Eiffel Tower. I’m going to have a lot of fun. I’m going to meet a
EDIT: Show relevant example
Need the latest mlx-lm 0.30.0, the tokenizer_config,json and tokenizer.json from here (these were manually updated), as well as the updated code from the GitHub pull request #693.
EDIT: Describe solution
I still can't get this model to work properly. The model's outputs are nonsense. What am I missing?
There has been more work done on this model the last few days.
Even yesterday one of the files in this model was updated.
- Download the latest
tokenizer_config.jsonof yesterday for this model. - Download and use the latest code
mlx-lmfrom GitHub. The latestpypipackage does not contain all the latest code changes.
For more help, see https://huggingface.co/mlx-community/GLM-4.6V-4bit/discussions/1#693c9b17aaa28c7276bdd747 to see how to use the latest code mlx-lm from GitHub (in that case it was for mlx-vlm instead of mlx-ml)