Phi-4-mini-instruct-ShopRAG

A post-trained Phi-4-mini-instruct (3.8B) for edge-device RAG against CNC machine tool documentation and live backup data. Built for FANUC 31i-Model B controllers. Two-stage training: SFT on ~600 grounded examples, then DPO on 102 contrast pairs targeting residual safety failures.

The model answers from retrieved documentation and decoded machine state only. It does not answer from memory. When backup data is absent, it says so instead of guessing.

Model Summary

Developer	Logan LeGrand
Base model	microsoft/Phi-4-mini-instruct
Parameters	3.8B
Context window	128K native, served at 4,096 tokens on edge hardware
Task	Grounded CNC parameter/manual lookup with live machine state fusion
Training method	SFT (Unsloth, LoRA r=64, rsLoRA) → DPO (beta=0.3, rpo_alpha=1.0)
Training hardware	Single RTX 5070 (12GB VRAM), ~15 min SFT + ~100 min DPO

---

Evaluation

148 held-out shop-floor questions, binary scoring on factual correctness and grounding.

Model	Score	Notes
This model (SFT + DPO)	135/148 (91.2%)	0 fabricated machine state claims
Claude Opus 4.8 (cloud)	126/148 (85.1%)	Imports outside FANUC knowledge not in the provided sources
Claude Sonnet 5 (cloud)	115/148 (77.7%)	Same — pre-training knowledge leaks into answers
Phi-4-mini-instruct base (untrained)	112/148 (75.7%)	Generic responses, fails exact table relay

The frontier models fail because they know too much — they inject real-sounding FANUC facts that aren't in the retrieved context and may not apply to the specific controller model or builder customization. The fine-tuned model has no FANUC knowledge to leak; it reads what's in front of it.

Deployed as GGUF via llama.cpp / LM Studio. Two quantizations shipped:

Variant	Size	Score (w/ guard)	Notes
q4_k_m + imatrix (primary)	2.49 GB	132/148	Domain-calibrated importance matrix. Clean verbatim relay.
q8_0 (fallback)	4.08 GB	130/148	Conservative fallback.

Naive q4_k_m (no imatrix) corrupted program lines — the domain-calibrated importance matrix fixed it by telling the quantizer which weights matter most for exact string relay.

---

Corpus

17,649 structured records extracted from six FANUC manual categories using NuExtract3 (NuMind, 4B params, Apache 2.0, Qwen3.5-4B base). NuExtract3 is a template-driven extraction model — you give it a JSON schema and source text, it fills in the fields verbatim. It extracts rather than paraphrases, which matters for technical manuals.

NuExtract3 runs locally via LM Studio (or any OpenAI-compatible server). Each call sends a system prompt with extraction rules and a user prompt containing the source text + a JSON template with empty fields. The model returns the template filled in. For parameter manuals the template includes number, name, data_type, bits (with when_0/when_1 per bit position), and text. For section-shaped manuals (operator, maintenance, PMC) it includes title, section_type, text, steps, and warnings.

PDFs are segmented into blocks (one per parameter or section heading), each block is extracted via NuExtract3 with category-specific prompts, validated for domain constraints (bit range, spurious boundaries, cross-contamination), then embedded into a LanceDB hybrid search index.

Category	Records
NC_Parameters	2,799
PLC_PMC	5,018
Operator	4,490
Maintenance	3,587
Programming	1,011
Other	744
Total	17,649

Record schema

Two shapes — parameter records carry bit-level definitions with 0/1 states, section records carry procedures/warnings/troubleshooting steps. Both use the same flat structure:

{
  "id": "<controller>::param::<number>::<manual_id>::pg<page>",
  "type": "parameter",
  "controller": "<controller_model>",
  "category": "NC_Parameters",
  "number": "9999",
  "name": "Example configuration bits",
  "data_type": "Bit",
  "text": "Bit 4 (EXMx): Example mode select ...",
  "bits": [
    {"bit": 4, "name": "EXMx", "when_0": "Mode A is selected", "when_1": "Mode B is selected"}
  ],
  "source": {"file": "<manual_pdf>", "page": [100, 101]}
}

Backup decoder

CNC backup files (e.g. CNC-PARA.TXT) store parameter values as raw strings. For bit-type parameters, the value is an 8-digit binary string where each digit maps to a bit position (bit 7 is leftmost, bit 0 is rightmost):

Raw backup value for parameter 9999:  00010000
                                      |||||||└─ bit 0
                                      ||||||└── bit 1
                                      |||||└─── bit 2
                                      ||||└──── bit 3
                                      |||└───── bit 4 = 1  ← EXMx
                                      ||└────── bit 5
                                      |└─────── bit 6
                                      └──────── bit 7

The decoder reads the bit value, matches it against the record's when_0/when_1 definitions, and produces a > **Current:** callout:

> **Current:** Bit 4 (EXMx) = 1 — Mode B is selected.

For non-bit parameters (word, real, etc.), the raw value is decoded directly — units are resolved against the machine's actual unit setting (mm vs inch) when the record documents both.

This decode step is done by software at query time, not by the model. The decoded callout is injected into the rendered context block so the model only needs to relay it.

At retrieval time, records are rendered into a fixed markdown shape with definition tables, > **Current:** callouts for live backup values, and > **Changed:** callouts for diff history. The model sees this same shape in every training example.

Note: The training dataset and extracted corpus are not included in this release — they contain proprietary FANUC documentation. To reproduce this approach for your own controllers, you would need to build your own extraction pipeline against your own manuals and backups.

---

Training Data

SFT — 523 train + 8 seed, 71 held-out eval

Every training example was generated through the real retrieval pipeline against real manuals and real backup data, then audited by a grounding verifier that rejected ~10–15% of attempts for fabrication. Questions are written the way machinists talk — terse, conversational, sometimes misspelled.

Category	Train	What it teaches
parameter_lookup	100	Relay documented definition + live value
refusal_backup_off	45	Give docs, refuse to state machine settings without backup
refusal_manual_off	45	Relay raw value, refuse to interpret without manual
multi_turn	55	Follow-up narrowing within shown context
operator_procedures	40	Step-by-step relay with verbatim warnings
bit_narrowed	40	Answer about one specific bit, not the full table
custom_macro	30	Code→program mapping, #variable identity
disambiguation	25	List candidates when query matches multiple records
machine_profile	25	Read from software-decoded machine config
alarm_lookup	20	Relay alarm cause + corrective actions
plc_reference	20	Relay typed signal/address entries
distractors_only	20	Say "not documented" when retrieval returns near-misses only
unit_disambiguation	20	Resolve mm/inch using machine's actual unit setting
layer_hierarchy	15	Priority ordering when multiple doc layers present
off_topic	15	Normal reply to non-CNC questions

Refusal categories make up ~18% of training — deliberately overweighted. An early pilot showed safety behaviors need dense repetition to stick.

DPO — 102 pairs

Targeted the failure modes SFT left behind. The main one: reading a documentation table and presenting it as live machine state when no backup was in context. After DPO, that failure was gone across all test runs.

Source	Pairs	Target
On-policy mining	30	Fabricated live-state claims, G/M-code conflation
Typo corruption	60	Digit transposition errors (2728→2278)
Verifier rejects	12	Ungrounded claims in connective prose

---

Prompt Template

Standard Phi-4 chat format. The **Active:** line tells the model what sources are present so it never has to guess.

<|system|>You are the CNC documentation and machine-state assistant for this shop.

Answer using only the documentation and machine data provided below — never invent parameter behavior, bit meanings, or machine settings that aren't shown to you.

If the information needed to answer isn't in what's provided, say so plainly, and name which source category the user could enable for a fuller answer.<|end|><|user|>**Active:** Parameter manual [✓/✗] · CNC-PARA backup [✓/✗]

[Rendered context blocks]

## Question
[User question]<|end|><|assistant|>

Example: backup active

<|user|>**Active:** Parameter manual ✓ · CNC-PARA backup ✓

### Parameter 9999 — Example configuration bits
**Layer:** Controller · **Type:** Bit path

| Bit | Name | Value 0 | Value 1 |
|---|---|---|---|
| 4 | EXMx | Mode A is selected | Mode B is selected |

> **Current:** Bit 4 (EXMx) = 1 — Mode B is selected.

*Source: Example Parameter Manual, p.100*

## Question
What's parameter 9999 bit 4 set to on this machine?<|end|><|assistant|>

Response:

Parameter 9999 bit 4 (EXMx) is set to **1** on this machine — Mode B is selected. When set to 0, Mode A would be selected instead.

Example: backup inactive (refusal)

<|user|>**Active:** Parameter manual ✓ · CNC-PARA backup ✗

### Parameter 9999 — Example configuration bits
[same documentation table, no Current: callouts]

## Question
Is Mode B active on this machine right now?<|end|><|assistant|>

Response:

Parameter 9999 bit 4 (EXMx) controls whether Mode B is active (1) or Mode A (0). I can't tell you the current setting — the CNC-PARA backup isn't active. Enable it to check the live value.

---

Inference

Sampler settings (critical)

Pin these exactly. Default repeat_penalty 1.1 in LM Studio / llama.cpp corrupts exact table relays and program-line reproduction.

Setting	Value
Temperature	0.0
Repeat penalty	1.0 (disabled)
Top_K	0
Min_P	0

llama-server

llama-server.exe -m phi4mini-shoprag-4bb-q4km-imx.gguf --port 8765 -ngl 99 --parallel 1 -c 4096 --no-webui

Python (local server API)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8765/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="phi4mini-shoprag-4bb",
    messages=[
        {"role": "system", "content": "You are the CNC documentation and machine-state assistant for this shop.\n\nAnswer using only the documentation and machine data provided below — never invent parameter behavior, bit meanings, or machine settings that aren't shown to you.\n\nIf the information needed to answer isn't in what's provided, say so plainly, and name which source category the user could enable for a fuller answer."},
        {"role": "user", "content": "**Active:** Parameter manual ✓ · CNC-PARA backup ✗\n\n### Parameter 5006 — Tool length compensation bits\n..."}
    ],
    temperature=0.0,
    extra_body={"repeat_penalty": 1.0, "top_k": 0, "min_p": 0},
    max_tokens=800
)

print(response.choices[0].message.content)

---

Hardware

Component	Size
Model (q4_k_m + imatrix GGUF)	2.5 GB
Manual index (17,649 records)	~0.2 GB
Embedding model (nomic-embed-text-v2-moe)	~0.6 GB
Machine backups	A few MB

~3.3 GB total. Runs on a single GPU with 8+ GB VRAM. CPU-only works as a slower fallback. No cloud, no internet, no subscription. Response time ~2–3 seconds on a mid-range GPU.