Paritok-4B-v1

The first open-source compression model trained specifically for coding agents.
Compresses individual long files to as little as 5% of their original size (typical content compression rate 25.7%), cutting end-to-end input token bills from ~25% on turn one to past 85% in context-saturated sessions — while retaining 86.5% of full-context solve quality on SWE-bench Lite.

---

🔥 TL;DR

~74% Content compression Typical CR = 25.7% Peak 95% on long files

86.5% Solve quality retained SWE-bench Lite Matches gpt-4.1-mini

4B Self-hostable LoRA Single 24GB GPU Apache 2.0, zero fee

---

---

✨ Why Paritok?

• 🎨 Code-native. Trained end-to-end on 45K real agent trajectories (file_read, bash_command, log_output...). Preserves function names, imports, paths, and error strings while compressing.
• 🚀 ~74% content compression on typical workloads, peaking at 95% on long files.
• 💰 End-to-end bill savings compound with session length — from ~25% at turn one to past 85% in context-saturated sessions on Claude Sonnet / GPT-4. Long-session teams save thousands per month.
• 🎯 86.5% of full-context solve quality retained on SWE-bench Lite — matching gpt-4.1-mini as compressor at less than half the token spend.
• 🪶 Small, fast, self-hostable. 4B LoRA adapter, bf16, single 24GB GPU. No SaaS, no lock-in, no per-token fee.
• 🔓 Fully open. Apache 2.0 weights, reproducible pipeline, real end-to-end benchmarks.

---

🚀 Quick Start

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
ADAPTER    = "paritok/paritok-4b-v1"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()
# Wrap your code as [SEG id=1 kind=file_read] ... [/SEG]
user_msg = "[SEG id=1 kind=file_read]\n<your code here>\n[/SEG]"
prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "<full system prompt — see GitHub>"},
     {"role": "user",   "content": user_msg}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

📎 Full runnable example + system prompt + reproduction pipeline: GitHub → Paritok-official/paritok-4b-v1

---

📊 SWE-bench Lite — Head-to-head

Context source	Quality retained ¹	Compression rate
Uncompressed baseline	100.0%	100.0%
gpt-4.1-mini (compressor)	85.6%	50.2%
gpt-5 (compressor)	93.6%	61.9%
Paritok-4B-v1 ⭐	86.5%	25.7%

_{¹ Quality retained = compressor solve rate ÷ uncompressed baseline solve rate. Higher is better.}

Paritok compresses ~2× harder than gpt-4.1-mini on the same task while keeping the same solve rate — the only open-source entry trained end-to-end on real coding-agent trajectories.

---

💰 Cost Impact (Claude Sonnet, $3/M input tokens)

Turn input size	Uncompressed input	With Paritok
Short (8K)	$0.024	$0.006
Typical (15K)	$0.045	$0.012
Long session (30K)	$0.090	$0.023

10-person team, 3-month project → save ~$10K on API bills.

---

📋 Model details (click to expand)

Property	Value
Base model	Qwen/Qwen3-4B-Instruct-2507
Adapter type	LoRA, r=32, α=64, dropout=0.0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training steps	2000 (selected from a 5-checkpoint sweep on OOD holdout)
Training precision	bf16
Effective batch	32 (per_device=2 × grad_accum=16)
Learning rate	1e-5, linear decay, 10% warmup
Optimizer	AdamW (8-bit)
Max seq length	16 384
Dataset size	45 000 samples (file_read, bash_command, log_output, ...)
Teacher	gpt-4.1-mini

⚠️ Limitations (click to expand)

1. ~6pp accuracy trade for 74% compression. Not free — buying context length with a small solve-rate drop. Bring your own uncompressed-fallback path for safety-critical turns.
2. Rare identifier loss (~40% preserved on hard OOD segments). Add a post-compression check that your target identifier is still in the compressed output before sending upstream.
3. English source code, Python-heavy training distribution. Non-English comments/strings and other-language codebases have not been benchmarked.

Recommended safeguards for production: format check, target-identifier presence check, and a session-level toggle to fall back to raw context on failure.

🗺️ Roadmap (click to expand)

• Paritok-4B-v2 — Next-generation training pipeline pushing compression to under 20% while closing the gap to uncompressed solve rate.
• Frontier-scale backbones (10B+ parameters) for multi-day sessions with 100K+ token histories.
• Multi-language expansion — TypeScript, Rust, Go, Java, C++, Kotlin.
• Native integrations — mcp add paritok plugin for Claude Code and Cursor, plus a hosted inference endpoint.

---

📖 Citation

Cite the paper for the method and results:

@misc{paritok4b,
  title         = {Paritok-4B: Intent-Conditioned Context Compression for Coding Agents},
  author        = {Shi, Jiayu and Chen, Luzhuo},
  year          = {2026},
  eprint        = {2608.24188},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2608.24188},
  url           = {https://arxiv.org/abs/2608.24188},
}

Cite the software for this release — the weights, the gateway, and the training and evaluation code:

@software{paritok-4b-v1,
  title   = {Paritok-4B-v1: An Open-Source Compression Model for AI Coding Agents},
  author  = {Shi, Jiayu and Chen, Luzhuo},
  year    = {2026},
  version = {4B-v1},
  license = {Apache-2.0},
  url     = {https://github.com/Paritok-official/paritok-4b-v1},
}

---

📄 License

Apache 2.0 — see LICENSE. Base model (Qwen3-4B-Instruct-2507) is under its own license; please review before commercial deployment.

Paper · GitHub · paritok.com · Contact