Here are some good sources for high-quality preference/RLHF datasets for alignment:

## Dedicated Repositories

- **[Hugging Face Hub](https://huggingface.co/datasets?search=preference)** — Search for "preference", "RLHF", or "reward" datasets. Largest open collection.
- **[OpenDataLab](https://opendatalab.com/)** — Mirrors many alignment datasets with easy access.

## Notable Datasets

### Human Preference / Reward Modeling
| Dataset | Description |
|---|---|
| **Anthropic HH-RLHF** | Human helpfulness & harmlessness preference pairs |
| **OpenAI WebGPT Comparisons** | Human preferences on web-assisted QA |
| **OpenAI Summarize from Feedback** | Preference data for summarization |
| **Stanford SHP** | Reddit-based human preference data (upvotes as proxy) |
| **UltraFeedback** | GPT-4 scored preferences across many models |
| **Capybara / OpenHermes** | High-quality instruction + preference mixes |
| **Nectar** | Multi-turn, multi-source ranked preferences |

### Constitutional AI / RLAIF
| Dataset | Description |
|---|---|
| **PKU-SafeRLHF** | Safety-focused preference pairs |
| **BeaverTails** | Harmlessness-focused with fine-grained labels |
| **HelpSteer / HelpSteer2** (NVIDIA) | Multi-attribute scored preferences |

## Academic / Research
- **[allenai/reward-bench](https://huggingface.co/datasets/allenai/reward-bench)** — Benchmark + data for evaluating reward models
- **[lmsys/chatbot_arena_conversations](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations)** — Real human head-to-head model comparisons

## Tips for Selection
1. **Check recency** — many datasets use older models (GPT-3.5 era); newer ones use GPT-4 or Claude 3
2. **Verify annotation quality** — look for inter-annotator agreement stats
3. **Domain match** — general vs. code vs. math vs. safety-specific
4. **License** — some have restrictive commercial terms (e.g., OpenAI datasets)

The **Hugging Face Hub** with the filter `task_categories:reinforcement-learning-from-human-feedback` is probably the most comprehensive single starting point.


What's the best small language model under 3B parameters?


Based on current benchmarks and community consensus, here are some of the best small language models under 3B parameters:

## Top Picks (Under 3B Parameters)

### 🥇 **Qwen2.5-1.5B / Qwen2.5-3B** (Alibaba)
- Exceptionally strong for their size, especially on coding and math
- Qwen2.5-3B punches well above its weight class
- Great multilingual support

### 🥈 **Gemma 3 1B / 2B** (Google)
- Gemma 3 2B is one of the best sub-3B models available
- Strong reasoning, instruction following, and multimodal capability (2B variant)
- Excellent efficiency for on-device use

### 🥉 **Phi-3.5-mini (3.8B) / Phi-4-mini (3.8B)** (Microsoft)
- Technically slightly over 3B, but worth mentioning
- Phi-3 Mini at 3.8B is phenomenally capable for its size

### **SmolLM2-1.7B** (Hugging Face)
- Purpose-built for on-device/edge deployment
- Surprisingly capable for everyday tasks at very low resource cost

## Quick Comparison

| Model | Params | Strengths |
|---|---|---|
| Qwen2.5-3B | 3B | Coding, math, multilingual |
| Gemma 3 2B | 2B | General tasks, multimodal |
| Qwen2.5-1.5B | 1.5B | Best ~1.5B overall |
| SmolLM2-1.7B | 1.7B | Edge/on-device efficiency |

## My Recommendation
**Qwen2.5-3B** is arguably the best at the 3B ceiling, while **Gemma 3 2B** is the best if you want something from a major lab with broad capability. For truly tiny deployments, **SmolLM2-1.7B** is a great choice.

> ⚠ Note: This is a fast-moving space — always check the latest [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) on Hugging Face for up-to-date rankings.


Best model for named entity recognition in production


Here's a breakdown of the **best models for Named Entity Recognition (NER) in production** as of April 2026:

---

## 🏆 Top NER Models for Production

### 1. **GLiNER** *(Best overall for flexibility)*
- Zero-shot NER — no task-specific fine-tuning needed
- Handles arbitrary entity types at inference time
- Efficient, small footprint (BERT-sized)
- Great for production when entity types evolve over time
- **Best for:** Dynamic entity schemas, low-maintenance pipelines

---

### 2. **spaCy (with `en_core_web_trf`)** *(Best for production reliability)*
- Battle-tested, production-grade NLP library
- Transformer-backed pipeline (RoBERTa underneath)
- Fast, well-documented, easy to deploy
- Excellent CPU/GPU support, REST-API-friendly
- **Best for:** Ops teams that want stability, speed, and ecosystem maturity

---

### 3. **BERT / RoBERTa fine-tuned for NER** *(Best accuracy on known entity types)*
- Models like `dslim/bert-base-NER` or `Jean-Baptiste/roberta-large-ner-english` on HuggingFace
- State-of-the-art F1 on CoNLL-2003 and similar benchmarks
- Fine-tunable on domain-specific data
- **Best for:** High-accuracy needs with a fixed entity taxonomy (PER, ORG, LOC, MISC)

---

### 4. **Flair NLP** *(Best for stacking contextual embeddings)*
- Contextual string embeddings + sequence labeling
- Strong benchmark scores, especially multilingual
- Slightly heavier than spaCy but very accurate
- **Best for:** Research-to-production teams who want SOTA accuracy

---

### 5. **LLM-based NER (Claude, GPT-4, Mistral, etc.)** *(Best for complex/nuanced entities)*
- Prompt-based extraction; no training needed
- Handles ambiguous, nested, or exotic entity types extremely well
- Higher latency and cost, but unmatched flexibility
- Use structured outputs (JSON mode) for reliability
- **Best for:** Low-volume, high-complexity extraction; rapid prototyping

---

## ⚖ Decision Matrix

| Need | Recommended Model |
|---|---|
| Fast & reliable, fixed entity types | **spaCy trf** or **BERT-NER** |
| New/dynamic entity types, no training | **GLiNER** |
| Highest accuracy, willing to fine-tune | **RoBERTa-large-NER** |
| Complex reasoning about entities | **LLM (Claude/GPT-4)** |
| Multilingual support | **Flair** or **mBERT/XLM-R** |
| Edge / CPU-only deployment | **spaCy sm/md** or **GLiNER-small** |

---

## 💡 Production Tips

- **Quantize** BERT-family models (INT8 via ONNX Runtime) for 3–4× speedup with minimal accuracy loss
- Use **batching** to maximize GPU throughput
- Pair any model with **post-processing rules** (regex, gazetteers) to catch domain-specific entities
- For critical pipelines, **ensemble** a fast model (spaCy) with a slow LLM as a fallback verifier

---

**TL;DR:** For most production use cases, **GLiNER** or **spaCy + transformer** hits the best accuracy/speed/maintainability tradeoff. If you need to handle unpredictable entity types without retraining, **GLiNER** is the standout pick in 2026.

Want a recommendation tuned to your specific domain (medical, legal, finance, etc.)?


What's the state of the art for text-to-speech models right now?


## State of the Art in Text-to-Speech (April 2026)

The TTS landscape has advanced rapidly. Here's a breakdown of where things stand:

---

### 🏆 Top Proprietary / API Models

- **Fish Audio S2** (March 2026) — Currently considered the best overall for most users. Leads on voice quality, emotion control, and multilingual voice cloning. Introduced *word-level voice direction* using plain-language inline tags.
- **ElevenLabs Eleven v3** (February 2026) — Still a top contender, supporting 70+ languages, multi-speaker dialogue, audio tags, and "speech-to-speech" with emotional control (whisper, shout, laugh on command).
- **Hume AI TADA** (March 2026) — "Text-Acoustic Dual Alignment," notable for claiming **zero hallucinations** across 1,000+ test samples. A significant reliability breakthrough.
- **Microsoft VibeVoice-1.5B** — Optimized for long-form, expressive, multi-speaker audio, capable of up to **90 minutes of speech** with four distinct speakers in a single generation.
- **OpenAI** — Continues to offer next-generation audio models via API, though Fish Audio and ElevenLabs are currently beating them on voice quality benchmarks.
- **Amazon Polly** — Enterprise staple with 100+ voices in 40+ languages; now includes a "Generative" voice tier at $30/1M characters.

---

### 🔓 Best Open-Source Models

| Model | Strengths |
|---|---|
| **Kokoro-82M** | Breakout star — tiny (82M params), runs on consumer GPUs/Apple Silicon, achieves 4.5 MOS (highest tested), 17% CER |
| **Fish Speech v1.5** | Best open-source for multilingual quality and benchmark performance |
| **CosyVoice2-0.5B** (FunAudioLLM) | Best for real-time streaming, ~150ms latency |
| **IndexTTS-2** | Leads on word error rate, speaker similarity, and emotional fidelity in zero-shot cloning |
| **ChatTTS** | Conversational use cases; trained on ~100K hours of Chinese + English |

---

### 📌 Key Trends
- **Zero-shot voice cloning** is now near-universal — clone any voice from a short sample.
- **Emotional and prosodic control** (via tags or direction) is the new battleground.
- **Lightweight local models** (Kokoro) are closing the quality gap with large cloud APIs.
- **Hallucination reduction** is emerging as a key reliability metric (Hume AI TADA).
- **Long-form generation** (Microsoft VibeVoice) is becoming viable for audiobooks, podcasts, etc.

---

Sources:
- [The Best Open-Source Text-to-Speech Models in 2026 – BentoML](https://bentoml.com/blog/exploring-the-world-of-open-source-text-to-speech-models)
- [Best TTS Models in 2026: Ranked & Compared – BeFreed](https://www.befreed.ai/blog/best-tts-models-2026)
- [Best TTS Model 2026: Top 9 AI Voice Generators Ranked – BeFreed](https://www.befreed.ai/blog/best-tts-model-2026)
- [Open-Source vs Proprietary TTS Models 2026 – Geeky Gadgets](https://www.geeky-gadgets.com/text-to-speech-tts-ai-models/)
- [The Ultimate Guide to AI Text-to-Speech (2026) – FatCow Digital](https://fatcowdigital.com/blog/ai-topics/ai-text-to-speech-guide-2026/)
- [Top Text-to-Speech (TTS) Models in 2026 – Trelis Research](https://trelis.substack.com/p/top-text-to-speech-tts-models-in)
- [Introducing next-generation audio models in the API – OpenAI](https://openai.com/index/introducing-our-next-generation-audio-models/)


Which model is best for zero-shot classification?


Your question isn't about building a Claude API app, so the `claude-api` skill doesn't apply here. Let me answer directly:

## Best Model for Zero-Shot Classification

For zero-shot classification tasks, the best choice depends on your priorities:

### **Best accuracy: Claude Opus 4.6** (`claude-opus-4-6`)
- Most intelligent model, best at nuanced classification without examples
- Ideal when accuracy matters most

### **Best balance of speed + accuracy: Claude Sonnet 4.6** (`claude-sonnet-4-6`)
- Strong classification performance at lower cost/latency
- Good default for most production classification pipelines

### **Best for high-volume / cost-sensitive: Claude Haiku 4.5** (`claude-haiku-4-5`)
- Fastest and cheapest
- Works well for straightforward classification tasks (sentiment, topic, intent)

---

## General Recommendation

| Use Case | Model |
|---|---|
| Complex/nuanced categories, edge cases | Opus 4.6 |
| Standard production classification | Sonnet 4.6 |
| Simple labels, high throughput | Haiku 4.5 |

For **bulk classification**, consider pairing any of these with the **Batches API** (50% cost reduction) since classification is typically non-latency-sensitive.

If you're asking about **non-Claude models** (e.g., open-source models like BART, DeBERTa, or SetFit for zero-shot classification), let me know and I can give guidance on those instead.


Best model for multi-turn conversational AI


For multi-turn conversational AI, here are the best models as of April 2026:

## Top Recommendations

### 1. **Claude 3.7 Sonnet** *(Anthropic)*
- Excellent at maintaining context across long conversations
- Strong reasoning, nuanced responses, and instruction-following
- Best balance of **speed, cost, and quality**
- Ideal for most conversational applications

### 2. **Claude 3.5 Opus** *(Anthropic)*
- Maximum capability for complex, high-stakes conversations
- Superior context retention and coherent multi-turn reasoning
- Higher cost — best when quality is paramount

### 3. **GPT-4o** *(OpenAI)*
- Strong multi-turn performance with multimodal support
- Good tool/function calling for agentic conversations
- Well-supported ecosystem

---

## Key Factors for Multi-Turn Conversational AI

| Factor | What to Look For |
|---|---|
| **Context window** | Larger = better memory (Claude: 200K tokens) |
| **Instruction following** | Stays on task across many turns |
| **Latency** | Matters for real-time chat UX |
| **Cost** | Scales with conversation length |
| **Tool use** | Needed for agentic/action-taking bots |

---

## My Pick

**Claude 3.7 Sonnet** is the best overall choice for most multi-turn conversational applications — it handles long context well, follows system prompts reliably, and is fast enough for interactive use.

If you're building with the Claude API, I can help you set that up — just ask!


What's the best open-source model for multilingual translation?


Here's a breakdown of the best open-source models for multilingual translation as of 2026:

---

## 🏆 Top Open-Source Multilingual Translation Models

### 1. **Qwen3-235B-A22B** (Alibaba)
The current top pick for broad multilingual coverage. It supports **100+ languages**, uses a Mixture-of-Experts (MoE) architecture (235B total / 22B active parameters), and excels particularly at Chinese, Japanese, Korean, and other Asian languages alongside European ones. Best for maximum language coverage.

### 2. **Meta Llama 3.1 (8B / 70B Instruct)**
A strong all-rounder, especially for European languages. The 8B variant is efficient and cost-effective for deployment, while the 70B version delivers higher accuracy. Great for general-purpose multilingual translation.

### 3. **Qwen2.5-VL-7B-Instruct**
Best for **visual/multimodal translation** — can translate text in images (signs, menus, documents, screenshots). Ideal for real-time or on-device use cases.

### 4. **StepFun Step3**
A strong option if you need **multimodal translation** (text + visual understanding combined).

---

## 🔑 Key Takeaways

| Need | Best Model |
|------|-----------|
| Max language coverage | Qwen3-235B-A22B |
| Efficient/self-hosted | Llama 3.1-8B-Instruct |
| Visual/image translation | Qwen2.5-VL-7B-Instruct |
| Asian language pairs | Qwen 2.5 / Qwen 3 family |
| Multimodal | StepFun Step3 |

For most use cases, **Qwen3** or **Llama 3.1** are the go-to choices depending on whether you prioritize language breadth or deployment efficiency.

---

Sources:
- [The Best Open Source Models for Translation in 2026](https://www.siliconflow.com/articles/en/best-open-source-models-for-translation)
- [Best Open Source Models for Multilingual Tasks in 2026](https://www.siliconflow.com/articles/en/best-open-source-models-for-multilingual-tasks)
- [Best LLM for Translation in 2026 (Tested & Ranked)](https://nutstudio.imyfone.com/llm-tips/best-llm-for-translation/)
- [Best Open-Source AI for Real-Time Translation in 2026](https://www.siliconflow.com/articles/en/best-open-source-AI-for-real-time-translation)


Best open-weight model for medical/clinical NLP tasks


## Best Open-Weight Models for Medical/Clinical NLP (2026)

Here's a breakdown by use case and capability:

---

### 🏆 Top Contenders

| Model | Best For | Notes |
|---|---|---|
| **Medical-LLM-78B** | Clinical QA, benchmarks | 94.72% on medical benchmarks — outperforms GPT-4 (86%); top specialized medical model |
| **Me-LLaMA** | General clinical NLP | Zero-shot + instruction-tuned; surpasses ChatGPT/GPT-4 on text analysis after fine-tuning |
| **DeepSeek-R1** (671B / distills) | Complex clinical reasoning | 164K context window; excellent for differential diagnosis & multi-step reasoning |
| **GPT-OSS-120B** (OpenAI) | Broad health tasks | MoE design, fits on single 80GB GPU; strong on health/reasoning benchmarks |
| **GLM-4.5V** | Medical imaging + text | Vision-language; strong on radiology, pathology, clinical charts |

---

### 🔬 Purpose-Built Medical Models

- **John Snow Labs' Medical LLMs** — purpose-trained for clinical NLP pipelines (NER, de-identification, ICD coding, clinical summarization); great for enterprise healthcare NLP workflows
- **Me-LLaMA (13B / 70B)** — open-source, built on LLaMA, fine-tuned on large medical corpora; excellent for text classification, extraction, and summarization tasks

---

### 📊 Key Benchmarks to Reference
- **Open Medical-LLM Leaderboard** (HuggingFace) — tracks performance on MedQA, MedMCQA, PubMedQA, etc.
- **CSEDB** — safety + effectiveness focused (30 metrics, 26 departments)
- **LLMEval-Medicine** — real-world EHR-based questions, physician-validated

---

### 💡 Practical Recommendations

| Task | Best Choice |
|---|---|
| NER, de-identification, ICD coding | John Snow Labs Medical LLMs |
| Clinical QA / USMLE-style | Medical-LLM-78B or Me-LLaMA |
| Radiology report analysis | GLM-4.5V |
| Complex reasoning / differential dx | DeepSeek-R1 |
| General-purpose + deployable | GPT-OSS-120B or DeepSeek-R1 distills |

---

**Bottom line:** For pure clinical NLP tasks (extraction, classification, summarization), **Me-LLaMA** and **John Snow Labs' stack** are the most purpose-fit. For reasoning-heavy tasks, **DeepSeek-R1** or **Medical-LLM-78B** lead on benchmarks.

---

Sources:
- [Best Open Source LLMs for Medical Industry in 2026 – SiliconFlow](https://www.siliconflow.com/articles/en/best-open-source-LLM-for-medical-industry)
- [Medical Language Models in 2026: Enterprise Guide – Picovoice](https://picovoice.ai/blog/medical-language-models-guide/)
- [Medical Foundation LLMs for Comprehensive Text Analysis – npj Digital Medicine](https://www.nature.com/articles/s41746-025-01533-1)
- [Open Medical-LLM Leaderboard – HuggingFace](https://huggingface.co/blog/leaderboard-medicalllm)
- [John Snow Labs – Medical LLMs](https://nlp.johnsnowlabs.com/docs/en/LLMs/medical_llm)
- [Clinical Information Extraction with Open-Source LLMs – JAMIA Open](https://academic.oup.com/jamiaopen/article/8/5/ooaf109/8270821)
- [LLMEval-Med: Real-world Clinical Benchmark – OpenReview](https://openreview.net/forum?id=UkEQ5TgEN2)
- [Benchmarking LLM-based Information Extraction for Medical Documents – medRxiv](https://www.medrxiv.org/content/10.64898/2026.01.19.26344287v1.full)


What open-source models support long context windows over 128k tokens?


Here are notable open-source models with context windows over 128k tokens:

## Very Long Context (1M+ tokens)
- **Gemma 3** (27B) — 128k... actually 1M token context
- **Qwen2.5** series — up to **1M tokens** (7B–72B variants)
- **Kimi** (Moonshot AI) — 1M token context

## 512k–1M tokens
- **Yi-34B-200K** — 200k tokens
- **MiniCPM-3** — up to 1M (via compression techniques)

## 128k–256k tokens
- **Llama 3.1 / 3.2 / 3.3** — **128k tokens** (8B, 70B, 405B)
- **Mistral Large / Mistral Nemo** — 128k tokens
- **Mixtral 8x22B** — 64k (just under, notable anyway)
- **DeepSeek-V2 / V3 / R1** — **128k tokens**
- **Qwen2.5-Coder** — 128k tokens
- **Phi-3 / Phi-4** (Microsoft) — 128k tokens
- **Command R / Command R+** (Cohere) — 128k tokens
- **InternLM2.5** — up to 1M (with special techniques), natively 32k+
- **GLM-4** — 128k tokens

## Practical Notes
- **Effective context** often differs from advertised max — many models degrade in quality ("lost in the middle" problem) at very long ranges
- **Qwen2.5** and **Llama 3.1** are among the most widely used and well-tested in the 128k range
- Models like **Jamba** (AI21) combine SSM+Transformer for efficient long context

Would you like recommendations filtered by use case (local inference, coding, RAG, etc.)?


Best open-source alternative to DALL-E for image generation


## Best Open-Source Alternatives to DALL-E

Here are the top open-source image generation models/tools:

### 🥇 Top Picks

1. **Stable Diffusion** (by Stability AI)
   - Most popular & widely used
   - Multiple versions: SD 1.5, SD 2.1, **SDXL**, **SD 3.5**
   - Runs locally on consumer GPUs
   - Huge ecosystem of fine-tuned models (via [CivitAI](https://civitai.com))
   - [github.com/Stability-AI/stablediffusion](https://github.com/Stability-AI/stablediffusion)

2. **FLUX** (by Black Forest Labs)
   - State-of-the-art quality, arguably beats Midjourney
   - Models: `FLUX.1-schnell` (fastest, Apache 2.0), `FLUX.1-dev` (non-commercial)
   - Excellent prompt adherence & text rendering
   - [github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux)

3. **Kandinsky** (by Sber AI)
   - Strong multilingual prompt support
   - Good aesthetic quality
   - [github.com/ai-forever/Kandinsky-3](https://github.com/ai-forever/Kandinsky-3)

---

### 🖥 Best UIs / Frontends

| Tool | Description |
|------|-------------|
| **ComfyUI** | Node-based workflow, very powerful |
| **Automatic1111 (AUTOMATIC1111/stable-diffusion-webui)** | Most feature-rich WebUI |
| **InvokeAI** | Polished, user-friendly UI |
| **Fooocus** | Simplified Midjourney-like experience |

---

### ☁ Hosted Open-Source Options
- **Hugging Face Spaces** — run models free in-browser
- **Replicate** — pay-per-use API for open models

---

### 🏆 Recommendation

| Use Case | Best Choice |
|----------|-------------|
| Best quality | **FLUX.1-dev** or **SD 3.5 Large** |
| Fastest/free commercial use | **FLUX.1-schnell** |
| Largest community & LoRA support | **Stable Diffusion XL** |
| Easiest local setup | **Fooocus** |

**FLUX.1** is currently the closest open-source rival to DALL-E 3 in terms of image quality and prompt following.


Which open-source speech recognition model is most accurate?


Based on current knowledge (as of early 2026), **OpenAI's Whisper** (specifically **Whisper Large v3**) is widely regarded as the most accurate open-source speech recognition model overall.

## Top Contenders

| Model | Developer | Notes |
|---|---|---|
| **Whisper Large v3** | OpenAI | Best overall accuracy, 99 languages |
| **Whisper Large v3 Turbo** | OpenAI | Nearly same accuracy, ~8x faster |
| **SeamlessM4T v2** | Meta | Strong multilingual + translation |
| **MMS (Massively Multilingual Speech)** | Meta | 1,100+ languages, lower per-language accuracy |
| **NVIDIA Canary-1B** | NVIDIA | Competitive on English/multilingual benchmarks |
| **Distil-Whisper** | HuggingFace | Distilled Whisper, fast with minimal accuracy loss |

## Key Takeaways

- **Whisper Large v3** consistently tops benchmarks (WER on LibriSpeech, Common Voice, etc.)
- For **English-only** tasks, some smaller specialized models can match or beat it
- For **multilingual** use, Whisper and SeamlessM4T are the strongest choices
- Speed vs. accuracy tradeoffs matter — **Whisper Large v3 Turbo** is often the practical winner

## Caveats

- Accuracy varies heavily by **language**, **accent**, **audio quality**, and **domain**
- Newer models may have emerged — the field moves fast
- Commercial APIs (like Deepgram, AssemblyAI) sometimes outperform open-source on specific benchmarks

For most use cases, **Whisper Large v3** or **Whisper Large v3 Turbo** is the recommended starting point.


How do I create an organization to manage shared ML assets?


To create an organization on **Hugging Face** to manage shared ML assets (models, datasets, Spaces, etc.), follow these steps:

---

## Creating a Hugging Face Organization

### 1. Go to the Organization Creation Page
- Log in to [huggingface.co](https://huggingface.co)
- Click your profile picture (top right) → **"New Organization"**
- Or navigate directly to: [huggingface.co/organizations/new](https://huggingface.co/organizations/new)

### 2. Fill in Organization Details
- **Organization name** – a unique handle (e.g., `my-ml-team`)
- **Display name** – a human-readable name
- **Description** – optional, but recommended
- **Avatar** – optional logo/image
- **Plan** – choose Free or a paid plan (for private repos, more compute, etc.)

### 3. Configure Access & Visibility
- Set the org to **Public** or **Private**
- Private orgs keep all repos hidden from non-members

### 4. Invite Members
- After creation, go to **Organization Settings → Members**
- Invite collaborators by username or email
- Assign roles:
  - **Admin** – full control
  - **Write** – can push to repos
  - **Read** – read-only access (useful for private orgs)

---

## Managing Shared ML Assets Within the Org

| Asset Type | How to share |
|---|---|
| **Models** | Create repos under `org-name/model-name` |
| **Datasets** | Create dataset repos under the org namespace |
| **Spaces** | Deploy Gradio/Streamlit apps under the org |
| **Collections** | Group related assets into a named collection |
| **Inference Endpoints** | Deploy shared endpoints from the org |

### Tips
- Use **Teams** (paid plans) to manage subgroups with different permissions
- Use **repo-level access tokens** for CI/CD pipelines
- Pin important repos to the org profile for visibility
- Use **Collections** to curate and document a set of related models/datasets

---

Would you like help with a specific part of the setup, such as automating repo creation via the Hugging Face API or configuring access tokens for your team?


Best workflow for reviewing model changes before merging


This is a great question for a TypeScript/Next.js project. Let me give you a practical, battle-tested workflow for reviewing model changes before merging — one that catches regressions early and builds confidence.

---

## Best Workflow for Reviewing Model Changes Before Merging

### 1. Maintain a Golden Test Suite

Before any model change review, you need a baseline. Create a suite of representative prompts that cover your core use cases:

```typescript
// tests/golden/cases.ts
export const goldenCases = [
  {
    id: "summarize-basic",
    input: "Summarize: The quick brown fox jumped over the lazy dog.",
    assertContains: ["fox", "dog"],
    maxTokens: 100,
  },
  {
    id: "classify-sentiment",
    input: "I love this product! It's amazing.",
    assertContains: ["positive"],
    maxTokens: 50,
  },
  // ... your real use cases
];
```

### 2. Run Side-by-Side Evals (Old vs New Model)

The core of your review is a diff between model outputs:

```typescript
// scripts/eval-model-diff.ts
import Anthropic from "@anthropic-ai/sdk";
import { goldenCases } from "../tests/golden/cases";
import fs from "fs";

const client = new Anthropic();

async function runCase(model: string, testCase: typeof goldenCases[0]) {
  const stream = client.messages.stream({
    model,
    max_tokens: testCase.maxTokens ?? 1024,
    thinking: { type: "adaptive" },
    messages: [{ role: "user", content: testCase.input }],
  });
  const message = await stream.finalMessage();
  const text = message.content
    .filter((b): b is Anthropic.TextBlock => b.type === "text")
    .map((b) => b.text)
    .join("");
  return {
    output: text,
    inputTokens: message.usage.input_tokens,
    outputTokens: message.usage.output_tokens,
    stopReason: message.stop_reason,
  };
}

async function main() {
  const OLD_MODEL = "claude-sonnet-4-6"; // baseline
  const NEW_MODEL = "claude-opus-4-6";   // candidate

  const results: Array<{
    id: string;
    old: Awaited<ReturnType<typeof runCase>>;
    new: Awaited<ReturnType<typeof runCase>>;
    passed: boolean;
    regressions: string[];
  }> = [];

  for (const tc of goldenCases) {
    console.log(`Running: ${tc.id}`);
    const [oldResult, newResult] = await Promise.all([
      runCase(OLD_MODEL, tc),
      runCase(NEW_MODEL, tc),
    ]);

    const regressions: string[] = [];

    // Check assertions on new model output
    for (const expected of tc.assertContains ?? []) {
      if (!newResult.output.toLowerCase().includes(expected.toLowerCase())) {
        regressions.push(`Missing expected content: "${expected}"`);
      }
    }

    // Flag if output changed significantly (>20% length difference)
    const lenRatio =
      newResult.output.length / Math.max(oldResult.output.length, 1);
    if (lenRatio < 0.5 || lenRatio > 2.0) {
      regressions.push(
        `Output length changed dramatically: ${oldResult.output.length} → ${newResult.output.length} chars`,
      );
    }

    results.push({
      id: tc.id,
      old: oldResult,
      new: newResult,
      passed: regressions.length === 0,
      regressions,
    });
  }

  // Write human-readable diff report
  const report = results
    .map((r) => {
      const status = r.passed ? "✅ PASS" : "❌ FAIL";
      return [
        `## ${r.id} — ${status}`,
        r.regressions.length ? `**Regressions:**\n${r.regressions.map((x) => `  - ${x}`).join("\n")}` : "",
        `**Old output:** ${r.old.output.slice(0, 200)}...`,
        `**New output:** ${r.new.output.slice(0, 200)}...`,
        `**Token usage:** ${r.old.outputTokens} → ${r.new.outputTokens} output tokens`,
      ]
        .filter(Boolean)
        .join("\n");
    })
    .join("\n\n---\n\n");

  fs.writeFileSync("eval-report.md", report);

  const failed = results.filter((r) => !r.passed);
  console.log(`\n${results.length - failed.length}/${results.length} cases passed`);
  if (failed.length > 0) {
    console.error(`Failed cases: ${failed.map((r) => r.id).join(", ")}`);
    process.exit(1);
  }
}

main();
```

Run this in CI: `npx ts-node scripts/eval-model-diff.ts`

---

### 3. Cost & Latency Regression Check

Model changes often affect cost and speed. Track this automatically:

```typescript
// Add to your eval script
interface PerfMetrics {
  avgInputTokens: number;
  avgOutputTokens: number;
  estimatedCostPer1kRequests: number; // USD
}

function calcMetrics(
  results: Array<{ inputTokens: number; outputTokens: number }>,
  inputPricePerM: number,
  outputPricePerM: number,
): PerfMetrics {
  const avgIn = results.reduce((s, r) => s + r.inputTokens, 0) / results.length;
  const avgOut = results.reduce((s, r) => s + r.outputTokens, 0) / results.length;
  return {
    avgInputTokens: avgIn,
    avgOutputTokens: avgOut,
    estimatedCostPer1kRequests:
      ((avgIn * inputPricePerM + avgOut * outputPricePerM) / 1_000_000) * 1000,
  };
}

// claude-opus-4-6: $5/M in, $25/M out
// claude-sonnet-4-6: $3/M in, $15/M out
```

---

### 4. PR Review Checklist

When someone opens a PR that changes the model string or prompt:

| Check | How |
|---|---|
| **Run eval diff script** | CI gate — must pass all golden cases |
| **Review output diffs** | Read `eval-report.md` artifact in the PR |
| **Cost delta** | Attach cost estimate to PR description |
| **Latency (if latency-sensitive)** | Add `Date.now()` timing around `stream.finalMessage()` |
| **Edge cases** | Manually test 2–3 adversarial inputs |
| **Rollback plan** | Is the old model string still in code / env var? |

---

### 5. Feature Flag the Model (For Safe Rollout)

Never hard-code a new model directly in production. Use an env var or feature flag:

```typescript
// lib/claude.ts
const MODEL = process.env.CLAUDE_MODEL ?? "claude-opus-4-6";

export function createClient() {
  return {
    client: new Anthropic(),
    model: MODEL,
  };
}
```

This lets you roll back instantly by changing one env var, without a redeploy.

---

### Summary

1. **Golden test suite** — representative prompts with assertions
2. **Side-by-side eval script** — diff old vs new model outputs in CI
3. **Cost/latency tracking** — attach numbers to every PR
4. **Human review** — read the `eval-report.md` diff artifact before approving
5. **Feature flag** — ship the model change behind an env var for easy rollback

This workflow catches both quality regressions (wrong outputs) and operational regressions (cost spikes, latency increases) before they hit users.


How do I track experiment runs and link them to model versions?


Here's how experiment runs and model versions work in this codebase:

---

## Experiment Runs

Runs are the top-level concept for tracking experiments. Each run is a batch of queries executed across one or more tool/model combinations.

### Creating a Run

Use the **Runs** UI (`/runs`) or `POST /api/runs`:
```json
{
  "name": "My Experiment",
  "toolsConfig": [
    { "tool": "claude-code", "model": "claude-opus-4-6", "effort": "high", "thinking": "enabled" },
    { "tool": "codex", "model": "gpt-4.1", "effort": "medium" }
  ]
}
```

### Run Lifecycle

A run moves through these statuses: `pending → running → completed` (or `failed`, `aborted`, `retrying`)

- **Start:** `POST /api/runs/[id]/start` (admin-only)
- **Abort:** `POST /api/runs/[id]/abort`
- **Retry failed jobs:** `POST /api/runs/[id]/retry` (optionally filter by tool)

---

## Linking Runs to Model Versions

Model versions are embedded directly in the `toolsConfig` of each run — there is no separate model registry. Each result row in the `results` table stores:

| Field | Purpose |
|---|---|
| `tool` | Tool used (e.g., `claude-code`, `codex`) |
| `model` | Exact model identifier (e.g., `claude-opus-4-6`) |
| `effort` | Effort level (`low`/`medium`/`high`/`max`) |
| `thinking` | Thinking mode (`enabled`/`disabled`) |
| `runId` | Foreign key back to the parent run |

This means every result is fully traceable to both the run and the exact model/parameter configuration used.

---

## Available Models

| Tool | Models |
|---|---|
| Claude Code | `claude-opus-4-6`, `claude-sonnet-4-6` |
| Codex | `gpt-4.1`, `gpt-4.1-mini` |
| GitHub Copilot | `claude-sonnet-4.6`, `gpt-5.4` |
| Cursor | `composer-2` |

---

## Viewing Results

- **Run detail page** (`/runs/[id]`): Per-run analytics — mention rates, costs, latency, token usage, filterable by tool/model/level/effort/product, with CSV export.
- **Dashboard** (`/`): Cross-run trends — coverage dynamics over time, product leaderboards, and a coverage map comparing tool/model performance.

---

## Key Limitation

There is **no semantic model versioning, artifact storage, or model lineage** system. Model version tracking is done implicitly by recording the model identifier string in each result. If you need baseline comparisons, you'd compare runs with the same `toolsConfig` across different time periods using the dashboard's coverage dynamics chart.


How do I browse dataset columns and statistics without downloading?


This project is a **HuggingFace DevTools Dashboard** — it tracks how coding assistants mention HF products in their responses. It doesn't include built-in dataset column/statistics browsing functionality.

However, if you want to browse HuggingFace dataset columns and statistics **without downloading**, here are the best approaches using HF's own tools:

---

## Browsing Dataset Columns & Stats Without Downloading

### 1. **HuggingFace Dataset Viewer (Web UI)**
Go to any dataset on the Hub and click the **"Dataset Viewer"** tab:
```
https://huggingface.co/datasets/<owner>/<dataset-name>
```
- Shows column names, types, and a preview of rows
- Shows dataset statistics (value counts, distributions) via the **"Stats"** sub-tab

### 2. **Datasets Server API (no download required)**
HF exposes a REST API to query metadata and statistics:

```bash
# Get column info / features
curl "https://datasets-server.huggingface.co/info?dataset=<owner>/<dataset>"

# Get first rows (preview)
curl "https://datasets-server.huggingface.co/first-rows?dataset=<owner>/<dataset>&config=default&split=train"

# Get statistics (histograms, nulls, value counts, etc.)
curl "https://datasets-server.huggingface.co/statistics?dataset=<owner>/<dataset>&config=default&split=train"
```

### 3. **In Python, using `load_dataset_builder`**
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("owner/dataset-name")
print(builder.info.features)      # column names + types
print(builder.info.splits)        # split sizes
# Nothing is downloaded — only metadata is fetched
```

### 4. **HF Hub Python client**
```python
from huggingface_hub import HfApi

api = HfApi()
info = api.dataset_info("owner/dataset-name")
print(info.cardData)   # dataset card metadata
```

---

The **Datasets Server API** (option 2) is the most powerful — it provides full column statistics including histograms, null counts, min/max, and top values, all without touching the actual data files.


How do I load a dataset directly into a pandas DataFrame?


You can load a dataset directly into a pandas DataFrame using several methods depending on your data source:

## From a File

```python
import pandas as pd

# CSV
df = pd.read_csv('data.csv')

# Excel
df = pd.read_excel('data.xlsx')

# JSON
df = pd.read_json('data.json')

# Parquet
df = pd.read_parquet('data.parquet')
```

## From a URL

```python
df = pd.read_csv('https://example.com/data.csv')
```

## From the Hugging Face `datasets` Library

```python
from datasets import load_dataset

ds = load_dataset('dataset-name', split='train')
df = ds.to_pandas()
```

## From a Dict or List

```python
# From a dictionary
df = pd.DataFrame({'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']})

# From a list of dicts
df = pd.DataFrame([{'col1': 1, 'col2': 'a'}, {'col1': 2, 'col2': 'b'}])
```

## From a SQL Database

```python
import sqlite3

conn = sqlite3.connect('mydb.db')
df = pd.read_sql('SELECT * FROM my_table', conn)
```

## Useful Common Options

| Option | Description | Example |
|---|---|---|
| `sep` | Custom delimiter | `read_csv(..., sep='\t')` |
| `header` | Row to use as column names | `read_csv(..., header=0)` |
| `usecols` | Load only specific columns | `read_csv(..., usecols=['a','b'])` |
| `nrows` | Limit rows loaded | `read_csv(..., nrows=1000)` |
| `dtype` | Specify column types | `read_csv(..., dtype={'id': str})` |

The most common starting point is `pd.read_csv()` for local or remote CSV files.


How do I find datasets with permissive licenses for commercial use?


To find datasets with permissive licenses for commercial use, here are the most effective approaches:

## Hugging Face Hub

- **Filter by license** on [huggingface.co/datasets](https://huggingface.co/datasets) using the sidebar — look for:
  - `apache-2.0` — very permissive, commercial use allowed
  - `mit` — permissive, commercial use allowed
  - `cc-by-4.0` — allows commercial use with attribution
  - `cc0-1.0` / `public domain` — no restrictions at all
- **Avoid** `cc-by-nc-*` (non-commercial), `gpl` (copyleft), or `other` (unknown restrictions)

## Other Dataset Repositories

| Source | Notes |
|---|---|
| [Kaggle](https://www.kaggle.com/datasets) | Filter by license type in search |
| [Google Dataset Search](https://datasetsearch.research.google.com/) | Searches across many sources |
| [UCI ML Repository](https://archive.ics.uci.edu/) | Most datasets are freely usable |
| [AWS Open Data](https://registry.opendata.aws/) | Curated open datasets |
| [data.gov](https://data.gov) / [data.europa.eu](https://data.europa.eu) | Government data, typically public domain |

## Key Tips

1. **Always read the full license text** — dataset cards sometimes misclassify licenses.
2. **Check the data source**, not just the license tag. A dataset labeled MIT might contain scraped data with upstream restrictions.
3. **Attribution requirements** — even permissive licenses like CC-BY-4.0 require you to credit the creator.
4. **Model outputs** — if the dataset was generated by a model (e.g., GPT-4), check that model's terms of service too, as those often prohibit commercial use regardless of the dataset license.

## On Hugging Face Specifically

You can also use the API to filter programmatically:

```python
from huggingface_hub import list_datasets

datasets = list_datasets(
    license="apache-2.0",  # or "mit", "cc-by-4.0", "cc0-1.0"
    sort="downloads",
)
```

The safest licenses for commercial use are **Apache 2.0**, **MIT**, **CC0**, and **CC-BY 4.0**.


How do I store large binary files alongside my dataset metadata?


I can answer this directly from my knowledge — this is a common question about dataset management.

## Storing Large Binary Files with Dataset Metadata

The best approach depends on your stack, but here are the standard patterns:

---

### 1. **Git LFS (Large File Storage)** — for git-tracked repos
Track binaries separately from metadata while keeping them logically together:
```bash
git lfs install
git lfs track "*.bin" "*.parquet" "*.arrow" "*.tar.gz"
git add .gitattributes
git add data/
git commit -m "Add dataset with LFS-tracked binaries"
```
- Metadata (JSON, YAML, CSV) stays in git as normal
- Binaries are stored in LFS, referenced by pointer files

---

### 2. **Hugging Face Datasets Hub** — if you're using HF
```python
from datasets import Dataset
import datasets

# Push dataset (handles large files automatically)
ds = Dataset.from_dict({...})
ds.push_to_hub("your-org/your-dataset")
```
- The Hub uses **LFS automatically** for files > 10MB
- Metadata (`dataset_card`, `README.md`, `dataset_info.json`) lives alongside
- Binary shards (`.parquet`, `.arrow`) are LFS-tracked

---

### 3. **Object Storage + Metadata Sidecar** — for production pipelines
Store binaries in S3/GCS/Azure Blob, keep metadata in a lightweight format:
```
s3://my-bucket/datasets/v1/
├── metadata.json          # schema, splits, version, checksums
├── shard-000.parquet      # binary data shards
├── shard-001.parquet
└── checksums.sha256       # integrity verification
```
Reference binaries from metadata by URI + checksum:
```json
{
  "splits": {
    "train": {
      "files": [
        {"uri": "s3://my-bucket/datasets/v1/shard-000.parquet", "sha256": "abc123..."}
      ]
    }
  }
}
```

---

### 4. **DVC (Data Version Control)** — git-native data versioning
```bash
dvc init
dvc add data/large_binary.bin   # tracks in .dvc file, ignores in git
git add data/large_binary.bin.dvc data/.gitignore
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push
```
- `.dvc` pointer files commit to git alongside your metadata
- Actual binaries sync to remote storage

---

### Key Principles

| Concern | Recommendation |
|---|---|
| **Integrity** | Always store checksums (SHA-256) in metadata |
| **Versioning** | Tag metadata versions separately from binary versions |
| **Discovery** | Keep a human-readable `README`/card next to binaries |
| **Reproducibility** | Metadata should fully describe how to reconstruct the dataset |

What's your current storage setup — are you using Hugging Face Hub, a cloud bucket, or a local/git workflow? I can give more targeted advice.


Best way to host and serve a 500GB+ dataset


## Hosting & Serving a 500GB+ Dataset

Here are the best approaches depending on your use case:

---

### 🏆 Hugging Face Hub (Best for ML datasets)
- **`datasets` library** with streaming: `load_dataset(..., streaming=True)` — no full download needed
- **Parquet format** on Hub enables SQL-like queries via the [Datasets Server API](https://huggingface.co/docs/datasets-server)
- Free for public datasets; paid plans for private
- Built-in versioning via Git LFS

---

### ☁ Cloud Object Storage (Best for raw serving)
| Service | Egress Cost | Notes |
|---|---|---|
| **Cloudflare R2** | **Free egress** | Best cost for high-traffic datasets |
| **Backblaze B2** | Very cheap | Good for large static files |
| **AWS S3** | ~$0.09/GB | Most ecosystem support |
| **GCS** | ~$0.12/GB | Good BigQuery integration |

**Serve via HTTP Range requests** so clients can fetch slices without downloading everything.

---

### ⚡ High-Performance Formats (Critical for 500GB+)
- **Parquet** — columnar, compressed, supports predicate pushdown; pair with DuckDB or Polars
- **Zarr / HDF5** — best for array/tensor data (genomics, satellite, etc.)
- **WebDataset (tar shards)** — ideal for image/audio ML training
- **Lance** — newer columnar format optimized for ML, supports random access

---

### 🔍 Query Layer Options
- **DuckDB** — query Parquet directly from S3/R2, extremely fast, zero infra
  ```sql
  SELECT * FROM read_parquet('s3://bucket/data/*.parquet') WHERE ...
  ```
- **Apache Arrow Flight** — high-throughput RPC for bulk data transfer
- **Trino/Presto** — distributed SQL over object storage at scale
- **ClickHouse** — if you need real-time analytics on the dataset

---

### 📦 Practical Recommended Stack

**For ML training data:**
> Shard into ~1GB Parquet or WebDataset files → upload to R2 or HF Hub → stream during training

**For analytical queries:**
> Parquet on S3/R2 + DuckDB (serverless, pay-per-query via MotherDuck)

**For API serving:**
> PostgreSQL/ClickHouse for hot data + object storage for raw files, with a CDN (Cloudflare) in front

---

### Key Tips
1. **Always shard** — don't store as one monolithic file; aim for 100MB–1GB shards
2. **Compress** — Parquet with Zstd typically achieves 3–10x compression
3. **Index/partition** — partition by date, category, etc. to enable partition pruning
4. **CDN** — put Cloudflare in front of any HTTP-served dataset to reduce origin load

What's your specific use case (ML training, analytics, API serving)? I can give more targeted advice.


How do I create a gated dataset that requires access approval?


I'll answer this question about creating a gated dataset on Hugging Face directly from my knowledge.

## Creating a Gated Dataset on Hugging Face

Gated datasets require users to request and receive approval before they can access the data. Here's how to set one up:

### 1. Via the Hugging Face Web UI

1. Go to your dataset repository on [huggingface.co](https://huggingface.co)
2. Click the **Settings** tab
3. Scroll to the **Gated dataset** section
4. Choose a gating mode:
   - **Auto-approval** — access is granted instantly after the user agrees to your conditions
   - **Manual approval** — you review and approve/reject each request yourself
5. Optionally customize the **access request form** (e.g., ask for name, affiliation, intended use)
6. Click **Save**

---

### 2. Via `dataset_card_metadata` in your `README.md`

Add the `extra_gated_*` fields to your dataset card's YAML front matter:

```yaml
---
license: cc-by-nc-4.0
extra_gated_prompt: "By accessing this dataset, you agree to the terms of use."
extra_gated_fields:
  Name: text
  Affiliation: text
  Country: country
  Intended use case: text
  I agree to the terms of use: checkbox
extra_gated_heading: "Request access to this dataset"
---
```

For **auto-approval**, add:
```yaml
extra_gated_button_content: "Acknowledge license and access dataset"
```

---

### 3. Via the `huggingface_hub` Python Library

```python
from huggingface_hub import HfApi

api = HfApi()

# Enable gating on an existing dataset
api.update_repo_settings(
    repo_id="your-username/your-dataset",
    repo_type="dataset",
    gated="auto"      # or "manual" for manual approval
)
```

---

### Managing Access Requests

Once gating is enabled, you can manage requests:

- **Web UI**: Go to Settings → Gated dataset → **Access Requests**
- **Python**:
```python
# List pending requests
requests = api.list_pending_access_requests(
    repo_id="your-username/your-dataset",
    repo_type="dataset"
)

# Approve a request
api.grant_access(
    repo_id="your-username/your-dataset",
    repo_type="dataset",
    user="requesting-username"
)

# Reject a request
api.reject_access(
    repo_id="your-username/your-dataset",
    repo_type="dataset",
    user="requesting-username"
)
```

---

### Key Notes

- 🔒 The dataset files remain **private/inaccessible** until a user is approved
- ✅ Approved users access the dataset normally with their HF token
- 📧 Users are **notified by email** when their request is approved or rejected
- 👥 You can also **grant access proactively** to specific users without them requesting it


How do I compare my model against top models on public leaderboards?


I can help you understand how to compare your model against top models on public leaderboards. Here are the main approaches:

## Public Leaderboard Platforms

### 1. **Hugging Face Open LLM Leaderboard**
- Go to [huggingface.co/spaces/open-llm-leaderboard/open_llm-leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm-leaderboard)
- Submit your model by clicking **"Submit here!"**
- Your model must be publicly hosted on Hugging Face Hub
- It gets evaluated on benchmarks like: MMLU, HellaSwag, ARC, TruthfulQA, GSM8K, etc.

### 2. **LMSYS Chatbot Arena**
- Go to [lmarena.ai](https://lmarena.ai)
- Human-preference-based ranking via blind A/B battles
- Contact the LMSYS team to add your model, or use the API

### 3. **EleutherAI LM Evaluation Harness** *(run locally)*
```bash
pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=your-org/your-model \
  --tasks mmlu,hellaswag,arc_easy,arc_challenge,truthfulqa \
  --device cuda \
  --output_path ./results
```
Then compare your JSON results against published scores.

### 4. **AlpacaEval**
```bash
pip install alpaca-eval
alpaca_eval evaluate \
  --model_outputs path/to/your_outputs.json \
  --annotators_config alpaca_eval_gpt4
```
Gives a win-rate vs. GPT-4 and ranks against a public leaderboard.

---

## Tips for Fair Comparison

| Factor | What to Check |
|---|---|
| **Prompt format** | Use the exact chat template your model was trained on |
| **Quantization** | Compare at the same precision (fp16, bf16, etc.) |
| **Few-shot count** | Match the `num_fewshot` used by the leaderboard |
| **Task version** | Pin benchmark dataset versions to avoid drift |
| **Decoding params** | Greedy decoding is standard for most benchmarks |

---

## Quickest Path

If your model is already on **Hugging Face Hub**, the fastest route is:
1. Submit to the **Open LLM Leaderboard** (automated, free)
2. Run `lm-eval` locally with the same tasks for a preview before your submission is processed

Would you like help setting up any of these evaluation pipelines specifically?


Best way to evaluate hallucination rates in a language model


## Evaluating Hallucination Rates in Language Models

Here are the most effective approaches, from rigorous to practical:

---

### 1. **Benchmark-Based Evaluation**

Use established datasets designed to probe factual accuracy:

| Benchmark | What it tests |
|-----------|---------------|
| **TruthfulQA** | Questions humans often answer falsely; measures truthfulness |
| **HaluEval** | Hallucination detection across QA, dialogue, summarization |
| **FActScoring** | Fact-level precision in long-form generation |
| **FEVER** | Fact verification against Wikipedia |
| **SimpleQA / PopQA** | Short-answer factual recall |

---

### 2. **Reference-Based Automatic Metrics**

- **FActScore** — breaks generated text into atomic facts, verifies each against a knowledge source
- **SelfCheckGPT** — samples multiple outputs; inconsistency across samples signals hallucination (no reference needed)
- **ROUGE/BERTScore** — weaker proxies; measure overlap but not factual correctness

---

### 3. **LLM-as-Judge**

Use a strong model (e.g., GPT-4, Claude) to score factual consistency:
- Prompt it to identify unsupported claims in generated text vs. a source document
- Works well for **RAG pipelines** (does the answer stick to retrieved context?)
- Tools: **RAGAS**, **TruLens**, **DeepEval**

---

### 4. **Human Annotation (Gold Standard)**

- Have domain experts label claims as *supported*, *unsupported*, or *contradicted*
- Expensive but necessary for high-stakes domains (medical, legal)
- Use **inter-annotator agreement** (Cohen's κ) to validate label quality

---

### 5. **Closed-Book vs. Open-Book Testing**

- Ask factual questions with and without retrieved context
- Measure how often the model **contradicts** the provided context (closed-form hallucination) vs. fabricates outside it

---

### 6. **Consistency / Self-Consistency Checks**

- Ask the same question multiple ways; measure **answer consistency**
- Ask the model to **cite sources** and verify those citations exist and are accurate
- Ask the model to **self-evaluate** its confidence (calibration analysis)

---

### Key Metrics to Report

| Metric | Description |
|--------|-------------|
| **Hallucination rate** | % of responses with ≥1 unsupported claim |
| **Claim-level precision** | Supported claims / total claims |
| **Faithfulness score** | How well output adheres to source (for RAG) |
| **Calibration (ECE)** | Does model confidence match actual accuracy? |

---

### Practical Recommendations

1. **Start with SelfCheckGPT or FActScore** — they require no ground-truth labels and scale well
2. **Use RAGAS** if you're evaluating a RAG system specifically
3. **Domain-specific probes beat generic benchmarks** — build a small curated test set for your use case
4. **Combine methods** — no single metric captures all hallucination types (intrinsic vs. extrinsic, open-domain vs. grounded)

The field is moving fast — FActScore and SelfCheckGPT are currently among the most cited scalable methods.


How do I create custom evaluation tasks for domain-specific testing?


Based on the codebase exploration, here's how to create custom evaluation tasks for domain-specific testing in this dashboard:

---

## Creating Custom Evaluation Tasks

This dashboard evaluates AI coding tools by running **queries** (developer questions) through them and measuring which Hugging Face products get mentioned. Custom domain-specific testing is done by adding queries to the database.

---

### 1. Add Queries Directly to the Database

Queries are stored in the `queries` table (`/src/lib/db/schema.ts`) with these fields:

| Field | Description |
|---|---|
| `text` | The developer question/prompt |
| `level` | `"General"` or `"Open Source"` |
| `category` | Domain label (e.g., `"Model Serving"`, `"Fine-Tuning"`) |

You can insert domain-specific queries directly via SQL or a migration/seed script:

```sql
INSERT INTO queries (text, level, category)
VALUES
  ('How do I deploy a custom medical NLP model at scale?', 'General', 'Healthcare AI'),
  ('What tools help fine-tune a model on private tabular data?', 'Open Source', 'Data Science');
```

---

### 2. Extend the Seed File

The canonical place to add queries programmatically is alongside the HF product seeding in **`/src/lib/db/seed.ts`**. Add a batch insert for your domain:

```ts
await db.insert(queries).values([
  { text: "...", level: "General", category: "Your Domain" },
  // ...
]);
```

Then re-run the seed script to populate the DB.

---

### 3. Create a New Run Targeting Your Queries

Once queries exist, create an evaluation run via the API:

```bash
POST /api/runs
{
  "toolConfigs": ["claude-code", "codex", "copilot"],
  "queryFilter": { "category": "Your Domain" }  # if filtering is supported
}
```

Then start it:
```bash
POST /api/runs/{id}/start
```

---

### 4. Add Domain-Specific HF Products to Track

If your domain has specific HF products that should be detected, add them to the **`hf_products`** table (also in `seed.ts`):

```ts
await db.insert(hfProducts).values([
  {
    name: "Lighteval",
    category: "Tool",
    keywords: ["lighteval", "light-eval"],
  },
]);
```

The keyword detector in **`/src/lib/engine/detector.ts`** uses word-boundary regex matching, so aliases are important for accurate detection.

---

### 5. Tune Concurrency & Retries Per Domain

If your domain queries are long/complex (more tool calls, more tokens), you may want to adjust per-tool concurrency in **`/src/lib/engine/runner.ts`**:

```ts
const CONCURRENCY = {
  "claude-code": 3,   // reduce for heavy queries
  "codex": 5,
  "copilot": 3,
  "cursor": 1,
};
```

---

### Summary Flow

```
Add queries (seed.ts or SQL)
       ↓
Add domain products to hf_products (seed.ts)
       ↓
POST /api/runs  →  POST /api/runs/{id}/start
       ↓
Runner distributes jobs across tool workers
       ↓
Extractor (Claude Haiku) identifies mentioned products
       ↓
Results visible in /api/dashboard with your category filter
```

The key extension points are the **`queries` table** (what to ask), the **`hf_products` table** (what to detect), and the **`seed.ts` file** (where both are managed).


What's the standard way to evaluate a model's safety and toxicity?


## Standard Ways to Evaluate Model Safety and Toxicity

Here are the main approaches used in the field:

---

### 1. **Benchmark Datasets**
- **ToxiGen** – Tests generation of toxic content targeting specific demographic groups
- **RealToxicityPrompts** – Prompts designed to elicit toxic completions; scored with Perspective API
- **BBQ (Bias Benchmark for QA)** – Evaluates social biases in question-answering
- **WinoBias / WinoGender** – Probes gender bias in coreference resolution
- **HateXplain** – Hate speech detection with human rationales

### 2. **Automated Scoring Tools**
- **Perspective API** (Google/Jigsaw) – Scores text on toxicity, identity attacks, insults, threats, etc.
- **Detoxify** – Open-source transformer-based toxicity classifier
- **OpenAI Moderation API** – Classifies content across harm categories

### 3. **Red-Teaming**
- **Manual red-teaming** – Human adversaries try to elicit harmful outputs
- **Automated red-teaming** – Use another LLM (or RL agent) to generate adversarial prompts at scale (e.g., Perez et al. 2022)
- **Structured attack taxonomies** – MITRE ATLAS, OWASP LLM Top 10

### 4. **Refusal & Policy Compliance Testing**
- Present the model with known-harmful prompts and measure **refusal rate**
- Test for **jailbreaks** (prompt injection, role-play exploits, encoding tricks)
- Evaluate **over-refusal** (false positives on benign content)

### 5. **Holistic Frameworks**
- **HELM** (Holistic Evaluation of Language Models) – Stanford; includes toxicity, bias, and fairness metrics
- **EleutherAI LM Evaluation Harness** – Pluggable benchmark runner
- **MLCommons AI Safety** – Standardized safety benchmarks across hazard categories
- **LMSYS Chatbot Arena** – Human preference + safety-aware ranking

### 6. **Human Evaluation**
- Crowdworker annotation of model outputs for harm, offensiveness, or bias
- Expert review for high-stakes domains (medical, legal, CBRN risks)

---

### Key Dimensions Typically Measured

| Dimension | What's Tested |
|---|---|
| **Toxicity** | Hate speech, slurs, harassment |
| **Bias / Fairness** | Differential treatment across demographics |
| **Misinformation** | Factual accuracy, hallucination rate |
| **Dangerous content** | Weapons, self-harm, CSAM |
| **Privacy** | PII leakage, memorization |
| **Robustness** | Behavior under adversarial prompts |

---

### Best Practice
Most rigorous evaluations combine **automated benchmarks** (for scale and reproducibility) with **human red-teaming** (for novel attack discovery), then track metrics over time across model versions.


How do I run reproducible benchmarks across different hardware?


Running reproducible benchmarks across different hardware requires careful attention to several key areas:

## 1. Control Your Environment

**Containerization**
- Use Docker or Podman to freeze your software stack (OS libs, drivers, runtime versions)
- Pin base images by digest, not just tag: `FROM ubuntu@sha256:abc123...`
- Avoid `latest` tags — they drift

**Dependency Locking**
- Lock all dependencies: `requirements.txt` with exact versions, `package-lock.json`, `Cargo.lock`, etc.
- Include the compiler/interpreter version in your lock strategy

---

## 2. Isolate the System

**Before each benchmark run:**
- Disable CPU frequency scaling / turbo boost (or enable it consistently):
  ```bash
  # Linux: set performance governor
  sudo cpupower frequency-set -g performance
  ```
- Disable NUMA balancing, address space layout randomization (ASLR), and hyperthreading if you need tight control
- Pin processes to specific cores with `taskset` or `numactl`
- Kill background services that cause noise (cron jobs, indexers, etc.)

---

## 3. Warm Up and Repeat

- **Always warm up** — run several throwaway iterations before recording
- **Run many iterations** — 30+ is a common minimum for statistical significance
- **Report statistics**, not just a single number:
  - Median and percentiles (p50, p95, p99) over mean (mean is sensitive to outliers)
  - Standard deviation / coefficient of variation
  - Min/max to catch anomalies

---

## 4. Use a Benchmarking Framework

Don't roll your own timing loops — use tools designed for this:

| Language | Tool |
|----------|------|
| Python | `pytest-benchmark`, `timeit`, `pyperf` |
| Rust | `criterion` |
| Go | `testing.B` (built-in) |
| C/C++ | `Google Benchmark` |
| JVM | `JMH` |
| General | `hyperfine` (CLI commands) |

These handle warm-up, iteration counts, and statistical reporting automatically.

---

## 5. Record Hardware Context

Log metadata alongside every benchmark result:
```
CPU model, core count, clock speed
RAM size and speed
OS + kernel version
GPU model (if relevant)
Compiler version and flags
Benchmark timestamp + git commit hash
```

Tools like `lscpu`, `dmidecode`, `nvidia-smi`, and `/proc/cpuinfo` are useful here.

---

## 6. Make Benchmarks Part of CI

- Store results as artifacts and **track regressions over time**
- Use tools like [Bencher](https://bencher.dev), [Codspeed](https://codspeed.io), or simple JSON files committed to git
- Run on **dedicated benchmark hardware** — shared CI runners have too much noise

---

## 7. Control Randomness and I/O

- Set fixed random seeds where applicable
- Pre-generate or cache any input data — don't benchmark I/O if you're measuring compute
- Use `tmpfs` (RAM disk) if disk I/O is unavoidable but not the focus

---

## 8. Cross-Hardware Comparison

When comparing results across machines:
- **Never compare raw times** without noting clock speed differences
- Prefer **throughput or ops/sec** normalized to a known workload
- Run the same benchmark on a reference machine as a calibration point
- Document all hardware differences explicitly in your results

---

## Quick Checklist

```
[ ] Pinned dependencies and base image
[ ] Consistent governor/frequency settings
[ ] Warm-up iterations discarded
[ ] 30+ timed iterations
[ ] Median/percentiles reported (not just mean)
[ ] Hardware metadata logged
[ ] Git commit hash recorded
[ ] Background noise minimized
[ ] Results stored and versioned
```

The single biggest mistake is trusting a single run. **Variance is your enemy** — measure it, report it, and minimize it systematically.


How do I download only specific model files instead of the full repo?


To download only specific files from a Hugging Face repository (instead of the full repo), you have several options:

---

## 1. `huggingface_hub` Python Library (Recommended)

### Download a single file
```python
from huggingface_hub import hf_hub_download

hf_hub_download(
    repo_id="meta-llama/Llama-3-8B",
    filename="config.json",
    local_dir="./my_model"
)
```

### Download multiple specific files
```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="meta-llama/Llama-3-8B",
    allow_patterns=["*.safetensors", "config.json", "tokenizer*"],
    local_dir="./my_model"
)
```

### Exclude unwanted files
```python
snapshot_download(
    repo_id="meta-llama/Llama-3-8B",
    ignore_patterns=["*.bin", "*.gguf", "original/*"],
    local_dir="./my_model"
)
```

---

## 2. CLI with `huggingface-cli`

```bash
# Download a single file
huggingface-cli download meta-llama/Llama-3-8B config.json

# Download multiple files
huggingface-cli download meta-llama/Llama-3-8B config.json tokenizer.json

# Download with glob patterns
huggingface-cli download meta-llama/Llama-3-8B --include "*.safetensors" --local-dir ./my_model

# Exclude patterns
huggingface-cli download meta-llama/Llama-3-8B --exclude "*.bin" --local-dir ./my_model
```

---

## 3. Direct URL Download (no library needed)

Files are accessible via a predictable URL structure:
```bash
wget https://huggingface.co/{repo_id}/resolve/main/{filename}

# Example:
wget https://huggingface.co/meta-llama/Llama-3-8B/resolve/main/config.json
```

For a specific branch/revision:
```bash
wget https://huggingface.co/meta-llama/Llama-3-8B/resolve/{branch_or_commit}/{filename}
```

---

## 4. Dataset Files

For datasets, use the `datasets` library:
```python
from datasets import load_dataset

# Load only a specific split or subset
ds = load_dataset("allenai/c4", "en", split="train", streaming=True)
```

---

## Tips

| Use Case | Best Option |
|---|---|
| Single known file | `hf_hub_download()` or direct URL |
| Multiple files by pattern | `snapshot_download()` with `allow_patterns` |
| CLI / scripting | `huggingface-cli download` |
| Large repos (avoid bulk download) | `ignore_patterns` to exclude large shards |

For **gated models** (like Llama), make sure you're authenticated first:
```bash
huggingface-cli login
```
or set `HF_TOKEN` as an environment variable.


How do I load a model directly from a repository by name?


Your question about loading "a model directly from a repository by name" doesn't seem to be related to this Next.js project. Could you clarify what you mean?

For example, are you asking about:

1. **A machine learning model** (e.g., loading a Hugging Face model by repo name like `"bert-base-uncased"`)?
2. **A Next.js or JavaScript concept** — e.g., importing a module/package by name from a repository?
3. **Something specific to this project** (`hf-devtools-dashboard`) — perhaps fetching model metadata from the Hugging Face Hub API?

Given that this project appears to be a **Hugging Face devtools dashboard**, I suspect you might be asking about loading/fetching a Hugging Face model by its repo name (e.g., `"meta-llama/Llama-3.1-8B"`). Let me know and I can point you to the right approach!


Where can I find GGUF or GPTQ quantized model files?


You can find GGUF and GPTQ quantized model files in several places:

## Hugging Face Hub (Primary Source)
- **[huggingface.co/models](https://huggingface.co/models)** — Search with filters:
  - Filter by library: **GGUF** or **GPTQ**
  - Or search terms like `"GGUF"`, `"GPTQ"`, or a model name + format (e.g., `"llama gguf"`)

### Notable Publishers on HF:
- **[TheBloke](https://huggingface.co/TheBloke)** — Historically the most prolific quantizer (GGUF & GPTQ)
- **[bartowski](https://huggingface.co/bartowski)** — Active GGUF quantizer for newer models
- **[LoneStriker](https://huggingface.co/LoneStriker)** — GGUF & GPTQ quants
- **[mradermacher](https://huggingface.co/mradermacher)** — Large collection of GGUF quants

## Format-Specific Notes

| Format | Best For | Common Tools |
|--------|----------|--------------|
| **GGUF** | CPU/GPU inference, llama.cpp | llama.cpp, Ollama, LM Studio, Jan |
| **GPTQ** | GPU-only inference | AutoGPTQ, text-generation-webui, vLLM |

## Other Sources
- **[Ollama Library](https://ollama.com/library)** — Curated GGUF models, easy pull/run
- **[LM Studio](https://lmstudio.ai/)** — Has a built-in search/download UI for GGUF models
- **[GPT4All](https://gpt4all.io/)** — Curated GGUF model catalog

## Tips
- On HF, you can filter by quantization level (e.g., Q4_K_M is a popular balance of size/quality for GGUF)
- Check the model card's README for benchmarks and recommended quant levels
- For GPTQ, look for `4bit` or `8bit` variants


How do I create a live demo for my model that anyone can try?


Here are the most popular ways to create a live demo for your model that anyone can try:

---

## 🚀 Options for Hosting a Live Model Demo

### 1. **Hugging Face Spaces** *(Most Popular)*
Since you're working in a HF devtools dashboard, this is likely your best bet.

- Go to [huggingface.co/spaces](https://huggingface.co/spaces) → **Create new Space**
- Choose a SDK:
  - **Gradio** – easiest, Python-native, great for ML demos
  - **Streamlit** – more flexible UI
  - **Docker** – full control
- Push your code to the Space repo (it's a git repo)
- It auto-deploys and gives you a public URL instantly

```python
# Example: app.py with Gradio
import gradio as gr

def predict(text):
    # your model inference here
    return result

gr.Interface(fn=predict, inputs="text", outputs="text").launch()
```

**Free tier available.** Upgrade to GPU Spaces if you need acceleration.

---

### 2. **Gradio** (Self-hosted or via Spaces)
- `pip install gradio`
- Run locally and share via `share=True` for a temporary public link:
  ```python
  gr.Interface(...).launch(share=True)  # gives you a *.gradio.live URL
  ```

---

### 3. **Streamlit**
- Deploy to [streamlit.io/cloud](https://streamlit.io/cloud) for free
- Connect your GitHub repo → auto-deploys
- Good for more dashboard-style demos

---

### 4. **Replicate**
- Package your model with [Cog](https://github.com/replicate/cog)
- Push to [replicate.com](https://replicate.com) — auto-generates a UI + API
- Great if others also want API access to your model

---

### 5. **Modal / RunPod / Banana** *(GPU-heavy models)*
- For large models needing beefy GPUs
- Modal is especially developer-friendly (`modal deploy`)

---

## Quick Decision Guide

| Need | Best Option |
|---|---|
| Fastest to launch | Hugging Face Spaces (Gradio) |
| Nice custom UI | Streamlit |
| Others need API access too | Replicate |
| Large model, GPU required | Modal or HF Spaces (GPU) |
| Full control | Docker on HF Spaces or self-host |

---

Since you're already in a **HF devtools dashboard**, I'd recommend **Hugging Face Spaces with Gradio** — it's the fastest path from model to shareable demo with zero infrastructure to manage. Want help setting that up?


How do I publish a model with gated access requiring approval?


I can answer this from my knowledge about Hugging Face, which is the context of this project. Here's how to publish a model with gated access requiring approval on Hugging Face:

## Publishing a Model with Gated Access on Hugging Face

### 1. Create or Go to Your Model Repository
- Go to [huggingface.co](https://huggingface.co) and create a new model repository (or navigate to an existing one).

### 2. Enable Gated Access
In your model repository settings:
1. Go to **Settings** tab of your model repo.
2. Scroll to the **Gated model** section.
3. Toggle **Enable gated access** on.

### 3. Configure the Gating Options
You'll have a few options:

- **Manual approval** – You review and approve/reject each access request individually.
- **Automatic approval** – Requests are automatically granted (useful when you just want to collect user info).

For requiring approval, select **Manual approval**.

### 4. Customize the Access Request Form
You can customize what information requesters must provide (e.g., name, organization, use case). Edit the `README.md` (model card) to include a `extra_gated_fields` section in the YAML front matter:

```yaml
---
license: llama3
extra_gated_prompt: "Please describe your intended use case."
extra_gated_fields:
  Name: text
  Organization: text
  Country: country
  Intended use case: text
  I agree to the terms: checkbox
extra_gated_button_content: "Request Access"
---
```

### 5. Set the Repo Visibility
Make sure the repository is set to **Public** — gated access only applies to public repos. Private repos use team/org permissions instead.

### 6. Managing Requests
Once gating is active:
- Go to **Settings → Access Requests** in your repo.
- You'll see a list of pending requests with the info each user submitted.
- Approve or reject them individually, or in bulk.

### 7. Via the Hub API (optional, programmatic)
You can also configure gating via the `huggingface_hub` Python library:

```python
from huggingface_hub import HfApi

api = HfApi()
api.update_repo_settings(
    repo_id="your-username/your-model",
    gated="manual",  # or "auto"
)
```

---

**Key points to remember:**
- Users who haven't been approved will see a "Request Access" button instead of download links.
- You'll receive email notifications for new access requests.
- You can revoke access at any time from the Access Requests panel.


Best way to upload large model checkpoints (100GB+)


For uploading large model checkpoints (100GB+) to Hugging Face, here are the best approaches:

## Best Methods

### 1. **Hugging Face CLI with `huggingface-hub`** (Recommended)
```bash
pip install huggingface-hub hf_transfer

# Enable fast transfers (uses hf_transfer under the hood)
export HF_HUB_ENABLE_HF_TRANSFER=1

huggingface-cli upload my-org/my-model ./checkpoint-dir . --repo-type model
```
`hf_transfer` is a Rust-based upload accelerator — significantly faster for large files.

---

### 2. **Python API with chunked/resumable uploads**
```python
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="./checkpoint-dir",
    repo_id="my-org/my-model",
    repo_type="model",
    multi_commits=True,        # splits into multiple commits to avoid timeouts
    multi_commits_verbose=True,
)
```
`multi_commits=True` is key for 100GB+ — it batches files across several commits so no single upload times out.

---

### 3. **Upload individual large files with `upload_file`**
```python
api.upload_file(
    path_or_fileobj="./model.safetensors",
    path_in_repo="model.safetensors",
    repo_id="my-org/my-model",
    repo_type="model",
)
```

---

### 4. **Git LFS (traditional, but slower)**
```bash
git lfs install
git clone https://huggingface.co/my-org/my-model
cp ./checkpoints/* my-model/
cd my-model && git add . && git commit -m "Add checkpoints" && git push
```
Works but is generally **slower and less reliable** than the CLI for very large files.

---

## Key Tips

| Concern | Solution |
|---|---|
| **Speed** | Set `HF_HUB_ENABLE_HF_TRANSFER=1` |
| **Timeouts** | Use `multi_commits=True` |
| **Resumability** | CLI/API uploads auto-resume; Git LFS does not |
| **Format** | Prefer **safetensors** over `.bin` — faster to load and safer |
| **Sharding** | Split checkpoints into ≤5GB shards (transformers does this automatically) |
| **Auth** | `huggingface-cli login` or set `HUGGING_FACE_HUB_TOKEN` env var |

---

## For Truly Massive Uploads (500GB+)

- Consider uploading from a **cloud VM co-located with HF's servers** (EU region) to maximize bandwidth
- Use **parallel shard uploads** via `upload_folder` with `num_workers` parameter
- Check your **repo's LFS quota** — private repos have limits; contact HF for increases

The CLI + `hf_transfer` combo is the go-to for most large checkpoint uploads.


How do I run ML inference directly in the browser with JavaScript?


Running ML inference directly in the browser with JavaScript has become quite capable. Here are the main approaches:

## 1. **Transformers.js** (Hugging Face)
The most feature-rich option — runs Hugging Face models directly in the browser via ONNX Runtime.

```js
import { pipeline } from '@huggingface/transformers';

// Sentiment analysis
const classifier = await pipeline('sentiment-analysis');
const result = await classifier('I love running models in the browser!');
// [{ label: 'POSITIVE', score: 0.999 }]

// Text generation
const generator = await pipeline('text-generation', 'Xenova/gpt2');
const output = await generator('The meaning of life is', { max_new_tokens: 50 });
```

Models are downloaded and **cached in the browser** on first use.

---

## 2. **ONNX Runtime Web**
Lower-level — run any `.onnx` model with full control over inputs/outputs.

```js
import * as ort from 'onnxruntime-web';

const session = await ort.InferenceSession.create('./model.onnx');
const input = new ort.Tensor('float32', Float32Array.from([...]), [1, 3, 224, 224]);
const results = await session.run({ input });
```

Supports **WebGL**, **WebGPU**, and **WASM** backends.

---

## 3. **TensorFlow.js**
Google's ML library with its own model format and ecosystem.

```js
import * as tf from '@tensorflow/tfjs';
import * as mobilenet from '@tensorflow-models/mobilenet';

const model = await mobilenet.load();
const img = document.getElementById('myImage');
const predictions = await model.classify(img);
```

---

## 4. **WebLLM** (in-browser LLMs via WebGPU)
Run full LLMs (LLaMA, Mistral, etc.) entirely in the browser using WebGPU.

```js
import * as webllm from '@mlc-ai/web-llm';

const engine = await webllm.CreateMLCEngine('Llama-3.2-3B-Instruct-q4f32_1-MLC');
const reply = await engine.chat.completions.create({
  messages: [{ role: 'user', content: 'Hello!' }]
});
```

> ⚠ Requires a **WebGPU-capable browser** (Chrome 113+).

---

## Key Concepts to Know

| Topic | Detail |
|---|---|
| **Backend priority** | WebGPU > WebGL > WASM — libraries auto-select the best available |
| **Model caching** | Use the Cache API or IndexedDB; models can be 100s of MB |
| **Web Workers** | Always run inference in a Worker to avoid blocking the UI thread |
| **Quantization** | Use `q4`, `q8`, or `int8` quantized models to reduce size/memory |

---

## Recommended Starting Point

**Transformers.js** is usually the best starting point — it covers the widest range of tasks (NLP, vision, audio) with minimal setup and direct access to thousands of Hugging Face models.


How do I build a multi-step AI agent that can use external tools?


Here's a guide to building a **multi-step AI agent with external tools** using the Claude API and TypeScript (since this is a TypeScript/Next.js project).

---

## Two Approaches

### Option A: Tool Runner (Recommended — automatic loop)

The SDK's beta tool runner handles the agent loop for you using **Zod schemas**:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import { betaZodTool } from "@anthropic-ai/sdk/helpers/beta/zod";
import { z } from "zod";

const client = new Anthropic();

// 1. Define tools with Zod — schemas are auto-generated
const searchWeb = betaZodTool({
  name: "search_web",
  description: "Search the web for current information",
  inputSchema: z.object({
    query: z.string().describe("The search query"),
  }),
  run: async ({ query }) => {
    // Your implementation: call a search API, scrape, etc.
    const results = await mySearchAPI(query);
    return JSON.stringify(results);
  },
});

const getWeather = betaZodTool({
  name: "get_weather",
  description: "Get current weather for a city",
  inputSchema: z.object({
    city: z.string(),
    units: z.enum(["celsius", "fahrenheit"]).default("celsius"),
  }),
  run: async ({ city, units }) => {
    const data = await weatherAPI.fetch(city, units);
    return `${data.temp}°, ${data.conditions}`;
  },
});

const readFile = betaZodTool({
  name: "read_file",
  description: "Read contents of a file",
  inputSchema: z.object({
    path: z.string().describe("Absolute file path"),
  }),
  run: async ({ path }) => {
    return fs.readFileSync(path, "utf-8");
  },
});

// 2. Run the agent — SDK handles the loop automatically
const finalMessage = await client.beta.messages.toolRunner({
  model: "claude-opus-4-6",
  max_tokens: 16000,
  thinking: { type: "adaptive" },   // lets Claude reason between tool calls
  tools: [searchWeb, getWeather, readFile],
  messages: [
    {
      role: "user",
      content: "Research the latest AI news, check the weather in SF, then summarize both.",
    },
  ],
});

// The runner loops until Claude stops calling tools
for (const block of finalMessage.content) {
  if (block.type === "text") console.log(block.text);
}
```

---

### Option B: Manual Loop (full control)

Use this when you need **approval gates**, **custom logging**, or **streaming per step**:

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const tools: Anthropic.Tool[] = [
  {
    name: "search_web",
    description: "Search the web for current information",
    input_schema: {
      type: "object",
      properties: {
        query: { type: "string", description: "The search query" },
      },
      required: ["query"],
    },
  },
  {
    name: "get_weather",
    description: "Get current weather for a city",
    input_schema: {
      type: "object",
      properties: {
        city:  { type: "string" },
        units: { type: "string", enum: ["celsius", "fahrenheit"] },
      },
      required: ["city"],
    },
  },
];

async function executeTool(name: string, input: unknown): Promise<string> {
  switch (name) {
    case "search_web":
      return JSON.stringify(await mySearchAPI((input as any).query));
    case "get_weather":
      const { city, units } = input as { city: string; units?: string };
      return await weatherAPI.fetch(city, units ?? "celsius");
    default:
      return `Unknown tool: ${name}`;
  }
}

async function runAgent(userMessage: string): Promise<string> {
  let messages: Anthropic.MessageParam[] = [
    { role: "user", content: userMessage },
  ];

  while (true) {
    const response = await client.messages.create({
      model: "claude-opus-4-6",
      max_tokens: 16000,
      thinking: { type: "adaptive" },
      tools,
      messages,
    });

    console.log(`Stop reason: ${response.stop_reason}`);

    // Done — no more tool calls
    if (response.stop_reason === "end_turn") {
      const text = response.content.find((b): b is Anthropic.TextBlock => b.type === "text");
      return text?.text ?? "";
    }

    // Server-side tool hit iteration limit — resume
    if (response.stop_reason === "pause_turn") {
      messages.push({ role: "assistant", content: response.content });
      continue;
    }

    // Claude wants to call tools
    const toolUseBlocks = response.content.filter(
      (b): b is Anthropic.ToolUseBlock => b.type === "tool_use"
    );

    // Append Claude's response (includes tool_use blocks — required!)
    messages.push({ role: "assistant", content: response.content });

    // Execute all tools (can be parallelized)
    const toolResults: Anthropic.ToolResultBlockParam[] = await Promise.all(
      toolUseBlocks.map(async (tool) => {
        console.log(`  → Calling: ${tool.name}(${JSON.stringify(tool.input)})`);
        const result = await executeTool(tool.name, tool.input);
        console.log(`  ← Result: ${result.slice(0, 100)}...`);

        return {
          type: "tool_result" as const,
          tool_use_id: tool.id,    // must match the tool_use block
          content: result,
        };
      })
    );

    // Feed results back to Claude
    messages.push({ role: "user", content: toolResults });
  }
}

const answer = await runAgent(
  "What's the weather in Tokyo, and what are the top AI news stories today?"
);
console.log(answer);
```

---

## With Streaming (see each step in real time)

```typescript
async function runAgentStreaming(userMessage: string) {
  let messages: Anthropic.MessageParam[] = [
    { role: "user", content: userMessage },
  ];

  while (true) {
    const stream = client.messages.stream({
      model: "claude-opus-4-6",
      max_tokens: 64000,     // higher limit safe with streaming
      thinking: { type: "adaptive" },
      tools,
      messages,
    });

    // Print text as it arrives
    stream.on("text", (delta) => process.stdout.write(delta));

    const message = await stream.finalMessage();  // collects the full response

    if (message.stop_reason === "end_turn") break;
    if (message.stop_reason === "pause_turn") {
      messages.push({ role: "assistant", content: message.content });
      continue;
    }

    const toolUseBlocks = message.content.filter(
      (b): b is Anthropic.ToolUseBlock => b.type === "tool_use"
    );

    messages.push({ role: "assistant", content: message.content });

    const toolResults: Anthropic.ToolResultBlockParam[] = await Promise.all(
      toolUseBlocks.map(async (tool) => ({
        type: "tool_result" as const,
        tool_use_id: tool.id,
        content: await executeTool(tool.name, tool.input),
      }))
    );

    messages.push({ role: "user", content: toolResults });
  }
}
```

---

## Key Rules to Remember

| Rule | Why it matters |
|---|---|
| Always append `response.content` (not just text) | Tool use blocks must be preserved in history |
| Each `tool_result` must include the matching `tool_use_id` | Claude uses this to correlate call → result |
| Loop until `stop_reason === "end_turn"` | `tool_use` means Claude wants another round |
| Handle `pause_turn` by re-sending without a new user message | Server-side tools can pause mid-loop |
| Use `thinking: { type: "adaptive" }` on Opus 4.6 | Lets Claude reason between tool calls — no `budget_tokens` needed |

---

## When to Use Which Approach

| Need | Use |
|---|---|
| Simple automation, few tools | **Tool Runner** (`betaZodTool`) |
| Human approval before tool execution | **Manual loop** |
| Custom logging / observability | **Manual loop** |
| Streaming each step to a UI | **Manual loop + `stream()`** |
| Long-running agent (Anthropic hosts the loop) | **Managed Agents** (`client.beta.agents`) |

The **Managed Agents** surface (via `client.beta.agents` / `client.beta.sessions`) is the right choice when you want Anthropic to fully host the agent loop and container — see the managed-agents docs if that fits your use case.


Best way to add image classification to a mobile app


## Adding Image Classification to a Mobile App

Here are the best approaches, ranked by ease of implementation:

---

### 1. **On-Device ML (Best for Privacy & Offline Use)**

**iOS**
- **Core ML + Vision framework** — Apple's native stack; fast, private, no network needed
- **Create ML** — Train custom classifiers directly on Mac, export to `.mlmodel`

**Android**
- **ML Kit (Google)** — Easiest option; pre-built image labeling API, works offline
- **TensorFlow Lite** — More control; run `.tflite` models on-device

**Cross-platform**
- **MediaPipe** — Google's cross-platform ML pipeline; strong image classification support
- **ONNX Runtime Mobile** — Run any ONNX model on iOS/Android

---

### 2. **Cloud API (Best for Accuracy & Low Dev Effort)**

| Service | Strengths |
|---|---|
| **Google Cloud Vision** | Broad label detection, OCR, object detection |
| **AWS Rekognition** | Deep AWS integration, custom labels |
| **Azure Computer Vision** | Strong custom model support |
| **Clarifai** | Purpose-built for visual AI |

Trade-offs: requires network, has latency, costs money at scale, raises privacy concerns.

---

### 3. **Hybrid Approach (Recommended for Production)**

Run a **lightweight model on-device** for common/fast cases, fall back to a **cloud API** for edge cases or higher accuracy needs.

---

### Practical Recommendations by Stack

| Your Stack | Best Choice |
|---|---|
| React Native | TensorFlow.js + `@tensorflow/tfjs-react-native` or ML Kit via community libs |
| Flutter | `tflite_flutter` plugin or Google ML Kit plugin |
| Swift / SwiftUI | Core ML + Vision (no brainer) |
| Kotlin / Jetpack Compose | ML Kit or TensorFlow Lite |
| Expo (React Native) | Cloud API is simplest; on-device requires bare workflow |

---

### Key Decision Factors

1. **Privacy** → On-device wins
2. **Accuracy** → Cloud or large on-device model
3. **Offline support** → On-device only
4. **Custom categories** → Train your own model (Create ML, Teachable Machine, or fine-tune MobileNet/EfficientNet)
5. **Speed to ship** → ML Kit or a cloud API

---

### Quick Start for Most Teams

> **ML Kit (Android) or Core ML (iOS)** for native apps, or **TensorFlow.js** for React Native — these have the best documentation, community support, and are free at reasonable usage levels.

What's your mobile stack? I can give more specific guidance.


How do I build a real-time translation feature using an open model?


Here's how to build a real-time translation feature using Claude as the open model backend:

---

## Real-Time Translation with Claude

This project uses TypeScript (detected from your `package.json`/`tsconfig.json`), so I'll use the `@anthropic-ai/sdk`.

### Architecture

For real-time translation the right surface is **streaming** — it makes the translated text appear word-by-word as Claude generates it, which feels instantaneous to the user.

---

### Installation

```bash
npm install @anthropic-ai/sdk
```

---

### Basic Streaming Translation

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env

async function translateStream(
  text: string,
  targetLanguage: string,
  sourceLanguage = "auto-detect"
): Promise<string> {
  const stream = client.messages.stream({
    model: "claude-opus-4-6",
    max_tokens: 4096,
    system: `You are a professional translator. Translate the user's text to ${targetLanguage}.
${sourceLanguage !== "auto-detect" ? `Source language: ${sourceLanguage}.` : "Auto-detect the source language."}
Output ONLY the translated text — no explanations, no labels, no quotes.`,
    messages: [{ role: "user", content: text }],
  });

  // Stream tokens to stdout in real time
  stream.on("text", (delta) => process.stdout.write(delta));

  const final = await stream.finalMessage();
  const textBlock = final.content.find((b): b is Anthropic.TextBlock => b.type === "text");
  return textBlock?.text ?? "";
}

// Example usage
await translateStream("The quick brown fox jumps over the lazy dog.", "French");
```

---

### Multi-Language Batch (Parallel Streams)

Translate to multiple languages simultaneously:

```typescript
async function translateToMany(
  text: string,
  languages: string[]
): Promise<Record<string, string>> {
  const results = await Promise.all(
    languages.map(async (lang) => {
      const stream = client.messages.stream({
        model: "claude-opus-4-6",
        max_tokens: 4096,
        system: `Translate the user's text to ${lang}. Output ONLY the translation.`,
        messages: [{ role: "user", content: text }],
      });

      const final = await stream.finalMessage();
      const textBlock = final.content.find(
        (b): b is Anthropic.TextBlock => b.type === "text"
      );
      return [lang, textBlock?.text ?? ""] as const;
    })
  );

  return Object.fromEntries(results);
}

const translations = await translateToMany(
  "Hello, how are you today?",
  ["Spanish", "Japanese", "Arabic", "German"]
);
console.log(translations);
```

---

### Streaming Translation API (Express / SSE)

For a web frontend, expose translations as Server-Sent Events:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import express from "express";

const client = new Anthropic();
const app = express();
app.use(express.json());

app.post("/translate/stream", async (req, res) => {
  const { text, targetLanguage, sourceLanguage = "auto-detect" } = req.body;

  // Set up SSE headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  try {
    const stream = client.messages.stream({
      model: "claude-opus-4-6",
      max_tokens: 4096,
      system: `You are a professional translator. Translate the user's text to ${targetLanguage}.
${sourceLanguage !== "auto-detect" ? `Source language: ${sourceLanguage}.` : "Auto-detect the source language."}
Output ONLY the translated text.`,
      messages: [{ role: "user", content: text }],
    });

    // Forward each text delta to the client
    stream.on("text", (delta) => {
      res.write(`data: ${JSON.stringify({ delta })}\n\n`);
    });

    await stream.finalMessage();
    res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
    res.end();
  } catch (error) {
    if (error instanceof Anthropic.RateLimitError) {
      res.write(`data: ${JSON.stringify({ error: "Rate limited, please retry" })}\n\n`);
    } else if (error instanceof Anthropic.APIError) {
      res.write(`data: ${JSON.stringify({ error: error.message })}\n\n`);
    }
    res.end();
  }
});

app.listen(3000, () => console.log("Translation server running on :3000"));
```

**Frontend consumption:**

```typescript
const source = new EventSource("/translate/stream"); // after POST sets it up
// or use fetch + ReadableStream for POST bodies:

async function streamTranslation(text: string, targetLanguage: string) {
  const response = await fetch("/translate/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text, targetLanguage }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let output = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const lines = decoder.decode(value).split("\n");
    for (const line of lines) {
      if (!line.startsWith("data: ")) continue;
      const payload = JSON.parse(line.slice(6));
      if (payload.delta) {
        output += payload.delta;
        // Update your UI here in real time
        document.getElementById("output")!.textContent = output;
      }
    }
  }
}
```

---

### With Prompt Caching (Cost Optimization)

If you're translating many strings with the same system instructions, cache the system prompt to cut costs ~90%:

```typescript
const response = await client.messages.create({
  model: "claude-opus-4-6",
  max_tokens: 4096,
  system: [
    {
      type: "text",
      text: `You are a professional translator specializing in technical documents.
[... extensive style guide, glossary, tone instructions ...]`,
      cache_control: { type: "ephemeral" }, // cached after first request
    },
  ],
  messages: [{ role: "user", content: textToTranslate }],
});

// Check cache savings:
console.log("Cache read tokens:", response.usage.cache_read_input_tokens);
```

---

### Key Design Notes

| Concern | Recommendation |
|---|---|
| **Latency** | Always stream — first tokens appear in ~300ms |
| **Quality** | Opus 4.6 produces excellent translation quality across 100+ languages |
| **Cost** | Cache the system prompt for bulk translation workloads |
| **Throughput** | Use `Promise.all` for parallel language targets |
| **Long documents** | Chunk by paragraph; translate each chunk in parallel |
| **Tone/formality** | Add to system prompt: *"Use formal register"* / *"Match the source tone"* |

For very long documents (books, manuals), consider the **Batches API** at 50% cost reduction — see `typescript/claude-api/batches.md` for the polling pattern.


How do I create an AI-powered code review tool?