What are the best datasets for fine-tuning a code generation model?


Great question! Here are some of the most well-regarded datasets for fine-tuning code generation models:

## Large-Scale Pretraining Datasets

- **The Stack (v2)** — A massive, deduplicated dataset from Software Heritage covering 600+ programming languages with permissive licensing info. One of the most comprehensive open datasets available.
- **StarCoder Data** — The curated dataset behind StarCoder/StarCoder2, sourced from GitHub with careful filtering, deduplication, and opt-out compliance.
- **RedPajama** — Includes a substantial code component alongside other data modalities.

## Instruction-Tuned / Fine-Tuning Datasets

- **CodeAlpaca** — ~20K instruction-following examples for code, generated in the style of Stanford Alpaca.
- **OASST (Code subset)** — Human-written, multi-turn conversations including code tasks from Open Assistant.
- **Evol-Instruct-Code** — Used to train WizardCoder; applies "Evol-Instruct" to evolve coding instructions into increasingly complex variants.
- **CommitPackFT** — Instruction-tuned dataset based on high-quality Git commits (used for OctoCoder).
- **Self-Instruct / OSS-Instruct** — Synthetic instruction datasets seeded from open-source code snippets (used for Magicoder).

## Benchmark / Evaluation Datasets (also useful for targeted fine-tuning)

- **HumanEval / HumanEval+** — OpenAI's 164 hand-written Python problems with unit tests.
- **MBPP / MBPP+** — ~1,000 crowd-sourced Python programming problems.
- **APPS** — 10,000 coding problems ranging from introductory to competition level.
- **DS-1000** — 1,000 data science problems grounded in real StackOverflow questions.
- **CodeContests** — Competitive programming problems from Codeforces, etc. (from DeepMind).
- **SWE-bench** — Real-world GitHub issues and their corresponding pull request fixes — great for training models on realistic software engineering tasks.

## Key Tips for Dataset Selection

1. **Match your use case** — If you want chat-style code assistance, use instruction-tuned data. If you want code completion, use raw code corpora.
2. **Language mix matters** — Ensure your dataset covers the languages you care about in the right proportions.
3. **Quality > Quantity** — Smaller, high-quality curated datasets (e.g., Evol-Instruct-Code, CommitPackFT) often outperform larger noisy ones for fine-tuning.
4. **Licensing** — Pay attention to licensing, especially if you're deploying commercially. The Stack v2 provides license metadata per file.
5. **Deduplication** — Ensure your dataset is properly deduplicated to avoid memorization and benchmark contamination.

Most state-of-the-art open code models (StarCoder2, CodeLlama, DeepSeek-Coder, Magicoder) combine a large pretraining corpus with a smaller, high-quality instruction-tuning dataset for the best results.


Where can I find high-quality instruction-tuning datasets?


Here are some of the best sources for high-quality instruction-tuning datasets:

## Major Hubs

- **Hugging Face Datasets Hub** (`huggingface.co/datasets`) — The largest open repository. Filter by task type (e.g., "text-generation", "question-answering") and sort by downloads/likes. Notable collections:
  - **Open Assistant** (`OpenAssistant/oasst1`, `oasst2`) — Human-generated, multi-turn conversations with quality rankings.
  - **Dolly** (`databricks/databricks-dolly-15k`) — 15K human-written instruction/response pairs across multiple categories.
  - **FLAN Collection** (`Muennighoff/flan`) — Massive collection of tasks reformatted as instructions.
  - **SlimOrca** / **OpenOrca** — Large-scale GPT-4 augmented instruction data.
  - **UltraChat** / **UltraFeedback** — High-volume multi-turn chat and preference data.

## High-Quality Curated Sets

| Dataset | Size | Notes |
|---|---|---|
| **Alpaca** (Stanford) | ~52K | GPT-generated, simple instructions |
| **ShareGPT / ShareGPT-Vicuna** | ~90K+ | Real user–ChatGPT conversations |
| **WizardLM Evol-Instruct** | ~250K | Complexity-evolved instructions |
| **Orca-Math** | ~200K | Strong for math reasoning |
| **MetaMathQA** | ~395K | Math-focused instruction tuning |
| **Code Alpaca / Code Feedback** | Varies | Code-specific instruction data |
| **Capybara** | ~16K | Multi-turn, high-quality |
| **Deita** | ~10K | Quality-filtered subset approach |

## Tips for Choosing

1. **Quality > Quantity** — Smaller, well-curated datasets (like LIMA's ~1K examples) can outperform massive noisy ones.
2. **Check licenses** — Some datasets (e.g., ShareGPT) have unclear licensing due to model-generated content. For commercial use, look for datasets with permissive licenses (Apache 2.0, CC-BY).
3. **Match your domain** — Use domain-specific sets for specialized models (medical, legal, code, math).
4. **Consider preference data** — For RLHF/DPO, look at **UltraFeedback**, **HH-RLHF** (Anthropic), or **Nectar**.
5. **Data decontamination** — Ensure your eval benchmarks aren't leaked into training data.

## Other Sources

- **Allen AI (ai2)** — Tülu datasets, high-quality research mixes
- **LMSYS** — Chatbot Arena conversation data
- **Papers With Code** — Search for datasets linked to instruction-tuning papers
- **GitHub awesome-lists** — e.g., `awesome-instruction-datasets`

Would you like recommendations tailored to a specific use case (e.g., code, math, multi-turn chat, a particular language)?


What open-source datasets are available for training a chatbot?


There are many open-source datasets available for training chatbots. Here's an overview of the most notable ones:

## Conversational / Dialogue Datasets

- **OpenAssistant Conversations (OASST)** — A large crowd-sourced dataset of multi-turn conversations with human-generated assistant responses, organized in conversation trees.
- **ShareGPT** — User-shared conversations with ChatGPT, widely used in the open-source fine-tuning community (e.g., Vicuna was trained on this).
- **Anthropic HH-RLHF** — Human preference data with chosen/rejected response pairs, useful for RLHF and DPO training.
- **Stanford Alpaca** — ~52K instruction-following examples generated from GPT, designed for fine-tuning LLMs.
- **Dolly (Databricks)** — ~15K human-written instruction/response pairs across multiple categories, with a permissive license.
- **LMSYS-Chat-1M** — 1 million real-world conversations from the Chatbot Arena platform.

## Instruction-Following Datasets

- **FLAN Collection (Google)** — Massive collection of tasks reformatted as instructions, used to train Flan-T5/PaLM.
- **Open Orca** — Millions of augmented FLAN data points with reasoning traces.
- **SlimOrca** — A curated, deduplicated subset of Open Orca.
- **UltraChat** — ~1.5M multi-turn instructional dialogues covering a wide range of topics.
- **WizardLM Evol-Instruct** — Instructions evolved for increasing complexity.

## Domain-Specific Datasets

- **Medical:** PubMedQA, MedDialog, ChatDoctor
- **Code:** The Stack, CodeAlpaca, CodeFeedback
- **Finance:** FiQA, FinGPT datasets
- **Science:** SciQ, SciFact

## Preference / Alignment Datasets

- **UltraFeedback** — Large-scale AI feedback dataset for preference learning.
- **Nectar (Berkeley)** — Multi-turn preference dataset with rankings.
- **HelpSteer (NVIDIA)** — Fine-grained human ratings on helpfulness, correctness, coherence, etc.

## Classic / Foundational Dialogue Datasets

- **Cornell Movie Dialogs Corpus** — ~220K conversational exchanges from movie scripts.
- **PersonaChat** — Conversations grounded in assigned personas.
- **DailyDialog** — ~13K human-written daily conversations with emotion/topic labels.
- **Wizard of Wikipedia** — Conversations grounded in Wikipedia knowledge.
- **MultiWOZ** — Multi-domain task-oriented dialogues (restaurants, hotels, etc.).

## Where to Find Them

- **[Hugging Face Datasets Hub](https://huggingface.co/datasets)** — The largest repository; search for "chat", "instruct", or "dialogue".
- **[Papers With Code](https://paperswithcode.com/)** — Links datasets to the papers that introduced them.
- **GitHub** — Many datasets are hosted directly in repos (e.g., Alpaca, Dolly).

## Key Considerations

| Factor | What to Check |
|---|---|
| **License** | Some datasets (e.g., ShareGPT) have unclear licensing; Dolly and OASST have permissive licenses |
| **Quality** | Curated human-written data generally outperforms synthetic data |
| **Size** | More data isn't always better — quality and diversity matter |
| **Format** | Match the dataset format to your training framework (e.g., ChatML, Alpaca-style) |

Would you like more detail on any of these, or help selecting the right dataset for a specific use case?


Best multilingual NLP datasets for production use


# Best Multilingual NLP Datasets for Production Use

Here's a curated overview of top multilingual NLP datasets, organized by task:

---

## 🔤 General Language Understanding

| Dataset | Languages | Use Case |
|---|---|---|
| **XNLI** | 15 languages | Natural language inference / cross-lingual transfer |
| **XTREME / XTREME-R** | 40+ languages | Comprehensive benchmark (QA, classification, retrieval, etc.) |
| **GLUE-X / SuperGLUE** (multilingual variants) | Varies | General NLU tasks |

## 📝 Text Classification & Sentiment

| Dataset | Languages | Use Case |
|---|---|---|
| **Amazon Reviews Multilingual** | 6 languages | Sentiment analysis at scale |
| **MLDoc** | 8 languages | Document classification |
| **Multilingual Sentiment (Twitter)** | 12+ languages | Social media sentiment |

## ❓ Question Answering

| Dataset | Languages | Use Case |
|---|---|---|
| **XQuAD** | 12 languages | Cross-lingual extractive QA |
| **MLQA** | 7 languages | Multilingual extractive QA |
| **TyDi QA** | 11 typologically diverse languages | QA across diverse language families |
| **MKQA** | 26 languages | Open-domain QA |

## 🏷 Named Entity Recognition (NER)

| Dataset | Languages | Use Case |
|---|---|---|
| **WikiANN (PAN-X)** | 282 languages | Broad-coverage NER |
| **MultiCoNER** | 11 languages | Complex, noisy NER |
| **MasakhaNER** | 10 African languages | Low-resource NER |

## 🌐 Machine Translation & Parallel Corpora

| Dataset | Languages | Use Case |
|---|---|---|
| **OPUS / OPUS-100** | 100 language pairs | Parallel text for MT |
| **CCAligned / CCMatrix** | 100+ languages | Web-mined parallel sentences |
| **FLORES-200** | 200 languages | MT evaluation benchmark |
| **Tatoeba** | 300+ languages | Sentence-level translation pairs |

## 🗣 Semantic Similarity & Retrieval

| Dataset | Languages | Use Case |
|---|---|---|
| **STS Multilingual** | 10+ languages | Sentence similarity |
| **MIRACL** | 18 languages | Information retrieval |
| **Mr. TyDi** | 11 languages | Multilingual dense retrieval |

## 📚 Large-Scale Pretraining Corpora

| Dataset | Languages | Use Case |
|---|---|---|
| **CC-100** | 100+ languages | Language model pretraining |
| **mC4** | 101 languages | Multilingual T5 pretraining data |
| **CulturaX** | 167 languages | Cleaned, deduplicated web text |
| **ROOTS** | 46 languages | BLOOM model training corpus |

---

## 🏭 Production Recommendations

1. **Start with XTREME-R** as a benchmark suite to evaluate cross-lingual transfer of your models.
2. **For NER**, WikiANN offers unmatched language coverage; pair it with domain-specific annotations.
3. **For retrieval/RAG pipelines**, MIRACL and Mr. TyDi are gold standards for multilingual evaluation.
4. **For low-resource languages**, look at MasakhaNER (African), IndicNLP (Indic), and FLORES-200.
5. **For pretraining/fine-tuning**, CulturaX and mC4 provide the cleanest large-scale multilingual web text.

### Key Tips for Production
- **License audit** — CC-100 and mC4 are derived from Common Crawl (broadly permissive); Amazon Reviews has specific terms. Always verify.
- **Data quality** — Web-mined datasets (CCAligned, mC4) benefit greatly from additional filtering and deduplication.
- **Evaluation across scripts** — Don't just test Romance languages; include CJK, Arabic, Devanagari, and agglutinative languages (Turkish, Finnish) to catch tokenization/encoding issues.

Most of these are available via [Hugging Face Datasets](https://huggingface.co/datasets) for easy loading with `datasets.load_dataset()`.


What datasets should I use for evaluating LLM reasoning?


Great question! Here are some widely-used datasets and benchmarks for evaluating LLM reasoning, organized by category:

## Mathematical Reasoning
- **GSM8K** — Grade-school math word problems (8.5K examples). A staple benchmark.
- **MATH** — Competition-level math (algebra, geometry, number theory, etc.).
- **MathQA / AQuA-RAT** — Math with multiple-choice and rationale annotations.
- **MINERVA / HENDRYCKS MATH** — More challenging math across difficulty levels.

## Logical & Formal Reasoning
- **LogiQA** — Logical reasoning from Chinese civil service exams.
- **ReClor** — Reading comprehension requiring logical reasoning.
- **FOLIO** — First-order logic reasoning with natural language.
- **ProntoQA / ProofWriter** — Synthetic deductive reasoning chains.

## Commonsense Reasoning
- **HellaSwag** — Sentence completion requiring commonsense.
- **WinoGrande** — Pronoun resolution (Winograd schema).
- **PIQA** — Physical intuition / everyday physics.
- **CommonsenseQA / OpenBookQA** — General commonsense Q&A.
- **ARC (AI2 Reasoning Challenge)** — Grade-school science questions (Easy & Challenge sets).

## Multi-Step & Complex Reasoning
- **BIG-Bench Hard (BBH)** — A curated subset of BIG-Bench tasks that require multi-step reasoning.
- **MMLU (Massive Multitask Language Understanding)** — 57 subjects, tests reasoning across domains.
- **GPQA** — Graduate-level science questions (very hard, expert-validated).
- **MuSR** — Multi-step soft reasoning over narratives.

## Code Reasoning
- **HumanEval / MBPP** — Code generation requiring algorithmic reasoning.
- **CRUXEval** — Code reasoning (predicting inputs/outputs).
- **SWE-bench** — Real-world software engineering tasks.

## Agentic / Tool-Use Reasoning
- **ToolBench / API-Bank** — Reasoning about when and how to use tools.
- **ALFWorld / WebArena** — Multi-step planning in simulated environments.

## Meta-Benchmarks & Leaderboards
- **LMSYS Chatbot Arena** — Human preference (ELO-based).
- **Open LLM Leaderboard (Hugging Face)** — Aggregates multiple benchmarks.
- **HELM (Stanford)** — Holistic evaluation across many axes.

---

## Recommendations by Goal

| Goal | Start With |
|---|---|
| Quick sanity check | GSM8K, HellaSwag, ARC |
| Rigorous reasoning eval | BBH, MATH, GPQA |
| Broad capability survey | MMLU, HELM |
| Cutting-edge difficulty | GPQA, SWE-bench, MATH (Level 5) |

**Tip:** Combine benchmarks across categories for a more complete picture — strong math reasoning doesn't guarantee strong commonsense reasoning, and vice versa. Also consider **contamination risks**: newer or private benchmarks (like GPQA) are less likely to appear in training data.

Would you like more detail on any specific category or guidance on setting up evaluations?


What's the best LLM for code generation right now?


That's a great question! The landscape evolves quickly, but here's a snapshot as of my knowledge:

## Top LLMs for Code Generation

**Frontier Models:**
- **Claude (Anthropic)** — Strong at complex reasoning, large codebase understanding, and following nuanced instructions. Excels at refactoring and multi-file changes.
- **GPT-4o / GPT-4.1 (OpenAI)** — Very capable at code generation across many languages, with strong instruction following.
- **Gemini 2.5 Pro (Google)** — Competitive on coding benchmarks, strong with long contexts.

**Code-Specialized Models:**
- **Codex (OpenAI)** — Purpose-built for code tasks within OpenAI's ecosystem.
- **DeepSeek Coder V2/V3** — Impressive open-weight models that punch above their weight on coding benchmarks.
- **Qwen2.5-Coder (Alibaba)** — Strong open-weight option, available in various sizes.

## It Depends on Your Use Case

| Use Case | Strong Picks |
|---|---|
| **Multi-file refactoring** | Claude, GPT-4.1 |
| **Quick completions / autocomplete** | Codestral, DeepSeek, Copilot |
| **Explaining / reviewing code** | Claude, GPT-4o |
| **Open-weight / self-hosted** | DeepSeek Coder, Qwen2.5-Coder, CodeLlama |
| **Agentic coding (autonomous)** | Claude, GPT-4.1 |

## Key Takeaway

There's no single "best" — it depends on the **task**, **language**, **context window needs**, and whether you need **open-weight vs. API**. The top frontier models (Claude, GPT-4, Gemini) are all very strong and trade benchmark leads frequently. For self-hosted or cost-sensitive use cases, DeepSeek and Qwen are excellent choices.

Want me to help evaluate one of these for a specific use case you have in mind?


Which model should I use for structured data extraction from text?


Great question! Here's a breakdown of the best models for structured data extraction from text:

## Best Overall: **Claude Sonnet**

For most structured extraction tasks (JSON output, filling schemas, parsing documents), **Claude 3.5/4 Sonnet** hits the sweet spot of accuracy, speed, and cost. It excels at:

- Extracting entities, dates, amounts, and relationships
- Outputting well-formed JSON matching a schema
- Handling messy, real-world text (emails, invoices, reports)

## When to Choose Other Models

| Model | Use When |
|---|---|
| **Claude Opus** | Extremely complex or ambiguous documents; multi-step reasoning needed to resolve extraction |
| **Claude Sonnet** | Best default — high accuracy, fast, cost-effective |
| **Claude Haiku** | High-volume, simple extractions (e.g., pulling names/dates from clean text); lowest cost & latency |
| **GPT-4o** | Already in an OpenAI ecosystem |
| **Fine-tuned smaller models** | Millions of identical extractions with a fixed schema (cost optimization) |

## Key Tips for Structured Extraction

1. **Use tool/function calling or JSON mode** — Forces the model to return valid structured output rather than free text.
2. **Provide a clear schema** — Define the exact fields, types, and descriptions you want extracted.
3. **Include examples** — 1–2 input/output examples dramatically improve consistency.
4. **Use Pydantic / Zod validation** — Parse and validate the output programmatically.

## Quick Example (Anthropic API)

```python
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    tools=[{
        "name": "extract_invoice",
        "description": "Extract structured data from an invoice",
        "input_schema": {
            "type": "object",
            "properties": {
                "vendor": {"type": "string"},
                "total": {"type": "number"},
                "date": {"type": "string", "description": "ISO 8601 date"},
                "line_items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "description": {"type": "string"},
                            "amount": {"type": "number"}
                        }
                    }
                }
            },
            "required": ["vendor", "total", "date", "line_items"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract_invoice"},
    messages=[{"role": "user", "content": f"Extract data from this invoice:\n\n{invoice_text}"}]
)
```

**TL;DR** — Start with **Claude Sonnet** + tool use for structured output. Move to Haiku if you need to scale up volume cheaply, or Opus if accuracy on complex documents is critical.


Best multimodal model for analyzing images and text together


Based on current capabilities, here are the top multimodal models for analyzing images and text together:

## Top Tier

1. **GPT-4o (OpenAI)** — Natively multimodal, strong at detailed image understanding, OCR, chart/graph analysis, and reasoning across visual + textual inputs.

2. **Claude (Anthropic)** — Excellent vision capabilities, particularly strong at document analysis, detailed image description, and reasoning about visual content with nuance. Handles complex multi-step visual reasoning well.

3. **Gemini 1.5 Pro / 2.0 (Google DeepMind)** — Natively multimodal with a very large context window, great for processing many images alongside text (e.g., video frames, long documents with figures).

## Best For Specific Use Cases

| Use Case | Best Pick |
|---|---|
| **Document/PDF analysis** | Claude or GPT-4o |
| **Chart & data extraction** | GPT-4o or Claude |
| **Many images + long context** | Gemini 1.5/2.0 Pro |
| **Creative image interpretation** | GPT-4o or Claude |
| **Scientific/medical imagery** | GPT-4o or Claude |
| **OCR & handwriting** | GPT-4o or Gemini |
| **Video understanding** | Gemini 2.0 |

## Open-Source Options

- **LLaVA-NeXT / LLaVA-OneVision** — Strong open-source multimodal model
- **InternVL 2.5** — Competitive with proprietary models on many benchmarks
- **Qwen2-VL** — Excellent multilingual multimodal support

## Key Takeaway

There's no single "best" — it depends on your task. For **general-purpose image+text analysis**, **GPT-4o**, **Claude**, and **Gemini 2.0** are all excellent and closely competitive. If you need a **large context window** for many images, Gemini stands out. For **careful reasoning and nuance**, Claude tends to excel.

Would you like a recommendation tailored to a specific use case?


What's the best embedding model for semantic search?


Great question! Here's a rundown of the top embedding models for semantic search:

## Top Tier (as of my knowledge)

### **OpenAI `text-embedding-3-large`**
- 3072 dimensions (can be truncated for efficiency)
- Excellent general-purpose performance
- Easy API access, but proprietary

### **Cohere `embed-v3`**
- Strong multilingual support
- Offers search-specific optimization (input types: `search_document` vs `search_query`)
- Compressed embeddings available

### **Google `text-embedding-004`**
- Competitive performance, good pricing
- Task-type hints for search vs classification

## Best Open-Source Options

### **`BAAI/bge-large-en-v1.5`**
- One of the best open models on MTEB benchmarks
- 1024 dimensions, runs locally

### **`sentence-transformers/all-MiniLM-L6-v2`**
- Only 384 dimensions — very fast and lightweight
- Great for prototyping or resource-constrained environments

### **`intfloat/e5-large-v2`** / **`intfloat/multilingual-e5-large`**
- Strong performance, especially with the `"query: "` / `"passage: "` prefix convention
- Multilingual variant is excellent for cross-language search

## How to Choose

| Factor | Recommendation |
|---|---|
| **Best overall quality** | `text-embedding-3-large` (OpenAI) or Cohere `embed-v3` |
| **Best open-source** | `bge-large-en-v1.5` or `e5-large-v2` |
| **Low latency / small footprint** | `all-MiniLM-L6-v2` |
| **Multilingual** | Cohere `embed-v3` or `multilingual-e5-large` |
| **On-device / private data** | Any open-source model via `sentence-transformers` |

## Practical Tips

1. **Always benchmark on *your* data** — MTEB leaderboard rankings don't always transfer to domain-specific corpora.
2. **Normalize embeddings** and use **cosine similarity** (most models are trained for this).
3. **Matryoshka models** (like OpenAI's) let you truncate dimensions for a speed/quality tradeoff.
4. **Asymmetric search** (short query → long document) benefits from models that distinguish query vs. passage inputs (e5, Cohere).

Check the [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for the latest benchmarks — the space moves fast!


Which LLM has the best long context window for processing large documents?


This is a general knowledge question, so I'll share what I know (keeping in mind my training data has a cutoff and the landscape evolves rapidly):

## Top LLMs for Long Context Windows

Here are some of the leading models known for large context windows:

| Model | Context Window |
|---|---|
| **Google Gemini 1.5 Pro** | **1M–2M tokens** — One of the largest available, strong at retrieving info across very long documents ("needle in a haystack"). |
| **Anthropic Claude (3.5 / Opus family)** | **200K tokens** — Known for strong recall and reasoning quality across its full context window. |
| **OpenAI GPT-4o / GPT-4 Turbo** | **128K tokens** — Widely used, though effective utilization can degrade in the middle of very long contexts. |
| **Mistral Large** | **128K tokens** |
| **Cohere Command R+** | **128K tokens** — Specifically optimized for RAG and document processing. |

## Key Considerations

- **Window size ≠ quality.** A model with a 1M-token window isn't necessarily *better* at understanding long documents — what matters is **effective context utilization** (how well it reasons over and retrieves information across the full window).
- **"Lost in the middle" problem:** Some models struggle with information placed in the middle of long contexts. Gemini 1.5 Pro and Claude have been noted for handling this relatively well.
- **Cost:** Longer contexts = more tokens = higher cost. For very large document sets, **RAG (Retrieval-Augmented Generation)** is often more cost-effective than stuffing everything into the context window.
- **Use case matters:** For summarization, Claude and Gemini tend to excel. For precise retrieval from legal/financial docs, benchmarks vary.

**As of my knowledge cutoff**, **Google Gemini 1.5 Pro** has the largest raw context window, while **Claude** is often praised for the quality of its reasoning across its full 200K window. I'd recommend checking recent benchmarks (like RULER or "needle-in-a-haystack" evaluations) for the most up-to-date comparisons, as this space moves fast.


What's the best open-source alternative to GPT-4 I can self-host?


Great question! Here are the top open-source alternatives to GPT-4 you can self-host, ranked by general capability:

## Top Tier

1. **Llama 3 (Meta)** — The 70B and 405B parameter versions are extremely capable. The 405B model is arguably the closest open-source match to GPT-4.
   - License: Llama 3 Community License (permissive, with usage thresholds)
   - VRAM: ~40GB (70B quantized), ~200GB+ (405B quantized)

2. **Mixtral 8x22B (Mistral AI)** — A mixture-of-experts model that punches well above its weight, with strong reasoning and coding ability.
   - License: Apache 2.0
   - VRAM: ~80-100GB (quantized)

3. **Qwen 2.5 (Alibaba)** — The 72B model is very competitive, with strong multilingual and coding performance.
   - License: Apache 2.0 (for most sizes)

## Strong Mid-Tier

4. **DeepSeek-V2 / DeepSeek-Coder-V2** — Excellent for coding and reasoning tasks, very efficient MoE architecture.
5. **Mistral Large (open-weight releases)** — Check for the latest open-weight Mistral releases.
6. **Command R+ (Cohere)** — Strong at RAG and enterprise tasks.

## Easiest to Run Locally

7. **Llama 3 8B / Mistral 7B / Qwen 2.5 7B** — Won't match GPT-4, but are surprisingly good for their size and run on a single consumer GPU.

---

## How to Self-Host

| Tool | Best For |
|---|---|
| **[vLLM](https://github.com/vllm-project/vllm)** | High-throughput production serving |
| **[llama.cpp](https://github.com/ggerganov/llama.cpp)** | CPU/low-VRAM inference with quantization |
| **[Ollama](https://ollama.com)** | Easiest setup (one-command install) |
| **[TGI](https://github.com/huggingface/text-generation-inference)** | HuggingFace's production server |

## Practical Recommendation

- **Best bang for buck:** **Llama 3 70B** (quantized to 4-bit) via **vLLM** or **Ollama** on a machine with ~40GB VRAM (e.g., 1× A100 or 2× RTX 3090/4090).
- **If you have serious hardware:** **Llama 3 405B** is the closest you'll get to GPT-4 level in open source.
- **On a budget / consumer hardware:** **Qwen 2.5 7B** or **Llama 3 8B** quantized via **llama.cpp** can run on 8GB VRAM.

Would you like help setting up any of these?


Best open-source model for code completion under 10B parameters


Based on what's known as of early 2026, here are the top contenders for **open-source code completion models under 10B parameters**:

---

### 🥇 **DeepSeek-Coder-V2-Lite (2.4B / 6.7B)**
- Excellent performance relative to size, strong on multi-language completion
- MoE architecture gives it punch above its weight class
- Great for inline/fill-in-the-middle (FIM) completion

### 🥈 **StarCoder2 (3B / 7B)**
- Trained on The Stack v2 (~600+ languages)
- Strong FIM support, purpose-built for code completion
- The **7B** variant is particularly well-regarded for its size tier

### 🥉 **CodeLlama (7B)**
- Meta's code-specialized Llama fine-tune
- Available in base, Python-specialized, and Instruct variants
- Solid all-rounder, large community and tooling ecosystem

### Honorable Mentions
| Model | Params | Notes |
|---|---|---|
| **Qwen2.5-Coder (7B)** | 7B | Very competitive, especially for Python/JS |
| **CodeGemma (7B)** | 7B | Google's code model, good FIM support |
| **Stable Code (3B)** | 3B | Best option if you need something *very* small |

---

### Recommendation

- **Best overall**: **StarCoder2-7B** or **Qwen2.5-Coder-7B** — both are strong across languages with native fill-in-the-middle support, which is critical for real-time code completion.
- **Best if constrained to ~3B**: **StarCoder2-3B** — surprisingly capable for its size.
- **Best for Python specifically**: **Qwen2.5-Coder-7B** edges ahead on Python benchmarks.

All of these support **FIM (fill-in-the-middle)** prompting, which is what you want for tab-completion style use cases (e.g., in VS Code with Continue, Tabby, or llama.cpp backends).


What open-source LLMs can I run on a single GPU?


Great question! Here's a practical breakdown of open-source LLMs you can run on a single GPU, organized by VRAM requirements:

---

## 🟢 8 GB VRAM (e.g., RTX 3070/4060)

| Model | Parameters | Notes |
|---|---|---|
| **Llama 3.1 8B** | 8B | Excellent general-purpose model |
| **Mistral 7B / Mistral Nemo 12B** | 7–12B | Strong reasoning, fast inference |
| **Gemma 2 9B** | 9B | Google's open model, great quality/size ratio |
| **Phi-3 Mini** | 3.8B | Microsoft; punches above its weight |
| **Qwen 2.5 7B** | 7B | Strong multilingual support |

*Use 4-bit quantization (GPTQ/AWQ/GGUF) to fit larger models.*

---

## 🟡 16 GB VRAM (e.g., RTX 4080/A4000)

| Model | Parameters | Notes |
|---|---|---|
| **Llama 3.1 70B** (4-bit quant) | 70B | Near-GPT-4 quality with quantization |
| **Mixtral 8x7B** | ~47B active 12B | MoE architecture, very capable |
| **CodeLlama 34B** | 34B | Excellent for code generation |
| **Qwen 2.5 32B** | 32B | Strong all-around |
| **DeepSeek-Coder-V2-Lite** | 16B | Great for coding tasks |

---

## 🔴 24 GB VRAM (e.g., RTX 3090/4090/A5000)

| Model | Parameters | Notes |
|---|---|---|
| **Llama 3.1 70B** (8-bit quant) | 70B | Better quality than 4-bit |
| **DeepSeek-V2** | 236B (21B active) | MoE; excellent quality |
| **Qwen 2.5 72B** (4-bit quant) | 72B | Top-tier open model |
| **Command R+ 104B** (4-bit) | 104B | Strong for RAG use cases |

---

## Key Tips

1. **Quantization is your friend** — 4-bit (AWQ/GPTQ) cuts VRAM by ~75% with modest quality loss. GGUF format works great with `llama.cpp`.
2. **Tools to run them:**
   - **Ollama** — easiest setup (`ollama run llama3.1`)
   - **vLLM** — best for serving/throughput
   - **llama.cpp** — CPU+GPU offloading, very flexible
   - **Text Generation WebUI (oobabooga)** — nice GUI
3. **MoE models** (Mixtral, DeepSeek) give you more capability per VRAM dollar since only a fraction of parameters are active per token.
4. **Context length** also eats VRAM — a 128K context window needs significantly more memory than 4K.

Would you like recommendations tailored to a specific use case (coding, chat, RAG, etc.) or a specific GPU you have?


Best open-source vision-language model for document understanding


# Best Open-Source Vision-Language Models for Document Understanding

As of early 2026, here are the top contenders:

## 🥇 **Qwen2.5-VL (Qwen team / Alibaba)**
- **Why**: State-of-the-art on document benchmarks (DocVQA, ChartQA, TextVQA, OCRBench). Handles arbitrary resolution images natively, excellent OCR and layout understanding.
- **Sizes**: 3B, 7B, 72B
- **Best for**: General document understanding, table extraction, chart reading, multilingual docs.

## 🥈 **InternVL 2.5 (Shanghai AI Lab)**
- **Why**: Very competitive on document and OCR benchmarks, strong multi-image and high-resolution support.
- **Sizes**: 1B, 2B, 4B, 8B, 26B, 38B, 78B
- **Best for**: High-res document parsing, multi-page reasoning.

## 🥉 **GOT-OCR2.0 (StepFun)**
- **Why**: Purpose-built for OCR/document tasks. Outputs structured text (Markdown, LaTeX) from documents, sheet music, charts, etc.
- **Best for**: Pure OCR and structured text extraction from complex layouts.

---

## Other Strong Options

| Model | Strength |
|---|---|
| **Florence-2 (Microsoft)** | Lightweight (0.2B/0.7B), great for OCR + region-level tasks |
| **Idefics3 (Hugging Face)** | Strong general VLM with good doc understanding |
| **Llama 3.2 Vision (Meta)** | Solid baseline, easy ecosystem integration (11B, 90B) |
| **Phi-3.5 Vision (Microsoft)** | Small but capable (4.2B), good for edge deployment |
| **Nougat (Meta)** | Specialized for academic PDF → Markdown conversion |

---

## Recommendation by Use Case

- **Best overall**: **Qwen2.5-VL-72B** (or 7B if resource-constrained)
- **Best lightweight/edge**: **Florence-2** or **Phi-3.5 Vision**
- **Best for structured extraction** (tables → Markdown/LaTeX): **GOT-OCR2.0**
- **Best for academic papers**: **Nougat**
- **Best for multi-page documents**: **InternVL 2.5**

**Qwen2.5-VL-7B** is generally the sweet spot — it punches well above its weight on document tasks while being runnable on a single consumer GPU.


Which open-source model is best for function calling and tool use?


Great question! Here's a rundown of the top open-source models for function calling and tool use as of early 2026:

## Top Tier

1. **Llama 3 (Meta)** — The Llama 3 70B and 405B models have strong native tool-use capabilities, especially the instruction-tuned variants. Meta specifically optimized for structured output and function calling.

2. **Mistral Large / Mixtral** — Mistral's models have excellent built-in function calling support with a well-defined tool-use format. Mistral Large in particular rivals proprietary models.

3. **Qwen 2.5 (Alibaba)** — The Qwen 2.5 series (especially 72B) performs very well on tool-use benchmarks like BFCL (Berkeley Function Calling Leaderboard) and supports parallel function calls.

4. **Command R+ (Cohere)** — Purpose-built for RAG and tool use, with native multi-step tool calling and strong structured output.

## Strong Contenders

- **Hermes 2 Pro (NousResearch)** — Fine-tuned specifically for function calling with a clean JSON-based tool format.
- **Gorilla OpenFunctions** — A specialized model from UC Berkeley trained explicitly on API calling; excellent if tool use is your *primary* need.
- **DeepSeek-V3** — Strong general reasoning that translates well to tool use.

## How to Choose

| Factor | Best Pick |
|---|---|
| **Overall quality + tool use** | Llama 3 405B or Qwen 2.5 72B |
| **Best bang for size** | Qwen 2.5 7B/14B or Mistral 7B |
| **Specialized tool-use only** | Gorilla OpenFunctions |
| **Multi-step agentic workflows** | Command R+ or Llama 3 70B |
| **Self-hostable on consumer HW** | Qwen 2.5 7B or Hermes 2 Pro 7B |

## Key Tips

- Check the **Berkeley Function Calling Leaderboard (BFCL)** for up-to-date benchmarks — rankings shift frequently.
- Make sure you use the model's **native tool-calling format** (not just prompting) for best results.
- For agentic use cases (multi-turn, multi-tool), larger models (70B+) significantly outperform smaller ones.

Would you like recommendations for a specific use case, or help integrating one of these into a project?


How do I share a fine-tuned model with my team?


# Sharing a Fine-Tuned Model with Your Team

There are a few common ways to share a fine-tuned model, depending on your platform:

## On Hugging Face Hub (most common)

### 1. **Push directly from your training code**
```python
# After training, push to the Hub
model.push_to_hub("your-org/model-name")
tokenizer.push_to_hub("your-org/model-name")
```

### 2. **Use the Hugging Face CLI**
```bash
# Login first
huggingface-cli login

# Upload a local model directory
huggingface-cli upload your-org/model-name ./path-to-model
```

### 3. **Control visibility**
- **Private repo** (team only): Set `private=True` when creating, or toggle in the Hub UI under **Settings → Visibility**
- **Organization-scoped**: Push to `your-org/model-name` instead of `your-username/model-name` — all org members get access automatically
- **Gated access**: Enable "gated" mode in repo settings to require approval before access

### 4. **Share with specific people**
On the Hub, go to your model repo → **Settings → Collaborators** and add team members by username.

## Best Practices

| Practice | Why |
|---|---|
| **Include a Model Card** | Document training data, hyperparameters, eval metrics, and intended use |
| **Tag the revision** | Use `revision="v1.0"` so teammates pin to a known-good checkpoint |
| **Use an Organization** | Keeps team models organized under one namespace |
| **Upload training configs** | Include `training_args.json` or equivalent so others can reproduce |

## Quick Example (end-to-end)
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load your fine-tuned model
model = AutoModelForSequenceClassification.from_pretrained("./my-finetuned-model")
tokenizer = AutoTokenizer.from_pretrained("./my-finetuned-model")

# Push to your org (private by default)
model.push_to_hub("my-team/sentiment-classifier", private=True)
tokenizer.push_to_hub("my-team/sentiment-classifier", private=True)
```

Your teammates can then load it with:
```python
model = AutoModelForSequenceClassification.from_pretrained("my-team/sentiment-classifier")
```

Would you like help with any specific part of this workflow, like setting up an organization or writing a model card?

transformers
from transformers
AutoTokenizer


Best way to version and collaborate on ML datasets


# Versioning & Collaborating on ML Datasets

Here's a rundown of the most popular approaches, from lightweight to full-featured:

---

## 1. **DVC (Data Version Control)**
The most widely adopted tool purpose-built for this.
- Works **alongside Git** — you version metadata in Git while large files live in remote storage (S3, GCS, Azure, etc.)
- Tracks data pipelines and experiments, not just files
- `dvc push` / `dvc pull` for team collaboration
- Great for **reproducibility** — locks dataset + code + model together

```bash
dvc init
dvc add data/training.parquet
git add data/training.parquet.dvc .gitignore
git commit -m "Add training dataset v1"
dvc push
```

## 2. **Hugging Face Datasets Hub**
Best for public/shared datasets and the broader ML community.
- Git-based repos with **Git LFS** under the hood
- Dataset viewer, cards, and versioning via branches/tags
- Easy integration with the `datasets` library
- Supports streaming for huge datasets

## 3. **LakeFS**
Best when you need **Git-like branching on data lakes**.
- Creates branches, commits, and merges on top of object storage
- Zero data duplication (copy-on-write)
- Works with existing tools (Spark, Presto, dbt, etc.)

## 4. **Delta Lake / Apache Iceberg**
Best for **tabular data at scale** with ACID transactions.
- Time-travel queries (`VERSION AS OF`)
- Schema evolution and enforcement
- Great for teams already using Spark/Databricks

## 5. **Git LFS (Lightweight)**
Simplest option if your datasets aren't enormous.
- Stores large files in a separate backend, pointers in Git
- Familiar Git workflow, but lacks ML-specific features

---

## Key Recommendations by Scenario

| Scenario | Best Tool |
|---|---|
| Small team, files < 10 GB | **DVC** or **Git LFS** |
| Sharing with the ML community | **Hugging Face Hub** |
| Data lake with branching workflows | **LakeFS** |
| Large-scale tabular data + analytics | **Delta Lake / Iceberg** |
| Full MLOps pipeline | **DVC + MLflow** or **Weights & Biases** |

---

## Best Practices (Tool-Agnostic)

1. **Semantic versioning** — tag datasets like releases (`v1.0`, `v1.1`) so models can pin to a version
2. **Data cards** — document schema, source, biases, preprocessing steps alongside each version
3. **Immutable snapshots** — never mutate a released version; create a new one
4. **Checksums** — always hash files (SHA-256) to detect corruption or drift
5. **Separate data from code** — version them in parallel but don't store large data in the code repo
6. **Automate validation** — run schema checks and data quality tests on every new version (e.g., Great Expectations, Pandera)

The most common stack I see in practice is **DVC + Git + cloud storage**, which gives you reproducibility without reinventing your workflow.


How do I contribute a dataset to the open-source community?


Great question! Here's a guide on how to contribute a dataset to the open-source community:

---

## 🗂 Contributing an Open-Source Dataset

### 1. **Prepare Your Dataset**
   - **Clean the data** — remove duplicates, fix errors, handle missing values.
   - **Document the schema** — describe each column/field, data types, and units.
   - **Choose a standard format** — CSV, JSON, Parquet, or Arrow are widely supported.

### 2. **Handle Legal & Ethical Considerations**
   - **Licensing** — Choose an open license (e.g., CC0, CC-BY 4.0, Apache 2.0, ODbL).
   - **Privacy** — Remove or anonymize any personally identifiable information (PII).
   - **Consent** — Ensure you have the right to share the data.
   - **Bias audit** — Document any known biases or limitations.

### 3. **Write a Dataset Card / README**
   - Description and motivation
   - How the data was collected
   - Size, splits (train/test/val), and statistics
   - Intended use cases and limitations
   - Citation information

### 4. **Choose a Platform**
   | Platform | Best For |
   |---|---|
   | **Hugging Face Hub** (`huggingface.co/datasets`) | ML/NLP datasets, easy versioning, wide community |
   | **GitHub** | Smaller datasets, code + data together |
   | **Kaggle Datasets** | Data science community, competitions |
   | **Zenodo** | Academic datasets, get a DOI for citation |
   | **Data.gov / data.world** | Public/government data |

### 5. **Upload & Publish**
   **Example: Hugging Face Hub (most popular for ML)**
   ```bash
   pip install huggingface_hub
   huggingface-cli login

   # Create and push a dataset repo
   huggingface-cli repo create my-dataset --type dataset
   git clone https://huggingface.co/datasets/<your-username>/my-dataset
   # Add your files + README.md (dataset card), then push
   ```

   **Example: GitHub**
   ```bash
   git init my-dataset && cd my-dataset
   # Add data files, README, LICENSE
   git add . && git commit -m "Initial dataset release"
   # Push to GitHub and tag a release
   ```

### 6. **Promote & Maintain**
   - Share on Reddit (r/datasets, r/MachineLearning), Twitter/X, and relevant forums.
   - Respond to issues and questions.
   - Version your dataset if you update it (use git tags or Hub revisions).
   - Add a **citation file** (`CITATION.cff`) so others can properly credit you.

---

### Quick Checklist
- ✅ Data is clean and well-formatted
- ✅ Open license chosen and included
- ✅ No PII or sensitive information
- ✅ Dataset card / README written
- ✅ Uploaded to a discoverable platform
- ✅ Citation info provided

Would you like more detail on any of these steps, such as writing a dataset card or choosing the right license?


What platforms support collaborative model training?


Several platforms support collaborative model training, where multiple participants contribute to building ML models together:

## Open-Source / Community Platforms
- **Hugging Face** — Offers collaborative model training through shared repositories, the `accelerate` library, and community-driven efforts.
- **OpenMined (PySyft)** — Focused on privacy-preserving collaborative/federated learning.
- **Flower (flwr)** — A framework for federated learning that enables collaborative training across distributed participants.
- **Hivemind** — Decentralized deep learning library (used in projects like BigScience/BLOOM) allowing volunteers to collaboratively train models over the internet.

## Cloud / Enterprise Platforms
- **Google Cloud Vertex AI** — Supports distributed and collaborative training workflows with team-based access controls.
- **AWS SageMaker** — Offers distributed training, shared notebooks, and team collaboration features.
- **Microsoft Azure ML** — Provides collaborative workspaces, shared compute, and distributed training.
- **Weights & Biases (W&B)** — Collaboration-focused experiment tracking, model registry, and shared dashboards.
- **Neptune.ai** — Team-based experiment tracking and model management.
- **Comet ML** — Collaborative experiment tracking and model management.

## Federated / Decentralized Training
- **NVIDIA FLARE** — Federated learning framework for collaborative training across institutions (especially healthcare).
- **Intel OpenFL** — Open federated learning framework.
- **Substra** — Privacy-preserving federated learning, often used in regulated industries.
- **PaddleFL (Baidu)** — Federated learning framework built on PaddlePaddle.

## Research / Notebook-Based
- **Google Colab** — Supports shared notebooks and (with external tooling) collaborative training.
- **Kaggle Kernels** — Community-driven collaborative environment with shared notebooks and competitions.
- **Databricks** — Collaborative workspace with distributed Spark-based training (MLflow integration).

The right choice depends on your use case — **federated learning** platforms (Flower, NVIDIA FLARE) are best when data can't leave its source, while **cloud ML platforms** (Vertex AI, SageMaker) suit teams sharing infrastructure, and **community platforms** (Hugging Face, Hivemind) work well for open, volunteer-driven efforts.


How do I set up access controls for shared model repositories?


This is a general question about access controls for shared model repositories, so let me give you a comprehensive overview. The answer depends on which platform/system you're using:

## Hugging Face Hub

On Hugging Face, you can control access at multiple levels:

### Repository Visibility
- **Public** — Anyone can view and download
- **Private** — Only you and explicitly granted collaborators can access

### Organization-Level Controls
1. **Create an Organization** at huggingface.co/organizations
2. **Add members** with roles:
   - **Read** — Can view and clone repos
   - **Write** — Can push changes
   - **Admin** — Can manage settings and members
3. **Set default permissions** for new repos in org settings

### Per-Repository Access
- Go to **Settings → Collaborators** on any repo
- Add users by username with a specific role (read/write/admin)

### Gated Repositories
For controlled distribution (e.g., models requiring license acceptance):
1. Go to **Settings → Gated access**
2. Enable gating — users must request access
3. Choose **automatic** (instant approval) or **manual** (you review each request)

### Access Tokens
- Create **fine-grained tokens** at huggingface.co/settings/tokens
- Scope tokens to specific repos and permissions (read/write)
- Use tokens for programmatic access (CI/CD, scripts)

---

## GitHub (Git LFS for models)

If you're hosting models on GitHub:
- Use **repository visibility** (public/private)
- Manage access via **Settings → Collaborators and Teams**
- Use **branch protection rules** to require reviews before merging
- Use **CODEOWNERS** files to require specific reviewers for model files

---

## General Best Practices

| Practice | Why |
|---|---|
| Use org-level permissions | Easier to manage at scale |
| Principle of least privilege | Grant only the access needed |
| Use scoped tokens (not personal) | Limits blast radius if leaked |
| Enable audit logs | Track who accessed/changed what |
| Gate sensitive models | Ensure license compliance |

Would you like help with a specific platform or with implementing access controls in the context of this project?


How do I download a large dataset in parquet format programmatically?


# Downloading Large Datasets in Parquet Format Programmatically

Here are several approaches depending on your ecosystem:

---

## 1. Hugging Face `datasets` Library (Python)

```python
from datasets import load_dataset

# Streams data without downloading everything into memory
dataset = load_dataset("username/dataset-name", split="train", streaming=True)

# Or download fully and save as parquet
dataset = load_dataset("username/dataset-name", split="train")
dataset.to_parquet("output.parquet")
```

For very large datasets, **streaming** avoids memory issues:

```python
from datasets import load_dataset

dataset = load_dataset("username/dataset-name", split="train", streaming=True)

for batch in dataset.iter(batch_size=10_000):
    # Process each batch incrementally
    ...
```

## 2. Direct Parquet Download via Hugging Face Hub API

Many HF datasets already store parquet files. You can download them directly:

```python
from huggingface_hub import HfApi, hf_hub_download

api = HfApi()

# List parquet files in the repo
files = api.list_repo_tree("username/dataset-name", repo_type="dataset")
parquet_files = [f.rfilename for f in files if f.rfilename.endswith(".parquet")]

# Download each file
for filename in parquet_files:
    hf_hub_download(
        repo_id="username/dataset-name",
        filename=filename,
        repo_type="dataset",
        local_dir="./data",
    )
```

## 3. Using `pandas` + `pyarrow` (any Parquet source)

```python
import pandas as pd

# Read from a URL
df = pd.read_parquet("https://example.com/data.parquet")

# Read in chunks for large files
import pyarrow.parquet as pq

parquet_file = pq.ParquetFile("large_file.parquet")
for batch in parquet_file.iter_batches(batch_size=50_000):
    df_chunk = batch.to_pandas()
    # process chunk...
```

## 4. Using `DuckDB` (great for querying without full download)

```python
import duckdb

# Query parquet directly from a URL — only reads needed columns/rows
result = duckdb.sql("""
    SELECT *
    FROM 'https://huggingface.co/datasets/username/dataset-name/resolve/main/data.parquet'
    WHERE column_a > 100
    LIMIT 1000
""").df()
```

## 5. CLI with `wget` / `curl`

```bash
# If you have direct parquet URLs
wget -P ./data/ "https://huggingface.co/datasets/username/dataset-name/resolve/main/data/train-00000-of-00004.parquet"

# Or all shards with huggingface-cli
huggingface-cli download username/dataset-name --repo-type dataset --include "*.parquet" --local-dir ./data
```

---

## Key Tips for Large Datasets

| Concern | Solution |
|---|---|
| **Memory limits** | Use streaming (`streaming=True`) or iterate in batches |
| **Disk space** | Download only specific splits/shards |
| **Speed** | Use `hf_transfer` for faster downloads: `pip install hf_transfer` then `HF_HUB_ENABLE_HF_TRANSFER=1` |
| **Resumable downloads** | `huggingface-cli download` supports automatic resume |
| **Only need certain columns** | Use DuckDB or `pq.read_table(path, columns=[...])` |

Which approach works best depends on whether you need to process incrementally (streaming), query selectively (DuckDB), or download everything to disk (Hub CLI).


Where can I search for pre-processed NLP datasets by task type?


Here are the best places to search for pre-processed NLP datasets by task type:

## 🥇 Top Resources

1. **Hugging Face Datasets Hub** — [huggingface.co/datasets](https://huggingface.co/datasets)
   - Filter by **task** (e.g., text-classification, question-answering, summarization, translation, NER, etc.)
   - Datasets are ready to use via the `datasets` library (`load_dataset(...)`)
   - Community-uploaded + canonical datasets

2. **Paperswithcode Datasets** — [paperswithcode.com/datasets](https://paperswithcode.com/datasets)
   - Browse by **task taxonomy** (300+ NLP tasks)
   - Links datasets to benchmarks and SOTA results

3. **TensorFlow Datasets (TFDS)** — [tensorflow.org/datasets/catalog](https://www.tensorflow.org/datasets/catalog/overview)
   - Pre-processed and loadable via `tfds.load()`
   - Filterable by task

## Other Notable Sources

| Source | Strength |
|---|---|
| **Kaggle Datasets** | Community datasets, many pre-processed with notebooks |
| **Google Dataset Search** | Broad search engine across the web |
| **Allen AI (Beaker/AI2)** | Research-grade NLP datasets |
| **NLTK Data** | Classic NLP datasets (older but well-documented) |
| **UCI ML Repository** | Some text/NLP datasets |

## Quick Example (Hugging Face)

```python
from datasets import load_dataset

# Browse by task programmatically
from huggingface_hub import list_datasets
datasets = list_datasets(task="text-classification")

# Load a specific dataset
dataset = load_dataset("glue", "sst2")  # sentiment analysis
```

**Hugging Face Datasets Hub** is generally the best starting point — it has the largest collection, strong task-based filtering, and seamless integration with training frameworks.

from datasets
load_dataset
hugging face datasets


How do I stream a dataset that's too large to fit in memory?


# Streaming Large Datasets

There are several approaches depending on your tooling:

## Hugging Face `datasets` (Streaming Mode)

The simplest approach — pass `streaming=True` to avoid downloading the full dataset:

```python
from datasets import load_dataset

dataset = load_dataset("large-dataset/name", streaming=True)

for example in dataset["train"]:
    # Process one example at a time — nothing stored in memory
    process(example)
```

Key points:
- Returns an `IterableDataset` instead of a `Dataset`
- Data is downloaded on-the-fly as you iterate
- Supports `.map()`, `.filter()`, `.shuffle()`, `.take()`, `.skip()`

```python
# Shuffle, filter, and batch — all lazily evaluated
stream = (
    dataset["train"]
    .shuffle(seed=42, buffer_size=1000)
    .filter(lambda x: x["label"] == 1)
    .map(preprocess_fn, batched=True, batch_size=64)
)
```

## Pandas — Chunked Reading

```python
import pandas as pd

chunks = pd.read_csv("huge_file.csv", chunksize=10_000)

for chunk in chunks:
    # Each chunk is a DataFrame with 10k rows
    process(chunk)
```

## PyTorch `DataLoader` with `IterableDataset`

```python
from torch.utils.data import IterableDataset, DataLoader

class StreamDataset(IterableDataset):
    def __iter__(self):
        with open("huge_file.jsonl") as f:
            for line in f:
                yield json.loads(line)

loader = DataLoader(StreamDataset(), batch_size=32)
```

## Parquet (Columnar + Lazy)

Parquet files are great for large data because you can read only the columns you need:

```python
import pyarrow.parquet as pq

# Read only specific columns without loading the whole file
table = pq.read_table("data.parquet", columns=["text", "label"])

# Or iterate in batches
parquet_file = pq.ParquetFile("data.parquet")
for batch in parquet_file.iter_batches(batch_size=5000):
    df = batch.to_pandas()
    process(df)
```

## General Tips

| Strategy | When to use |
|---|---|
| `streaming=True` | Hugging Face Hub datasets |
| `chunksize=N` | CSV/JSON with pandas |
| `IterableDataset` | PyTorch training pipelines |
| Parquet batches | Column-oriented analytics |
| Memory-mapped files | Random access without full load (`numpy.memmap`, `datasets` default mode) |

The **Hugging Face streaming mode** is usually the easiest starting point — it works with any dataset on the Hub and requires just a one-argument change.


Best way to preview dataset contents before downloading


# Previewing Dataset Contents Before Downloading

Here are the best approaches, from quickest to most thorough:

## 1. **Hugging Face Dataset Viewer (Easiest)**
Every dataset on the Hub has a built-in viewer at its page:
```
https://huggingface.co/datasets/<owner>/<dataset-name>
```
- Browse rows, columns, and splits directly in the browser
- Filter and search without downloading anything
- Works for most formats (Parquet, CSV, JSON, Arrow, etc.)

## 2. **Dataset Viewer API**
Programmatically preview rows without installing anything heavy:
```python
import requests

API_URL = "https://datasets-server.huggingface.co/rows?dataset=imdb&config=plain_text&split=train&offset=0&length=5"
response = requests.get(API_URL)
rows = response.json()["rows"]
```

## 3. **Streaming with `datasets`**
Load rows on-demand — **no full download**:
```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train", streaming=True)

# Peek at the first 5 examples
for i, example in enumerate(ds):
    print(example)
    if i >= 4:
        break
```

## 4. **Dataset Card & Metadata**
Check the `README.md` / dataset card for:
- Schema (column names, types)
- Size, number of rows/splits
- Example rows and intended use

```python
from huggingface_hub import dataset_info

info = dataset_info("imdb")
print(info.card_data)       # Metadata
print(info.dataset_size)    # Size in bytes
```

## 5. **Partial Parquet Download**
Many Hub datasets are stored as Parquet. You can read just the schema or first N rows:
```python
import pandas as pd

url = "https://huggingface.co/datasets/imdb/resolve/main/plain_text/train-00000-of-00001.parquet"
df = pd.read_parquet(url, columns=["text", "label"]).head(10)
print(df)
```

---

## Quick Comparison

| Method | Needs Code? | Downloads Data? | Best For |
|---|---|---|---|
| Dataset Viewer (web) | No | No | Quick browsing |
| Viewer API | Minimal | No | Automation/scripts |
| Streaming | Yes | No (on-the-fly) | Row-level inspection |
| Dataset Card | No | No | Schema & metadata |
| Partial Parquet | Yes | Minimal | Column-level analysis |

**TL;DR** — Start with the **web viewer** for a quick look, use **streaming** when you need to inspect data programmatically without committing to a full download.

dataset viewer
hugging face dataset viewer


How do I filter and download only a subset of a large dataset?


# Filtering & Downloading a Subset of a Large Dataset

There are several approaches depending on your tooling:

---

## 1. **Hugging Face `datasets` library (streaming)**

The most memory-efficient way — rows are fetched on-demand, never downloading the full dataset:

```python
from datasets import load_dataset

# Stream instead of downloading everything
dataset = load_dataset("bigcode/the-stack", split="train", streaming=True)

# Filter on the fly
filtered = dataset.filter(lambda x: x["language"] == "Python")

# Take only the first 1,000 matching examples
subset = filtered.take(1000)

for example in subset:
    print(example)
```

## 2. **Download only specific splits or slices**

```python
# Download only the first 5% of the training split
dataset = load_dataset("imagenet-1k", split="train[:5%]")

# Or a specific row range
dataset = load_dataset("imagenet-1k", split="train[1000:2000]")
```

## 3. **Filter after loading (small-to-medium datasets)**

```python
dataset = load_dataset("imdb", split="train")

# Filter in-memory (fast, uses Arrow under the hood)
positive_reviews = dataset.filter(lambda x: x["label"] == 1)

# Select specific columns only
positive_reviews = positive_reviews.select_columns(["text"])
```

## 4. **SQL-like filtering with DuckDB / the HF Hub API**

For datasets stored as Parquet on the Hub, you can query directly without downloading everything:

```python
import duckdb

url = "hf://datasets/mstz/adult/adult/train-00000-of-00001.parquet"

result = duckdb.sql(f"""
    SELECT age, occupation, income
    FROM '{url}'
    WHERE age > 30 AND occupation = 'Tech-support'
    LIMIT 500
""").fetchdf()
```

## 5. **From the CLI**

```bash
# Download a specific config/split
huggingface-cli download bigcode/the-stack --repo-type dataset \
    --include "data/python/*" \    # only Python files
    --local-dir ./subset
```

---

## Key Tips

| Goal | Approach |
|---|---|
| Never download full dataset | `streaming=True` |
| Download a fixed slice | `split="train[:1000]"` |
| Filter by column value | `.filter(lambda x: ...)` |
| Only need certain columns | `.select_columns([...])` |
| SQL-style queries on Parquet | DuckDB with `hf://` URLs |
| Download specific files only | `huggingface-cli download --include` |

The **streaming + filter + take** combo is usually the best starting point for very large datasets since it avoids downloading data you'll never use.


How do I upload a custom dataset for public use?


This is a general question about uploading datasets to the **Hugging Face Hub** — here's how to do it:

---

## Uploading a Custom Dataset for Public Use

### Option 1: Via the Hugging Face Web UI
1. Go to [huggingface.co](https://huggingface.co) and sign in.
2. Click your profile picture → **New Dataset**.
3. Give it a name, set visibility to **Public**, and click **Create dataset**.
4. Use the **"Add file"** button to upload your files (CSV, JSON, Parquet, etc.).
5. Add a **Dataset Card** (`README.md`) describing the data, its format, license, and intended use.

### Option 2: Via the `huggingface_hub` Python Library
```python
from huggingface_hub import HfApi

api = HfApi()

# 1. Create the repo
api.create_repo("your-username/my-dataset", repo_type="dataset", private=False)

# 2. Upload files
api.upload_folder(
    folder_path="./my_local_data",
    repo_id="your-username/my-dataset",
    repo_type="dataset",
)
```

### Option 3: Via the `datasets` Library
```python
from datasets import Dataset

# Create from a dict, DataFrame, CSV, etc.
ds = Dataset.from_dict({"text": ["hello", "world"], "label": [0, 1]})

# Push to the Hub (public by default)
ds.push_to_hub("your-username/my-dataset")
```

### Option 4: Via Git (for advanced users)
```bash
# Install Git LFS (needed for large files)
git lfs install

# Clone the dataset repo
git clone https://huggingface.co/datasets/your-username/my-dataset
cd my-dataset

# Add your files and push
git add .
git commit -m "Add dataset files"
git push
```

---

## Best Practices
- **Add a Dataset Card** (`README.md`) with YAML metadata — this enables auto-detection of format, splits, and features on the Hub.
- **Use Parquet** for best performance with the Hub's dataset viewer.
- **Specify a license** so others know how they can use your data.
- **Use Git LFS** for files over 10 MB (the `huggingface_hub` library handles this automatically).
- **Organize splits** in folders like `train/`, `test/`, or name files `train.csv`, `test.csv` for automatic split detection.

Would you like help with any specific part of this process?


Best way to host a private dataset for my organization


# Hosting a Private Dataset for Your Organization

Here are the best options, depending on your needs:

---

## 1. **Hugging Face Hub (Recommended for ML/AI teams)**
- Create a **private dataset repository** under your HF Organization
- Supports access control via org membership and fine-grained tokens
- Built-in dataset viewer, versioning (Git LFS), and `datasets` library integration
```bash
huggingface-cli repo create my-dataset --type dataset --organization my-org
huggingface-cli upload my-org/my-dataset ./local-data --repo-type dataset --private
```
**Best for:** Teams already using HF for models/training, datasets up to ~300GB

---

## 2. **Cloud Object Storage (S3, GCS, Azure Blob)**
- Store data in a private bucket with IAM policies
- Pair with a data catalog (e.g., AWS Glue, DataHub) for discoverability
- Works well with Spark, DuckDB, or any processing framework

**Best for:** Large-scale datasets (TBs+), teams with existing cloud infrastructure

---

## 3. **Self-Hosted Data Platform**
- **MinIO** — S3-compatible object storage you run yourself
- **DVC (Data Version Control)** — Git-like versioning for datasets, backed by any remote storage
- **LakeFS** — Git-like branching for data lakes

**Best for:** Strict compliance requirements (data can't leave your network)

---

## 4. **Managed Database / Warehouse**
- **Snowflake, BigQuery, Redshift** — structured data with SQL access
- **Postgres + pgvector** — if you need embeddings/vector search too

**Best for:** Structured/tabular data with frequent querying

---

## Quick Decision Guide

| Factor | Best Choice |
|---|---|
| ML/AI workflow integration | **Hugging Face Hub** |
| Very large scale (TBs) | **Cloud Object Storage** |
| Air-gapped / on-prem requirement | **MinIO + DVC** |
| Structured data + SQL queries | **Data Warehouse** |
| Need versioning & branching | **DVC or LakeFS** |
| Simplest setup | **Hugging Face Hub** |

---

## Key Considerations
- **Access control:** Decide between org-level, team-level, or per-user permissions
- **Versioning:** Critical for reproducibility — HF Hub and DVC handle this natively
- **Size:** HF Hub works great up to ~300GB; beyond that, cloud storage or a data lake is more practical
- **Compliance:** If you have HIPAA/SOC2/GDPR constraints, self-hosted or a compliant cloud provider may be required

Would you like a deeper dive into any of these options?


How do I add a dataset card with proper documentation?


This is a general Hugging Face question rather than something specific to your codebase, but here's a comprehensive guide:

## Adding a Dataset Card with Proper Documentation

A **dataset card** is a `README.md` file at the root of your dataset repository that uses YAML metadata + Markdown documentation.

### 1. YAML Metadata Header

```yaml
---
license: mit
task_categories:
  - text-classification
language:
  - en
size_categories:
  - 1K<n<10K
tags:
  - sentiment-analysis
pretty_name: My Dataset Name
---
```

### 2. Markdown Body (recommended sections)

```markdown
# Dataset Card for [Dataset Name]

## Dataset Description
- **Homepage:** [URL]
- **Repository:** [URL]
- **Paper:** [URL]
- **Point of Contact:** [name/email]

### Dataset Summary
A 1–2 paragraph description of what the dataset contains and its intended use.

### Supported Tasks
- `text-classification`: Describe the task and expected performance.

### Languages
English (en-US)

## Dataset Structure

### Data Fields
| Field   | Type   | Description            |
|---------|--------|------------------------|
| text    | string | The input text          |
| label   | int    | 0 = negative, 1 = positive |

### Data Splits
| Split      | Examples |
|------------|----------|
| train      | 8,000    |
| validation | 1,000    |
| test       | 1,000    |

## Dataset Creation

### Source Data
Describe where the data comes from.

### Annotations
Describe the annotation process (manual, automated, etc.).

### Personal and Sensitive Information
Note any PII or sensitive data considerations.

## Considerations for Using the Data

### Social Impact
Potential benefits and harms.

### Bias
Known biases in the data.

### Licensing
Detailed license information.

## Citation
```bibtex
@misc{yourdataset2026,
  title={Your Dataset},
  author={Author Name},
  year={2026}
}
```
```

### 3. How to Create It

**Option A — Hugging Face Hub UI:**
1. Go to your dataset repo on `huggingface.co`
2. Click **"Create Dataset Card"** or edit `README.md` directly
3. The UI has a metadata editor that fills in the YAML for you

**Option B — `huggingface_hub` Python library:**
```python
from huggingface_hub import DatasetCardData, DatasetCard

card_data = DatasetCardData(
    license="mit",
    language="en",
    size_categories="1K<n<10K",
    task_categories=["text-classification"],
)

card = DatasetCard.from_template(
    card_data,
    pretty_name="My Dataset",
    dataset_summary="A brief description of the dataset.",
)

card.push_to_hub("your-username/your-dataset")
```

**Option C — CLI:**
```bash
# Clone, edit, push
git clone https://huggingface.co/datasets/your-username/your-dataset
cd your-dataset
# Edit README.md with any text editor
git add README.md && git commit -m "Add dataset card"
git push
```

### Key Tips

- **YAML metadata matters** — it powers search, filtering, and the dataset viewer on the Hub.
- **Be specific about data fields** — downstream users need to know column names, types, and meanings.
- **Document limitations and biases** — this is increasingly expected and builds trust.
- **Use the [official template](https://github.com/huggingface/datasets/blob/main/templates/README.md)** as a starting point if you want full coverage.

Would you like help creating a dataset card for a specific dataset in this project?


What format should I use to upload a dataset for maximum compatibility?


This is a general data science question, so here's a concise guide on dataset formats ranked by compatibility:

## Recommended Formats (Most → Least Compatible)

### 🥇 **Parquet** (`.parquet`)
- **Best overall choice** for structured/tabular data
- Columnar format — fast reads, efficient compression, schema-preserving
- Native support on Hugging Face Hub (auto-previews, streaming, SQL queries)
- Works with Pandas, Polars, Spark, DuckDB, Arrow, etc.

### 🥈 **CSV** (`.csv`)
- Universally readable by virtually any tool
- Downsides: no schema enforcement, slow for large files, string-encoding ambiguities
- Good fallback if consumers are non-technical

### 🥉 **JSON Lines** (`.jsonl`)
- One JSON object per line — great for nested/variable-schema data
- Streamable and append-friendly
- Preferred for NLP tasks, chat logs, and semi-structured data

---

## By Data Type

| Data Type | Best Format |
|---|---|
| Tabular / structured | **Parquet** |
| Text / NLP | **JSONL** or Parquet |
| Images + metadata | **Parquet** (with image bytes) or image folders + `metadata.csv` |
| Audio + metadata | **Parquet** or audio folders + `metadata.csv` |
| Conversational / chat | **JSONL** |

## Key Tips

1. **Always include a schema** — column names, types, and a dataset card (`README.md`) describing fields.
2. **Shard large datasets** — split into multiple files (e.g., `train-00000-of-00005.parquet`) for parallel loading and streaming.
3. **Use UTF-8 encoding** everywhere.
4. **Avoid Excel (`.xlsx`)** — it's lossy, size-limited, and poorly supported in data pipelines.

**TL;DR — Use Parquet.** It gives you the best mix of performance, type safety, compression, and ecosystem support.


How do I set up automated dataset updates from a data pipeline?


Based on exploring your codebase, here's a tailored answer:

---

## Setting Up Automated Dataset Updates for the HF Devtools Dashboard

Your project currently has **no automated scheduling** — all data updates are either user-triggered (via the dashboard UI to start runs) or manual (via scripts in `scripts/`). Here are your main options:

### Option 1: Cron-Based Scheduled Runs (Simplest)

Use a system cron job or PM2's built-in cron to periodically trigger a new benchmark run:

```bash
# Example: run benchmarks nightly at 2am
0 2 * * * curl -X POST http://localhost:3000/api/runs -H 'Content-Type: application/json' -d '{"tools":["claude","copilot","codex","cursor"]}'
```

Or with PM2 (since you already use it for deployment):
```js
// ecosystem.config.js - add a scheduled worker
{
  name: "nightly-benchmark",
  script: "scripts/run-benchmarks.ts",
  cron_restart: "0 2 * * *",
  autorestart: false
}
```

### Option 2: Script-Based Pipeline (Most Flexible)

Chain your existing scripts into a pipeline. You already have:
- `scripts/update-queries-v2.ts` — refresh the query set
- `scripts/backfill-products.ts` — extract products from new results
- `scripts/retag-hf-products.ts` — re-categorize products

You could create a master pipeline script:

```ts
// scripts/pipeline.ts
import { updateQueries } from "./update-queries-v2";
import { backfillProducts } from "./backfill-products";

async function runPipeline() {
  // 1. Trigger a new run via the API
  const res = await fetch("http://localhost:3000/api/runs", { method: "POST", ... });

  // 2. Poll until the run completes
  // (your runner.ts already has stale-run recovery and auto-abort)

  // 3. Backfill product extraction for new results
  await backfillProducts();
}
```

### Option 3: GitHub Actions (If Deployed Externally)

```yaml
# .github/workflows/benchmark.yml
on:
  schedule:
    - cron: '0 2 * * *'  # nightly
jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - run: curl -X POST ${{ secrets.DASHBOARD_URL }}/api/runs ...
```

### Key Things to Keep in Mind

1. **Your runner already handles concurrency** — it limits parallel jobs per tool (Claude: 5, Codex: 8, Copilot: 5, Cursor: 3) with 5-minute timeouts and auto-abort after 10 minutes of no progress.

2. **Stale run recovery** exists in `src/instrumentation.ts` — if a scheduled run crashes, it'll be marked as failed on next server restart.

3. **Product detection is two-layered** (regex + LLM extraction) — the `backfill-products.ts` script handles results that haven't been processed yet, so you can safely run it after each benchmark completes.

---

**Which approach interests you?** I can help implement any of these — whether it's a cron job, a pipeline script, or a GitHub Actions workflow.


How do I benchmark my fine-tuned model against standard evaluations?


# Benchmarking a Fine-Tuned Model Against Standard Evaluations

There are several well-established approaches and tools for this:

## 1. Use an Evaluation Harness

The most popular option is **EleutherAI's `lm-evaluation-harness`**:

```bash
pip install lm-eval

lm_eval --model hf \
  --model_args pretrained=your-org/your-fine-tuned-model \
  --tasks hellaswag,mmlu,arc_challenge,truthfulqa_mc2,winogrande,gsm8k \
  --batch_size auto \
  --output_path ./results
```

This gives you scores on widely-reported benchmarks so you can directly compare against published baselines.

## 2. Common Benchmark Suites

| Benchmark | What it measures |
|---|---|
| **MMLU** | Broad knowledge across 57 subjects |
| **HellaSwag** | Commonsense reasoning / sentence completion |
| **ARC** (Challenge) | Grade-school science Q&A |
| **TruthfulQA** | Resistance to generating false claims |
| **GSM8K** | Grade-school math (chain-of-thought) |
| **HumanEval / MBPP** | Code generation |
| **WinoGrande** | Coreference / commonsense |
| **MT-Bench** | Multi-turn conversational quality (LLM-as-judge) |

## 3. Compare Base vs. Fine-Tuned

Run the same eval suite on **both** your base model and your fine-tuned model to measure the delta:

```bash
# Base model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3-8B \
  --tasks mmlu,hellaswag,gsm8k \
  --output_path ./results_base

# Fine-tuned model
lm_eval --model hf \
  --model_args pretrained=your-org/your-fine-tuned-model \
  --tasks mmlu,hellaswag,gsm8k \
  --output_path ./results_finetuned
```

This helps you detect **catastrophic forgetting** — if general benchmark scores drop significantly, your fine-tuning may have overfit to the narrow task.

## 4. Hugging Face Open LLM Leaderboard

You can submit your model to the [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) to get standardized scores and appear alongside other models.

## 5. Task-Specific Evaluation

Beyond standard benchmarks, evaluate on **your actual use case**:

- **Hold-out test set** — split your fine-tuning data and evaluate on unseen examples
- **Domain-specific metrics** — e.g., F1 for classification, ROUGE/BLEU for generation, pass@k for code
- **LLM-as-Judge** — use a stronger model (e.g., Claude or GPT-4) to rate outputs:

```python
# Example with MT-Bench style evaluation
from fastchat.llm_judge import run_judge
```

## 6. Key Tips

- **Always report few-shot settings** (0-shot, 5-shot, etc.) — they dramatically affect scores.
- **Use the same prompt template** for base and fine-tuned comparisons.
- **Watch for contamination** — if your fine-tuning data overlaps with benchmark test sets, scores are inflated.
- **Log generation parameters** (temperature, top-p) for reproducibility.

Would you like more detail on any of these approaches, or help setting up a specific evaluation pipeline?


What metrics should I use to evaluate an LLM for coding tasks?


# Metrics for Evaluating LLMs on Coding Tasks

Here are the key metrics, organized by category:

## Functional Correctness
- **pass@k** — The probability that at least one of *k* generated solutions passes all unit tests. **pass@1** is the most common benchmark metric (used by HumanEval, MBPP, SWE-bench, etc.).
- **Test pass rate** — Percentage of test cases passed across a suite, giving a more granular view than binary pass/fail.

## Code Quality (Beyond "Does it Run?")
- **Syntactic validity** — Does the output parse/compile without errors?
- **Linting compliance** — Warnings or errors from static analysis tools (ESLint, Pylint, Clippy, etc.).
- **Code similarity metrics** — BLEU, CodeBLEU, or edit distance against reference solutions (useful but not sufficient alone — many correct solutions look nothing like the reference).

## Task-Specific Metrics
| Task | Key Metrics |
|---|---|
| **Code generation** | pass@k, functional correctness |
| **Bug fixing / SWE-bench** | % of issues resolved, patch correctness |
| **Code translation** | Computational accuracy, test pass rate in target language |
| **Code completion** | Exact match, edit similarity, prefix match |
| **Code explanation** | Human evaluation, factual accuracy |
| **Refactoring** | Behavioral equivalence, improvement in quality metrics |

## Efficiency & Practicality
- **Token efficiency** — How many tokens (input + output) does it take to reach a correct solution? Lower cost per solved problem matters in production.
- **Latency / time-to-solution** — Wall-clock time, especially for interactive use (autocomplete, chat).
- **pass@1 vs. pass@k tradeoff** — A model with high pass@1 is more useful in practice than one that needs many samples to get lucky.
- **Self-repair rate** — Can the model fix its own errors when given feedback (compiler errors, test failures)?

## Robustness & Generalization
- **Performance across languages** — Does it work well beyond Python? (MultiPL-E benchmark)
- **Sensitivity to prompt phrasing** — Does rephrasing the same task cause failures?
- **Handling of edge cases** — Does it generate code that handles boundary conditions?
- **Novel problem performance** — Accuracy on problems released *after* training data cutoff (to avoid data contamination).

## Popular Benchmarks (for reference)

| Benchmark | What It Measures |
|---|---|
| **HumanEval / HumanEval+** | Function-level generation (164 problems) |
| **MBPP / MBPP+** | Basic Python programming (974 problems) |
| **SWE-bench (Verified/Lite)** | Real-world GitHub issue resolution |
| **LiveCodeBench** | Contamination-free competitive programming |
| **BigCodeBench** | Complex, multi-library function calls |
| **Aider polyglot** | Multi-language edit/refactor tasks |

## Recommendations

1. **Start with pass@1 on a functional correctness benchmark** — it's the strongest single signal.
2. **Add SWE-bench** if you care about real-world repo-level tasks (not just isolated functions).
3. **Measure cost-efficiency** ($ per solved task) — a cheaper model that solves 90% of tasks may beat an expensive one that solves 95%.
4. **Don't rely on code similarity metrics alone** — they correlate poorly with correctness.
5. **Test on your actual workload** — build a private eval set from real tasks in your codebase to avoid benchmark overfitting.


Best open-source eval framework for comparing multiple models


Here are the top open-source eval frameworks for comparing multiple models, ranked by maturity and adoption:

---

### 1. **lm-evaluation-harness** (EleutherAI)
- **GitHub:** `EleutherAI/lm-evaluation-harness`
- **Best for:** Standardized academic benchmarks (MMLU, HellaSwag, ARC, etc.)
- **Strengths:** 200+ built-in tasks, HuggingFace integration, used by the Open LLM Leaderboard
- **Weakness:** More focused on benchmarks than custom/subjective evals

### 2. **Evals** (OpenAI)
- **GitHub:** `openai/evals`
- **Best for:** Custom eval creation with structured grading
- **Strengths:** Easy to define new evals as YAML/JSONL, supports model-graded evals
- **Weakness:** Originally OpenAI-centric (though extensible)

### 3. **promptfoo**
- **GitHub:** `promptfoo/promptfoo`
- **Best for:** Side-by-side model comparison with a nice UI
- **Strengths:** Built-in support for OpenAI, Anthropic, Ollama, HuggingFace, etc. Great CLI + web UI for comparing outputs. Supports assertions, LLM-as-judge, and custom scoring.
- **This is probably the best fit if your goal is literally "compare Model A vs B vs C"**

### 4. **Braintrust**
- **GitHub:** `braintrustdata/braintrust-sdk`
- **Best for:** Production-grade eval pipelines with logging and scoring
- **Strengths:** Nice UI, supports custom scorers, tracing, and datasets

### 5. **DeepEval**
- **GitHub:** `confident-ai/deepeval`
- **Best for:** LLM-as-judge metrics (faithfulness, relevancy, hallucination, bias)
- **Strengths:** Pytest-style interface, 14+ built-in metrics, CI/CD friendly

### 6. **inspect_ai** (UK AI Safety Institute)
- **GitHub:** `UKGovernmentBEIS/inspect_ai`
- **Best for:** Safety and capability evals with reproducibility
- **Strengths:** Clean Python API, task-based design, built-in sandboxing

---

### Quick recommendation:

| Goal | Use |
|---|---|
| **"Which model is best on my prompts?"** | **promptfoo** |
| **Standard benchmarks (MMLU, etc.)** | **lm-evaluation-harness** |
| **Custom LLM-as-judge scoring** | **DeepEval** or **promptfoo** |
| **Safety / alignment evals** | **inspect_ai** |
| **Production eval pipelines** | **Braintrust** |

For most people doing practical model comparison (running your own prompts across Claude, GPT-4, Llama, Mistral, etc. and seeing results side-by-side), **promptfoo** is the strongest starting point. It has the lowest friction and the best multi-provider support out of the box.


How do I run MMLU and HumanEval benchmarks on a custom model?


# Running MMLU and HumanEval Benchmarks on a Custom Model

## MMLU (Massive Multitask Language Understanding)

MMLU tests a model across 57 subjects (math, history, law, medicine, etc.) using multiple-choice questions.

### Using the `lm-evaluation-harness` (recommended)

```bash
# Install EleutherAI's evaluation harness
pip install lm-eval

# Run MMLU against a HuggingFace model
lm_eval --model hf \
    --model_args pretrained=your-org/your-model \
    --tasks mmlu \
    --batch_size auto \
    --output_path results/mmlu
```

**For a custom/local model:**
```bash
# Point to a local checkpoint
lm_eval --model hf \
    --model_args pretrained=/path/to/your/model,tokenizer=/path/to/tokenizer \
    --tasks mmlu \
    --num_fewshot 5 \
    --batch_size 8 \
    --output_path results/mmlu
```

**For an API-served model (e.g., vLLM, TGI):**
```bash
# If your model is behind an OpenAI-compatible API
lm_eval --model local-completions \
    --model_args model=your-model,base_url=http://localhost:8000/v1,tokenizer_backend=huggingface \
    --tasks mmlu \
    --num_fewshot 5
```

---

## HumanEval (Code Generation)

HumanEval tests a model's ability to generate correct Python functions from docstrings (164 problems).

### Using the official `human-eval` package

```bash
# Install
pip install human-eval

# 1. Generate completions from your model (you write this script)
python generate_completions.py  # outputs: samples.jsonl

# 2. Evaluate (runs code in a sandbox)
evaluate_functional_correctness samples.jsonl
```

**`generate_completions.py` example:**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from human_eval.data import write_jsonl, read_problems
import torch

model_path = "/path/to/your/model"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16, device_map="auto")

problems = read_problems()
samples = []

for task_id, problem in problems.items():
    prompt = problem["prompt"]
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.8, do_sample=True)
    completion = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    samples.append({"task_id": task_id, "completion": completion})

write_jsonl("samples.jsonl", samples)
```

### Using `bigcode-evaluation-harness` (alternative)

```bash
pip install bigcode-eval-harness

accelerate launch main.py \
    --model /path/to/your/model \
    --tasks humaneval \
    --n_samples 200 \
    --temperature 0.8 \
    --batch_size 10 \
    --allow_code_execution
```

---

## Key Tips

| Topic | Detail |
|---|---|
| **MMLU few-shot** | Standard is 5-shot (`--num_fewshot 5`) |
| **HumanEval metric** | Report **pass@k** (typically pass@1, pass@10, pass@100) |
| **HumanEval safety** | Code execution is sandboxed, but use Docker/VMs for extra safety |
| **GPU memory** | Use `--batch_size auto` or quantization (`load_in_4bit=True`) for large models |
| **Reproducibility** | Set seeds and report temperature, top-p, and number of samples |

Would you like more detail on any of these steps, such as setting up a specific inference backend or interpreting the results?


How do I set up automated evaluation pipelines for model development?


# Setting Up Automated Evaluation Pipelines for Model Development

This is a broad ML engineering topic, so here's a structured overview of the key components and best practices:

## 1. Core Pipeline Components

**Data Management**
- **Versioned eval datasets** — Use tools like DVC, Hugging Face Datasets, or cloud storage with versioning to ensure reproducibility.
- **Hold-out & golden sets** — Maintain curated test sets that never leak into training. Rotate periodically to avoid overfitting to benchmarks.

**Evaluation Harness**
- Define a standard `eval` interface that every model checkpoint can be run through:
  - **Task-specific metrics** (accuracy, F1, BLEU, ROUGE, etc.)
  - **Safety/alignment evals** (toxicity, bias, refusal rates)
  - **Latency & throughput benchmarks** (tokens/sec, P50/P99 latency)
  - **Regression tests** — a fixed set of input/output pairs that must pass

## 2. Orchestration Patterns

| Pattern | Tool Examples | When to Use |
|---|---|---|
| **CI-triggered** | GitHub Actions, GitLab CI, Jenkins | On every commit/PR to model code |
| **Scheduled** | Cron jobs, Airflow, Prefect | Nightly or weekly full-suite runs |
| **Event-driven** | Pub/Sub, SQS + Lambda | When a new checkpoint lands in storage |
| **Experiment tracker** | W&B, MLflow, Neptune | Log metrics alongside training runs |

## 3. Typical Architecture

```
┌──────────────┐     ┌───────────────┐     ┌──────────────┐
│  Training     │────▶│  Artifact     │────▶│  Eval        │
│  Pipeline     │     │  Store        │     │  Pipeline    │
│  (fine-tune,  │     │  (S3, HF Hub, │     │  (harness +  │
│   RLHF, etc.) │     │   GCS)        │     │   benchmarks)│
└──────────────┘     └───────────────┘     └──────┬───────┘
                                                   │
                                           ┌───────▼───────┐
                                           │  Dashboard /   │
                                           │  Leaderboard   │
                                           │  (compare runs) │
                                           └───────┬───────┘
                                                   │
                                           ┌───────▼───────┐
                                           │  Alerting /    │
                                           │  Gating        │
                                           │  (pass/fail)   │
                                           └───────────────┘
```

## 4. Key Frameworks & Tools

- **[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)** — The de facto standard for running LLM benchmarks (MMLU, HellaSwag, ARC, etc.)
- **[Hugging Face `evaluate`](https://huggingface.co/docs/evaluate)** — Modular metrics library with 100+ metrics
- **[HELM](https://crfm.stanford.edu/helm/)** — Stanford's holistic eval framework
- **[promptfoo](https://github.com/promptfoo/promptfoo)** — Eval framework for prompt engineering & LLM apps
- **[Braintrust](https://www.braintrust.dev/)** / **[Humanloop](https://humanloop.com/)** — Managed eval platforms

## 5. Best Practices

1. **Treat evals as code** — Version control your eval configs, prompts, and scoring rubrics alongside model code.
2. **Gate deployments** — No model ships unless it passes a minimum eval threshold (like a CI test suite).
3. **Use LLM-as-judge carefully** — Automated LLM grading scales well but needs calibration against human ratings. Track inter-rater agreement.
4. **Track multiple dimensions** — A single "accuracy" number hides regressions. Measure capability, safety, latency, and cost independently.
5. **Run evals at multiple stages:**
   - **Pre-training** — loss curves, perplexity
   - **Post-training** — benchmark suites
   - **Pre-deployment** — red-teaming, adversarial evals
   - **Post-deployment** — monitoring real-world performance drift
6. **Budget for compute** — Full eval suites can be expensive. Use a tiered approach: fast smoke tests on every commit, full suite nightly.

## 6. Minimal Starter Example

```python
# eval_pipeline.py
from datasets import load_dataset
from evaluate import load
import json, datetime

def run_eval(model, tasks=["mmlu", "hellaswag", "truthfulqa"]):
    results = {}
    for task in tasks:
        dataset = load_dataset(task, split="test")
        metric = load(task)
        predictions = model.predict(dataset["input"])
        results[task] = metric.compute(
            predictions=predictions,
            references=dataset["label"]
        )

    # Log results with metadata
    report = {
        "model": model.name,
        "timestamp": datetime.datetime.utcnow().isoformat(),
        "results": results,
        "passed": all(
            v["score"] >= THRESHOLDS[k] for k, v in results.items()
        ),
    }

    with open(f"evals/{model.name}_{report['timestamp']}.json", "w") as f:
        json.dump(report, f, indent=2)

    return report
```

Would you like me to go deeper on any specific aspect — like integrating evals into a CI/CD pipeline, setting up LLM-as-judge scoring, or building a results dashboard?


How do I download model weights in safetensors format?


# Downloading Model Weights in Safetensors Format

There are several ways to download model weights in **safetensors** format:

## 1. Using the `huggingface_hub` Python Library

```python
from huggingface_hub import hf_hub_download, snapshot_download

# Download a single safetensors file
hf_hub_download(
    repo_id="meta-llama/Llama-3-8B",
    filename="model.safetensors",
    local_dir="./model"
)

# Download all safetensors files from a repo
snapshot_download(
    repo_id="meta-llama/Llama-3-8B",
    allow_patterns=["*.safetensors"],
    local_dir="./model"
)
```

## 2. Using the `huggingface-cli`

```bash
# Install the CLI
pip install huggingface_hub

# Download a specific file
huggingface-cli download meta-llama/Llama-3-8B model.safetensors --local-dir ./model

# Download only safetensors files from a repo
huggingface-cli download meta-llama/Llama-3-8B --include "*.safetensors" --local-dir ./model
```

## 3. Using `transformers` Directly

```python
from transformers import AutoModel

# Transformers will prefer safetensors by default if available
model = AutoModel.from_pretrained("meta-llama/Llama-3-8B")
```

## 4. Direct URL Download

Most Hugging Face model repos offer safetensors files directly:
```
https://huggingface.co/<org>/<model>/resolve/main/model.safetensors
```

## Tips

- **Authentication**: For gated models (e.g., Llama), log in first:
  ```bash
  huggingface-cli login
  ```
- **Sharded models**: Large models split weights across multiple files like `model-00001-of-00004.safetensors`. Use `snapshot_download` or `--include "*.safetensors"` to grab them all.
- **Why safetensors?** It's faster to load, memory-efficient, and safer than pickle-based formats (like `.bin`), since it doesn't allow arbitrary code execution.

transformers
from transformers
AutoModel


Where can I find pre-trained models for my specific use case?


Great question! Here are the main places to find pre-trained models, depending on your use case:

## 🏆 Major Model Hubs

1. **Hugging Face Hub** (huggingface.co/models)
   - Largest open-source model repository (hundreds of thousands of models)
   - Covers NLP, computer vision, audio, multimodal, and more
   - Filter by task, framework, language, license, etc.

2. **TensorFlow Hub** (tfhub.dev)
   - Pre-trained models optimized for TensorFlow/Keras

3. **PyTorch Hub** (pytorch.org/hub)
   - Models optimized for the PyTorch ecosystem

4. **ONNX Model Zoo** (github.com/onnx/models)
   - Framework-agnostic models in ONNX format

## 🔍 By Domain

| Use Case | Where to Look |
|---|---|
| **NLP / Text** | Hugging Face (BERT, GPT, T5, Llama, Mistral, etc.) |
| **Image Classification** | Hugging Face, TF Hub, torchvision |
| **Object Detection** | Hugging Face, Ultralytics (YOLO), TF Model Garden |
| **Audio / Speech** | Hugging Face (Whisper, Wav2Vec2) |
| **Code Generation** | Hugging Face (StarCoder, CodeLlama) |
| **Generative AI / LLMs** | Hugging Face, Ollama, Together AI |
| **Embeddings / Search** | Hugging Face (Sentence Transformers) |

## 💡 Tips for Finding the Right Model

1. **Start with the task**: Identify your task (e.g., "text classification," "image segmentation") and filter by it on Hugging Face.
2. **Check the leaderboards**: Hugging Face's Open LLM Leaderboard, Papers With Code, etc.
3. **Consider size vs. performance**: Smaller models (distilled/quantized) are often good enough and much cheaper to run.
4. **Check the license**: Make sure the model license fits your use (commercial vs. research-only).
5. **Look at model cards**: Good models have documentation on training data, limitations, and benchmarks.

Would you like a more specific recommendation based on your particular task or domain?


How do I search for models by license type?


Based on my exploration of the codebase, **license-based model search is not currently implemented** in the dashboard. Here's what I found and how you could approach it:

---

## Current State

The dashboard tracks Hugging Face product **mentions** in tool responses (via keyword matching), but it doesn't store or filter by license metadata. The `hfProducts` table in `src/lib/db/schema.ts` has `name`, `keywords`, `category`, and `isActive` — no license field.

---

## How to Add License-Based Search

You'd need changes in three areas:

### 1. Schema — add a `license` column
In `src/lib/db/schema.ts`, add a field to `hfProducts`:
```ts
license: text("license"), // e.g. "MIT", "Apache-2.0", "OpenRAIL"
```

### 2. API — add a `license` filter parameter
In `src/app/api/dashboard/queries/route.ts`, read a `license` query param and add it to the SQL conditions, similar to how `tool`, `model`, and `category` filters already work.

### 3. UI — add a license dropdown
In `src/app/page.tsx` (the query explorer), add a filter dropdown populated with distinct license values.

---

## Alternative: Use the Hugging Face Hub API Directly

If you want to search HF models by license without building it into this dashboard, you can query the Hub API:

```bash
# Search for models with a specific license
curl "https://huggingface.co/api/models?filter=license:mit&limit=10"
curl "https://huggingface.co/api/models?filter=license:apache-2.0&limit=10"
```

Common license filter values: `mit`, `apache-2.0`, `openrail`, `cc-by-4.0`, `llama3`, `gemma`, etc.

---

Would you like me to implement the license filter feature in the dashboard, or is the Hub API approach sufficient for your needs?


Best way to compare model performance before downloading


# Comparing Model Performance Before Downloading

Here are the best approaches, roughly ordered from quickest to most thorough:

## 1. **Hugging Face Model Cards**
Every model repo on HF has a model card (`README.md`) that typically includes:
- Benchmark scores (accuracy, F1, BLEU, etc.)
- Training data and methodology
- Known limitations

## 2. **Open LLM Leaderboard**
- [open-llm-leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) — standardized benchmarks (MMLU, ARC, HellaSwag, TruthfulQA, etc.) across hundreds of models
- Great for apples-to-apples comparison of LLMs

## 3. **Papers With Code**
- Links models to their original papers **and** benchmark results
- Lets you filter by task (e.g., text classification, object detection) and sort by metric

## 4. **Chatbot Arena / LMSYS**
- Elo-style rankings based on **human preference** (blind side-by-side comparisons)
- More reflective of real-world quality than synthetic benchmarks

## 5. **HF Dataset Viewer + Eval Results**
- Many models publish evaluation result datasets — you can browse them directly in the browser without downloading anything

## 6. **Inference API / Spaces**
- Test models **live** via HF's free Inference API or Gradio Spaces before committing to a download
- Example: `curl` the Inference API or use the widget on the model page

## 7. **Compare Specific Metrics**
| Consideration | Where to Look |
|---|---|
| Accuracy / benchmarks | Leaderboard, model card |
| Latency / size | Model card (`safetensors` file sizes), `config.json` for param count |
| License | Model card sidebar |
| Community trust | Downloads count, likes, discussions tab |
| Real-world quality | Chatbot Arena, community blog posts |

## Pro Tip

Before downloading a large model, check:
```
# See model size without downloading
huggingface-cli repo info <model-id>
```

This gives you file sizes, last commit date, and tags — so you don't waste bandwidth on a 70B model when a 7B variant scores nearly as well on your target task.


How do I download a quantized version of a large model?