benchmarks
62 talks
Latency Is a Budget. Humanlike Is the Goal.
Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell
We Let Claude Code and Codex Race Human Researchers
Are LLM Performance Benchmarks Reliable?
Homa: The End of TCP for AI Clusters
Can LLMs Write Fast Multi-GPU Kernels?
Einstein Arena: Harnessing Collective Agent Intelligence for Open Science
Computer Use at the Edge of the Statistical Precipice
Computer-use models will agentify the web, not APIs
Beyond Static Intelligence: Evaluating Continual Learning
Multiplayer agentic engineering
Benchmarking Coding Agents on New vs Legacy Codebases
When Will the Benchmaxxing Plague End?
Rethinking Environments for Long-Horizon Work
Teaching AI to Find Real Vulnerabilities
Benchmarks: The Good, the Bad, and the Ugly
Verifiable Environments for AI in Biology
Persona Engineering: A Field Guide to AI Synthetic Personas
DeepSWE: A Contamination-Resistant Coding Benchmark
State of Data
Training Frontier Models to Out-Think Hackers
Vending-Bench: Long-Horizon Agent Evals
Special Topics in Kernels, RL, Reward Hacking in Agents
Computer-Use 2.0: Agents Just Got Multi-Cursor
Stop Evaluating Models Like It's the 50s
The Agentic Web and the Bazaar Era of AI
SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale
Evals Are Broken, Use Them Anyway
SWE-rebench: Lessons from Evaluating Coding Agents
The Art & Science of Benchmarking Agents
Benchmarking Semantic Code Retrieval on Claude Code
Task Fidelity Scaling Laws
Rethinking What State-of-the-Art Means
Run Frontier AI at Home
Agentic Evaluations at Scale, For Everybody
What Do Models Still Suck At?
Running LLMs Locally: Practical LLM Performance on DGX Spark
How METR Measures Long Tasks and Experienced Open Source Developer Productivity
Why Agent Hype Can Fall Short of Reality
Coding Evals: From Code Snippets to Codebases
Hard Won Lessons from Building Effective AI Coding Agents
Context Platform Engineering to Reduce Token Anxiety
What Data from 20m Pull Requests Reveal About AI Transformation
Vision AI in 2025
Building Metrics that Actually Work
Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
Does AI Actually Boost Developer Productivity? (100k Devs Study)
Measuring AGI: Interactive Reasoning Benchmarks for ARC-AGI-3
Benchmarks Are Memes: How What We Measure Shapes AI and Us
The Wild World of AI: 6 Months That Changed Everything
Trends Across the AI Frontier
New York Times' Connections: A Case Study on NLP in Word Games
How fast are LLM inference engines anyway?
Vector Search Benchmark[eting]
Agents reported thousands of bugs, how many were real?
The Benchmarks Game: Why It's Rigged and How You Can (Really) Win
The Current State of Browser Agents
Evaluating Domain-Specific LLMs for Real-World Finance
Unveiling the Latest Gemma Model Advancements
A Practical Guide to Efficient AI
LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries
Open Challenges for AI Engineering