Notes from the field
What we learn building benchmarks and measuring agentic engineering — written down as it happens.
Simranjit Singh12 min read
Building a coding benchmark from your own repository
The full process, run on Apache Airflow: mining merged pull requests into validated tasks, then the three refinements that decide which tests are fair, which tasks are answerable as written, and how to freeze a number you can trust.
ReadBharath Bhat5 min read
A look at coding agent benchmarks, and what may be interesting next
How current coding agent benchmarks and eval frameworks are designed — from task sourcing to verifiers — and where we think they're heading.
ReadBharath Bhat4 min read
What would it take to match model intelligence to the task?
We're working on offering the right level of model intelligence for the right task. Before we can build it, there's a set of interlocking questions to answer. This post is the map.
ReadSimranjit Singh7 min read
Frontier models go out of their way to cheat
Six rounds of cat-and-mouse against models that would rather find the answer than solve the task — and what it did to their benchmark scores.
Read