Insights on product thinking and engineering excellence • 17 articles published
Everyone agrees you should "look at your data." Then they open a hundred traces, scroll for ten minutes, feel vaguely worried, and reach for a tool. The looking has a method — open coding, then axial coding into a ranked taxonomy — and the taxonomy, not your assumptions, is what decides which evals to write.
Most teams build one corner of the eval map — offline, end-to-end, on the inputs they imagined — and call it "evals," the way you'd say "we have tests." Eval is plural. Here's the map: Hamel Husain's three levels, and the three questions that decide which kind of eval you're actually writing.
The first eval post said pull a hundred production traces and read them. But what if you haven't launched? Here's how to build your first eval set from dimensions, scenarios, and synthetic data — and why that set is scaffolding, not the building.
Most teams buy an eval tool, run it once, and call it done. They confuse benchmarks with evals — and ship AI that confidently produces wrong outputs no one will catch. The work the tool can't do for you is the work that matters: looking at your traces, naming the failure modes, and writing assertions that fire when they recur.
CODEOWNERS, blame, and branch protection were built around named humans with reputation. When the committer becomes an agent, the trust scaffolding stays standing on a foundation that quietly disappeared — and the load migrates to a single reviewer's signature.
AI code passes review on day one and ages worst by month six — not because it's wrong, but because the design intent that makes it refactorable was never part of the diff. The fix is encoding intent in artifacts the next iteration must read.
When an AI agent works on your codebase, your test suite is the only artifact of intent it consistently sees. Tickets, Notion pages, and Slack threads do not enter the loop. If your tests describe what the system should do, the agent is constrained. If they only describe what the code already does, the agent ships whatever it generated.
AI agents and offshore teams break in exactly the same way. They are execution accelerators, not thinking substitutes. The work that doesn't compress — specs, architecture, test design, critique — is the work you have to keep in-house. Outsource the typing. Keep the thinking.
AI didn't take your job. It exposed it. The work was never typing characters into an editor — it was translating ambiguous business intent into something that adds value to users. Code was always the byproduct. The translation was always the work.