Three Questions I Ask Before Trusting AI Output
AI generates answers in seconds. Confidence is not accuracy. Three questions that separate effective AI users from ones who amplify mistakes.
Insights on product thinking and engineering excellence • 8 articles published
AI generates answers in seconds. Confidence is not accuracy. Three questions that separate effective AI users from ones who amplify mistakes.
Your juniors are shipping more code and learning less from it. The agent answers the question before they form the question. The debugging muscle never gets built. The cost lands eighteen months from now when those juniors are the mid-levels and nobody can reason from first principles.
Your lead time is not slow because of code review or approvals or sprint ceremonies. It is slow because three files do everything, four modules depend on each other in a circle, and every meaningful change has to pass through the same engineer who understands the wiring. Process is the symptom. The architecture is the disease.
Your engineering agent reviews PRs, writes tests, and ships code. Nobody can tell you whether it is better than rubber-stamping. Without evals on the agent itself, you are flying blind on the tool that is making 40% of your engineering decisions.
A PM and an engineer sit down with four colors of sticky notes. Sixty minutes later they have surfaced every business rule, every edge case, and every unanswered question — and produced acceptance tests an AI agent can build against. No prose. No translation layer. No "the AI guessed wrong" post-mortem.
The demo works. The prototype impresses. Then it hits production and quietly fails — wrong answers at scale, cascading LLM timeouts, no way to know when the prompt regressed. This is not an AI problem. It is an engineering problem with a known solution.
In most teams, every architecture decision, every risky PR, and every onboarding conversation routes through three or four senior engineers. That is not seniority — it is a single point of failure. Here is how to move that judgment out of their heads and into the workflow.

Most teams treat engineering quality reactively — ship fast, catch problems in production, fix when it hurts. That cycle keeps you busy without making you better. Prevention, Detection, and Correction form a closed loop where each phase reduces the work the next must do.