By Sunday, I had a problem the final answer could not show me.
I had spent much of the weekend developing and validating Think → Build → Prove 2.0.1. The outputs often looked strong. But a polished result could still hide a failed workflow.
I thought I was testing whether the system could produce good work. The real test became harder:
Can I prove the workflow did what it says it did?
A specialist could be selected. Its method could be opened. That still did not prove the method was applied to the artifact that shipped. A late correction could leave an earlier evaluation attached to the wrong version. An acceptance test could also become easier if the executor could see hints from the answer key.
None of this is as entertaining as asking AI to build something in 30 seconds. Neither is reconciling an expense report. Both become interesting when they are wrong.
A good answer can hide a bad process
Selecting a capability is not the same as executing it.
If an AI workflow says it used a specialist method, I now want evidence I can inspect. At minimum:
- the artifact and version that were evaluated;
- the method or requirement that was applied;
- the issue that was found;
- the correction that was made; and
- the affected area that was rechecked afterward.
That receipt matters because a polished artifact can look correct even when an important step never happened.
Microsoft Foundry’s agent evaluators separate tool selection, tool-call success, and tool-output utilization instead of treating the final answer as the whole test. Google Cloud makes a similar distinction between evaluating the final response and evaluating the trajectory, the sequence of actions and tool calls used to reach it.
For an operator, the rule is simple:
Do not ask only, “Is the answer good?” Ask, “Can I see that the required work actually happened?”
The builder should not be the only judge
I also became stricter about separating creation from acceptance.
The producer’s job is to create the work. The evaluator’s job is to compare it with explicit requirements and be free to reject it.
OpenAI’s evaluation guidance warns against “vibe-based evals,” where something is judged to work because it looks right, and recommends task-specific evaluation early and often. NIST’s 2026 draft TEVV-Athlon framework takes the broader view: testing, evaluation, verification, and validation should create evidence that an AI system can meet the goals that matter to the organization.
You do not need to turn the company into a software lab.
For consequential AI work, define what must be true before you build. Then evaluate the finished work against that standard.
A passed test can expire
Suppose version two passes. Then version three changes a claim, opening, CTA, dependency, or source.
The old evaluation may still be accurate about version two. It no longer proves that version three passes.
So I stopped treating “passed” as permanent.
OpenAI recommends continuous evaluation as systems change. My operating rule is narrower: after a material correction, rerun the checks that the correction could invalidate. Do not automatically rerun everything.
That gives you rigor without turning governance into bureaucracy.
Speed comes from knowing what not to redo
The bad shortcut is skipping a control because you want the work finished: trust the attractive answer, count routing as execution, or carry an old approval onto a changed artifact.
The other mistake is treating rigor as a reason to reopen every decision and rerun every test.
The better shortcut is subtraction.
Keep supported decisions locked. Recheck the areas a change could actually affect. Use the lightest model and reasoning effort that still meets the quality bar. OpenAI’s current model-selection guidance recommends comparing configurations and keeping the lightest setting that meets the requirement.
Then stop.
That is what slow is smooth, and smooth is fast now means to me.
The goal is not more process. It is moving from decision → execution → verification without reopening finished work or discovering that the thing you approved was not the thing you tested.
The first pass may feel slower.
The fifth pass should not exist.
Run a second pass before you call the work done
The Second Pass Audit is the practical check I use to compare the source, the output, the correction, and the recheck before treating work as complete.
Get the Second Pass Audit Kit →
Resources
- OpenAI — Evaluation Best Practices
- NIST — TEVV-Athlon Framework
- Microsoft Foundry — Agent Evaluators
- Google Cloud — Evaluate Gen AI Agents
- OpenAI — Model Selection
Related reading
- AI Buyer Simulator: I Built One, Then Realized I Rigged the Test
- How I Stopped Chasing AI Tools and Built an AI Workflow That Actually Works
- I Put Loop Marketing to Work. The First Thing I Did Was Refuse to Build Another Marketing System.
