How to Verify AI-Generated Work: The Four-Tier Done Test

By Ryan Vanshur

An agent says the job is finished, the file is in the repository, the page loads, and the deploy went green, and all of that can be true while the feature still does nothing useful. AI output rarely looks broken. It looks done, so that look is exactly what we end up trusting, and that's why the look is the thing to test.

To verify AI-generated work is to refuse "done" until someone checks the claim at the depth it asserts, and this guide gives you the working method. It covers four tiers of done, the words that signal an unchecked claim, what counts as evidence, and a roster for who grades the work when its author can't.

The Four Tiers of Done

Take PlotIQ, an invented deathcare software company we use as the running example across the Guild. Its agent reports that the renewal-reminder job is done. The Verify skill sorts that claim into one of four tiers, and each tier proves more than the one before it.

1. Exists. The file sits at the expected location and isn't empty, and most claims of done stop right here, with the reminder job file sitting in the PlotIQ repository.
2. Substantive. The content is real, and a job that returns an empty list every time exists and looks finished, so it passes tier one and fails this one.
3. Wired. Something actually calls it, because finished code that nothing reaches is dead weight that reads as progress. For PlotIQ, the scheduler has no entry for the job.
4. Functional. It works end to end, and someone has watched it do so. For PlotIQ, the job ran against the live renewal table, and the accounts it should have reminded got reminded.

Red-Flag Words When You Verify AI-Generated Work

The rule underneath the tiers is short, and we keep it short on purpose, so no completion claim goes out without fresh verification evidence in the same message. That means not evidence from earlier in the session, not evidence from a similar case, and not a claim followed by a request that someone else go look.

The words give an unchecked claim away, and they're easy to catch once you know them. Should, probably, and seems to all show up in the wrong places, and each one admits that nobody ran the thing. So if the PlotIQ agent writes "the reminder job should work now," it has told you where it stopped. It reached tier one and is guessing at tier four.

What Counts as Evidence

Vague sentences are how tier one gets dressed up as tier four, so we ask for evidence you can paste into the message:

• A test run with its output pasted in, not described
• A row count from the live table, not from a sample
• A screenshot of the rendered page, not a sentence saying it renders
• An exit code, a file size, or the exact line that changed

For PlotIQ, "the reminder job works" is not evidence. "The job matched 37 accounts due this month against the live renewal table, and the reminder log shows 37 sends" is, because anyone can check it against the table, and so can we in a minute.

If we can't paste it, we haven't checked it, so the honest sentence is "built, not yet verified." That's a respectable place to be, as long as we say so out loud and don't let the word "done" slip in ahead of the proof.

Report Fixes in a Fixed Format

The same discipline applies to the message that reports a fix, so name the root cause at a file and a line, name the fix at a file and a line, and put the evidence from this session, the test that passed or the output that changed, in the same message as the claim. A fix reported that way can be checked in a minute, and we ask for that shape every time, but most fixes reported any other way can't be checked at all.

"Should be fixed now" isn't an outcome, and neither is "refresh the page and let me know." Either the proof sits in the message, or the message says which step is still unproven. Stacking a second fix on top of the first is the same failure in another form, because it leaves two wrong things that cancel each other out until one of them changes. Patching a symptom now and hunting the root cause later just plants a new bug somewhere else.

Who Grades the Work?

The question underneath the whole method is simple and worth asking every time, namely who grades the work if not the thing that did it? A model that built something shares the builder's assumptions, so asking it to review its own output sends those assumptions back to you dressed as evidence. We give the grading to someone else, and the roster looks like this:

• A reviewer from a different vendor. A model from another company reads the finished diff. Two vendors trained differently miss different things, and a person gives every finding a verdict before anything changes.
• An assigned opponent. Before a plan commits, one person builds the strongest case against it and gets graded on how convincing it is.
• A context check. Before building, someone searches what already exists and writes the result down, because "I looked" doesn't count as a result.
• A circuit breaker. After three failed attempts at a fix, work stops. Write down what each attempt proved, then re-plan rather than guessing a fourth time.
• You, at tier four. Only the person who can run the proof, or read the pasted output, can sign off on functional work.

The reviewer deserves a closer look, because it will be wrong often. It may object to a choice that only makes sense once you know why the customer's data arrives in that shape. It may also miss a bug anyone on the team would catch, because the bug lives in the business rule rather than the code. That's fine, because you aren't buying its verdict. You're buying a set of eyes trained on different mistakes than yours. The minutes you spend arguing with its findings are the minutes when your own blind spot gets said out loud, and nothing should be applied automatically, because the reviewer sees the diff and not the business.

How to Verify AI-Generated Work Across All Five Habits

Verification is the habit that audits the rest, because a context search can be shallow and still get reported as thorough. A debate can end with "the panel agreed" while the decision matrix sits empty. A debug run can claim a root cause with no test behind it. A second opinion can get skimmed and relayed as agreement. Each of those is a claim of done at tier one dressed up as tier four, and only a habit that refuses the claim until it sees proof can catch that shape.

So we treat the other four habits as claims, not conclusions, and each one has to leave an artifact behind. The search has to produce a written bucket and a next step, the debate has to produce a filled matrix with both columns on every row, the debug run has to name the test that proved the cause, and the review has to show a verdict on each finding. If the written artifact is missing, the habit didn't happen, whatever the message says.

Go Deeper

• Trust But Verify is the essay this guide grew out of.
• The Verify skill watches for the red-flag words and refuses claims without evidence.
• The Second Opinion skill sends finished work to a reviewer from another vendor.
• The Context Gap skill searches what already exists before anything gets built.
• AI Agent Governance Checklist lists the five fields to fill for every agent you run.
• Your AI Just Lied to You. It Didn't Even Know It. covers the difference between an AI scanning something and actually reading it.

The Scene When It Works

Picture the reminder job on a Monday morning, when the agent's message never says "should" and instead pastes the matched count from the live renewal table, 37 accounts, next to the reminder log showing 37 sends, and marks the job at tier four. A reviewer from a second vendor left two findings in the thread, and we agreed with one and disagreed with the other, because the renewal date comes from the contract and not from the table.

Nobody had to ask whether it worked, because the proof was in the message, and the proof was the part you could paste, so the question never came up. That's what done looks like once the habit is running, and it's the only version of done we're willing to sign.

Get the weekly operator brief and the next guide in your inbox. Subscribe to the newsletter, free.