Generator-Evaluator

Pattern: A named solution to a recurring problem.

Split code creation and code critique into separate agents so that neither role can blind the other.

If you’ve ever proofread your own writing and missed a typo that a colleague spotted in seconds, you already know the problem. You read what you meant to write, not what’s on the page. A model reviewing its own code does the same thing: it sees the intent behind each line and passes the bug sitting in it. Generator-Evaluator answers this by giving the work to two agents. One writes; the other judges, and it never sees how the work was made.

Understand This First

• Verification Loop — the single-agent feedback cycle that Generator-Evaluator extends across two agents.

• Subagent — the generator and evaluator are specialized subagents with distinct roles.

• Feedback Sensor — the evaluator is a feedback sensor with judgment authority.

Context

At the agentic level, Generator-Evaluator is a multi-agent architecture for producing higher-quality output than any single agent achieves alone. It sits above the Verification Loop, which runs generate-test-fix inside one agent’s context. Generator-Evaluator separates those responsibilities into two agents with independent context windows: one writes, one judges.

The pattern draws on a principle that predates AI: the person who creates the work shouldn’t be the only one who reviews it. Code review, editorial review, adversarial red-teaming, peer grading in education — they all exploit the same structural insight. When the critic is separate from the creator, the critique is harder to dismiss and harder to game.

Problem

How do you get reliable quality from an agent when the agent can’t evaluate its own output honestly?

LLMs exhibit a consistent self-review bias. Ask a model to generate code, then ask it whether that code is correct, and it will tend to say yes. The same context window that produced the output also produces the review, so the model’s reasoning stays anchored to its own prior choices. It finds reasons to defend what it wrote rather than reasons to doubt it.

The output looks confident. It reads well. But it hides bugs, missed requirements, and architectural drift behind fluent prose.

Forces

• Self-review bias means a single agent rates its own work too favorably.

• Context contamination makes it hard for one agent to both generate and critique, because the generation reasoning occupies the same window as the critique.

• Correlated blind spots survive a fresh context. An evaluator from the same model family shares the generator’s training. A clean window clears the two biases above, but not the failure modes the two models learned together.

• Quality thresholds are easier to enforce when the judge can’t be swayed by the author’s intent.

• Cost and latency increase with every additional agent in the loop, so the architecture must earn its overhead.

Solution

Assign two agents distinct, non-overlapping roles. The generator writes code, builds features, or produces whatever artifact the task requires. The evaluator grades the output against explicit criteria, produces structured critique, and decides whether the work meets the bar.

The two agents operate in a loop:

1. The generator produces output based on the task specification and any prior feedback.

2. The evaluator inspects the output against acceptance criteria and returns a structured verdict: pass or fail, with specific reasons.

3. If the evaluator fails the work, the generator receives the critique and tries again.

4. The loop repeats until the evaluator passes the output or a maximum iteration count is reached.

A planner agent often sits upstream of both. The planner breaks a high-level goal into discrete tasks with explicit acceptance criteria, giving the evaluator something concrete to grade against. Without clear criteria, the evaluator defaults to vague judgments (“looks good”) that don’t drive improvement.

Four design choices matter most:

Independent context windows. The generator and evaluator each get their own context. The evaluator never sees the generator’s internal reasoning, draft attempts, or abandoned approaches. It sees only the finished artifact and the acceptance criteria. This prevents the evaluator from rationalizing the generator’s mistakes.

Structured feedback. The evaluator doesn’t just say “try again.” It returns specific, actionable critique: which tests failed, which requirements weren’t met, which edge cases were missed. The generator treats this feedback as its primary input for the next iteration, not its own self-assessment.

Concrete grading criteria. The acceptance criteria should be as specific as possible: expected behavior, required test coverage, edge cases to handle, constraints to satisfy. Vague criteria produce vague evaluations. When the evaluator can run tests, check types, or interact with a live application, the grading gets sharper.

How independent the evaluator is. A fresh context hides the generator’s reasoning from the evaluator, but it can’t hide their shared training. Models tend to recognize and favor their own output, so an evaluator from the generator’s family is likely to miss what the generator missed. LLM-as-Judge gives the same warning about graders.

An evaluator from a different family can catch defects a same-family one won’t, but the gain depends on who reviews whom. A 2026 controlled study ran 116 LiveCodeBench problems through Claude and Codex in every pairing, with reviewers who saw the problem and the draft but couldn’t run tests. Claude reviewing Codex drafts lifted the pass rate from 71.6% to 89.7%. Codex reviewing its own drafts in a fresh pass reached 84.5%, at lower cost and latency than the cross-model pass. In the other direction, Codex reviewing Claude drafts dropped the pass rate from 91.4% to 82.8%: its changes broke more working solutions than they repaired. Claude reviewing its own drafts left the pass rate unchanged.

So treat a cross-family evaluator as the stronger and costlier option, and measure the pairing on your own work before you trust it. One benchmark in which the reviewer couldn’t execute anything doesn’t tell you how a given pair behaves on your codebase. A reasonable trigger for trying one is a same-model evaluator that has stopped finding anything while defects still reach production.

Tip: The evaluator doesn’t have to be a more capable model. It can be the same model, or even a cheaper one, running in a fresh context with a grading rubric. That default is cheap, and it works. Capability isn’t the axis to worry about. Independence is: a same-family evaluator escapes the generator’s reasoning but not its training.

How It Plays Out

A team builds an internal tool using a three-agent harness. The planner reads the product spec and decomposes it into feature tasks, each with a checklist of acceptance criteria: required endpoints, expected UI behavior, error handling requirements. The generator picks up each task and writes the implementation. The evaluator loads the running application through a browser automation tool, navigates the pages, fills out forms, clicks buttons, and checks whether the behavior matches the spec. When the evaluator finds that a form submission silently drops validation errors, it returns a structured report: “The /register endpoint accepts empty email fields. Expected: validation error with HTTP 422.” The generator reads the critique, adds the validation, and resubmits. On the next pass, the evaluator confirms the fix and moves on.

A solo developer working on a data pipeline separates generation from evaluation without a framework. She uses one agent conversation to write transformation functions and a second conversation to review them. The review conversation gets only the function signatures, the docstrings, and a set of sample inputs with expected outputs. The review agent runs the samples, flags two functions that produce incorrect output on edge cases, and returns the failures. She pastes the feedback into the generation conversation, which fixes the issues. The separation is manual, but it catches bugs that the generation agent missed on its own.

A payments team runs its generator and evaluator on the same model, and for a month the evaluator passes nearly everything while race conditions keep turning up in staging. Instead of switching every review to a second vendor, the lead replays 60 recently merged changes through an evaluator from a different model family and scores each verdict by hand. On the ledger service’s concurrency code it flags three real defects the first evaluator passed. Everywhere else, half of its proposed fixes fail tests that had been green. The team routes only the concurrency-heavy directories to the second evaluator and keeps the cheaper one for the rest.

Consequences

Benefits:

• Output quality improves because critique comes from an independent context that the generation process can’t bias.

• Failure modes become visible. The evaluator’s structured feedback creates an audit trail of what went wrong and when, making debugging easier for humans.

• The pattern scales naturally. You can increase iteration depth (more passes through the loop) or tighten evaluator rigor (stricter criteria, more tools) without changing the architecture.

Liabilities:

• Cost and latency roughly double at minimum, since every piece of work goes through at least two agent passes. For simple tasks where a single agent gets it right on the first try, the evaluator pass is pure overhead.

• The pattern requires well-defined acceptance criteria. If the criteria are vague, the evaluator can’t grade meaningfully and the loop degenerates into wasted iterations.

• Iteration limits need tuning. Too few passes and the generator can’t converge. Too many and you burn tokens on diminishing improvements, or the generator starts cycling between equally mediocre alternatives.

• A cross-family evaluator can make the work worse. It may rewrite correct code it doesn’t recognize as correct, and whether it helps depends on which model reviews which, so a pairing has to be measured rather than assumed.

Complements: Code Review — The review is the evaluation half of the generate-evaluate cycle.

Complements: Plan Mode — A planner agent produces the specs and acceptance criteria that the evaluator grades against.

Contrasts with: Ralph Wiggum Loop — Where Generator-Evaluator uses structural separation to produce honest critique, the Ralph Wiggum Loop is what happens when self-review has no teeth.

Contrasts with: ReAct — Generator-Evaluator is two agents criticizing each other; ReAct is one agent criticizing its last observation.

Contrasts with: Reflexion — Reflexion is single-agent self-critique; Generator-Evaluator is two-agent independent review. Choose Reflexion when you have a machine-checkable oracle; choose Generator-Evaluator when you do not.

Contrasts with: Trusted Monitoring — Generator-Evaluator puts the critic inside the construction loop; trusted monitoring watches the loop from outside it.

Depended on by: Dark Factory — One agent writes, another critiques — the evaluator stands in for the missing human reviewer in a Dark Factory.

Extended by: Evaluator-Driven Code Search — Evaluator-Driven Code Search turns the evaluator's score into selection pressure across a population of candidate programs.

Extends: Verification Loop — The verification loop runs generate-test-fix in one agent; Generator-Evaluator distributes those roles across two agents for stronger critique.

Mitigates: Compounding Error — A separate evaluator catches the generator's errors before later steps build on them, provided its blind spots differ from the generator's.

Related: Agent Teams — Agent teams coordinate multiple agents across many tasks; Generator-Evaluator is a specific two-role architecture for quality within a single task.

Related: Permission Classifier — A generator-evaluator loop judges the *quality* of an agent's output; the classifier judges the *safety* of an agent's action — different questions, similar shape.

Related: Prompt Chaining — A generator-evaluator loop is a two-step chain that repeats until the judge accepts the output.

Related: Steering Loop — The steering loop decides whether to continue, adjust, or stop; Generator-Evaluator is a specific instantiation focused on output quality.

Specializes: Belt-and-Suspenders — A separate evaluator judging a generator output is Belt-and-Suspenders in agentic form.

Used by: Loop Engineering — The maker/checker split is the loop's internal division of labor: one agent proposes, another independently grades.

Uses: Feedback Sensor — The evaluator is a feedback sensor that grades output against criteria.

Uses: LLM-as-Judge — The evaluator is often an LLM judge grading against a rubric, and it inherits the judge's biases, including self-preference.

Uses: Subagent — The generator and evaluator are specialized subagents with distinct contexts and roles.

Violated by: Review Overcorrection — Review Overcorrection breaks the loop when the evaluator keeps finding problems in work that was already correct.

Sources

• Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Yoshua Bengio, Aaron Courville, and Sherjil Ozair introduced Generative Adversarial Networks in Generative Adversarial Nets (NeurIPS 2014). The GAN’s core insight that pairing a generator against a discriminator produces stronger output than either alone inspired the adversarial structure adapted here for code generation.

• Anthropic described a three-agent harness (planner, generator, evaluator) for long-running application development in Harness design for long-running application development (March 2026). The evaluator used browser automation to interact with live applications and grade output against spec-derived criteria, demonstrating the pattern at production scale.

• Zuodong Xiang, Yike Zhang, YueMing Zhang, and Hailu Xu measured same-model and cross-model review in Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa? (accepted at Agentic SE @ KDD 2026). Their six-condition experiment on 116 LiveCodeBench tasks supplies the directional gains, the regression, and the cost comparison in the Solution’s discussion of evaluator independence.

• Arjun Panickssery, Samuel R. Bowman, and Shi Feng showed in LLM Evaluators Recognize and Favor Their Own Generations (2024) that models can tell their own text from other models’ and that this self-recognition tracks a measurable preference for their own output. The bias belongs to the model rather than the conversation, which is why a fresh context can’t remove it.

• Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui introduced AgentCoder in AgentCoder: Multi-Agent-Based Code Generation with Iterative Testing and Optimisation (2023). Their framework split code generation into three specialized agents (programmer, test designer, test executor) and showed that multi-agent separation outperformed single-agent generation on competitive coding benchmarks.

• The separation of code authoring from code review is a longstanding software engineering practice. Michael Fagan’s software inspection process in Design and Code Inspections to Reduce Errors in Program Development (1976) established that independent review by someone other than the author catches defects that self-review misses, a principle that Generator-Evaluator applies to autonomous agents.