Playyy is a design agent that turns a user’s prompt into an editable design. For example, a user can write, “Turn this launch brief into a 10-slide presentation.” Before the editor shows a result, the agent has to choose an output type, decide whether it needs more information, select its tools, and create something the user can keep editing. A wrong route at the first step sends every later tool call down the wrong branch.
An agent eval is a repeatable test of that sequence. It gives the agent a defined task, records what it does, and checks the result against expected behavior. We need these evals because unit tests cover only individual tools. A unit test can verify that a tool rejects bad arguments, but it cannot verify that the model picks the right tool, calls it the intended number of times, and hands the result to the next step. Manual checks also miss the variation between two runs with the same input.
Our regression evals use versioned, scripted tasks and check the agent’s route, tool calls, arguments, and handoff. We rerun them after an instruction, tool schema, or skill changes. These tests use fixtures written for evaluation. Diagnostic replay never runs on the path that handles a live request.
During one poster regression, an engineer opened the trace expecting one bad argument. The request asked for one landscape poster divided into three visual regions. Instead, the trace contained 7 generation calls and no image-placement action. Four parts of the reconstructed context could have produced the extra calls: the system prompt, the tool description, a loaded workflow, or the fixture’s scripted tool results.
Reading the trace suggested several fixes, each in a different place. Editing one and getting a pass would not settle the matter. The pass might come from the edit, or it might be another sample from a variable model.
We added a procedure we call diagnostic replay to narrow the next test. It rebuilds the observable context around the decision, challenges several separately produced explanations, and returns a hypothesis and a smaller test for an engineer to run.
01 / Verify the failure
Check the failure before you explain it
Before starting diagnostic replay, we check the fixture and rerun the case. The replay may be incomplete, the evaluator may cover a stubbed part of the system, or the first failure may be normal variation between runs.
We run this workflow on versioned evaluation fixtures inside the test environment. A fixture is a scripted test case, including any attachments and tool results needed for that case. The workflow does not inspect live product requests.
Replay completeness
- Question
- Does the fixture contain the same context, attachments, tool results, and result states as the failed turn?
- Next action
- Repair the fixture before drawing a conclusion about the agent.
Repeatability
- Question
- Does the same fixture fail again, or was the first result one sample from a noisy distribution?
- Next action
- Choose a repeat count and decision rule, then record the observed results.
Observable scope
- Question
- Is the failed check about a model decision, or about pixels, persistence, billing, or delivery?
- Next action
- Use diagnostic replay for the former and a targeted full-stack run for the latter.
02 / Reconstruct the context
Reconstruct the context the model saw
The eval records six groups of inputs and outputs. If the replay drops a fixture attachment, changes a scripted tool result, or loads a newer schema, it is no longer investigating the same decision.
Each role receives a different set of fields from the stored bundle.
Runs for real
The agent model, system instructions, tool schemas, skills, and sandbox.
Stubbed in tests
The application backend and generated assets. The suite records tool results, not rendered or persisted output.
Because the tests stub the backend and generated assets, the standard suite can check tool selection and arguments. A targeted full-stack run still has to check pixels, persistence, billing, final delivery, and provider compatibility.
Replay doesn’t reproduce the activations or internal state of the original turn. It starts a new session under the reconstructed context, and any account it produces may contain invented details. Work on unfaithful model explanations found that a plausible explanation may omit a biasing input. That work studies chain-of-thought explanations rather than agent replay. We still apply its warning by checking every account against the trace.
This also differs from Causal Agent Replay. That method intervenes on a trajectory, runs it again, and compares outcome distributions. Our replay reconstructs context for a new conversation and helps an engineer choose an intervention. Until that intervention has a control and a declared decision rule, its result remains screening evidence. Causal attribution requires the later experiment.
03 / Separate the roles
Give each role only the context it needs
Four roles take part. The replayed agent can read the instructions and recorded steps but not the verdict. The interrogator knows which check failed but cannot inspect the prompt and tool-schema source files. The synthesizer receives the completed session records. The engineer is the only role that checks those claims against source and changes an input.
Replayed agent
- What it can see
- The reconstructed context, without the failed check
- Responsibility
- Answers questions about why the recorded steps looked reasonable
Interrogator
- What it can see
- The recorded steps and failed check, without the prompt source files
- Responsibility
- Questions the replayed agent one turn at a time
Synthesizer
- What it can see
- The completed session records
- Responsibility
- Reports agreement, contradictions, and claims that need checking
Engineer
- What it can see
- The trace, source, diagnosis, and regression suite
- Responsibility
- Checks the facts, designs the rerun, and decides where to edit
In the first wave, the same replayed-agent configuration enters 3 separate sessions. Each session pairs it with an interrogator from a different model family. The family change applies to the interrogator, not the replayed agent. An interrogator asks one question, reads the answer, and decides what to ask next. It can ask at most 5 questions.
The 3 sessions cannot see one another’s notes during that wave. We hide the failed check from the replayed agent so it cannot work backward from the evaluator’s verdict. We hide the source files from the interrogator so a suspicious sentence does not become the answer before questioning starts. These boundaries reduce two obvious forms of anchoring. They do not make the resulting accounts unbiased.
After the sessions finish, the synthesizer compares their records and keeps both agreement and contradiction. An engineer then checks every concrete claim, such as a quoted instruction or a tool result, against the stored trace before using it to design a rerun.
04 / Define a disconfirming test
Make the explanation predict an observable result
Asking a model “why did you fail?” usually produces an apology, a summary of the trace, or a prompt rewrite. None of those tells us whether the explanation is correct.
We ask it to name the mechanism that made the recorded action look reasonable and an observation that would count against that account. The JSON contract calls that observation falsifier. We use it to describe a disconfirming probe.
A single contrary sample cannot falsify an explanation of a stochastic model. Before the rerun, the engineer chooses the number of repeats and the result that would make the hypothesis less credible. If the original input was not repeated under the same design, the result is useful for screening but cannot estimate how much the intervention changed behavior.
{
"mechanism": "What made the action look correct, and where it took hold",
"evidence": "Session answers and recorded steps that support the account",
"falsifier": "What observation would count against this account"
}The session contract has no production_fix field. A replay of one decision cannot show where a durable rule belongs or which neighboring behaviors depend on the current wording.
05 / Challenge the accounts
Agreement earns a hypothesis a rerun
After wave 1, the synthesizer records where the 3 sessions agree and where they conflict. A second wave receives that summary and tries to produce a competing mechanism or a result the first account cannot explain. We synthesize again after the challenge.
The records are separately produced, but they are not statistically independent. They share the same trace and the same replayed-agent configuration. Their models may also share training data and evaluation habits.
A 2026 study of correlated LLM judges used 9 judges from 7 model families but measured about 2 independent votes of information. Accuracy was 8 to 22 percentage points below the ideal that assumes independent votes. Those figures come from judge-ensemble experiments. For our replay sessions, agreement only raises a hypothesis high enough to spend a rerun on it.
06 / Run the intervention
Change one input and keep the rest fixed
The interrogator can produce a testable claim. It can’t see where a durable rule belongs or which neighboring behaviors depend on the current wording. The engineer reads the source and regression suite before choosing where to make the change.
For the controlled rerun, the engineer changes one declared input and keeps the model, task, remaining fixture data, and evaluator fixed. They write down the intervention, repeat count, and decision rule before running it. We compare results across those repeats. A mismatch sends the explanation back for revision. A match justifies inspecting the source and testing a possible production patch against neighboring regressions.
01
Mechanism
Name what made the action look correct.
02
Disconfirming probe
State what result would count against the account.
03
Intervention
Have an engineer change one relevant input.
04
Rerun
Compare the observed behavior with the prediction.
A concrete example
When 3 panels became 3 image calls
The poster regression from the opening produced at least 2 different traces. One called the image tool 3 times and then continued to page assembly. Another called it 7 times and recorded no image-placement action. These were separate runs, not two stages of one run.
The trace also contained route and aspect-ratio problems. Those required separate tests. Here we focus only on why one requested poster became multiple image calls.
- Recorded trace
- “Split it into three blocks” was followed by three generation calls even though the fixture requested one poster.
- Session account
- The sessions proposed that the agent had converted the three internal regions into a deliverable count before it loaded the image-set workflow. The workflow then carried that count forward.
- Probe design
- Keep one poster with the same three-region composition, but remove the phrase that looks like a count. The account predicts fewer generation calls.
- Probe result
- All 3 probe reruns issued 1 generation call. We had not repeated the original wording with the same design, so this 3 of 3 result was a reason to inspect the source, not an estimate of effect size.
- Source change
- Source inspection found that the one-deliverable rule came after workflow selection. We moved it before that choice so regions inside one picture stay part of one image.
The wording probe and the production change answered different questions. The probe told us the proposed mechanism was worth following. Moving the rule tested whether the actual task could keep its wording while the agent made the deliverable decision earlier. The patch still had to pass the original regression, neighboring evals, and a targeted full-stack check.
Where the method stops
The test environment can be wrong, and the diagnosis can invent facts
The poster investigation found 2 fixture states that production can’t produce. Every generated image reused the same mocked asset ID, and an unlisted upload call returned an empty object. The model responded by retrying work that the real backend would not have asked it to repeat. We repaired the fixture before treating later reruns as evidence about the product prompt.
Diagnostic sessions can invent details too. In one run, 2 of 3 sessions said a pinned aspect ratio was absent from the model context. The remaining session said it was present. The recorded eval trace showed that it was present. The majority was wrong.
For facts about the original run, the trace wins. Session accounts only suggest how those facts may have shaped the decision. A controlled rerun can show whether behavior moves in the predicted direction under the declared test design. Pixels, persistence, billing, and delivery still need a targeted full-stack run.
Passing evals do not trigger this workflow. A difficult failure can still take close to 15 minutes because it may use 3 sessions, up to 5 questions per session, and a second wave. We run it only when reading the trace and checking the fixture have not produced a useful next test.
How we use the result
Write the test design before changing the prompt
Every concrete claim from the sessions is checked against the stored trace. Agreement and contradiction then help us choose which hypothesis to test first. Before the rerun, we write down the variable to change, the behavior we expect, the repeat count, and the decision rule. A result that misses the prediction sends us back to the trace. A result that matches it earns a source inspection and a possible patch that still has to pass the original regression and neighboring evals.
References and related work
Language Models Don’t Always Say What They Think (2023)
Shows that a plausible chain-of-thought explanation can omit a biasing input. The result is why we check replay accounts instead of trusting their fluency.
Nine Judges, Two Effective Votes (2026)
Finds that nine judges from seven model families carried about two independent votes of information in the experiments.
Which Agent Causes Task Failures, and When? (2025)
Locates the responsible agent and error step in a multi-agent trajectory. Playyy starts from one recorded decision and asks what to test next.
Causal Agent Replay (2026)
Re-executes trajectories under causal interventions and compares outcome distributions. Playyy reconstructs context and leaves the intervention to a later engineer-run test.
