Fix Broken AI Apps | Expert Repair for Replit, Lovable, Bolt & More

TL;DR

Traditional unit testing fails when applied to AI agents due to their stochastic nature, multi-step workflows, and stateful context. Relying on simple, end-to-end checks misses hidden failures and grants false confidence. To achieve true production reliability, engineers must adopt a structured State-Transition Evaluation (STE) framework that breaks down complex workflows into deterministic, verifiable steps and measures both the final outcome and the correctness of the intermediate states.

The Problem: Why Simple Evals Grant False Confidence

An AI agent's value lies in its ability to execute complex, multi-step tasks autonomously. Unfortunately, this complexity makes standard evaluation techniques virtually useless. Developers frequently face challenges that obscure true reliability:

1. The Stochasticity Trap

Due to the nature of LLMs (even with temperature set to zero), agents exhibit non-deterministic behavior. A single prompt can yield different tool calls, reasoning steps, or final outputs across runs. If 9 out of 10 runs pass a simple evaluation, the system appears reliable, but that 1 failure, the result of a slight token shift, is unpredictable and impossible to reproduce consistently without a structured framework.

2. Hidden Failures in Multi-Step Workflows

In a four-step workflow (e.g., Plan → Search → Synthesize → Act), an error might occur in Step 2 (Search), but the agent manages to hallucinate or recover partially in Step 3 (Synthesize), leading to a result that is wrong but plausible. A simple end-to-end evaluation that only checks the final answer will mark this run as a success, masking a core system failure.

3. State-Dependent Context Drift

Agents are stateful: what they learn in step $N$ affects step $N+1$. Evaluating a task in isolation fails to capture the cumulative effects of erroneous memory or tool usage that build up over a long interaction. A perfect performance on a simple query doesn't guarantee success when the agent is operating on stale or polluted internal state (e.g., having remembered a fabricated fact from 20 steps ago).

4. Difficulty of Ground Truth Definition

It's easy to define the expected output for simple classification, but hard for a complex agent that interacts with external APIs. Is the "correct" output the final user-facing response, or the sequence of correct API calls, or both? Lack of clear ground truth for intermediate steps renders testing incomplete.

The Core Concept: State-Transition Evaluation (STE)

The fix for unreliable evaluation is to move beyond judging the final answer to verifying the entire decision path. The core concept is State-Transition Evaluation (STE), which treats the agent's workflow as a finite sequence of verifiable actions and internal state changes.

STE is based on two pillars:

Intermediate Checkpoints: The evaluation pipeline must intercept and verify the agent's internal state at every critical juncture (e.g., after planning, after tool call arguments are generated, after memory retrieval).
Deterministic Oracles: We must replace subjective human scoring with automated, deterministic "oracles" (e.g., specialized LLM classifiers, JSON Schema validators, or structured output checkers) that confirm the correctness of the output format and decision logic at each checkpoint.

This ensures that even if the agent produces a slightly different sentence structure, the underlying logic and system state remain reliably correct.

Step-by-Step Implementation of the STE Framework

To implement a reliable State-Transition Evaluation framework, follow these steps:

Step 1: Decompose Workflows into Verifiable State Transitions

Break down the agent's complex task into sequential, named steps that correspond to internal state changes (e.g., "Initial Plan," "Tool Call Selection," "Argument Generation," "Memory Update").

Example Workflow Decomposition (Task: "Schedule a meeting"):

State 1: Plan Generation: Input $\rightarrow$ LLM outputs a structured plan (e.g., JSON list of steps).
State 2: Tool Selection & Args: Plan $\rightarrow$ LLM selects schedule_meeting and outputs JSON arguments.
State 3: Tool Execution Result: Arguments $\rightarrow$ API call returns a success or failure state.
State 4: Final Response Synthesis: Result $\rightarrow$ LLM confirms the scheduling to the user.

Step 2: Define Deterministic Checkpoint Oracles

For each state transition, define a simple, machine-readable check that acts as your ground truth oracle.

For State 1 (Plan Generation): Use a JSON Schema validator to ensure the plan structure is correct. Use a lightweight LLM oracle to check the intent (Is the plan relevant to scheduling?).
For State 2 (Tool Selection & Args): Use a pre-defined JSON Schema to validate that the selected tool is correct and all required arguments are present and correctly typed (see previous post on RTIL).
For State 3 (Tool Execution Result): The oracle is the boolean success/failure of the API call itself.

Step 3: Implement Causal Tracing and Logging

Your evaluation harness must capture the entire agent history, every prompt, every internal thought, and every tool output.

Trace ID: Assign a unique Trace ID to every full evaluation run.
Structured Logs: Log all inputs and outputs at every state transition. This logging is not just for debugging; it is the data source for your oracles.

Step 4: Run Failure-Injected and State-Dependent Tests

Reliable evaluation requires testing both successful paths and failure recovery.

Non-Deterministic Evaluation (N=20): Run your test suite a large number of times (e.g., $N=20$) to understand the distribution of successful vs. failed runs. If success rate is $<95%$, the component is not production-ready, even if some runs were perfect.
Failure Injection Tests: Deliberately inject failures at specific checkpoints (e.g., force the tool execution in State 3 to return a 500 error). Verify that the agent transitions to a correct recovery state (e.g., State 4 involves an apology and a suggestion for retry, not a hallucinated success).
State Initialization Tests: Write tests that initialize the agent with pre-polluted memory (e.g., inject a contradictory "fact") and ensure the agent's reasoning in the face of conflict is reliable.

Verification & Testing

The effectiveness of your evaluation framework itself needs to be verified:

Oracle Agreement Testing: Test your LLM oracles (if used) against human-labeled ground truth for a small subset of decisions. If the oracle agreement is below 90%, the oracle itself is too noisy and must be refined with clearer prompts or replaced with structured validators.
Reproducibility Test: Take five failed runs identified by your STE framework. If you cannot reproduce the exact sequence of steps, tool calls, and failures when re-running the test, your environment or logging is non-deterministic. Fix the underlying non-determinism (e.g., setting seeds, mocking external APIs).
End-to-End Sanity Check: Ensure that tests that pass the STE framework always also pass a simple end-to-end check, but critically, verify that a failure in any intermediate state (as identified by an oracle) always leads to the overall run being marked as a failure, preventing hidden failures.

Key Considerations for Engineering Leaders

Evaluation is an Architecture: Treat your evaluation harness as an integral part of your agent's architecture, not just a set of scripts. It should be version-controlled, documented, and have its own test suite.
Mock Everything: To achieve the necessary level of determinism, mock all external dependencies. Tool APIs, databases, and external search engines must return pre-defined, controlled responses during testing. Stochasticity should be isolated to the LLM core, where it can be measured statistically.
Cost vs. Coverage: High-coverage STE frameworks can be expensive due to multiple LLM calls for oracles and high run counts ($N=20$). Balance this by using smaller, cheaper models for the oracle checks and focusing the most expensive, high-N runs on your agent's most critical workflows.

By implementing State-Transition Evaluation, you stop guessing about agent performance and start proving its reliability through verifiable, deterministic steps.

Get a reliability audit for your AI agent →

Evaluating AI Agents Is Harder Than It Looks: A Framework for Real-World Testing