I broke an agent on purpose to see what an AI reviewer would catch
Artificial Intelligence (AI) & Machine Learning (ML)

I broke an agent on purpose to see what an AI reviewer would catch

August 11, 2026/10 min read

There has been a lot of urgency and conversation around Product Builders. But the no-code and low-code tools we'd need for that haven't caught up. I want a tool that root causes the problem and generates a fix based on my error traces and the evaluation criteria I've defined. Not just a list of what's wrong.

So when Arize Signal became generally available on July 29, 2026, I built a broken agent to test it. In this article I break down how I tested Signal, what it caught, what it got wrong, and whether I'd trust it.

Signal is a worker that scans your production traces on a scheduled basis and ranks what's broken. Before I ran it, I expected Signal to blame the model or the prompt.

I also distrusted it. Signal has no idea what my agent is supposed to do. It has never seen my documents, my users, or my definition of a good answer. It reads spans. How does a tool with no knowledge of my domain tell a search that failed from a search that worked?

Arize is an AI observability company, which means teams send it data about how their AI applications behave. The paid platform is AX. The free open-source version is Phoenix. Most product people know this field as evals, that is, scoring what a model says against a set of criteria. This piece is about the layer underneath that.

When an agent runs, it leaves a trace, which is a tree of steps. Each step is called a span. One span records what the agent searched for and what came back. Another records a tool call and its result. Another records the prompt and the model's reply. Every span carries a status, OK or ERROR, inherited from OpenTelemetry, a standard built for web services that either answer or fail.

Arize was built on an idea that now sounds obvious: you cannot tell whether a model is working by checking whether it responded. A model can return a smooth, confident, completely wrong answer and still report success. That gap is why evals exist, and why an industry grew up scoring output instead of counting errors.

The same problem shows up one layer down. A step that reports OK has not necessarily done its job. My search tool never returned usable data, in four different ways. Only one of them turned a span red. Across 30 runs, 22 of those broken calls were reported healthy.

The tools to catch this do exist. Arize ships trajectory evaluations that score an agent's whole sequence of tool calls, and the prebuilt versions landed on July 1, three weeks before Signal. Every major eval framework has some version of it. But you have to know to reach for them. The status field is what you get by default, it is what your alerts are wired to, and it is a yes-or-no flag inherited from a world where services either answered or crashed.

What I built, and what I broke on purpose

I built a broken agent whose only job is to produce bad but well-formed traces, so that trace-quality tooling would have something to chew on. It answers questions using a search over a small document set, plus one tool call. Every step is instrumented. Every step also has a way to fail.

run_turn AGENT (root)

├─ retrieve RETRIEVER

├─ plan LLM

├─ search_web TOOL

└─ answer LLM

Signal is useful as a triage product and unreliable as a diagnosis.

  1. retrieve() fails in two ways. In 16 of the 30 runs it returns nothing at all. In the other 14 it returns three documents, scored between 0.06 and 0.19. The corpus is the trick. It holds four documents: cafeteria hours, parking validation, the holiday party, and bike storage. The agent is being asked about SSO provisioning, trace retention, and API key rotation. Not one of the four documents answers any of the eight questions.
  2. search_web() never works at all. 22 of 30 broken calls produce a green span. Only raises turns a span red. Triage filtering on status_code = 'ERROR' misses approximately 73% of the broken calls.

Figure 1. The search tool returned None. The span is not marked as an error, and the trace reports OK.

What Signal found

Signal scanned the project and opened three issues. As I'll get to, they covered only four of the six failure modes.

The top issue, marked High severity: search_web returns valid JSON marked "unparseable", answers silently fail. 22 matching traces. The second: search_web failing with upstream 502 on about 20% of calls. 14 traces. The third: RAG retrieve step always returns unrelated context, knowledge base not serving queries. 30 traces, also High. The largest of the three, by trace count, is ranked last.

All three issues landed on my tool and my retriever. I had expected Signal to go after my prompt instructions or the model itself. It never mentioned either, and it was right not to because the prompt and the model were doing exactly what I asked. I had also predicted Signal would struggle without domain knowledge about my agent. It found all three planted bugs without any.

Let’s think about what Signal had to work with. It has never seen my document set. It does not know what a good answer to an SSO question looks like. It cannot tell whether the agent's final response was true. What it can see is the shape of the runs: a tool that returns the same useless thing over and over, and a step whose output does not support the answer that follows it. That turned out to be enough to find both bugs, and it did it without any of the domain knowledge I assumed it would need.

Signal put the 502 error rate at roughly 20% of calls. The real number is 8 of 30, or 27%. It worked that out from traffic, with no access to the code. However, putting my PM hat on, this analysis is not close enough to quote in a leadership status update.

Figure 2. Three issues, three planted bugs, all marked High. None of them name the model or the prompt.

Right neighborhood, wrong house

Signal's top identified issue is that search_web returns valid JSON marked "unparseable". That is backwards compared to the error I actually introduced in the agent.

If I open the issue, the body is correct. It separates two modes: malformed output, three traces, where the payload is structured so the answer step can't use it. Null output, three traces, where output.value is the string "None" and the empty body is silently swallowed as no-results.

But the headline describes a bug I don't have. My tool doesn't return valid JSON that the parser rejects. It returns nothing, None gets coerced to the string "null" at line 170, and json.loads("null") succeeds. Unusable content, marked parsed. This is the exact inverse of what Signal’s headline says.

The same pattern shows up in issue #3: RAG retrieve step always returns unrelated context, that the knowledge base is not serving queries. It’s the retrieval that is broken. The knowledge base is fine. Four documents about cafeteria hours and bike storage are doing exactly what four irrelevant documents should do.

There's a line going around that evals are the new PRD. It's mostly right. Eval criteria are where you write down what "working" means, and that's a product decision. A spec describes correct behavior; an eval suite makes that description executable. Signal offers to help write it. Every issue carries a Create Evaluator button that turns a detected failure into a criterion you score against from then on.

Which is where the naming problem stops being cosmetic. Click that on issue #1 and I generate an evaluator for a bug I don't have. The headline says my parser is too strict. My parser is too loose. I'd be watching for rejected-but-valid payloads while accepted-but-useless ones keep shipping.

Figure 3. The body gets it right. The headline, which is the part most people read, gets it backwards.

What I'd tell a PM starting Evaluation and Agent Tracing today

Arize.ai Signal is good at telling you where to look. It's not reliable at telling you what's wrong.

Here's what it did well. I had planted six different failure modes. Signal found four of them. It did that without knowing anything about my documents, my users, or what a good answer would even look like. It read the traces and worked it out. It showed me the exact requests that failed. And it told me my prompt and my model were fine. If I had 3,000 traces instead of 30 and no idea where to start, this is the exact list I'd want.

However, everything in the product is presented at the same level of confidence: the headline, the evidence, the counts, the proposed fix. The counts were directionally right, but numerically wrong. If you can't tell which parts to trust, you either verify everything, which defeats the purpose of a ranked list, or you trust all of it, which is how you end up auditing a healthy corpus and wasting time.

So I'd use it as a pointer, but never as a finding. Open the issue. Read the evidence. Ignore the headline until you've formed your own view of what the traces say. That's a slower workflow than the product is selling.

Say you're running a health chatbot. Someone asks whether two medications interact. The knowledge retrieval step comes back empty. However, the model answers anyway, from whatever it picked up in training. Each span is green. The error rate is zero. Your dashboard says the feature is healthy, and it just gave your user medical advice with no knowledge behind it.

Signal catches that. It tells you that the retrieval step returned nothing. It shows you the exact questions where it happened. Your monitoring would never have flagged it, because nothing errored. That's the part worth paying for. But then it names the cause, and that's where it fails. It told me my knowledge base wasn't serving queries. My knowledge base was fine. The bug was in my retrieval code. So the headline sent me to the wrong place while the evidence underneath it was correct.

Signal is accurate about where to look and unreliable about what's wrong. If you read the headline and act, you audit healthy content for a day. If you open the issue and read the evidence, you find the actual bug in ten minutes.

The thing it did that I couldn't have done myself is read thirty traces with no knowledge of my domain, my documents, or my definition of a good answer, and it found four of the six failure modes I planted.

What it didn't do is close the loop. I wanted the next step: this is the root cause, here is the change, here is the diff. If PMs are supposed to be builders now, the tools that would let us build at a high quality bar, for e.g., Evaluation Tracing and Debugging tools, are still missing key features, and what's left keeps us at triage. I can tell you what's broken. I still can't fix it myself.

Further reading: