Devil In The Details: Housing Deeper Telemetry to Spot Agentic AI Risks from Hallucinated Inputs, to Tool Calls Starting with a Prompt
Published
Question: As AI-driven systems become more autonomous, teams struggle to understand how prompts translate into downstream actions. What kinds of runtime behaviors, decision chains, and hidden operational patterns must be tracked? How can deeper telemetry help identify risks?
Ganesh Narasimhadevara, Director Solutions Consulting at New Relic
Enterprise transition to agentic AI fundamentally breaks the assumptions that traditional engineering teams have built their practices around. While traditional systems follow a predictable script, AI agents improvise and adapt, completely changing how software behaves.
An LLM might decide to call different tools based on subtle variations in its reasoning, a retrieval system might surface different documents depending on semantic similarity scores, and a multi-agent workflow might take completely different execution paths based on how agents interpret their instructions. That is what precisely makes these systems powerful but it also means that without the right visibility teams lack visibility.
So what needs to be tracked? At the most fundamental level, teams must move well beyond surface-level logging. When an agent receives a prompt and initiates a chain of actions, every node in that decision tree carries risk.
- The specific tool calls an agent
- The order in which it makes them
- The reasoning it applies at each fork
- The state it carries forward
- All of this needs to be captured.
While traditional logging might tell you that an LLM call was completed and that a database query failed, it will not tell you
- Why the database query failed
- What the LLM actually decided to do
- Which tool it tried to call
- Whether the prompt was well-formed
- Whether it hallucinated an input that caused the failure downstream
- That gap is where incidents hide.
Hidden operational patterns are equally important to surface. In a unified monitoring environment,
- metrics,
- logs,
- traces, and
- events
- Each tells us something different and together paints a full picture.
For agentic systems, this means correlating token consumption rates against task completion outcomes to detect
- Runaway loops,
- Mapping latency spikes across tool-calls to identify bottlenecks that compound across agent steps, and
- Tracking inter-agent communication to understand where one agent's error cascades into another's context
The financial stakes sharpen the urgency. High-impact outages now cost a median of two million dollars per hour. That is over $33,000 dollars for every minute that systems are down.
A single misconfigured agent can burn through thousands of dollars in API calls before anyone notices, and hallucinations or cascading errors can slip through silently because teams cannot see what is happening inside their agent mesh.
Deeper telemetry addresses this directly by making the invisible visible. Rather than collecting more data, the priority is about housing all telemetry in one place
- So teams can see what is happening across the stack,
- Understand why it is happening, and
- Act quickly.
Dependency mapping plays a critical role here. Understanding:
- Which services talk to each other
- Which data stores support workloads
- How traffic flows through a system allows teams to understand the blast radius when something degrades.
Applied to agentic systems, this means being able to trace a prompt from its point of entry through every downstream tool call, API interaction, and model invocation, and identifying exactly where a deviation from expected behaviour originated.
The bottom line is this: observability for agentic AI is not an operational nicety. It is the foundational requirement for anyone building systems that think, plan, and act autonomously.
Related
