AI Observability: What Defenders Need When Systems Execute What They Read and Act Without Input
Published
Key Takeaways
- Security tooling was built for static infrastructure and deterministic code. AI agents are neither.
- Noma Security warns of zero-click vulnerabilities that enable account takeover by opening a malicious message.
- A single message can be read and acted on, triggering system-level actions without any user input.
- Braun notes that most organizations lack visibility into what their models generate and what those outputs trigger downstream.
- Agents don't show up in network traffic the same way a rogue SaaS subscription does.
Niv Braun, CEO and Co-Founder of Noma Security, explains what changes when AI systems start taking actions inside enterprise environments, and what those systems can access and trigger.
Braun led cyber intelligence teams in Unit 8200, where he built and managed security-focused systems, received certificates of excellence for creativity and initiative, and later led a 50-person team to build a cybersecurity product line from concept through delivery.
Security risks in AI are changing at a massive speed, and teams are not fully equipped yet. The biggest exposure is not the chatbots but AI agents that can access your data and operate across systems, triggering actions.
They don’t follow security assumptions. They lack visibility and ignore access control logic. What’s worse? They behave unpredictably at runtime.
From prompt injection attacks to poisoned data pipelines, enterprises are dealing with a growing surface they can’t monitor or fully audit. The result is a security model that cannot safeguard the infrastructure it was designed to protect.
Vishwa: Where are security teams seeing new risks emerge in the AI application stack?
Niv: The real exposure is at the agent layer. An AI chatbot that answers questions is manageable. An AI agent that can query your database, send emails, or call external APIs is a completely different risk surface.
The problem is that security tooling was built for static infrastructure and deterministic code. AI agents are neither. Most teams don't yet have visibility into what's running, what it can access, or what it's doing.
Vishwa: What security issues appear when AI retrieval pipelines touch sensitive enterprise data?
Niv: The core problem is that the model has no concept of access controls. It works with whatever gets retrieved. If your data hygiene isn't clean, and it rarely is, the model will surface things users shouldn't see.
There's also a more deliberate threat: attackers can poison the documents in a knowledge base with hidden instructions that change how the model responds. You're no longer just protecting data at rest. You're protecting it as it moves through a system that doesn't reason about authorization.
Vishwa: How can a prompt injection change the behavior of an AI application in practice?
Niv: Think of it as social engineering for AI. An attacker embeds instructions inside something the model reads as trusted input, a document, an email, or a web page. The model follows those instructions instead of the developer's.
We found a zero-click vulnerability in Google Gemini where an attacker could take over an account just by sending a message with a hidden instruction inside it. The user didn't have to do anything wrong. They just opened something.
Vishwa: What risks come from letting model-generated outputs execute inside enterprise environments?
Niv: When a model generates code or a query that then runs automatically, every mistake the model makes, or every manipulation an attacker pulls off, has real consequences. SQL injection through AI-generated queries is a real-world example. Code execution agents are another.
The human checkpoint that traditional security relied on is gone. Most organizations have no visibility into what their models are generating, let alone what those outputs trigger downstream.
Vishwa: What gaps surface when AI copilots are deployed inside enterprise environments?
Niv: Discovery is usually the first gap. Teams stand up copilots connected to internal systems with no central inventory of what's running, what data it can reach, or what actions it can take.
Permissions get set once at deployment and never revisited. You end up with agents carrying far more access than they need, in configurations nobody documented.
Shadow AI at the agent layer is harder to find than shadow IT ever was; agents don't show up in network traffic the same way a rogue SaaS subscription does.
Vishwa: What changes when defenders need to monitor how AI models behave at runtime?
Niv: Traditional security is largely static. Scan the code, check the config, and identify known vulnerabilities. AI behavior doesn't work that way. The same model given slightly different input can produce very different outputs. You can't audit it once and move on.
Runtime monitoring for AI means watching every prompt, every response, every tool call in real time. The detection logic also has to change.
Prompt injection doesn't look like SQL injection. Data leaking through an LLM response doesn't look like a file transfer. Defenders need a different observability layer entirely
Vishwa: Which parts of the AI pipeline still have almost no security visibility today?
Niv: The local agent layer is mostly dark for most organizations. Coding agents running on developer laptops, MCP servers, local AI tools, and security teams have no idea what these are connected to or what data they're touching. Supply chain is another major gap. Most enterprises have no systematic way to evaluate the models they're pulling from public registries or the third-party toolsets their agents are loading.
And agent-to-agent communication, where AI systems are calling other AI systems, is almost entirely unmonitored. That's where the next serious incidents are going to come from.
Related
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular