Attackers Break In, Governance Breaks Down: AI Knows What’s Relevant, But Not What’s Appropriate

Gidi Cohen, CEO and Co-Founder of Bonfy.AI, explores what happens when systems understand relevance better than they understand appropriateness. That gap may become one of the defining governance challenges of the AI era.

Cohen founded Skybox Security and helped pioneer cyber risk management, attack simulation, and security posture modeling for enterprises.

He says many organizations are focused on visibility, access, and adoption, while a more difficult question is not whether AI can access information, but whether it should use that information in a particular context.

AI can retrieve information using legitimate permissions and generate outcomes that conflict with policy intent and customer expectations.

That is where Cohen places the idea of "Shady AI." Neither unauthorized AI. Nor malicious AI. Approved AI is moving beyond the boundaries that organizations intended.

The deeper concern is the accumulation of thousands of seemingly legitimate decisions made at machine speed. Each one appears reasonable in isolation. But create a widening gap between relevance and appropriateness.

Vishwa: You recently introduced the term ‘Shady AI’ to describe AI activity that can drift beyond business intent even inside approved and trusted workflows. What kinds of scenarios pushed you to frame the problem that way?

Gidi: The term emerged from a pattern I kept encountering in conversations with CISOs and data security teams. They were rightly focused on Shadow AI:

• employees using unsanctioned tools, 
• business units adopting AI SaaS without security review

But the more I talked with practitioners already running pilots and early production deployments, the more I noticed a different kind of discomfort. The systems they were worried about were not the unauthorized ones. They were the approved ones.

That tension pushed me to separate the risks more precisely. Shadow AI is a visibility and governance problem — you don't know where AI is being used, so you can't enforce policy consistently. 

Shady AI is something else. 

• The platform is sanctioned. 
• The deployment is intentional. 
• The workflow is in production. 
• And the outcome is still unacceptable.

The scenarios that crystallized this were deceptively ordinary. An AI assistant authorized to generate customer responses, prompted in a way that aggregates details beyond what should be shared. 

Nothing unauthorized:

• The tool is approved, 
• The access is permitted
• Yet the outcome is a policy violation.
  • No alert fires. 
  • Nothing looks broken. 
  • And yet customer trust has been compromised.
  • What makes these scenarios hard to dismiss is that they can't be addressed by hardening the system or sanctioning the platform more carefully. 

The risk wasn't at the edges of approved use. It was inside trusted workflows, on legitimate access, doing exactly what the system was designed to do. Authorized AI can still produce unauthorized outcomes — not because the platform is untrustworthy, but because trust in a platform doesn't govern how data is used once reasoning and automation operate at scale.

Underlying all of it is what I think of as the missing "who." Most data security is built around identifying what data is. But customer protection isn't fundamentally about what the data is — it's about:

• who it belongs to, 
• who's receiving it, and 
• whether a specific use aligns with the relationship between those parties. 

An AI system optimizes for relevance. It doesn't inherently understand appropriateness. And relevance is not permission.

"Shady AI" was the term that captured this most honestly. Not unauthorized, not malicious — but gradually drifting from business intent and customer trust, often without any clearly identifiable violation. 

You can materially reduce unsanctioned AI use and still face hard questions about whether your approved workflows are operating within intended policy boundaries. Solving one doesn't solve the other.

Vishwa: Security teams focused on protecting data at rest and in transit. How should that change when AI systems summarize and reason over enterprise data?

Gidi: The classic framing — protect data at rest, protect data in transit — was built for a world where enterprise data primarily existed as stored objects moving between defined locations. 

• A file sits in a repository. 
• A user retrieves it. 
• It travels across a network. 
• Controls wrap those moments.
  • That model was never perfect, but it was workable.

AI breaks the underlying assumption. Modern AI systems don't primarily retrieve and transmit stored objects. 

• They retrieve fragments from multiple sources, 
• combine them with user prompts, 
• apply reasoning across them, and 
• generate outputs that never existed anywhere as a single document. 

The information being evaluated — and the risk being created — is assembled dynamically, in motion, often ephemerally. By the time a traditional control has something to inspect, the meaningful moment has already passed.

The failure modes this creates are subtle precisely because nothing looks obviously wrong. A document may be owned by the correct user, stored in the correct repository, accessed with valid permissions. 

From a metadata perspective, everything is normal. But the generated output may reference the wrong customer, combine insights from multiple accounts, or expose internal reasoning that was never meant to leave the organization. 

The sensitivity controls did their job. The contextual risk was invisible to them. This points to what I think is the core shift security teams need to internalize: the problem is no longer just identifying sensitive data and controlling its movement.

The more important question is whether the meaning of information remains correct as it's reconstructed across AI-driven workflows. That requires understanding which entities appear in the content, what relationships those entities have, and whether a specific use aligns with those relationships in that context.

The same piece of information can be entirely appropriate in one interaction and a serious violation in another. Static classification can't make that distinction. Metadata about the file can't make that distinction. Only semantic context — the "who" embedded in the content itself — can.

Security architectures weren't designed to reason at that level. They were designed to answer whether access is allowed. They were not designed to govern whether the output that results from that access remains within policy intent. 

As AI moves from isolated use to deeply embedded, connected workflows, that gap becomes the defining exposure. Protecting data at rest and in transit is still necessary. It is no longer sufficient.

Vishwa: You've argued that detection without accuracy is insufficient — what ideas in your current work felt ahead of the curve?

Gidi: The one I'd point to most directly is the argument that detection without accuracy isn't actually security — it's observation. That distinction sounds obvious when you say it out loud, but the industry spent a long time not saying it out loud.

For years, there was an unspoken acceptance that data security controls were inherently imprecise. Everyone knew it. Detection systems generated noise, policies were blunt, and context was missing. 

But because the industry operated under the same constraints collectively, inaccuracy stopped being treated as a problem to solve. It became part of the landscape — something you managed rather than fixed. 

Large enterprises would deploy powerful platforms and then deliberately leave enforcement turned off, not because they didn't need it, but because they didn't trust it enough to let it touch real workflows.

What I kept arguing was that this compromise only held because enforcement was optional. Humans were still in the loop. Mistakes could be caught and corrected. You could afford ambiguity when the pace was human. AI removes that tolerance entirely. 

• In agent-driven workflows, generation is instantaneous, 
• Automation amplifies mistakes, and 
• There is no natural pause for manual review.
  • If a control isn't applied in-line with sufficient confidence, it effectively isn't applied at all. 

That shifts the entire frame: the question is no longer how much you can detect, but what you can enforce reliably without breaking the business.

The second idea that met resistance was what I call the "who" problem — the argument that entity context, not just content classification, is the foundational requirement for real data governance. 

The industry built extraordinarily capable tools around one question: what type of data is this? 

Pattern matching, classification engines, regex — all optimized around the "what." My argument was that this was the wrong primary question. Most privacy regulations and customer trust expectations are not written around abstract data types. 

They are written around specific people — customers, policyholders, patients. The obligation attaches to a person, not a pattern. And the same piece of data can be entirely appropriate in one interaction and a serious violation in another, with the difference coming down entirely to who it refers to, who is sending it, and who is receiving it.

That was a harder sell when the world was mostly files moving between humans. AI made it undeniable. 

When copilots and agents are continuously assembling and generating content across systems at machine speed, you can correctly identify every sensitive pattern in a piece of output and still make the completely wrong decision about whether it should exist, because you don't know whose data it is, whether the right customer is referenced, or whether the entity relationships in the generated content are even correct.

The through-line between both ideas is the same: data security has been operating as an observation discipline when the actual requirement is control. Detection tells you what might be happening. 

Control requires knowing what is happening, in this specific context, with enough confidence to act on it — instantly, at scale, without constant human intervention. That is a materially higher bar. And it is the only bar that matters.

Vishwa: What assumptions about trust are AI systems quietly breaking?

Gidi: The deepest assumption AI is breaking is one most organizations never had to make explicit: that approved means governed.

For years, security models treated trust and control as largely traveling together. 

• If a platform was sanctioned, 
• If access was properly configured, 
• If the right people had the right permissions — governed outcomes were presumed to follow.
  • Approval was the proxy for safety. 

That assumption was never perfect, but in a world of human-paced workflows and discrete data objects, it was workable enough that nobody was forced to confront it directly.

An organization can deploy a fully sanctioned platform, configure access correctly, and run workflows that look entirely legitimate — and still produce outcomes that violate policy, mix customer data across relationships, or expose internal reasoning that was never meant to leave the organization. 

• Nothing unauthorized occurred. 
• No obvious rule was broken. 

The system did exactly what it was designed to do. And yet the outcome is not what policy intended.

The second broken assumption is that access controls and data controls are the same problem. They are not. Access controls govern whether a user or system can reach data. They say nothing about what happens once reasoning and automation begin operating on that data at scale. 

• An AI system may retrieve information it is entirely permitted to access, 
• combine it with other permitted context, and 
• generate an output that crosses a line no individual access decision was ever designed to prevent. 
• The permission model looks clean throughout. The outcome is still wrong.
  • This is where the "who" dimension becomes critical. 

Traditional controls were built to answer what the data is — identifying patterns, classifying content, detecting sensitivity. But the actual trust obligation most organizations carry, toward customers, regulators, and counterparties, is not about data types. It is about people. 

A customer sharing information with an organization is placing trust that it will be used for a specific purpose, shared only with appropriate parties, and not combined in ways they didn't anticipate. 

That expectation doesn't attach to a pattern. It attaches to a relationship. And AI systems, optimizing for relevance rather than appropriateness, do not inherently understand the difference.

The third assumption is that the user is the meaningful unit of accountability. Security architectures were built around a tight coupling: 

• a user acts, 
• a control observes, 
• enforcement follows. 

AI agents break that coupling structurally. A single prompt can trigger multi-step execution across retrieval systems, orchestration layers, and downstream services the user never directly touches. 

The user initiates — but the work executes somewhere else entirely. Attribution becomes less reliable, inspection points miss meaningful activity, and the controls that were positioned around the user are simply in the wrong place to catch what matters.

Taken together, these aren't edge cases or theoretical risks. They are structural properties of how AI systems work. And they will not be addressed by hardening existing controls or sanctioning platforms more carefully. 

The trust assumptions being broken are baked into the architecture of how data security has been practiced — detection over enforcement, classification over context, access over governance. AI didn't introduce those gaps. It just made them impossible to ignore.

Vishwa: Do you think the current pace of enterprise AI adoption is sustainable from a security perspective?

Gidi: The honest answer is that sustainability depends on whether organizations are willing to confront a gap they have mostly been avoiding. The pace of adoption itself isn't the core problem. 

AI is being embedded into enterprise workflows because it delivers real value — that pressure isn't going away, and it shouldn't. The sustainability question is whether security architecture is evolving fast enough to govern what's actually being deployed. Right now, in most organizations, it isn't.

What's being exposed is a structural mismatch that was always latent but is now impossible to ignore. Enterprise data security was built around a clear model: protect where data resides, control where it moves, observe what users do. 

The controls — endpoints, DLP, CASB, access permissions — were broadly aligned with where meaningful data activity occurred. Connected AI moves part of that execution surface into AI reasoning environments, connector pipelines, and agent orchestration layers that sit outside those traditional control assumptions. 

The enterprise still owns the data. It increasingly doesn't own the execution path through which that data is used. That's the architectural break.

The deeper problem is that most organizations haven't separated two questions that AI forces apart: 

• Whether a user is allowed to access information, and 
• Whether an AI system should be permitted to use that information in a specific context for a specific purpose.
  • Those are not the same question. 

What makes this unsustainable isn't adoption speed per se. It's the compounding effect of deploying systems at scale before the governance layer catches up. In human-paced workflows, 

• inaccuracy and incomplete controls were manageable — humans were in the loop,
• mistakes could be caught and corrected, 
• enforcement could be deferred. 

AI removes that tolerance. Generation is instantaneous. Automation amplifies mistakes. Agents act across systems without natural checkpoints. If a control isn't applied in-line with sufficient precision, it effectively isn't applied at all. The margin for error shrinks exactly as the scale of exposure grows.

There's also a false comfort embedded in the current moment. Many organizations assume that moving from unsanctioned AI tools to approved enterprise platforms resolves the hard governance questions. It doesn't. 

Sanctioned AI can still 

• produce outcomes that violate policy, 
• mix customer context across relationships, or 
• expose internal reasoning in ways that erode trust — without any obviously unauthorized act occurring. 

Approval is not governance. It never was, but AI makes that distinction consequential in ways it wasn't before.

None of this means adoption should slow. But it does mean the industry needs to be honest about what it's actually deploying. 

The next phase of AI security won't be defined by better detection of shadow usage. It will be defined by whether organizations can govern how approved AI systems use data — at the moment of generation, at the scale of automation, with enough contextual understanding to distinguish between what's permitted and what's appropriate. 

That capability doesn't exist at scale yet. Closing that gap, while adoption continues accelerating, is the real sustainability question.

Vishwa: Your background spans military intelligence, enterprise software, SaaS, and AI governance. Which environment taught you the most about how humans actually behave around sensitive information and trust?

Gidi: The honest answer is: every one of these environments had its own meaningful contribution. 

In the military intelligence, you learn very quickly that policies and classifications describe how information should be handled, and the risk associated with leakage and mishandling of classified information. 

Enterprise software and SaaS then showed me similar patterns, but with less explicit structure. In a big company, almost everyone thinks they are the exception: 

• “I’ll just export this data to a spreadsheet.” 
• “I’ll send this file to my personal email.” 
• “I’ll reuse this report for another customer.”
  • The intent is almost always positive—move faster, help a client, hit a deadline
    • But the net effect is a slow erosion of the original trust boundaries

AI governance is where all of that comes together and accelerates. Now you have systems that are incredibly helpful, incredibly persuasive, and very easy to trust—while also being opaque and probabilistic. 

Humans are still humans; they will overshare with tools that help them, and they will assume “if it’s inside the company, it must be safe.” 

What I took from earlier environments is that you cannot rely on policy or training alone. You have to design for how people actually behave around sensitive information and trust, not how you wish they would.

Vishwa: If you look five years ahead, what concerns you more: attackers using AI more effectively, or organizations deploying AI faster than they can meaningfully govern it?

Gidi: The second one. And it's not particularly close.

Attackers using AI more effectively is a real and serious problem — but it's a problem the security industry is oriented to fight. It's adversarial, it's visible, it triggers investment and response. 

The industry has spent decades building muscle around external threats. When the threat evolves, the defense eventually evolves with it. That race is uncomfortable, but it's a known race.

The governance gap is different in kind. It doesn't announce itself. It accumulates inside systems organizations already trust, in workflows they've already approved, through access they've already granted. 

There's no alert, no incident, no obvious moment where something went wrong. Just a slow widening of the distance between what policy intended and what AI is actually doing — at scale, at machine speed, across every corner of the enterprise.

What concerns me is the compounding effect. Right now, most organizations are making a category error: treating approval as governance. 

Once a platform is sanctioned, the hard questions are presumed answered. But approval governs access. It doesn't govern use.

It doesn't

• determine whether an AI system should combine customer context across relationships,
• surface internal reasoning in external outputs, or
• take agent-driven actions that are technically permitted but outside what policy was designed to allow. 

Those are different questions — and most enterprises don't yet have the architecture to answer them.

Five years of deployment at the current pace, without closing that gap, means the exposure isn't theoretical. It's structural and embedded. 

Millions of AI interactions will have occurred across customer data, regulated information, and sensitive business context — governed by controls designed for a world where humans were the primary actors and data moved in discrete, observable objects. 

That mismatch doesn't create one large visible breach. It creates thousands of subtle, contextual violations that are nearly impossible to unwind.

The deeper issue is that the traditional security model was built around a tolerance for inaccuracy that AI-driven workflows simply don't support. Detection could be imprecise because humans were in the loop and enforcement could be deferred. 

In agentic workflows there is no pause, no checkpoint, no moment to catch and correct. If governance isn't precise enough to operate in-line, at the moment of generation, it isn't operating at all.

I'm not saying the attacker problem is solved — it isn't. But the attacker problem is visible, it's adversarial, and it mobilizes organizations to respond. 

The governance problem is invisible, it's internal, and it gets deferred precisely because the systems creating it are the ones delivering the most business value. 

That asymmetry is what makes it the harder problem. And the more consequential one over the next five years.