The Same Traits That Make AI More Human Also Make It Easier to Socially Engineer
Published
Key Takeaways
- Humanix can evaluate whether the caller was verified and whether the agent is being pressured to shortcut the process.
- Stewart believes stopping social engineering means treating it like every other major attack class.
- In help desk environments, the context behind a conversation often matters as much as the conversation itself.
- Deepfakes are less about Hollywood-level deception and more about creating urgency around a convincing pretext.
- Help desk requests may appear routine until semantic context reveals missed procedures or policy conflicts.
Keith Stewart, CEO and Founder of Humanix, believes the industry's effort to make AI more human and helpful has also increased its vulnerability to social engineering. With a career that includes security, networking, product, and executive leadership roles across Cisco, Brocade, Riverbed, and vArmour, he brings a long-term perspective on how attackers adapt to new technologies.
Stewart argues that AI-generated trust manipulation is more dangerous than AI-generated phishing. While AI can make phishing campaigns more convincing, businesses ultimately run on trust between employees, customers, and suppliers, making that trust itself the target.
Most social engineering attacks have nothing to do with deepfakes, often succeeding through simple phone-based manipulation. Groups like Scattered Spider and Shiny Hunters show that simple phone calls remain highly effective attack vectors.
Semantic context helps distinguish routine help desk requests from interactions designed to pressure users into bypassing established safeguards.
The rapid adoption of agentic AI will transform all aspects of a company's relationships with its customers, employees, and suppliers. More insights from Stewart in the interview below.
Vishwa: You’ve repeatedly worked through transition moments: SD-WAN, SaaS, cloud-native security, and now human-focused defense. How do you identify which shifts are truly foundational?
Keith: Major shifts always involve some unmet need that people have. They may initially struggle to put their problem into words, but the need itself is clear if you dig into it with them. The problem ‘makes sense’.
For a shift to be really impactful, the solution itself should have the same attribute of ‘making sense’. When you say the solution out loud to someone with the problem, it should sound kind of obvious.
That implies a rich enough understanding of the problem to be able to visualize a simple, clear solution. When someone hears it, it ‘makes sense’. The transition to human-focused defense has both of these attributes.
On the clear, unmet need, one only has to look at the marquee breaches of the last few years. Virtually all of the large loss events came as a direct result of one person manipulating another into doing something they shouldn’t.
As cyber people, we say things like:
- “Humans are the weakest link”, and
- “We know training doesn’t really solve the problem, but what else can we do?”
Everyone agrees with those statements because there’s a common underlying problem they share.
So how do we solve it? The same way we do every other cyber problem - we should detect these attacks when they happen, and respond to protect our organizations. AI and large language models let us treat this as a detection and response problem, not as a training problem.
Said even more simply, ‘we should use AI to detect social engineering’. Makes sense.
Vishwa: When you say “detect and respond to attacks on humans,” what does that operationally mean in practice?
Keith: Stopping social engineering means treating social engineering like every other major attack class. We should deploy active detection and response, not just policy and awareness training.
Just as EDR and NDR emerged when endpoint and network attacks outgrew perimeter controls alone, human-targeted attacks also require technologies that can detect attacks in natural language and help people respond in the moment.
Operationally, Humanix adds observability and attack detection to the voice, chat, email, and ticketing systems people use to interact. We look at the live interaction and natural language, the workflow, and the required procedure before a sensitive action is completed.
In a help desk call, for example, we can evaluate whether the caller was verified, whether the agent is being pressured to shortcut the process, and whether the change would affect access.
When an interaction looks like an attack, we can alert the security team, helping them see a class of attacks that they’re currently blind to.
Vishwa: What signals indicate an organization is vulnerable to social engineering even if its technical controls look mature?
Keith: The clearest signal that an organization is vulnerable to social engineering is that it has a help desk or service desk with a defense strategy that still relies on training to stop social engineering attacks.
If a help desk or service desk agent can reset a credential, enroll a new MFA factor, or grant an exception, and the main control is expecting that agent to spot the attack, the organization is vulnerable.
A mistake can be to think that this is a ‘big company’ problem. As we’ve clearly seen in recent incidents, attackers are having lots of success in mid-sized companies as well as large.
For every Fortune 500 attacked, there are dozens of law firms, local banks, health care providers, and government offices who’ve been hit. That said, larger companies do have some additional challenges here.
The overall exposure to the social engineering threat grows in:
- distributed workforces,
- high-volume support environments,
- outsourced help desks & service desks, and
- companies with frequent onboarding and offboarding.
Retail, hospitality, healthcare, and financial services often have several of those dynamics at once. This results in a large, complex attack surface that requires more focus and attention.
Vishwa: You’ve worked with graph analytics and behavioral models. Do you think relationship mapping becomes critical for detecting social engineering and trust abuse?
Keith: Yes. Social engineering is about exploiting trust. Trust is an attribute of the relationships between individuals and the context of that relationship.
Analyzing the progression of an interaction against the context of its relationship is a key mechanism to detect social engineering.
As an example, a given help desk interaction involves a set of steps, and a set of procedures and policies that govern those steps. By understanding the context of the relationship (often by extracting semantically meaningful context from the interaction as it happens), we can assess whether the interaction is progressing in conformance with those procedures, or in conflict with them.
Did the agent actually verify the caller’s identity correctly, or were they misled or tricked into thinking they had when really they had not? It’s these kinds of signals that become available when you understand the context of the relationship.
Vishwa: Are organizations too focused on deepfakes while missing lower-tech but more scalable manipulation tactics?
Keith: Yes. The overwhelming majority of social engineering attacks have nothing to do with deep fakes. It’s groups like Scattered Spider or Shiny Hunters picking up the phone and calling people.
I worry that the conversation about deepfakes is distracting because it makes the threat feel exotic. But a deepfake is just another medium for the same manipulation we’ve long seen in social engineering attacks — most of which do not require Hollywood-level deception to succeed.
Success comes from urgency, the right pretext, and a workflow where a person can be pressured into an unsafe action that benefits the attacker. Detecting those signs of manipulation, regardless of medium, should be the defensive priority.
A real voice is as dangerous as a cloned voice. The important question is not whether the interaction is synthetic. It is whether someone is being pressured, impersonated, or guided around a required safeguard.
Organizations should stop treating deepfakes as a special case and focus on the manipulation patterns that make these attacks work.
Vishwa: What worries you more: AI-powered phishing or AI-generated internal trust manipulation?
Keith: AI-generated internal trust manipulation is a far more dangerous problem than simple AI-generated phishing. On phishing, AI certainly helps attackers eliminate simplistic faults in phishing emails like spelling errors, or silly stories about Nigerian Princelings.
But the email security companies do a pretty good job of spotting the suspicious email with the suspicious link anyway. AI can increase the volume and noise level, but not the fundamental nature of the vector.
On the other hand, businesses run on trust. Trust between employees, customers, and suppliers. That trust is essential for businesses to efficiently operate and communicate. It’s how business gets done.
That’s why there’s so much risk here. The most sensitive parts of a business take place over these relationships and interactions. That makes those interactions a prime target for attackers.
Exploiting those vulnerabilities effectively tends to require industry knowledge and environment-specific context - two major barriers to attackers. In a world of AI, those barriers are lowered or eliminated, making manipulation of those processes much more accessible.
When you couple that with the historic opacity (to the security team) of most of those interactions, you’ve got a dangerous recipe.
Vishwa: What does future cybersecurity look like if attackers continue targeting humans instead of infrastructure?
Keith: The rapid adoption of agentic AI will transform all aspects of a company’s relationships with its customers, employees, and suppliers. Today, those relationships are between humans. The social engineering threat of today targets humans to exploit those relationships.
The same problem gets far worse as companies adopt AI agents. The business services attackers already target – customer support, help desk, service desk, finance operations – are moving from human-delivered to AI-delivered. This will increase a company’s risk profile dramatically.
We have spent billions training the AI models that agents are built on to be more human and more helpful. From an AI product standpoint, that’s an essential element of a ‘good’ model. From a security standpoint, it looks a lot like a conscious effort to increase the agent’s vulnerability to social engineering.
The transition to Agentic AI will create a much larger natural language attack surface, but one where we’ve removed all the human skepticism and suspicion that have classically been our best defenses against social engineering. If one agent can talk another agent into revealing a secret or routing around a safeguard, that is the same social engineering pattern in a new form. It is likely to get worse before it gets better.
So, the future of cybersecurity is not just protecting infrastructure. It is protecting the conversations and workflows that cause infrastructure to change – where trust is established, authority is asserted, and a human or agent is persuaded to take an action.
Related
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular