Why Security Teams Should Expect Failure Before They Expect Prevention
Published
Key Takeaways
- Carignan makes decisions with the expectation that systems will fail and unknown risks already exist.
- Executives should understand that generative AI, anomaly detection, behavior analytics, and agentic AI are not the same.
- Darktrace observes that AI can increase workload when analysts constantly validate unclear or poorly integrated outputs.
- Government and intelligence environments assume prevention will never be perfect and continuously test defenses.
- Resilience comes from adapting operations over time rather than treating security as a fixed state.
Nicole Carignan, Senior Vice President, Security and AI Strategy, Field CISO at Darktrace, discusses what nearly three decades of operational, intelligence, and cybersecurity experience have taught her about risk, resilience, and security decision-making.
Working close to real adversaries and real incidents changes how you think about control and prevention. Carignan explains that experience teaches security leaders to assume systems will fail, unknowns already exist, and adversaries will exploit gaps that have not yet been identified.
She also explains why responsible AI must be operational rather than aspirational, why trust depends on transparency and explainability during incidents, and why executives need to understand that different types of AI bring different strengths, limitations, and risks.
Carignan discusses why some of the most significant breaches have occurred inside organizations with extensive security investments, and why resilience ultimately depends on continuously adapting how security operates.
Read on to learn why Carignan believes risk is not something to eliminate, but something organizations must understand well enough to act on with speed and confidence when it inevitably materializes.
Vishwa: Your journey seems deeply operational and intelligence-driven. How has that shaped the way you approach risk and decision-making?
Nicole: My career has been deeply operational and intelligence‑driven because it has always been mission‑oriented. Starting at NASA and then moving into the intelligence community, I entered cybersecurity at its infancy and have spent nearly three decades operating through successive waves of technological change.
That environment required constant learning; new domains, new technologies, and, critically, an understanding of the underlying principles beneath them. That depth of understanding is what allows risk to be assessed realistically, not abstractly.
Because most of my career has been rooted in operations, my approach to risk is grounded in experience rather than theory. Working close to real adversaries and real incidents changes how you think about control and prevention.
You stop assuming systems are fail‑safe, that policies will hold under pressure, or that threats will follow predictable paths. I make decisions with the expectation that systems will fail, that unknowns already exist, and that adversaries will exploit the gaps we haven’t identified yet.
That mindset drives a bias toward visibility, behavioral understanding, and resilience over static rules or checkbox compliance. It also shapes how I prioritize decisions: preserving optionality matters. Detect early, contain quickly, and adapt continuously.
Rather than betting on a single point of control, I focus on defense in depth as well as building the ability to respond decisively as conditions change. Risk isn’t something to eliminate—it’s something to understand well enough to act with speed and confidence when it inevitably materializes.
Risk mitigation or reduction really comes down to layered defenses that protect against a variety of threats, but that are well integrated to optimize intelligence gathering, analysis, containment, and the SOC workflow.
Vishwa: “Field CISO” can mean very different things depending on the company. In practical terms, what does your day-to-day role actually involve at Darktrace, and where do you spend most of your time?
Nicole: In practical terms, my role sits at the intersection of customers, technology, strategy, and the market. A lot of my time is spent speaking with CISOs, CIOs, security teams, boards, and industry stakeholders about how the threat landscape is changing, how AI is reshaping both offense and defense, and what that means operationally.
That can include helping organizations think through AI adoption, cyber resilience, security operations, risk reduction, governance, and how to safely deploy autonomous or AI-driven capabilities.
I also spend a lot of time translating between technical and executive audiences. Security teams want to know how something works, where it fits, and what risks it introduces. Executives want to understand business impact, governance, accountability, and whether the organization can trust the technology under pressure.
It is about bringing field insight back into strategy and helping customers connect emerging technology to real-world security outcomes. In rapid innovation periods like this, organizations are seeking guidance, technology understanding, risk and safety considerations, architecture, and well-integrated technology and security stacks.
Vishwa: Which indicators should security teams watch most closely as AI tools and agents become embedded across users, applications, and workflows?
Nicole: As organizations adopt AI and agentic systems, CISOs need to look beyond traditional security metrics and ask whether they actually have visibility and control over how AI is being used.
The first indicators I would look at are:
- what AI tools and agents are active in the environment,
- who or what has access to them,
- what data they can reach, and
- whether their behavior is staying within expected boundaries.
For agentic AI especially, access and behavior matter more than simple adoption metrics. These systems can act across workflows, connect to APIs, move data, and make decisions with varying degrees of autonomy.
So leaders should be measuring whether agents are over-permissioned, whether sensitive data is being exposed, whether API and communication boundaries are enforced, and whether there is anomalous behavior that could indicate misuse, prompt injection, or risky drift.
The most important metrics are the ones that show whether the organization can detect and contain risks of AI usage, misuse, or abuse in real time. That includes visibility into shadow AI usage, data exposure, identity and access risk, model or agent behavior, and time to investigate and contain suspicious activity.
AI governance can’t be a static policy exercise. It has to be tied to operational signals that tell security leaders whether AI is being used safely, responsibly, and within the organization’s risk tolerance.
Vishwa: What does “responsible AI” look like operationally, especially when teams are expected to trust automated systems during incidents?
Nicole: Responsible AI has to be operational, not just aspirational. In a security context, that
- means transparency,
- explainability,
- testing,
- evaluation,
- validation,
- verification, and
- clear controls over when AI recommends action versus when it acts autonomously.
During an incident, trust is not built by asking teams to accept a black-box decision. Trust is built when analysts can understand why the system reached a conclusion, what evidence it used, how confident it is, and what the potential impact of an action will be. That is especially important when AI is being used for response or containment.
Responsible AI also starts much earlier, with data. Organizations need to understand how data is sourced, classified, governed, secured, and protected throughout the AI lifecycle. If the data foundation is weak, no governance framework will be enough. AI systems are only as reliable as the data and controls behind them.
Vishwa: Based on your experience, what are the misconceptions executives might have about AI-powered cybersecurity tools?
Nicole: One misconception is that all AI is the same. It is not.
- Generative AI,
- supervised learning,
- unsupervised learning,
- behavior analytics,
- anomaly detection,
- neural networks,
- graph neural networks, and agentic systems all behave differently and carry different strengths, limitations, and risks.
Executives need to understand what kind of AI is being used, what it is designed to do, and where human oversight is still required.
Another misconception is that AI automatically reduces headcount or replaces security expertise. In reality, AI changes the work.
It can automate repetitive tasks, accelerate triage, and support autonomous investigation, but it also creates new requirements around
- data governance,
- model behavior,
- system architecture,
- technology and security stack integrations,
- testing and validation, and
- oversight.
- Security practitioners remain critical.
The third misconception is that AI security can be solved with static guardrails alone. Guardrails, access controls, and least privilege are necessary, but they are not sufficient.
Organizations also need visibility into how AI systems and agents behave in practice, especially when they have access to data, workflows, identities, and business systems.
Organizations should learn from the very hard lessons with rapid cloud adoption. Detection and response is a critical component to securing AI assets.
Orchestration, the control plane, and security posture management are all foundational, but behavioral-based advanced threat detection will mitigate the risk of adopting AI.
Vishwa: Having supported both government and commercial environments, what differences stand out most in how each sector approaches cyber resilience and incident response?
Nicole: The most striking difference is how each sector internalizes failure and uncertainty. In government and intelligence settings, you operate with the assumption that
- prevention is never perfect,
- controls will be stressed, and
- adversaries are already exploiting gaps you haven’t identified yet.
- That expectation fundamentally shapes cyber resilience and incident response.
- Security is treated as a living system that is continuously reviewed, tested, and refined across technology, people, and processes.
- The work is never finished, and it is always evolving toward stronger defense‑in‑depth and more effective risk mitigation.
In contrast, many commercial organizations place too much confidence in
- the tools they have purchased,
- assuming those investments alone will protect them.
Some of the most significant past breaches did not occur in under‑resourced environments, but in organizations with substantial security stacks that were poorly deployed, insufficiently integrated, or never rigorously tested.
In many cases, teams believed they had visibility and response capabilities that, under real conditions, simply didn’t exist.
Others failed to evolve their SOC operating models to absorb and act on new intelligence signals, leaving advanced threats undetected despite having the data in front of them.
This gap becomes even more pronounced as organizations adopt AI into their security operations. AI requires a shift away from deterministic, rules‑based classification toward probabilistic, behavioral‑based intelligence.
That transition is not just a tooling change, but it demands a rethinking of SOC workflows and a retraining of analysts to interpret, trust, and operationalize anomaly‑driven signals.
When used effectively, behavioral intelligence is critical for
- detecting novel threats,
- advanced persistent actors,
- insider risk,
- credential compromise,
- supply‑chain intrusions, and
- the misuse or compromise of AI systems themselves.
Resilience, in both sectors, ultimately comes down to whether organizations are willing to adapt how they operate and consider security not as a static state, but a dynamic organization and investment that must continue to improve and get better.
Vishwa: What’s one operational habit you picked up from government/intelligence work that commercial security teams should also adopt?
Nicole: Think like a criminal. As you are learning new technology, a new domain, a new product, think like an adversary.
- How would I exploit this?
- How could a user abuse this?
- What is the worst thing this technology could accomplish?
- This is where security and safety starts.
Domain subject matter expertise is critical to securing technology and mitigating the risk of its adoption. Think creatively, because you know the adversaries are.
Vishwa: After more than two decades in cybersecurity and intelligence-related work, what skills or instincts have remained consistently valuable despite all the technological shifts?
Nicole: Curiosity has been the most consistent one. Technology changes constantly, but the ability to stay curious, keep learning, and ask better questions never stops mattering.
The second is communication. Technical depth is critical, but if you cannot explain risk clearly to a board, a customer, a peer, or a junior analyst, you will struggle to drive action. Cybersecurity is a team sport, and diverse perspectives make teams stronger.
Creativity, empathy, and collaboration are not soft skills in this field; they are operational advantages. Everyone has blind spots and biases, so fill your team with diverse perspectives that close those blind spots and minimize biases.
Vishwa: Do you think AI is reducing analyst burnout, or are we mostly shifting the workload around
Nicole: It depends entirely on how AI is implemented. When applied responsibly, AI can reduce burnout by taking on repetitive, high-volume tasks like alert triage, initial investigation, correlation, and prioritization.
That gives analysts more time for higher-value work such as
- threat hunting,
- resilience planning,
- testing and validation,
- incident response, and
- strategic risk reduction.
But AI can also shift or even worsen the workload if it is not explainable, accurate, or well integrated into the SOC. If analysts have to validate every AI-generated finding without understanding how the system reached its conclusion, that is not reducing burnout. That is creating a new burden.
- The goal should not be automation for its own sake.
- The goal should be better human-AI collaboration:
- systems that improve speed and scale while giving analysts the
- context,
- confidence, and
- control they need to make better decisions.
- systems that improve speed and scale while giving analysts the
Vishwa: What kinds of cyber-attacks do you think defenders still systematically fail to model correctly?
Nicole: Defenders still tend to underestimate attacks that use legitimate access and normal-looking behavior. Identity abuse, credential theft, insider risk, supply chain compromise, agent misuse, SaaS abuse, and living-off-the-land techniques can all look legitimate to traditional controls because the account, tool, or workflow may actually be authorized.
That problem becomes even more complex with AI agents and non-human identities.
- Agents can have employee-like reach:
- they can access systems,
- move data,
- execute workflows, and
- interact across domains.
If an attacker compromises or manipulates that agent, the activity may not look malicious in a traditional sense. It may look like the agent doing what it was allowed to do.
Related
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular