Deployment and Recovery at Par: Scaling Delivery Pipelines With Clear Ownership Without Burning Out Engineers
Published
Key Takeaways
- Durkin believes that automation only scales if it’s auditable.
- Senior engineers often are the only people able to connect changes across systems during incidents.
- Harness advocates for teams to balance speed and governance by embedding policies directly into pipelines.
- Shared templates and guardrails help organizations maintain delivery speed without creating fragmented pipelines.
- There is a need to check whether incidents require large group chats to reconstruct changes, and senior engineers remain at escalation points.
Nick Durkin, Field CTO at Harness, details how delivery pipelines are becoming harder to understand as systems scale and automation expands in engineering. Fast deployment means little without fast recovery.
An active investor and advisor, backing technology startups and mentoring founders building data, cloud, and AI-driven platforms, Durkin says the real operational pressure is not just deployment speed but the cognitive load placed on engineers who must trace what changed across pipelines, services, environments, and policies during incidents.
With organizations adopting AI-assisted development, codes are moving faster, and making ownership harder to track. He argues that automation should reduce toil, and not understanding. When engineers cannot trace why the pipeline made a decision, they start working around the system rather than trusting it.
Durkin also notes that many teams optimize for deployment speed but underestimate recovery readiness. During incidents, rollback paths often exist but depend on manual steps or the one engineer who remembers how the system works.
Vishwa: How has the growing complexity of delivery pipelines affected the cognitive load on senior engineers and on-call teams?
Nick: As organizations scale, delivery pipelines tend to multiply. Over time, the system starts to feel like a library without a card catalog. The information exists, but finding the right piece at the right moment depends on who happens to remember it. And that’s where cognitive load spikes.
Senior engineers get pulled into incidents because they’re the only ones who can connect the dots across services, pipelines, environments, and policies. On-call turns into rebuilding the story of what changed while the incident is still unfolding.
Change is also coming faster now. AI-assisted development is pushing more code through the system, which raises the bar on change hygiene and makes ownership easier to lose. In real incidents, the earliest clues still come from humans – someone remembers a feature flag flip or heard a customer report.
That context is valuable, but it usually lives in chat threads and incident bridges, disconnected from the change history. We kept hearing this from customers, and it’s influenced where we’re investing across AI SRE and automation.
Vishwa: How should organizations think about training engineers as delivery decisions become more abstracted through automation and tooling?
Nick: Automation should reduce toil, not understanding. I’ve watched teams go from “we’re finally shipping faster” to “why did the pipeline do that” in about a week, and while the tooling got smarter, the shared understanding didn’t keep up. If teams can’t trace why the system made a decision, they work around it.
Training has to shift away from tool operation and toward system literacy. Engineers need to understand how the pipeline is wired, what the automation is optimizing for, how policies get enforced, and how signals from testing, security, cost, and production fit together. That’s also where critical thinking starts to matter more than ever, because the job becomes asking better questions about risk and customer impact, not just chasing a green checkmark.
The strongest teams are doing three things consistently:
- Teaching engineers how the guardrails work and why they exist
- Making policies visible and explainable inside the workflow
- Upskilling around system-level and business-level thinking, not just code
We hear this constantly from customers, and it drives where we invest. Automation only scales if it’s auditable.
Vishwa: Are you seeing mature teams become more selective about what they automate, and what tends to drive those decisions?
Nick: Most organizations I talk to fall into one of two categories. One group hesitates because they’re worried about risk and trust. The other group goes all-in and tries to automate everything at once. Neither works well.
When a team gets more selective, it’s a sign they’ve learned where automation actually pays off. Every automation becomes something you have to maintain under pressure, so they start with repeatable, high-volume work where outcomes are easy to validate and audit, like policy checks and standard pipeline paths. They stay hands-on where context matters, like risk calls and customer impact decisions.
Teams tend to ask three questions.
- Does this happen all the time?
- Is it painful when it fails?
- And is it predictable enough to standardize?
- When the answer is yes across the board, automation makes sense.
The goal is simple: remove the worst parts of people’s jobs, not the best.
Vishwa: What delivery practices tend to work better in regulated environments without turning pipelines into approval bottlenecks?
Nick: In regulated environments, manual approvals feel safe at first. At scale, they turn into bottlenecks.
The teams that scale well treat governance like a shared rulebook. Security, compliance, and engineering agree on what “good” looks like, then build it directly into the pipeline so enforcement is consistent and automatic.
That’s how you make it easy to do the right thing and hard to do the wrong thing. If a change meets policy, it moves. If it doesn’t, the engineer gets fast, actionable feedback. Regulation doesn’t have to mean friction when the rules are clear, automated, and shared.
Vishwa: What delivery practices look effective early on but tend to break down as organizations scale?
Nick: Early on, letting every team build whatever works feels fast and empowering. As the organization grows, that turns into inconsistency and confusion.
I like to use an airport analogy for this. You wouldn’t build a runway for every aircraft, or let every airline invent its own procedures and safety rules. You standardize the runway and the operating model so people can focus on flying.
In software, we often do the opposite by letting every application team build a bespoke pipeline, and it works until scale makes that complexity unmanageable.
The fix is straightforward:
- shared templates and guardrails,
- enforced automatically,
- with a clear path for the real exceptions.
That keeps speed without the chaos and gives engineers time back for work that moves the needle.
Vishwa: Many teams optimize for deployment speed. Where do you still see gaps when it comes to recovery and rollback?
Nick: A lot of teams are really good at getting changes out, but they’re less prepared for the moment something goes wrong and everyone asks, “Can we undo this quickly and safely?” That’s when MTTR and blast radius start to matter more than deploy frequency.
Rollback often exists, but not as a routine action. When an incident hits, the rollback path can be unclear, manual, or depend on the one person who remembers how it works. In modern systems, there's rarely a clean “go back to the last version” button, especially once multiple services and data changes are involved.
Many of our customers have made rollback part of how they ship every day. They use flags and gradual rollouts to reduce impact quickly, and they make it obvious when things are healthy again.
For example, Harness customer Ancestry.com saw a 50% decrease in overall production outages and significantly reduced outages related to deployment issues. If you can deploy fast but you can’t recover fast, it doesn’t feel like speed when it matters.
Vishwa: How do teams assess whether their delivery systems are actually making day-to-day work more sustainable for engineers?
Nick: You can start with the usual metrics like lead time, deploy frequency, and change failure rate, because they tell you if the machine is moving. But they don’t tell you what it feels like to work inside the machine.
So I look for the small signals. Are engineers still babysitting pipelines? Are incidents turning into long group chats because no one can tell what changed? Are senior engineers constantly pulled in because the system only makes sense to them?
The numbers help, but you still have to ask people how it’s going.
- Ask where the friction still lives,
- what still feels manual, and
- what keeps pulling people out of flow.
- Over time, you should see fewer escalations and calmer on-call rotations.
If engineers are getting their nights and weekends back, you’re heading in the right direction.
Related
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular
Get expert insights on threats, breaches, scams, and security trends — delivered every Monday.
Most Popular