That was the title of the talk. I co-presented it with Swapnil Jadhav at our Grainger Tech Conference, and the whole thing came out of one idea: most people have Ansible filed under the wrong category.

Ask a developer what Ansible is and you’ll get some version of the same answer. It configures servers. Inventory, SSH, packages, files, services. That’s not wrong. It’s how Ansible got popular in the first place. But it’s about a decade out of date, and it stops people from reaching for something that’s already sitting right there.

That’s the part I actually cared about. We already have Ansible Automation Platform. We already have Event-Driven Ansible running. Nobody has to build it, buy it, or get it approved. It’s available today, and hardly anyone outside the infrastructure org is using it.

I’m honestly not sure why. It could be that people just don’t know it exists. Internal platform capabilities have a way of staying invisible to the teams that would benefit most from them. Or it could be the mental model thing, where they know Ansible, they decided years ago what Ansible is for, and they never went back to check whether that was still true. My guess is it’s both, in different amounts depending on who you ask.

And here’s what surprises people the most: Ansible already knows how to talk to all of it. There are supported collections for Kubernetes, AWS, Azure, GCP, ServiceNow, Vault, GitHub, network gear, Windows. The list is long and it keeps growing. kubernetes.core will read pod and deployment state for you. There’s a module that opens the ServiceNow ticket. You are not writing a pile of shell scripts wrapped around curl and hoping the error handling holds up. Somebody already wrote it, it’s maintained, and it behaves like the rest of your automation.

So the talk was our attempt at fixing that.

Here’s the argument. Kubernetes is already an automation system, and a good one. A pod crashes, it restarts. A node dies, workloads get rescheduled. Replica count drops, it reconciles. I have zero interest in competing with any of that.

But self-healing is local. Recovery is cross-system.

When a pod goes into CrashLoopBackOff, Kubernetes handles its part in seconds. Everything after that is still a person. Somebody has to pull the previous container’s logs before they rotate. Somebody has to figure out which deployment change caused it. Somebody has to know who owns the service, post in the right channel, decide whether to roll back, open the ticket. None of that is in the cluster’s job description, and all of it is exactly the kind of repetitive, well-understood work that automation is good at.

There’s a timing problem underneath it too. Kubernetes moves faster than people do. The crash happens, the pod restarts twenty seconds later, the logs rotate a few minutes after that, and by the time a human opens a terminal the scene has already changed. You end up investigating the aftermath instead of the event. The evidence you wanted most (previous logs, the pod description, recent events, the image tag that was running, the rollout state) was all sitting there at the moment it broke and gets harder to find every minute after.

That’s the whole case for reacting at event time. Event-Driven Ansible sits subscribed to cluster events. Something matches a rule, and a playbook runs while the evidence is still fresh.

The pattern itself is boring, which is kind of the point. Event source, rulebook, action. Kubernetes emits the signal. The rulebook is where the decision lives: match the condition, apply the guardrails, figure out who owns this. The playbook does the work: read the pod and deployment state, capture current and previous logs, validate the dependencies the app actually needs, notify the team, open a ticket, or run an approved action.

For the demo we kept it deliberately dumb. A FastAPI app with a /crash endpoint. Hit the endpoint, the process dies, the pod fails, a watcher catches the condition and sends an event to the AAP event stream, the rulebook matches, the playbook collects context and notifies. Nobody needs a clever demo to understand a pod crashing. What we wanted people watching was the seam. The moment a Kubernetes event turns into trusted automation doing the next five things a human would have done. It ran clean, which is more than you can usually say about a live demo.

The part I’d push hardest on, though, is the guardrail, and I’d say this whether or not it was on a slide. Start with observe, validate, and notify. That’s it.

I know the exciting version of this talk is auto-remediation. Pod crashes, automation rolls back the deployment, nobody gets paged. You can get there. But if the first thing your reaction layer ever does is take an action in production, you’re asking people to trust something they’ve never watched work. Let it capture evidence and route notifications for a while. Let people see it be right, over and over. Then promote the remediations that are proven, bounded, and reversible. And only those.

Same reasoning applies to permissions. The service account we used has narrow RBAC, nothing close to cluster-admin. A reaction layer with broad cluster permissions is a very efficient way to turn a small incident into a large one.

And none of this is really about pod crashes. That was just the example that fit in ten minutes. Change the signal and change the action and the model holds. Dependency health, security drift, rollout health, access gaps. The dependencies almost never live where you want them to, either. They’re in another namespace, another cluster, on EC2, in some managed cloud service, on a server sitting on-prem. One playbook can check all of them, because it already has a way to reach every one of them, and what it does when it gets there is version-controlled and reviewable.

If you take one thing from the talk, I’d want it to be the question we closed on. What are the steps people on your team still do by hand after an event fires? Not the incident itself. The ritual after it. The logs you always pull. The dashboard you always check. The channel you always post in. The person you always have to go track down. That list is your backlog, and if you’re somewhere that already has AAP running, you’re a rulebook away from working through it.

The thing I keep coming back to is that we didn’t have to build any of this. It was already installed, already running, already approved. The only missing piece was people knowing they could use it.