From postmortems to guardrails at scale

Say it’s Thursday morning and your login service has been throwing 500s for 45 minutes. Support is fielding complaints. The team is staring at a release that passed testing in staging yesterday and is still running happily there. Production has other ideas.

Eventually, you find it: the release pipeline rebuilt the same source before deploying it. A fresh pull of python:latest brought in a base image without a system package the service needed. Same application code, different image. Staging was running yesterday’s build.

The service is back. Now comes the follow-up

The team explicitly installs the missing package, pins the base image by digest, and tests the rebuilt application image before deploying it. Logins work again. Everyone can breathe.

There are two lessons to carry forward: pin external base images by digest, and deploy the application image that actually passed testing. A version tag can still move. A digest identifies a specific image, which also means someone must deliberately update it to pick up future fixes. Docker’s guidance covers that tradeoff.

Then comes the Slack message. You explain what happened, update the wiki, and file tickets for the other teams you know are affected. A few Dockerfiles get fixed that afternoon. Finding the rest becomes the platform team’s problem.

Why the lesson gets lost

There are hundreds of repos, each with its own history. One team uses the current shared workflow. Another pinned an older version. A third has a perfectly reasonable custom build path that nobody on your team knew about. They might all use the same CI vendor. You still have to find out which ones need the fix.

The repeat-incident cycle when follow-up depends on reminders and tickets alone.

Shared workflows can enforce the rule wherever teams use them. Tickets help track the cleanup. But someone still has to chase the gaps, and the next sprint has already started. Months later, a new service copies an old Dockerfile. The engineer who creates it wasn’t in the incident channel.

For a platform leader, closing the incident leaves an uncomfortable question: where is this rule actually being checked? A completed ticket tells you a fix happened once. You need a check that keeps running when someone changes the code again.

That is the job of a guardrail: check the requirement, tell the author what failed, and block the change when enforcement is enabled. You also need to see which services it evaluated and which it couldn’t assess. Otherwise, the gaps are still yours to chase.

Give the action item something to check

“Improve incident communication” needs human follow-through. “Pin external base images by digest” gives you a condition a machine can evaluate. Start there: what would the check look for, and what should cause it to fail?

Writing the rule may be the easy part. You need to find the Dockerfiles and understand the image references they use. To verify that production gets the tested application image, you need evidence from testing and deployment too. Across teams with different tools and build paths, gathering that evidence can become a project of its own.

That work has to hold up beyond the repos you spot-checked after the outage. And someone needs to own the rule as the build process changes.

Turning the lesson into a Lunar guardrail

Earthly Lunar handles central collection and normalization of signals from your repos and CI/CD. It turns them into structured software development lifecycle (SDLC) posture data, so the platform team can write and manage guardrails centrally without copying their logic into every repo.

The guardrails engine evaluates that evidence deterministically: the same inputs produce the same result. For the Dockerfile incident, the important choice is exactly which requirement you enforce.

Catch the failure you mean to prevent

Lunar’s no-latest guardrail catches FROM python:latest and FROM python, either of which can pull a different base image on a later build. digest-pinned goes further: it requires base images to be pinned by digest rather than relying on a tag alone. Even a full version tag can move; a digest identifies the exact image content, enforcing the stricter standard from this postmortem. The policy checks base-image references collected by the Docker collector.

To require digest-pinned base images, add this to your lunar-config.yml:

collectors:
  - uses: github://earthly/lunar-lib/collectors/docker@v1.23.0
    on: ["production"]
    include: [dockerfile]

policies:
  - uses: github://earthly/lunar-lib/policies/container@v1.23.0
    on: ["production"]
    include: [digest-pinned]
    enforcement: block-pr-and-release

With Lunar’s PR checks and release gates configured, this applies the digest requirement to components tagged production across the company. A failing check blocks merges and releases for those components. It checks base-image references; verifying that the tested application image is the one deployed remains a separate requirement.

Roll it out without surprising every team

Start in draft mode and inspect the results before blocking changes. Did the collector find and parse the expected Dockerfiles? Can it handle your private registry references and build arguments? Investigate errors and skips. A green result won’t tell you about a file the check never saw.

Then use PR feedback to help teams fix existing violations, and enable blocking for the scope you’ve validated. Keep an owner for the guardrail. Once it’s running, a copied Dockerfile can trigger feedback at the next evaluated change, without someone remembering to forward the postmortem.

Keep the lesson close to the work

Lunar supports enforcement during AI-assisted authoring, on PRs, and at deploy gates. When a check’s inputs are available during authoring, agent hooks can return failures for the AI agent to address before opening a PR. The same engineering standards apply whether a person or an agent writes the code.

There are 300+ pre-built guardrails across testing, security, compliance, deployment, and operational readiness. For a requirement the library doesn’t cover, Lunar’s built-in AI skills can help draft a custom guardrail for your team to review and test.

Put your next postmortem finding to work

Take an engineering requirement from your latest postmortem and paste it into Lunar’s guardrail matcher. It finds relevant checks in the library and generates code for organization-specific standards. Review the results, pick a check, and validate it across the affected repos while the lesson is still fresh.

Find matching guardrails