Blog

Field notes from the graveyard.

Writing on infrastructure debt, the anthropology of on-call rotations, and the slow work of cleaning up after the people who came before.

~$

The cron job that ran for nine years before anyone noticed

An archaeology dig through one customer's /etc/crontab — a parable for what happens when "set it and forget it" is the architecture.

Read post →
[+]

Adoption ceremonies: a tiny ritual for surfacing infrastructure

Why every onboarding week should end with each new engineer "adopting" three live production processes. We've seen it work in three orgs.

Read post →
404

What happens to your services when you off-board the only person who understood them

Notes from five companies that lost a key engineer and discovered, six months later, what they didn't know about their own infrastructure.

Read post →

Why we made the agent read-only by default (and what we lost)

Letting Fernwake reach into your hosts and stop things would have made the demo better. Here's why we drew the line where we did, and how we make up for it.

Read post →
$$$

The hidden cost of cron: how a $4/month Lambda became a $1,400 invoice

A friend of ours runs a SaaS. He thought his AWS bill was high. The investigation took two months. What he found will sound familiar.

Read post →
/x/

Designing for sunsets: the case for an "owner field" in every deploy manifest

A proposal: every Helm chart, every Terraform module, every cron entry should require a human owner at submit time. Here's how three teams retrofitted this.

Read post →

No newsletter pop-ups, ever. RSS or nothing.

Subscribe via RSS