Chime: cutting on-call from a full-time job to a couple hours a week

At Chime, I led a project to reduce the burden of on-call for the engineering team. On-call had become “basically a full-time job manually repairing data errors.” The volume of transactions meant that ordinary network anomalies produced a steady stream of data errors in a complex system, and correcting them required real investigation — work that was both laborious and error-prone.

Over three months, we built a system to catalogue the different failure types, document them, and automate the response: detect a failure, determine the cause, map it to a correction, apply the correction, and log the action so it could be audited.

The result was a step change in the burden of the on-call rotation:

“When we were done, the burden of on-call went from a full-time job, often more than 40 hours a week, down to just a couple hours per week of verification and monitoring.”

— Chris Dwan, interview, 2026-08-07

That figure — 40+ hours a week down to a couple of hours a week — is by my own account, not an audited number.