The failure that never pages you

Every alert on your dashboard is built the same way. Something happens, the something is bad, an alert fires. Error rate over a threshold. Latency over a threshold. A health check that stops returning 200. All of them need an event to hang off.

The outage that costs you a week of reconciliation has no event in it. The nightly export ran for two years and then the worker exited cleanly on a deploy that removed its process from the supervisor. Nothing crashed. Nothing retried. The error rate is zero, because zero jobs ran, and zero of zero failed.

Absence is not an event

You cannot alert on an absence with the tools that alert on events, because there is nothing to count. What there is instead is a shape: this queue receives a job roughly every four minutes, has done for months, and has now not received one for two hours. That is a statement about the gaps between arrivals, and gaps are exactly what an event-shaped monitor throws away.

The usual answer is a heartbeat: the job pings a URL when it finishes, and something shouts if the ping does not arrive. It works, and it has one flaw that shows up on exactly the day you need it. The heartbeat is code inside the job. If the job never starts, the heartbeat never fires, which is correct. If the job is removed from the schedule, the heartbeat goes with it, and the monitor quietly stops expecting anything.

What to measure instead

Measure the interval, not the event, and measure it somewhere the job cannot delete.

Bidewell keeps the last two hundred arrival intervals for every queue and takes the median. When four times that median passes with nothing arriving, the queue is raised. There is nothing to configure, because the queue has already told us what normal is by running for a fortnight.

Two deliberate limits are worth stating, because both are the sort of thing that makes an alert untrustworthy if it is hidden:

It will not say anything until it has seen forty arrivals. A queue created this morning has no median worth having, and an alert built on six samples fires on a quiet Sunday and gets muted before it is ever right.

It works on the median and not the mean. A queue that runs every four minutes and once a month does a two-hour backfill has a mean interval that no longer describes anything. The median ignores the backfill, which is the correct behaviour, because the backfill is not what you are watching for.

The unhelpful truth about thresholds

Four times the median is arbitrary, and we are not going to pretend otherwise. It is chosen because it is far enough above the ordinary spread of a healthy queue to survive a slow afternoon, and close enough that a broken hourly job is raised inside half a working day. If your queue is regular to the second, four times is generous; if it is bursty, it is tight. It is a single number doing a job that would ideally be done by a distribution, and it is the honest place to start rather than the finished answer.

What matters more than the multiple is where the measurement lives. It is on our side of the wire. Deleting the queue’s code does not delete the expectation that the queue produces work, and that is the whole of the difference between this and a heartbeat.