An alert should need two sightings and a fresh look

We run a small internal health page for our own services. A collector on a schedule gathers three kinds of results: background jobs and whether they ran, smoke tests of public sites, and endpoint probes. It posts a snapshot to a NestJS backend, which stores it, works out what is failing, and notifies the owner. The first version of the notification logic was four lines: take the keys that are failing now, remove the keys that were failing in the last snapshot, and alert on whatever is left.

That is a reasonable first version and it has four problems. One bad snapshot sends an alert. A check that flickers fails, passes and fails again, and sends an alert each time it fails. A result that was never re-checked gets counted twice. And a probe that could not run at all looks the same as a probe that ran and found the service down.

We replaced the four lines with a function that takes the previous and current snapshot state and returns three lists: what to alert on, what has recovered, and what is currently an open incident. It does no I/O. Everything it needs is passed in, so the tests are a table of inputs and expected outputs.

Two sightings #

A key alerts only if it is failing in the current snapshot and was failing in the previous one.

const alert = input.currentFailing.filter(
  key => previousFailing.has(key) && !previousAlerted.has(key) && !repeated.has(key),
);

A single failed snapshot between two good ones sends nothing. The cost is delay: with snapshots on a schedule, a real failure is noticed one snapshot later than it could have been. We took that trade because for our own services a late alert is cheap and a false one trains the owner to ignore the next.

A fresh look #

The third condition in that filter is the one that is easy to forget. Some results are not collected every time. A smoke test might have run an hour ago and the collector sends its last result again with every snapshot. If the function only compares keys, that stale failure appears in two consecutive snapshots and counts as two sightings, when it was one observation.

So each result carries the time it was actually checked, and the collector's snapshot time is only the fallback when none is given. A key whose observation time has not changed since the last snapshot goes in the repeated list and cannot alert. One check cannot vouch for itself.

One alert per incident #

Once a key has alerted, it is recorded in a column on the snapshot row, so the next run knows it is already part of an open incident. A key that stays failing does not alert again. A key that fails while another incident is still open does alert, because the owner has not been told about it.

The incident also survives a brief pass. If an alerted key passes in one snapshot and fails in the next, it is the same incident and nothing is sent. Without that rule a flapping check produces a fresh alert on every flap, which is the same noise from a different direction.

Two passes to recover #

Recovery mirrors the alert rule: an alerted key is recovered only when it passes in the current snapshot and the previous one. One green result does not end the incident, so the recovery message and the next alert do not alternate.

What counts as a pass #

An endpoint probe that could not be verified, for example because the check itself was blocked from reaching the target, is neither failing nor passing. We store it separately as unverified. It does not raise an alert, it does not close an open incident, and the page shows a warning with its own count. A probe that is down and a probe we could not see through are different facts, and collapsing them in either direction gives the owner wrong information: a false alarm, or a false all clear.

Two smaller rules follow the same idea. A smoke site marked as pending, such as one waiting on DNS, is skipped entirely. And if an alerted key disappears from the monitored set, it is dropped from the open incidents without a recovery message, since nothing observed it get better.

Testing it as a table #

Because the function is pure, each rule is one short case: the previous failing keys, the previous alerted keys, the current failing keys, and the expected alert, recovered and alerted lists. The cases cover waiting for a second snapshot, a single bad snapshot between good ones, the open incident surviving a brief pass, recovery after two passes, unverified keys not counting as recovered, repeated observations not alerting, and a new key alerting while an old incident is open.

The test that earns its place is the repeated observation one. It is the rule nobody thinks of while writing the happy path, and it is the one that would let a single stale result page the owner twice.

What we would copy elsewhere #

If you are building alerting for a small system, the sequence that served us was: make the decision a pure function, store which keys have already alerted so the function is not stateless in the wrong place, record when each result was observed, and keep "could not check" as its own state. The thresholds themselves, two sightings and two passes, are a judgement for your system. The structure is what made them cheap to change.