Wait for the lock, not the clock

One of our NestJS backends lets a member answer "I'm in" to a plan, and lets a member delete their account. Those two requests can arrive together, and the rule is that a say-in racing a deletion must be refused, so the plan never keeps a row for someone who has gone. The fix for that race went in on 16 October. This post is about the test that proves it, and the one line in it that we changed the next day.

The test as first written #

A race test needs two transactions in a known order. Ours plays the deletion by hand:

await erasure.startTransaction();
await erasure.query("UPDATE users SET deletedAt = NOW(3) WHERE id = ?", [user.id]);
const sayIn = service.respond(code, user, "in").then(() => "in", (e) => e.status ?? "error");
await new Promise((resolve) => setTimeout(resolve, 300));
await erasure.query("DELETE FROM external_plan_members WHERE userId = ?", [user.id]);
await erasure.commitTransaction();

The erasure transaction holds a row lock. The say-in is started while it is held, so it should block. The test then finishes the erasure and checks that the say-in comes back refused.

The sleep in the middle is there to give the say-in time to reach the lock. Everything depends on that guess.

What a guessed sleep gets wrong #

If 300 milliseconds is too short, the erasure commits before the say-in has touched the database. The say-in then runs entirely after the deletion, which is a different and easier case, and the test passes without ever exercising the race. If the machine is busy, a test run in parallel with other suites can land there. Nothing fails, so nobody notices that the test stopped testing what its name says.

If 300 milliseconds is too long, we pay for it on every run, forever.

The worse property is that both outcomes look identical from the outside: a green test.

Ask the database instead #

On MySQL 8 with InnoDB, a transaction that is blocked on a row lock shows up in performance_schema.data_lock_waits. The test can poll that table and continue the moment a waiter exists:

async function untilWaitingOnLock(database: string) {
  for (let attempt = 0; attempt < 1000; attempt++) {
    const [row] = await db.query(
      "SELECT COUNT(*) AS n FROM performance_schema.data_lock_waits w JOIN performance_schema.data_locks l ON l.ENGINE_LOCK_ID = w.REQUESTING_ENGINE_LOCK_ID WHERE l.OBJECT_SCHEMA = ?",
      [database],
    );
    if (Number(row.n) > 0) return;
    await new Promise((resolve) => setTimeout(resolve, 10));
  }
  throw new Error("the say-in never waited on the erasure's lock");
}

Three details carry the weight:

• The query is scoped to the test's own schema. Without the OBJECT_SCHEMA filter, any other suite waiting on any lock on the same server would satisfy the check.
• The loop has a ceiling and a message. If the say-in never blocks, for example because someone later changes the code so it no longer takes that lock, the test fails with a sentence that says so, instead of hanging.
• The poll still sleeps, but for 10 milliseconds, and it exits early. The delay is no longer a bet about timing. It is only how often we look.

The test now fails in the situation where the old one passed quietly: when the say-in does not wait.

The same habit for clocks #

The same week we removed a 1.6 second sleep from another test that had to get past a rounding window in a verification check. The fix there was a fake clock that steps past the window. The test verifier had been reading a real time source, so it did not follow the fake clock until we made it call Date.now on every use.

When this applies, and when it does not #

This only works where the system under test exposes the state you are waiting for. data_lock_waits is specific to InnoDB, and the query above is the shape, not a drop-in for other databases. PostgreSQL has pg_locks and pg_stat_activity for the same job.

For anything without such a window, the next best thing is an explicit signal from the code you control: a promise the test resolves from a hook, or a counter the service increments when it reaches the contested step.

The question to ask of any setTimeout in a test is what condition it is standing in for. If you can name the condition, wait for the condition.