The lock I added and reverted four minutes later

I have a scheduled agent that keeps a kanban board in sync. Every two hours it wakes up on an LXC on the home cluster, starts a pi session, reads a pile of sources, and writes back a drafts file, a note in my vault and a finding on the board. It is boring, which is the point.

On September 18 two of those runs overlapped. Both wrote the same drafts file, the same note and the same board finding, and the second one won. Nothing caught fire. I just lost a run's worth of work without noticing until later, which is the worst way to lose anything.

So this morning I had an agent working through the issue list, and the overlap was on it.

The obvious lock, and why I didn't want it

The standard fix is a mkdir lock. mkdir is atomic, so whichever run creates the directory first owns the job, and the other one backs off. It's four lines of bash and it works right up until a run gets killed halfway. A killed run never releases anything, so the lock sits there and every future fire politely skips. Forever.

The usual patch is "break the lock if it's older than N minutes." I have a standing rule against that. Age tells you nothing about whether the owner is still working. A slow run is not a dead run, and guessing wrong means two writers again, which was the bug.

When the agent asked how stale locks should be handled, I picked the option that sounded most correct: check whether the owner is alive.

Keyed on a live process

The first version recorded which pi process owned the lock. A pid alone is not enough, because Linux reuses pids, so it also stored the process start time from /proc/<pid>/stat. The lock counted as held only if a process with that pid existed, was named pi, and had the same start time:

owner_alive() {
  local dir="$1" pid st
  pid="$(cat "$dir/pid" 2>/dev/null)" || return 1
  st="$(cat "$dir/pid-start" 2>/dev/null)" || return 1
  [ -n "$pid" ] && [ -d "/proc/$pid" ] || return 1
  [ "$(comm_of "$pid")" = "pi" ] || return 1
  [ "$(start_time "$pid")" = "$st" ]
}

A precheck hook ran before each fire. If a live pi owned the lock, it exited 1 and the runner recorded a skipped run. If the owner was provably gone, it reclaimed the lock. Fifteen self-test checks passed on the cron host. It went in at 09:12.

It came out at 09:16.

The process that never dies

The automation runner starts every fire as an interactive pi session in its own terminal, and when the run finishes it just leaves the terminal open. The pi process sits at its prompt, idle, alive. When I looked, there were 20 of them on that LXC.

Read the lock logic again with that in mind. The owner of any lock is a pi process. That process never exits. So owner_alive returns true for every lock, the precheck skips every fire, and the job never runs again. I had built a very careful detector for a dead process on a machine where nothing ever died.

The revert went in before the 09:45 fire could exercise it, which is the only part of this I'm proud of. The design wasn't wrong. It was keyed on an event the platform never produces.

Fix the premise first

The next move was to make "the run is over" and "the process is gone" mean the same thing again. That meant a reaper.

By the time the reaper was written the count was 21 idle pi processes, holding about 4 GB of RSS on a 12 GB container. One per past fire of two scheduled jobs, accumulating quietly since I don't know when. The reaper closes a terminal only when all three of these hold:

• it belongs to a run of one of the named automations, so my own terminals are never touched;
• the runner reports its TUI idle, meaning pi finished its turn and is sitting at the prompt;
• it has printed nothing for at least 10 minutes, so a run that is only between steps is left alone.

It also always exits 0, because it runs as a precheck too, and a reaper bug should never be the reason a fire gets skipped. The first manual pass closed 18 terminals. The pi count went from 21 to 3 and used memory from 2970 MB to 513 MB. One of the three survivors was a backlog worker that was genuinely mid-run. The reaper read it as busy and kept it, which is exactly what I wanted to see from a program whose job is killing things.

Review found problems within minutes. The worst was that the 60 second budget was checked only between terminals, so the setup calls plus one slow terminal could creep toward the 120 second precheck timeout. It became a single 90 second deadline that caps every call the reaper makes.

The lock, second attempt

With the reaper in place, a live pi means a live run again, so the lock came back at 10:17. The review of the first attempt had some notes, and they all landed in the second:

• acquiring the lock now reclaims a dead owner's lock by itself, so recovery doesn't depend on the precheck, which manual fires skip;
• acquire and precheck serialise on flock, so a reclaim can't delete a lock another run took a millisecond ago;
• the precheck skips only on proof of a live owner, and fails open on every error;
• the owner record is written to a temp file, moved into place and read back before the lock reports acquired.

Three minutes after that, one more round. Exit code 1 meant both "skip this fire" and "bash hit a fatal error," which is a terrible thing for one number to mean, so the skip got its own code, 75, and only 75 maps to a skip. And there was a last race hiding in the handoff between the two scripts: closing a terminal returns before pi has actually exited. The lock check right after the reaper could still see a finished run as alive and skip a whole two hour cycle. The reaper now waits up to five seconds for the pi count to drop.

The self-test went from 15 checks to 22.

I wrote the lock twice and the reaper once, and the reaper was the real fix. Any lock that asks whether its owner is alive depends on processes dying when their work is done. On this box nothing did, and 21 idle pi prompts were sitting on 4 GB of memory to prove it.