Three ways to break a rendered video in one week
A pixel aspect ratio off by one part in 8,000, a GPU admission counter read one await too early, and a form field that sent null when it meant nothing. Three bugs in one video pipeline, September 18 to 20.
I run a video pipeline for a few clients. Jobs come in, GPU render workers cut and composite a stack of clips into a finished video, and the result gets scheduled on YouTube. Most weeks it's boring. This week three unrelated bugs showed up in three days, and once I'd fixed all of them I noticed they were the same bug wearing different clothes. In each one, two values looked identical and meant different things.
One part in eight thousand
A client reported three renders stuck in error. All three had died the same way:
the CUDA filter graph failed on segment 1 ... exit=218
[fc#0] Error reinitializing filters!
It wasn't a one-off. Once the counting was done properly, the worker log held 14 of these failures over two days, 11 on the first and 3 on the second. Exit 218 is AVERROR(ENOSYS), and my first theory was GPU contention. That theory turned out to be half right, which is the most expensive kind of right. More on it below.
The better clue came from reproducing it on the render box against a real intermediate file from the same job. ffmpeg printed Reconfiguring filter graph because hwaccel changed right before the failure. With -v debug, the H.264 decoder also announced it was reinitialising its context. In the middle of the file.
A decoder only does that when the stream's parameters change under it. The intermediate carried two H.264 sequence parameter sets. The body clips are joined with a -c copy concat, so each clip brings its own SPS along, and scale and pad had been carrying each source's sample aspect ratio straight through to the encoder. The job's 187 clips split 111 and 76 across two SPS that differed in exactly one field: SAR 8384/8385 against 3839/3840.
Both of those are square pixels to within about 0.012%. Nobody could see the difference. The decoder could. It reinitialised on the seam, the CUDA frames context changed, ffmpeg decided the hardware acceleration had changed, and the CUDA filter graph could not survive being rebuilt. Running the real composite command on the old join failed at 7.91 seconds, which is exactly where the clip seam sits.
The first fix was setsar=1 at the end of the shared normalize chain. One SPS per file, zero reinit lines, exit 0. A review pass then found that fix was wrong for a different input. Pinning SAR at the end relabels whatever came before it. Take a 1440x1080 source at SAR 4:3, which is a 16:9 picture stored anamorphically. It gets fitted into 1440 of the 1920 output columns, and a trailing setsar=1 then declares those columns square. That clip comes out squashed by 25%.
The version that shipped squares the pixels first, before the fit:
scale=round(iw*sar/2)*2:ih,setsar=1
It uses round, not trunc, so those two nearly-square production SARs round back to the source's own width. Those clips keep their exact pixel grid. No resample, no hairline of padding. The same round also deleted a comment in the code claiming that a -c copy concat keeps one SPS for the whole file. It doesn't. That belief is probably why a two-SAR join looked harmless for so long.
A counter read one await too early
Back to the contention theory. The worker has a rule that only one CUDA filter graph runs at a time, and the log showed three segments of one composite admitted within 20 ms of each other, then running three overlay_cuda graphs at once.
The admission function built its decision from the active-graph counter, then awaited a VRAM probe, then incremented the counter. The counter was read before the suspension and written after it. When the waiters were woken in the same tick, every one of them read a stale 0 and every one of them admitted. The counter did block anyone who showed up later. It just never stopped a crowd that woke together, and in that box's log 171 of 181 admissions were bursts of two or more.
The fix is dull: probe VRAM first, then decide and increment in one synchronous block. The new test starts three concurrent waiters and expects one admission. On the old code it failed with 3 !== 1.
Then I didn't ship it. My own instruction had been to fix the failures without hurting throughput, and honouring the one-graph rule for real measured at minus 50% on the composite step with 20 jobs queued. So the admission fix stayed on a local branch waiting for a decision, while the hunt went on to the SAR seam, which cured the failures by itself.
The next day a scheduled backlog agent I'd set up found that branch. It held 420 lines of work, never pushed, with no PR. The card asked whether to land it or drop it, since the exit 218 symptom was already fixed. The agent's answer was that the defect was still on master, and the SAR fix covered a different half of the same error code. It proved the tests were real by reverting the source files to master, and three of the four new cases went red. With the fix back, the full suite ran 4,565 tests with zero failures, apart from two cancellations in a file with a known flake. It landed.
A counter that is 0 and a counter that was 0 a moment ago look identical to the code reading them.
The form that sent null
The third one is in the admin UI, not the renderer. The video edit form always sends publishAt. When it shows no schedule, it sends null.
That night I'd landed a change so that saving the form would mirror the schedule state it actually pushed to YouTube. I pushed back on the review, asking for the diff to be read again through an operator's actual workflow. That re-pass found a regression in a commit that was eight minutes old.
Some videos get stuck private: YouTube has lost the schedule, but our record still holds one. An overdue pass looks for exactly those videos and force-publishes them. On a stuck video the form shows no schedule, because YouTube has none. So an operator fixes a typo in the title and hits save. The form echoes publishAt: null. The sync reads that as the operator clearing the schedule and wipes the record, and the rescue that should have caught this video never fires. Nobody gets told. The only visible change is the new title.
The controller could not tell a deliberate clear from a form repeating what the screen showed. It needed to know what YouTube held before the edit. Now the service returns that held schedule, and the sync clears the record only when the operator's clear actually removed a schedule YouTube had. A null against a video with nothing held leaves the record alone. The reconcile pass owns that decision. A title edit doesn't.
Three values passing for something else: a SAR that was nearly 1, a counter that was 0 a moment ago, and a field that said "nothing" because the screen had nothing to show. Each one arrived where the code expected a value it could trust, and none of them was the value it looked like.