Timing language models with a bicycle essay and thirty facts about the Moon
Three throwaway prompts, a speed meter that lied by a factor of five, and ten days of changing my agent's default model. A feel test, written down honestly.
On the morning of September 15, in a scratch directory called pi-nitro-debug under /tmp, I ran three prompts through my terminal agent. One per session, nothing else.
Reply with exactly this line: hello there, this is a short reply for speed measurement purposes only.
Write a 30-line numbered list of short facts about the Moon. Plain text only.
Write a detailed 2500-word essay on the history of the bicycle, in plain text with no lists.
I did not want to know about the Moon. I did not want the essay either. I wanted to know whether a routing change I had made the night before did anything at all.
What I was testing
The agent is pi, and I send most of its model traffic through OpenRouter. At 22:41 on the 14th I had pushed a small extension that rewrites every OpenRouter request to use the :nitro variant of the model. Nitro tells OpenRouter to sort a model's endpoints by measured throughput and to let priority-tier endpoints serve the request, so you get the fastest endpoint instead of the cheapest acceptable one. I already had sort: "throughput" in my routing settings, which does half of that. The README I wrote the same night admits the two knobs pull the same lever.
So the question for the morning was simple. Does it feel faster?
Why three prompts
Each prompt stresses a different part of a response. The one-line reply is almost all waiting: queueing, network, time to first token. The Moon list is short enough that startup still matters but long enough to stream for a moment. The bicycle essay runs long enough for the decode rate to settle into a number you can actually read.
That last point is less of a design choice than it sounds. My own routing code throws away any speed sample under 40 tokens or 750 milliseconds as noise, and the footer meter refuses to show a rate at all when less than 250 milliseconds is measurable. A one-line reply can't produce a tokens-per-second number in my own tool. You need an essay about bicycles for that.
The stopwatch was broken
Here is the part that makes this a feel test and not a benchmark. The only instrument I had was the tokens-per-second readout in pi's footer, and that meter had been wrong twice that week. Once it claimed 232,000 tok/s. The second fix landed at 09:19 on the 15th, the same morning as the prompts, after it showed 705.5 tok/s for a response that really ran at 142.5.
I cannot tell you now whether my bicycle essays ran before or after that fix. I also did not save a single reading. There is no spreadsheet, no table of models against providers, no median. I watched a number in a footer and formed an opinion. Please weigh everything below with that in mind.
The only latency numbers I actually wrote down
Ironically, the careful measurement that week was about something else. On the night of the 14th I turned off automatic context compaction, and the commit for that change records turn latency from one session, same model and same provider, bucketed by how full the context was:
| Context size | Median turn latency |
|---|---|
| Under 50k tokens | 4.9 s |
| 50k to 150k | 7.9 s |
| 150k to 300k | 7.5 s |
| Over 300k | 14.8 s |
Three times slower at the far end, with no change of model at all. The middle two buckets are basically noise around each other. That table did more to change how I run the agent than any of the model shopping did: a day later I turned compaction back on and set it to fire at 30% of the window.
Ten days of changing my mind
The feel test fed into a week of flip-flopping that the git log records without mercy. These are the default-model changes in my pi config between September 13 and September 22:
- Sep 13, late: GLM 5.3 Flash on OpenRouter, because that copy allowed 131.1K output tokens against 16.4K elsewhere and accepted images.
- Sep 14, morning: DeepSeek V4.1 Flash, first for personal sessions, then everywhere by mid-afternoon.
- Sep 14, 15:48: the full GLM 5.3 at its deepest thinking level, noted in the README at nine times DeepSeek's per-token price.
- Sep 15, 09:04: back to DeepSeek V4.1 Flash.
- Sep 20: GLM 5.3 Flash again, for personal directories.
- Sep 21: DeepSeek V4.1 Flash for everything, one model on every directory rule.
- Sep 22, 09:10: MiMo V2.6 Pro on a subscription provider, after checking that it actually served the model and could describe a 16 by 16 test image as blue.
- Sep 22, 21:23: DeepSeek V4.1 Flash. Again.
DeepSeek won in the end, but not because a benchmark said so. It kept winning because every alternative I tried, I eventually switched back from.
Choosing providers by vibes, with instrumentation
The model is only half of it. OpenRouter was load-balancing DeepSeek V4.1 Flash across 16 upstream providers by price, so the same prompt could land somewhere different every run. On the afternoon of the 14th I went through this sequence in forty minutes:
- 15:05, prefer BaseTen. A live check routed 4 of 6 requests there, with the rest falling back to Fireworks, Morph and DeepInfra after rate limits.
- 15:23, add a speed floor. If two responses in a row decode under 30 tok/s, the preferred provider gets benched for a cooldown. Against a fake provider streaming at 16.3 tok/s it demoted on the third turn. At 165 tok/s it never did.
- 15:41, drop the BaseTen preference. My exact words were "ok now i dont preferbaseten". That also quietly disabled step 2, because pi never says which provider answered a stream, so with no preferred provider there is nobody to blame.
- 15:45, add a footer badge that asks OpenRouter which provider served the last response. The answer only exists four to ten seconds after the response ends, so it polls.
In its first live test, the badge said Morph.
On the night of September 22 I set the preference back to BaseTen, Fireworks and DeepInfra, in that order, and put Morph on the ignore list. No commit records why. There is no measurement behind it that I can find, only a bicycle essay and a feeling.