Teaching an AI to play an old Flash game so I don't have to
A Playwright bot drives Onslaught 2 inside Ruffle and asks a reasoning model where to put turrets. It never looks at the screen, and the bugs that mattered were about timing and keyboard focus.
Onslaught 2 is a Flash tower defense game. Creeps walk a winding sand track, you park turrets beside it, you upgrade them, you press the button for the next wave. Flash has been dead for years, but Ruffle, an emulator written in Rust, runs the old files in a browser tab, and some archive sites still host the game that way. The copy I play is a modded build. The bottom right corner says so, and the cash counter starts close to a million.
With unlimited money there is no economy left to manage. What remains is clicking the same places in the same order for thirty waves. On September 18 I spent a morning making a program do that.
Attaching to a browser that's already open
The bot does not launch its own browser. Chrome runs with a remote debugging port open, and the bot attaches to it over the Chrome DevTools Protocol with Playwright's connect_over_cdp. It looks through the open tabs for the game, opens one if there isn't one, and pulls it to the front. The upside is that it plays in my normal profile, cookies and all. The downside is that it shares a browser with a person, which came back to bite me later.
Ruffle renders the whole game onto one <canvas> inside a ruffle-embed element's shadow DOM, so Playwright's usual locators find nothing useful. The bot reaches in with a bit of JavaScript, clicks Ruffle's play button if it's showing, grabs the canvas, and asks for its bounding box. Every click after that is canvas origin plus a hardcoded offset.
Those offsets live in a constants file. The game canvas is 720 by 535. Map 1 is at (280, 260) on the map picker. The next wave dial is at (688, 500). There is a list of ten spots on the first map where a turret fits beside the track, in the order they should be filled. Choosing a turret type means pressing the right arrow a fixed number of times, because the build menu is a carousel: seven presses from a fresh start for the rocket turret, eight for the laser, nine for the cannon.
Placing a turret works like this:
- click the build tab
- click the turret icon in the rack
- click the map spot twice, once to position and once to drop
- click an empty corner to clear the cursor
Upgrading presses Shift+Q, Shift+W and Shift+R for max damage, range and rate, then clicks the three on-screen Max buttons as well, in case the hotkeys didn't land. Belt and braces.
It never looks at the game
That's the part I find funny in hindsight. The bot takes screenshots, one every five waves and one at the end, but only for me. Nothing reads them. Its idea of the game state is a Python dict it keeps itself: the wave number it thinks it's on, how many turrets it thinks it placed, which types, whether it thinks they're all maxed. If a click misses, the bot's world and the real one drift apart and nobody notices.
That dict is what gets sent to the model.
Asking a model where to put a rocket launcher
The tactician module sends the state to a reasoning model through a small SDK and asks three structured questions. One is a choice between placing a turret, upgrading the existing ones, or sending waves. One picks which turret or booster to build, from rocket, laser, cannon and three aura boosters. The last is a yes or no score for whether the defense is strong enough to hurry the waves along, and anything at 0.5 or above counts as yes.
The prompt for the turret choice includes the current counts and some rules I wrote in plain English: keep all three shooting turrets on the map, mix in boosters, don't pick the same booster twice in a row. The code also has one rule it doesn't trust the model with. If there are no shooting turrets yet and the model picks a booster, it gets overruled and the bot builds a rocket turret. A booster next to nothing is a very expensive decoration.
If there's no API key, or the call throws, the tactician falls back to a rule engine that does the obvious thing. Place turrets until you hit the target count, cycling through a fixed list of eight types. Then upgrade. Then send waves. Honestly, for a game with infinite money, the fallback is probably as good as the model.
Five waves at once
The first version worked in the sense that it placed turrets and started waves. It also had this line in the loop:
batch = 5 if decision.safe_to_spam else 1
Once the model decided the defense looked solid, the bot pressed next wave five times, slept three and a half seconds, and did it again. Five waves of creeps landed on the track together. The turrets were fine against one wave and useless against five stacked on top of each other, and the bot, which can't see, carried on confidently queueing more.
The fix went in at 13:56. The batch size became a flag that defaults to 1, and the wait became --wave-delay, 8 seconds by default, trimmed to 75 percent when the model says it's safe to hurry. One wave, a pause long enough for the turrets to clear it, then the next.
The space bar scrolled the page
Four minutes earlier was the other fix, and it was the more annoying bug. The game starts a wave on the space bar. On a normal web page, the space bar scrolls down.
When the canvas had focus, space went to the game. When focus slipped to the page body, which is easy when the bot shares a browser with a person, space scrolled the page. The canvas moved, the cached bounding box went stale, and every hardcoded offset after that clicked a few hundred pixels away from where it meant to.
The fix does three things. It sets overflow: hidden on the body so the page can't scroll. It adds a keydown listener that swallows space when the target is the body. And before each wave it dispatches the space key events straight to the canvas through JavaScript, then sends a real key press and clicks the dial anyway. It also gained a focus_game() helper that brings the tab forward, scrolls the canvas to the center and focuses it, called before every action. Oh, and it disables every link on the page, because a stray click on the host site's navigation would take the bot somewhere with no canvas at all.
Where it ended up
Six commits, between 10:05 and 13:56 on the same day. Eight unit tests, all against mocked browser calls, which can't tell you whether a click lands.
The one screenshot still in the repo is from a debugging run while I was checking the Shift hotkeys. Five turrets on the Area 51 map, wave 4, four kills, cash at $978,704. Not a victory screen. The code does say it will log "Successfully reached target Wave" at the end, and I have no record of it ever getting to say that.
I also found, writing this, that --turret-type tazer is an accepted option and the tazer counts as a shooting turret, but it has no position in the carousel table. The lookup falls back to the rocket's offset, so asking for a tazer quietly builds a rocket launcher.