Two Machines, Nine Models, One Number That Mattered

By Randy Armknecht and Mia

Three things were wrong with my desktop. The Radeon RX 7900 XT's 20 GB ceiling kept newer image models out of ComfyUI. Tuesday night gaming with friends meant unloading whatever model was resident first. And the scripts that summarize articles for me before I'm up needed a machine awake overnight, which meant a full desktop drawing full power around the clock. I wanted a low-power box, dedicated in purpose, on 24/7. AMD's Ryzen AI developer platform, a mini PC with 128 GB of memory shared between CPU and GPU, fit the description.

I knew going in that its memory bandwidth was a fraction of the card's. The YouTube reviews said so. The blog posts said so. What they couldn't tell me was how that gap plays out on my models, on my hardware, for my workloads, so I wrote a benchmark harness and ran the same 9 models through both machines. The card generates tokens twice as fast on each of the 6 models it can hold. The APU runs the 3 it can't.

Every Token Drains the Whole Tank

A dense model produces a token by reading every one of its weights from memory, once, and then reading them all again for the next token. Picture a water tank with a pipe at the bottom. The tank's size decides which model fits inside. The pipe's width decides how fast tokens come out, because generating a token means pumping the entire tank through the pipe. A bigger tank with the same pipe takes longer to empty.

BabelStream measured the card's pipe at 687 GB/s and the APU's at 236 GB/s, a 2.9x gap. The tokens-per-second gap across the 6 models both machines could hold landed between 2.0x and 2.2x. Qwen3.8 27B: 24.2 tokens a second on the card, 11.8 on the APU. The arithmetic is plain. Divide 236 GB/s by 17.8 GB of weights and the ceiling is 13 tokens a second; the machine delivered 11.8, which is 89% of its pipe. The card used 57% to 63% of its pipe on the same models. I don't know why yet, and the number is in the data either way.

The Small Tank Ran Out Before the Pipe Did

Capacity decides what runs at all. 3 of the 9 models never loaded on the 7900 XT: Qwen3.5 35B, Llama 3.3 70B, and gpt-oss 120B. Out of memory, all 3. The APU ran each of them. It also ran the 27B model with 262,144 tokens of context at 6.4 tokens a second, where the card fell over past 16K. Conversation history lives in the tank too. A long context takes the space a bigger model would have used, and the card had no space to give.

Nobody Is Waiting on a Cron Job

The result I keep coming back to is gpt-oss 120B. 63 GB of weights, and the APU produced 50.8 tokens a second, 3 times the speed of a 12 GB Gemma at Q8 on the same machine. Mixture-of-experts models route each token through a fraction of their weights, so they pump part of the tank instead of all of it. Size on disk stops predicting speed. A bigger, better model ran faster than a smaller one because of how it's built.

Which model is right depends on whether a human is waiting. When I'm chatting, I'm waiting on each token, and 11 tokens a second reads like a slow typist. Every morning a script on the APU summarizes the front page of Hacker News and pulls the key points out of the YouTube videos I flagged the day before. It runs Gemma 4 31B at Q4_K_M with a quarter million tokens of context, which is what conference talks and long articles need. The results are in Telegram when I wake up. Nobody is waiting on that script. The benchmark put that model at 10.7 tokens a second with an empty context and the 27B at 6.4 with the full quarter million loaded, so a 2,000-token summary takes somewhere between 3 and 5 minutes. I'm asleep for all of them. The card ran out of memory at 64K context on a smaller model.

The desktop is still my daily driver and where most of my development happens. Everything that runs while I sleep moved to the APU, and ComfyUI went with it. The cloud models get the most complex tasks. The local models handle summarization and knowledge extraction, and that split is what keeps my time and my cloud spend pointed at the work that needs the big model.

None of this came from a spec sheet. I downloaded the models, built llama.cpp, wrote the harness, and ran the sweep on both boxes. I've written before that reading about it won't build it, and the two measurements behind every number above take about 15 minutes to reproduce on any ROCm machine:

# memory bandwidth ceiling: BabelStream, HIP build, 100 iterations, 2 GB arrays
hip-stream -n 100 -s 268435456

# tokens per second: 512-token prompt, 128 generated, flash attention on, 5 repetitions
llama-bench -m Qwen3.8-27B-Q4_K_M.gguf -p 512 -n 128 -d 0 -fa 1 -ctk f16 -r 5 -o json

Divide the second number's weight file size into the first and the prediction is on the page before the run finishes.

I read the bandwidth line before I bought the box. The benchmark existed to replace what I had watched and read with numbers from my own models on my own hardware. Big tank, narrow pipe, and for the work that runs while I sleep, that is the right shape.

🤖 AIL LEVELS: This content's AI Influence Levels are AIL3 for the writing, and AIL4 for the images. AI Influence Level (AIL) framework