I run Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on one RTX 3060 with 12 GB of video memory and 30 GB of RAM. Nothing leaves my PC. This post is about the fork I made to get there, what I measured, and what I have not tested.
Starting point: Strata
The model is about 78 GB of files at the quantization I use (IQ3_S). It does not fit on a 12 GB card. Strata by Niko1221 and contributors (MIT) solves that by keeping the most-used experts on the graphics card, putting more in RAM, and streaming the rest from the SSD. My fork, Strata-HotRod, makes that path faster and checks that answers stay correct.
What I measured
All of this is on one machine: RTX 3060 12 GB, Intel i5-12400F, 30 GB RAM, a 500 GB NVMe on a PCIe 3.0 link, Rocky Linux 9.6.
| Answer (131K context) | tokens/s |
|---|---|
| Code, 500-650 tokens | 52-55 |
| Reasoning, about 1,000 tokens | 66-71 |
| Prose, 100-150 tokens | 50-53 |
Long prompts, 262K context: an 8,554-token prompt is read in 16.9 s and answered at 52-54 tokens/s. A 45,398-token prompt takes 67-68 s to read, then the answer comes at 38-41 tokens/s. The next turn of that same conversation starts in 0.4-0.5 s, because the saved state is reused. The first message of a chat is the slow part: it waits on the SSD for the model's experts.
What I changed
- A learned look-ahead. The engine guesses which experts the next layers will need and loads them early. Against the plain router, the guesses are right 85-88% of the time one layer ahead (router: 77-79%), 76-79% two layers ahead (66-69%), and 66-71% four layers ahead (55-58%).
- Three tiers: card, RAM, SSD. The hottest experts sit on the card, the next ones in a RAM tier sized from free memory, the rest on the SSD. Budgets and reserves are measured on your PC and kept across restarts.
- Expert placement from routing traces, instead of a fixed guess.
- Stand-in experts with a re-check. To stay fast, some of a token's experts are replaced with similar ones already on the card.
- Pinned RAM segments (on by default): about 9% faster on this PC.
Every change and its measurement is in the repo's docs, including ideas that did not help. For example, a learned read-ahead of lower-ranked guesses made decoding 23% slower on this PC, because the drive was already at its limit, so it stays opt-in.
Is the answer quality still good?
In a small test (50 maths questions, 30 coding questions, answers read by hand), the stand-ins gave 47/50 and 30/30 correct, against 48/50 and 30/30 with nothing replaced. A test this small cannot show a difference of one or two answers either way. On MMLU-Pro (15 questions from each of its 14 subjects, 210 in all, thinking on, temperature 0) it scored 87.1%, 183 of 210. That is a small sample, so read it as a sanity check, not a leaderboard result.
What I have not tested
One card, one model, one PC, Linux only. Other GPUs, AMD, Windows, macOS and other model sizes may work in upstream Strata, but I have not run them, so I will not claim them.
Try it, and tell me what you get
The repo has the install steps and the full change log: github.com/sangharshadhyeta/Strata-HotRod. If you run it on different hardware, I would like your numbers, good or bad. The repo has a short guide for how to report them.
Top comments (2)
Nice that you kept the failed ideas in the docs — the 23% slower read-ahead is exactly the negative result that gets dropped from most fork write-ups. The stand-in expert tradeoff is what I'd want to know more about: how is expert similarity scored at placement time, and does a token that routed to a replaced expert ever poison the KV cache in a way that shows up a few tokens later instead of immediately? The 47/50 vs 48/50 makes me wonder if that one miss was a stand-in or just model noise.
Also, is the RAM tier budget re-measured on every boot or does it trust stored reserves after a hardware change (someone pulls a stick)? Guessing the SSD tier is what to watch when the PCIe 3.0 link is shared with anything else reading the drive.
Thanks Max, these are the right questions. Similarity is just the router's own scores: a missing expert only gets swapped for one already on the card if its weight times the logit gap stays under a budget, and the budget tunes itself while it runs.
Your KV point is the reason the top two ranks are always exact now. When they were allowed stand-ins, an agent session broke after a restart and a 94K conversation leaked a turn marker into the answer, so we did see that one. For the lower ranks, a token a stand-in could flip gets recomputed exactly (about 1 in 5 of those re-checks had really flipped a word), and a request turns its own budget down when it sees flips. MMLU-Pro came out at 87.1% on 210 questions with thinking on, which is a long chain, but that's not the same as tracing a delayed effect. The 47/50 vs 48/50 I can't pin on a stand-in, 50 questions is just too few.
The RAM reserve is measured on the PC and saved between restarts, and every request checks free RAM first and hands experts back if it's short. The SSD does matter too: read-ahead used to queue in front of the answer's own reads, so it's down to 2 I/O threads now.
If you try it on your card I'd like to hear how it goes.
Sangharsh