I ran a 122-billion-parameter model on a $400 GPU without changing a single weight

The silent trade everyone makes

Every practical way to run a big model on small hardware quietly changes the model: int4/int8 quantization, truncated router top-k, speculative decoding whose greedy output diverges from the real thing. For chat, nobody cares. But for research, auditing, evals, and interpretability, it's a methodological bug: you end up publishing findings about a model nobody is actually running.

Tarfa's contract is the opposite, and it's written down in EXACTNESS.md:

• Expert weights are read bit-identical from the original BF16 checkpoint, verified by SHA-256, per expert, per file.
• The router keeps its native top-8. No truncation, ever.
• Anything that trades exactness for speed is a separate, labeled mode, never a default.
• Every performance change ships a validation script proving exactness, or it gets rejected.

How it works

A 122B Mixture-of-Experts doesn't use 122B parameters per token. Qwen3.5-122B-A10B has 48 layers × 256 experts, and each token routes through only 8 of them per layer, about 10B active parameters. So don't shrink the model. Stream exactly the experts each token asks for, from NVMe through pinned RAM into VRAM, as one managed memory hierarchy:

NVMe (JNF v1: aligned, SHA-256-verified BF16 experts)
  └─ parallel pread → pinned RAM ring
       └─ non-blocking H2D → VRAM
            ├─ learned expert-heat ledger (residency)
            └─ per-layer top-8 routed compute

The on-disk format (JNF v1) makes every expert projection independently addressable at a 4096-byte boundary, with a hash for every expert and a --verify mode that re-reads every emitted byte against the source checkpoint. Full design: ARCHITECTURE.md.

The numbers, honestly

Everything below was measured on one RTX 4060 Ti (16 GB, ~$400) with a PCIe 4 NVMe, by validation/bench_repo.py, with the JSON receipts committed in the repo.

Exactness costs ~5× in speed, because exact mode moves ~19 GB of verified expert weights from NVMe per decoded token. I think 0.35 tok/s for a provably-unmodified 122B on a $400 GPU is remarkable. I also think you deserve to know it's 0.35 and not “up to 2.”

A detail that cuts both ways: on my benchmark prompt, the fast int4 mode's greedy output matched exact BF16 token-for-token. Quantization often agrees. But the project's own Phase 9 audit proved it doesn't always, which is precisely why fast mode carries a permanent label instead of a footnote.

The chart that surprised me

While running a 122-billion-parameter model, the GPU averages 26% utilization at ~32 W, a fifth of its 160 W budget. The GPU is waiting. The bottleneck is the storage path, which is why the engineering effort went into preadv at high queue depth, pinned buffers, and transport overlap, not kernels. The ceiling on consumer hardware is engineering, not silicon.

And the full energy bill, measured at 500 ms resolution (receipts):

91 joules per exact token vs 22.5 fast. Exactness costs 4× the energy, not because it draws more power (it draws less), but because each token takes longer.

An engine that keeps its failures

The part of this project I'm most attached to is the ledger of rejected optimizations, published with their numbers in EXACTNESS.md:

• A GPU expert cache that “obviously” should have helped: 3.3% hit rate for 1.8 GB of VRAM. Rejected.
• A 4 GB RAM arena that avoided 24.7 GB of NVMe traffic per request, and still lost 9.9% end-to-end. Rejected.
• Speculative decoding: batched-verification numerics diverge from single-token decode, so its greedy output can differ. Demoted to the labeled fast mode.
• An 11% serving win from completion overlap, with zero numeric change (chart). Accepted.

Benchmarks that only show the wins are marketing. The full phase-by-phase log is in docs/PHASE_LOG.md, and the whole 23-day timeline, from checkpoint download to validated runtime, is in docs/STORY.md.

Try it

git clone https://github.com/syncroot/tarfa && cd tarfa && pip install -e .
tarfa convert --layers 0-47   # build SHA-256-verified JNF from the HF checkpoint
tarfa convert --verify        # re-verify every emitted byte against the source
tarfa serve                   # exact BF16, OpenAI-compatible API
tarfa start-fast              # the labeled non-exact fast mode

You'll need an NVIDIA GPU with ≥12 GB, an NVMe with the checkpoint (~234 GB), Python 3.11, and patience measured in seconds-per-token.

Why I'm releasing this

I'm not a company, and this isn't a product looking for users. There's no support contract, no roadmap, no Discord to answer at 3 a.m. I built this to see if it could be done, and it could.

I'm releasing it, MIT licensed at github.com/syncroot/tarfa, for the people who understand what they're reading: the ones who can look at a phase log, a rejected-optimization table, and an expert-streaming transport and see what's worth keeping. If there's anything here to learn, take it and go further than I did.

And there's a quieter hope underneath. Most of the industry answers “the model doesn't fit” with “buy more hardware”, and the price of that answer is set accordingly. Every proof that a $400 GPU can do honest work on a 122-billion-parameter model is a small argument that the ceiling is engineering, not silicon. If enough of those arguments land, maybe the hardware market calms down a little, and gamers can just buy RAM again.