Soup is an open source tool that enables fine-tuning an 8 billion parameter model on a 4 GB laptop GPU. Llama-3.1-8B-Instruct on RTX 3050, 3.32 GB peak VRAM, 119.6 tokens per second. The numbers are reported by the authors on the GitHub repository.

The technique is called layer streaming. Instead of loading the entire model into video memory (which wouldn’t fit on 4 GB), Soup keeps the base model out of VRAM and feeds it to the GPU one decoder layer at a time. The result, according to the authors, is bit-exact compared to a normal run with the model resident in memory.

For anyone who wants to adapt a model to their own data without sending it to the cloud, the point is direct. Fine-tuning required dedicated GPUs, often rented by the hour. If any ordinary laptop can do the job, the gap between “I use a generic model” and “I train my own model” becomes shorter. It’s the same thread as local inference in the browser: useful AI gets closer to the hardware you already have, except here we’re talking about training, not just running inference.

The tool is in beta. Layer streaming is opt-in (stream_layers: true). The numbers are measured on a single hardware setup (RTX 3050 Laptop 4 GB) and not yet independently replicated. Version 0.72.4 added support for DPO, ORPO, SimPO, and KTO, all via layer streaming, with the DPO reference model obtained for free by reusing the same base stream.

Apache-2.0 license, available on GitHub.

In detail

Fine-tuning an LLM has a simple bottleneck: video memory. An 8 billion parameter model in NF4 precision (4 bits per parameter) takes up about 4 GB just for the weights. On a 4 GB GPU like the RTX 3050 Laptop, there’s no room left for gradients, the optimizer, and the input batch. The model doesn’t fit.

Layer streaming bypasses the problem directly. The base model stays frozen (not updated) and loaded into CPU memory or disk. Soup feeds it to the GPU one decoder layer at a time: load the layer, compute the activation, unload it, move to the next. The GPU sees only one layer at a time, not the entire model. What stays in VRAM are the LoRA adapter parameters (small, a few tens of millions) and the current batch.

The authors report 3.32 GB peak on RTX 3050 4 GB with Llama-3.1-8B-Instruct quantized NF4, LoRA, batch 1, 512 token sequence, at 119.6 tokens per second. The strongest claim is that the result is bit-exact compared to a run with the model resident in VRAM: same outputs, zero difference. If confirmed, this means layer streaming introduces no approximations, just a reordering of when data transits through the GPU.

Version 0.72.4 extended layer streaming to preference learning methods (DPO, ORPO, SimPO, KTO). The interesting point is the DPO reference model. DPO normally requires two copies of the model: the one you train and a frozen reference copy for comparison. Two copies on 4 GB are impossible. Soup reuses the same base stream with the LoRA adapters disabled: one set of weights, one stream. The cost is paid entirely in compute time, without hitting memory: DPO reads the layer stack 1.52 times more often per step than supervised fine-tuning. Peak VRAM for streamed DPO is 0.914 times that of SFT, still within the 4 GB limit.

The authors are honest about what doesn’t work. GRPO and PPO remain excluded: token generation requires rereading every layer, and layer streaming becomes prohibitive in terms of time. KTO, often described as reference-free, actually uses the same reference scheme as DPO, and Soup treats it accordingly.

The limits are clear. The numbers come from a single hardware setup, measured by the authors themselves. Layer streaming is marked BETA and opt-in. The 119.6 tok/s speed is for batch 1 and 512 sequence: on real data, with longer sequences or larger batches, performance changes. The repo has 186 stars on GitHub and 60 open issues, signaling early adoption and real friction in daily use.

The signal, though, is interesting. If layer streaming delivers on the bit-exact promise, local fine-tuning on consumer GPUs stops being an experiment and becomes a practical option. For anyone with sensitive data they don’t want to send to the cloud, or who wants to iterate quickly without paying for GPU hours, the difference matters.