A researcher ran a language model with 28.9 million parameters on an ESP32-S3, a microcontroller that costs about 8 dollars. It generates text at roughly 9.5 tokens per second, entirely on-chip, with no connection to any server.

The previous record for a chip in this category was 260 thousand parameters. The hundred-fold jump comes from an idea borrowed from Google’s Gemma models: most of an LLM’s parameters live in an embedding table that the model reads, not computes. If you keep it in the chip’s slow flash memory and pull only the few rows needed for each token (about 450 bytes), the “large” model costs almost nothing in RAM.

As we reported on July 20th with 1-bit LLMs in the browser, local inference is moving down the device chain. From Mac to browser, now to the 8-dollar microcontroller without Wi-Fi.

For those building on constrained hardware or doing offline DIY work, the signal is that the threshold for an agent running on-device is dropping. This specific model only writes short stories (trained on TinyStories); it doesn’t answer questions or follow instructions. But the architecture (the memory trick that fits a model a hundred times larger on a chip without expansion) is what matters.

In Detail

The classic problem with microcontroller inference is RAM. The ESP32-S3 has 512KB of SRAM, the fast memory where the chip does calculations. A language model, even a small one, normally needs to fit entirely in that memory to run. 512KB is enough for about 260 thousand parameters, and that’s where the previous record on this hardware type had stopped.

The trick comes from Google’s Gemma 3n and Gemma 4 models, called Per-Layer Embeddings. In an LLM, most parameters (in this case 25 million out of 28.9 total) form an embedding table: the vocabulary the model consults to transform words into numbers and back. The model reads from this table but doesn’t do heavy computation on it. So you can keep it in the chip’s slow but spacious flash memory (16MB), and load into SRAM only the few rows needed for the current token: about 450 bytes per step, six rows of the table.

The part that does the actual reasoning, the core with 3.9 million parameters, stays in SRAM. The table with 25 million lives in flash and gets sampled one piece at a time. The result is a 14.9MB 4-bit model that generates text at 9.5 tokens per second, end-to-end, on a chip consuming a fraction of a watt.

The model is trained on TinyStories, a dataset of short, simple stories designed for tiny models. It writes coherent sentences like “the dog runs in the park,” but doesn’t answer questions, follow instructions, or know facts. The limitation isn’t memory; it’s the reasoning part of the model, and the memory trick doesn’t change that. The project is honest about this: the interest is in the architecture, not in what the model can say.

What changes for readers? Until yesterday, “local inference” meant a laptop, or a browser with WebGPU. Now the threshold drops to an 8-dollar chip with 512KB of RAM and no network connection. For those designing embedded devices, sensors with some logic, offline prototype voice interfaces, the architecture shows that a language model can live inside the device itself. No cloud, no network latency, no API key.

The question remains how replicable this is for useful tasks. The repo includes firmware, wiring and flashing instructions, plus training and quantization code in Python. But the jump from “writes stories” to “follows instructions” requires a more capable model. And a more capable model requires more active parameters in SRAM, which brings us back to the original bottleneck. The memory trick expands the embedding table, not the brain.