A 28.9M PARAMETER LANGUAGE MODEL RUNS ENTIRELY ON A CHEAP ESP32-S3 MICROCONTROLLER

This is a striking demonstration of a twenty-nine-million parameter language model running entirely on a microcontroller with no connectivity. The ESP32-S3 has 512KB of SRAM and 8MB of PSRAM, so the model should not fit — but slvDev borrowed Per-Layer Embeddings from Google's Gemma 3n and parked 25 million parameters in a lookup table in slow flash. Each token reads roughly 450 bytes out, activations stay in SRAM, and the dense core sits in PSRAM. The whole 28.9M parameter model quantized to 4 bits is 14.9MB. It generates at roughly 9.5 to 9.9 tokens per second end to end, displayed on a small screen. The measured benefit over a same-core baseline that fits in SRAM is 0.098 nats, about 9.3 percent perplexity improvement. For scale, earlier independent work on the same chip family ran 260 thousand parameters, so this is roughly 110 times larger. The honest limit is that the published language models were trained on TinyStories or a narrow espresso Q&A dataset — they will not answer general questions, follow instructions, write code, or know facts. If you want to understand how small models are deployed on constrained hardware or explore architectural tricks at the memory boundary, this is a very clean example. If you want a general-purpose assistant, this is not it and training your own will take more than a weekend. The single thing most likely to go wrong is expecting ChatGPT behaviour from a 28.9M parameter TinyStories model.