Running Large Models on Tiny Devices
Smolmaxxing optimizes large language models to run on resource-constrained devices—old tablets, TV boxes, routers, single-board computers—that would normally lack the memory to handle modern AI. The system achieves a notable milestone: running 8 billion and 16 billion parameter mixture-of-experts models with just 1GB of RAM.
The optimization employs several techniques patched into llama.cpp. It builds custom kernels for specific ARM processors to improve efficiency, reduces memory footprint by eliminating rarely-accessed vocabulary entries, and implements flash-based paging for mixture-of-experts layers. Only the most frequently used experts stay in RAM; others load from storage on demand. The approach works across different ARM architectures since it only requires basic NEON instruction support.
Released under MIT licensing, the project includes setup guides for various platforms ranging from Android TV boxes to older iOS devices. The maintainers invite community contributions: users can benchmark models on their own hardware and submit results for a shared database, expanding collective knowledge of practical LLM deployment on constrained systems.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.