Ditching cloud APIs for a self-hosted LLM stack on a 48GB Mac mini

A developer details a fully local LLM setup running on an M4 Pro Mac mini with 48GB of unified memory. The stack pairs a Qwen3.6-35B-A3B mixture-of-experts model for reasoning tasks with a lightweight Gemma-4-E4B model for routine work, served through the oMLX inference server and reached from an iPhone and MacBook over a private Tailscale mesh. Client apps like Telegram, Raycast, and an iOS chat tool all hit the same endpoint, giving every device a shared backend and conversation history without exposing anything to the public internet.

The motivation is largely about control rather than raw capability. The author frames commercial APIs as ‘rented land’ subject to unannounced price hikes, rate limits, and silent model swaps, and raises data-privacy and ‘AI sovereignty’ concerns—pointing to government-imposed restrictions on model availability as a risk you can’t mitigate without owning the hardware. Local inference also brings flat costs, low latency, offline operation, and no throttling. The stated goal isn’t to replace frontier models like GPT-5 or Claude Opus, but to absorb the roughly 80% of daily queries that don’t need them.

The more technically useful portion is a primer on sizing models for consumer hardware. The key insight is that MoE models are cheaper to run than their parameter counts suggest: the 35B-total Qwen model activates only ~3B parameters per token, so its memory footprint (~20GB at 4-bit) behaves closer to a much smaller dense model. The author walks through decoding model identifiers, estimating RAM needs (quantized file size plus OS overhead plus KV-cache room, with a 10–15% buffer to avoid SSD swapping), and notes that OptiQ 4-bit quantization costs only 1–2 benchmark points versus 16-bit. Swapping in newer models is reduced to a download-and-restart via oMLX’s built-in HuggingFace browser.