mixture-of-experts
2 posts
Article
Rewriting weights live on 8GB of VRAM
mini-AGI trains a byte-level MoE model from scratch on an 8GB GPU, keeping weights on disk and paging experts in, learning as it reads.
Article
One billion fire, eight billion sit in memory
Liquid AI's LFM2.5-8B-A1B is an 8B edge MoE firing ~1B active params per token, with 128K context, reasoning-only output, and fast CPU and GPU inference.