Janas-LLM Outperforms Llama.cpp in Running Large Language Models
Janas-LLM, a tool designed to run large language models (LLMs) on standard computers, has demonstrated superior performance compared to llama.cpp. On a laptop with 32 GB of RAM, Janas-LLM achieved 37 tokens per second (tok/s) with the Qwen3-Next-80B model, significantly outperforming llama.cpp’s 2.8 tok/s. The tool streams model weights from an NVMe SSD, keeping frequently used experts in memory, allowing it to handle models larger than the system’s RAM. Janas-LLM was tested on various models and consistently performed better than llama.cpp, except for the 3-bit Qwen3-30B, which reads prompts slower due to GPU limitations. The tool is available for Linux and Windows, with performance metrics provided for both operating systems.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.