Astra Weight Claims Conflate Distinct Techniques

A recent Astra paper claims the model can read raw weights to understand what networks were trained to do, generate game-playing networks that exceed PPO-trained baselines, and improve Qwen3.5-0.8B’s mathematics performance by editing 0.016% of weights (raising MATH-500 from 7.9% to 48.1%).

The critique argues this presentation is misleading because it conflates different experimental methods. Direct weight-reading without tools appears only in early experiments on small networks (17K–172K weights). Every real language model experiment requires Python coding. The Qwen improvement additionally involves running the model on up to 50 batches of practice problems. The paper suggests these demonstrate a single unified capability, but they actually represent distinct techniques. Astra’s practical achievement is writing ML engineering code that humans might write—valuable but fundamentally different from raw weight analysis.