How Language Models Carve Up Vector Space Across Languages
The author built an interactive visualization of Qwen3-1.7B’s internal representations by tracking how token vectors evolve through the model’s residual stream across thousands of conversations. By generating question-answer pairs in English, Chinese, German, Croatian, and Python, the researcher recorded the final vector states after the model processed each example and projected them into three-dimensional space.
The results revealed striking spatial patterns. English conversations compressed into a remarkably thin two-dimensional sheet, with a single plane accounting for 99.3% of all variation in the 3D projection. Chinese conversations clustered in a neighboring but distinctly offset region, while German and Croatian occupied overlapping areas with English across a broader neighborhood. Python code appeared in the same general region as English rather than forming a separate domain.
These spatial separations raise interpretive questions. The model’s representations simultaneously encode language choice, writing system, tokenization strategy, vocabulary structure, and answer quality—factors too entangled to isolate through geometry alone. When the analysis shifted to different dimensionality reduction methods, the visible relationships changed, suggesting that any single geometric picture reflects the analysis choices as much as the model’s fundamental organization.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.