Decade-Old CNN Outperforms Frontier LLMs on Skin Cancer Diagnosis
A dermatologist compared two frontier LLMs (GPT-6 Astra and Claude Fable 5.1) with a fine-tuned Inception v3 CNN from 2015 on 200 biopsy-confirmed skin lesions from the MILK10k dataset. The older CNN achieved 62% accuracy identifying the correct diagnosis versus 42% for both LLMs. On melanomas specifically, the CNN identified 13 of 30 versus 8–11 for the frontier models. Each system failed differently: the LLMs showed systematic bias toward particular diagnoses (one consistently overdiagnosed basal cell carcinoma, the other actinic keratosis), while the CNN tended to misclassify melanomas as benign lesions.
The CNN was fine-tuned in roughly twenty minutes on 68,011 dermoscopy and clinical images from the ISIC Archive, with test lesions excluded beforehand. It weighted dermoscopic images more heavily in its final decision. The frontier LLMs received identical dual-image inputs and default prompts without declining any cases.
This gap between general-purpose models and specialized medical AI reveals a limitation of current frontier LLMs in medical contexts. The author acknowledges constraints: small sample size, all lesions underwent biopsy (making diagnosis harder than typical practice), and potential overlap between MILK10k and LLM training data. Modern state-of-the-art medical imaging systems using ensembles and transformers reportedly perform substantially better than either approach tested.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.