Specialist CNNs vs Frontier LLMs for Image Classification

A dermatologist recently put 200 biopsy-proven skin lesions in front of two frontier LLMs and a fine-tuned copy of Inception v3, a convolutional network Google published a decade ago. The CNN got 62% of diagnoses right. The LLMs managed 42% each. On melanoma specifically — the diagnosis you least want to miss — the old network won too. If your job involves image classification at any scale, this result should recalibrate how you think about the specialist-model-versus-LLM decision, because it's not really a story about medicine. It's a story about picking the right tool.

The problem: classification looks easy for LLMs, and isn't

Here's the trap that smart engineers keep falling into. Frontier models write code, summarise contracts, and pass bar exams, so it feels almost insulting to suggest they can't look at a photo and pick one of six labels. Classification is the "hello world" of machine learning — surely the biggest models can do it.

But look at what classification actually is. You have a fixed, usually small label space. You have a repeatable input distribution. And you need a decision boundary that puts subtle, high-stakes cases on the right side of it, consistently, every time. That last part is where multimodal LLMs fall apart. They weren't trained to carve fine distinctions inside one narrow visual domain; they were trained to say something plausible about essentially any image ever posted online.

The skin cancer results make this concrete in an uncomfortable way. One LLM diagnosed 75 of 200 lesions as basal cell carcinoma when only 25 actually were — including 14 of the 30 melanomas. The other developed a fondness for actinic keratosis, calling it 53 times against 20 real cases. These aren't random errors. They're the signature of a model that has read a great deal about dermatology, has seen whatever lesion images happen to be in web-scale training data, and is pattern-matching to the answer that sounds most likely rather than computing one.

A model that has read every textbook is not the same as a model that has seen 68,000 labelled examples of the exact thing you're asking it about.

Failed attempt one: just ask the frontier model

The natural first move is the one the dermatologist tried: give the LLM the images, ask for a diagnosis, a confidence score, and a second choice. It's a fair test — that's genuinely how a clinician would use these tools. And the failure modes are instructive.

First, confidence is meaningless. In one case an LLM called a thin, low-pigment invasive melanoma a basal cell carcinoma with 95% confidence. The number is generated the same way the diagnosis is: plausibly. There's no calibrated probability under it. If your downstream logic thresholds on confidence — and most triage logic does — you're thresholding on fiction.

Second, the errors aren't conservative. Each run happens once and answers vary between runs, which matters enormously for anything you'd put in a pipeline. A specialist classifier gives you the same logits for the same input every time; an LLM gives you a sample from a distribution over answers. For a decision support tool, nondeterminism is a liability, not a quirk.

Third — and this is the subtle one — the LLMs were excellent at the adjacent task. Asked as a patient, "do I need to see a doctor?", they said yes to 98–99 out of 100 biopsy-proven melanomas, and to nearly every harmless lesion too. That's the behaviour of a model trained on text about medicine: err toward the doctor, always. Useful for a chatbot triage screen. Useless for discrimination, which is the actual task.

Failed attempt two: assuming bigger means better at everything

There's a deeper failure here that isn't in the experiment at all: the assumption that scale is a substitute for fit. A frontier LLM has, conservatively, thousands of times more parameters than Inception v3's ~24 million. None of that capacity is allocated to telling a seborrhoeic keratosis from a squamous cell carcinoma at the pixel level. For the model, dermatology is one drop in an ocean of training data, and dermoscopic images — the modality that carries most of the diagnostic signal — are rarer still.

This is the same reason a 30-million-parameter Go network beats any LLM at Go by an absurd margin. It's also why benchmarks keep misleading teams about where LLMs actually help in production — a point we made when looking at why AI benchmarks miss most of real work. Capability in the modal sense ("the model can sometimes do this") is not capability in the engineering sense ("the model does this reliably enough to build on").

There's a useful heuristic hiding in the discussion around this experiment: a frontier LLM is likely to be genuinely good at a task if that task is massively overrepresented in pretraining (code, everyday English) or it was specifically post-trained on it with reinforcement learning. Otherwise, expect the performance of a bright generalist who has read a few books and is confident anyway. Visual diagnosis of rare things sits squarely in the second bucket.

What actually works: a small model fine-tuned on the exact distribution

Now for the part that should embarrass anyone who thinks specialist models require a research lab. The CNN that won wasn't state of the art. It was Inception v3 — a baseline architecture — pretrained on ordinary photographs, then fine-tuned end-to-end on 68,011 dermatology images from a public archive, on a single H100, in about twenty minutes. Care was taken to remove every photo of the test lesions and their patients from the training data. The result runs in a phone browser via WebGPU without uploading anything.

Twenty minutes on one GPU. Compare that to the cost of running a frontier multimodal model over your image corpus at production volume, per image, forever. The specialist model isn't just more accurate; it's cheaper by orders of magnitude, deterministic, auditable, and small enough to ship to edge devices. If you can frame your problem as N labels over a stable input distribution, and you have (or can get) labelled data, the fine-tuned classifier wins on nearly every axis that matters in production.

The modern version of this playbook doesn't even require an old CNN. Small vision transformers, embedding models like Google's medical image encoders, or a fine-tuned head on a pretrained backbone will get you further. The point isn't Inception v3 nostalgia; it's that a 2015 architecture is already sufficient to beat models with four orders of magnitude more compute, which tells you where the bottleneck actually is. It's the fit to the task, not the size of the network.

# The specialist path in miniature: a frozen backbone + a small head
import torch
import torch.nn as nn
from torchvision import models
# A decade-old backbone is fine. Seriously.
backbone = models.inception_v3(weights="IMAGENET1K_V1")
in_features = backbone.fc.in_features
backbone.fc = nn.Linear(in_features, 6)  # your six labels
# Fine-tune everything on your labelled domain images.
# On one H100, tens of thousands of images: tens of minutes.
optimizer = torch.optim.AdamW(backbone.parameters(), lr=1e-4)
loss_fn = nn.CrossEntropyLoss()  # a *calibrated* probability lives here

And if your label space is fixed but you'd rather keep an LLM in the loop for the surrounding workflow, the honest way to do that is to constrain its output to a decision rather than trust its prose — the same thinking behind constrained decoding for turning LLMs into decision models. But for the classification itself, a trained head still wins.

A decision framework: when the specialist wins, when the LLM wins

After a few years of watching teams make this call badly, here's the framework I'd actually defend. Ask four questions:

1. Is the label space fixed and small? If you can enumerate the answers, a classifier is almost always the right shape. LLMs earn their keep when the output space is open-ended: summaries, explanations, dialogue, code.
2. Is the input distribution stable? A model fine-tuned on dermoscopy images is superb on dermoscopy images and lost on a photo containing a tattoo, a ruler, and a coffee cup. If your inputs are wild, a generalist's robustness has real value.
3. Do you have labelled data, or can you get it? This is the real fork. With 10k+ labelled examples, fine-tuning is nearly free and will dominate. With none, the LLM's zero-shot guess might be your best starting point — but treat it as a bootstrapping tool for labelling, not the production answer.
4. What does an error cost, and do you need calibration? High-stakes, thresholded decisions need real probabilities and deterministic outputs. That rules out vanilla LLM calls, full stop.

Run the skin cancer case through it: fixed six-label space, stable clinical imaging pipeline, 68k labelled public images, life-or-death errors. Four for four. The LLM never stood a chance, and the surprise isn't that it lost — it's that anyone expected otherwise.

The honest caveats, because pragmatism cuts both ways

I'd be doing you a disservice if I left it at "small models win." The fine-tuned classifier's strengths are also its brittleness. It was benchmarked against an un-fine-tuned generalist on data drawn from exactly the distribution it trained on. Take it into the wild — different cameras, different skin tones underrepresented in the archive, images with occlusions or annotations — and performance degrades in ways the LLM's broad training sometimes handles better. The generalist fails predictably within a distribution; the specialist fails catastrophically outside it.

There's also a task the classifier can't do at all: the rest of medicine. The same experiment's most interesting secondary result was that LLMs, given journal notes, patient history, and the ability to ask questions, could plausibly contribute to interpretation, communication, and management — everything downstream of the label. A CNN outputs six logits. It will never explain its reasoning to a worried patient or notice that the referral letter contradicts the image.

And note the study's own honesty: state-of-the-art dermatology AI isn't a lone Inception v3, it's ensembles of vision transformers trained on far more data, and regulated products built that way perform considerably better. The direction of travel in the field is bigger specialist models, not generalists — which reinforces the point rather than undermining it.

The real insight: LLMs are for building the classifier, not being it

The sharpest observation from the whole episode is one I keep coming back to: LLMs have driven the cost of building dedicated classifiers toward zero. Writing the fine-tuning script, cleaning the dataset, building the evaluation harness, generating the labelling interface — this is exactly the work LLMs are genuinely superhuman at accelerating. Even the dermatologist's analysis was produced with an AI coding tool. The generalist built and scored the experiment that the specialist won.

That's the architecture I now recommend by default: LLM in the loop for development, data work, edge-case triage, and the human-facing explanation layer; a small fine-tuned model for the repeated, high-volume, high-stakes decision itself. We've seen the same pattern in efficiency terms — the interesting story is often not the flagship model but the cheap tier doing real work, as with flash-tier models and inference efficiency. Match the model to the shape of the problem, and most of the "AI doesn't work" complaints evaporate.

A calculator outperforms every LLM at division, and nobody writes breathless posts about it. A fine-tuned classifier outperforming a chatbot at classification is the same story with more parameters. The only reason it surprises anyone is that we've spent two years being told scale dissolves the need for specialisation. It doesn't. It just makes specialisation cheaper to build — if you're willing to reach for the boring tool.