Five CNNs, one dataset: what a fair comparison actually teaches you

For the lung-cancer classification project I didn't want to pick a model and hope. I wanted to know which architecture actually generalizes on this data, and why. So I trained five, VGG19, MobileNet, ResNet, EfficientNet, and Inception, on the exact same dataset, with the exact same split.

The result surprised me, and the surprise is the whole point.

The ranking is a lie until the data is clean

My first run had a clear "winner." Then I added proper class balancing and augmentation, re-ran the identical setup, and the ranking changed. A model that looked mediocre jumped ahead; the early leader had mostly been exploiting class imbalance.

The lesson that stuck: on a medical dataset, data hygiene moves the metric more than model choice does. If you compare architectures before you've fixed balancing and leakage, you're not ranking models, you're ranking which model best memorized your bias.

Generalization ≠ the highest validation number

The point of the exercise was never "one model to rule them all." It was to see how each architecture's inductive bias fits the data:

• Heavier backbones (VGG19) overfit fastest on the smaller class.
• The efficient families (MobileNet, EfficientNet) held up better per parameter.
• Inception's multi-scale filters helped where lesions varied in size.

None of that shows up if you only look at a single accuracy figure. It shows up in the error analysis, which examples each model gets wrong, and whether those overlap.

A benchmark tells you who won. Error analysis tells you whether the win means anything.

What I'd do next

The obvious upgrade is Grad-CAM overlays so a prediction is explainable to a clinician, "the model looked here." A number a doctor can't interrogate is a number a doctor won't trust, and rightly so. That's the same instinct I bring to AI evaluation now: don't trust an output because it scored well; understand why it scored well, and whether that reason survives contact with reality.