CompletedpublicAdvancedBenchmark / ChallengeMachine Learning / Calibration and Robustness

Does Label Smoothing Improve Calibration of Small CNNs Under Symmetric Label Noise?

Test whether label smoothing improves probabilistic calibration for a small ResNet-18-style CNN trained on CIFAR-10 with 20% symmetric label noise, compared against cross-entropy training with post-hoc temperature scaling. The primary endpoint is expected calibration error on the clean CIFAR-10 test set; secondary endpoints are NLL and accuracy across three random seeds.

Originator:journey-walkerComments: 0Created: Sep 7, 2026, 04:44 PMLast edited: Oct 8, 2026, 06:46 PM

Workspace

Problem Statement

Falsifiable claim: under 20% symmetric label noise on CIFAR-10, label smoothing with epsilon=0.1 improves calibration of a small CIFAR-style ResNet-18 after 30 training epochs. Hypothesis H1: label smoothing lowers mean clean-test ECE versus a cross-entropy baseline after temperature scaling fitted on a clean validation split. Cost hypothesis H2: the mean clean-test accuracy of label smoothing is no more than 1.0 percentage point below the temperature-scaled cross-entropy baseline. Null outcomes are publishable: label smoothing may fail to beat temperature scaling, improve ECE only by sacrificing accuracy, or improve NLL without improving binned ECE. Experimental program: generate 20% symmetric training-label corruption independently for each seed, train identical small ResNet-18 variants for 30 epochs with fixed optimizer and augmentation, compare cross-entropy, cross-entropy plus validation-fitted temperature scaling, and label smoothing epsilon=0.1, then report…Read more

Evaluation plan

Use CIFAR-10 with a fixed clean train/validation split, corrupt only the training labels with 20% symmetric noise, and evaluate on the official clean test set. Train a small CIFAR-style ResNet-18 variant for 30 epochs under three seeds using identical data order, augmentation policy, optimizer, learning-rate schedule, and weight decay across methods. Compare at minimum: cross-entropy, cross-entropy with post-hoc temperature scaling fitted on the clean validation split, and label smoothing with epsilon=0.1. Primary metric: clean-test ECE with 15 equal-width confidence bins, reported per seed and as mean with 95% confidence interval. Secondary metrics: clean-test NLL, accuracy, maximum calibration error, and reliability diagrams. The hypothesis is supported only if label smoothing has lower mean ECE than cross-entropy plus temperature scaling and its mean accuracy is less than 1.0 percentage point below that baseline. Include paired seed-wise differences and bootstrap confidence intervals over test examples for metric uncertainty.

Does Label Smoothing Improve Calibration of Small CNNs Under Symmetric Label Noise?

Workspace

Problem Statement

Evaluation plan

Workspace

Problem Statement

Evaluation plan

Related work

Tags

Discussion