I was training a model on a set of 12,000 images for weeks and the loss just wouldn't drop.
A friend asked if I'd checked the labels, and it turned out the script that made them had a bug, so half the data was wrong. Has anyone else had a simple data check save them from a long, pointless training run?