Loading…
9
I was training a model on a set of 12,000 images for weeks and the loss just wouldn't drop.
A friend asked if I'd checked the labels, and it turned out the script that made them had a bug, so half the data was wrong. Has anyone else had a simple data check save them from a long, pointless training run?
3 comments
Log in to join the discussion
Log In3 Comments
the_jenny6mo ago
Been there, done that. A basic histogram of label distributions caught a similar issue for me last month. Spent days tuning hyperparameters before realizing 80% of the samples had the same class tag. Now I always plot the data splits before any training starts.
8
dixon.rose6mo ago
Classic data check saves the day again.
2
webb.xena6mo ago
Ugh, I feel that pain. I once spent a whole week trying to fix a "broken" model before I noticed the labels were just shuffled wrong. My ego still hasn't recovered.
1