9
I was training a model on a set of 12,000 images for weeks and the loss just wouldn't drop.
A friend asked if I'd checked the labels, and it turned out the script that made them had a bug, so half the data was wrong. Has anyone else had a simple data check save them from a long, pointless training run?
2 comments
Log in to join the discussion
Log In2 Comments
the_jenny3d ago
Been there, done that. A basic histogram of label distributions caught a similar issue for me last month. Spent days tuning hyperparameters before realizing 80% of the samples had the same class tag. Now I always plot the data splits before any training starts.
8
dixon.rose3d ago
Classic data check saves the day again.
2