Loading…
22
Our team bet on bigger models, but the tiny 7B one beat us at 94% on our own test set
I argued for 3 weeks that scaling up was the only path, then a stripped down model running on a single old GPU in our Chicago office matched our big setup on 94% of real tickets, so now I'm the guy eating crow in standup and honestly wondering if half of us are just burning cash on compute for bragging rights.
1 comments
Log in to join the discussion
Log In1 Comment
harperperry4d ago
Three months ago I was the loudest person in our Slack saying a 7B model could never handle our support queue, and then we ran the same test you did on 12k tickets from our Austin team and the small model hit 91% versus 93% for the giant one. The gap was basically noise once we cleaned up our prompts and stopped shoving every bit of context into the window. I felt pretty stupid after all those arguments, but the truth is the big model was mostly covering for messy data we should have fixed anyway. Now we run the small one on a single card and put the money into better evals and labeling, which actually moved our numbers more than any size jump ever did. Funny how nobody ever got promoted for saying we need less compute.
5