Three months ago I was the loudest person in our Slack saying a 7B model could never handle our support queue, and then we ran the same test you did on 12k tickets from our Austin team and the small model hit 91% versus 93% for the giant one. The gap was basically noise once we cleaned up our prompts and stopped shoving every bit of context into the window. I felt pretty stupid after all those arguments, but the truth is the big model was mostly covering for messy data we should have fixed anyway. Now we run the small one on a single card and put the money into better evals and labeling, which actually moved our numbers more than any size jump ever did. Funny how nobody ever got promoted for saying we need less compute.

Ran the same short trips and my analysis showed fuel dilution too, trust the severe schedule.