Loading…
18
Pro tip: a guy at a Denver hackathon told me to stop fine tuning and just do retrieval, and he was right
Back in March I was at a small AI hackathon in Denver and this guy named Raj who builds search tools for a living watched me burn four hours trying to fine tune a small model on 900 rows of my own notes. He said flat out, stop that, just embed the notes and pull the right ones into context. I ignored him for two weeks because fine tuning felt like real AI work. Then I finally tried it, took me about 20 minutes to chunk my notes and set up a plain vector store, and the answers were better than anything my fine tuned model ever gave me. The model didn't know my stuff, it just read the right three chunks and answered from those. Cost me about $4 in embedding calls versus the $60 I blew on GPU time for training runs that mostly overfit. I still fine tune sometimes but only when I need a specific tone or format, not for facts. Anyone else find that retrieval beats fine tuning way more often than the hype suggests, or am I just bad at training?
1 comments
Log in to join the discussion
Log In1 Comment
craig.michael3d ago
The line about ignoring him for two weeks because fine tuning felt like real AI work really hit me. I did the exact same thing last year with my recipe collection, spent a whole weekend messing with learning rates and batch sizes and my model kept spitting out garbage that sounded confident. Finally broke down and just embedded everything, and suddenly it was pulling the right recipe and actually answering questions about substitutions. The part that stung was realizing I wasn't bad at training, I was just training for the wrong problem. Facts live in the documents, the model doesn't need to memorize them. That $60 vs $4 gap is real too, I think people just like the feeling of "building" something even when throwing chunks at context does the job better.
2