My buddy Liam spent three hours last week cutting "like" and "you know" from a ten minute podcast clip. He got it all cleaned up, played it back, and it sounded so choppy and weird. The flow was totally gone, just these sharp little sentences. He had to go back and put half of them in again.
Seriously, those filler words are what make talking sound normal. If you edit them all out it just feels robotic and weird. @joeljones, same with your ums, you gotta leave a few in for it to sound like a person.