Wait, do you mean leaving the flue open without a chimney cap? Because that's the real issue. A decent cap keeps animals and almost all rain out. If you leave the flue closed all the time, the moisture just gets trapped inside the brickwork, which is way worse for the masonry. The draft you get from an open flue with a cap is actually a good thing, it keeps air moving so things don't get damp and smelly.