I have never used one for hunting but would suspect that it could be an issue under the right (wrong) weather conditions.
I would try, as much as possible, to NOT use it during hunting. Last thing you want is that buck of a lifetime to come by and you cannot see becasue your peep fogged up.