I rarely wear hats, but when I do find myself wearing a hat I generally take it off before doing anything at all, including using the bathroom, eating, walking somewhere, checking my phone, turning a page in my book, etc. It just feels weird to do something while wearing a hat.
I also generally ruin my hat pretty early in the day by throwing it in the air and then failing to catch it, letting it fall onto the sandy and/or wet ground or into the firepit or something stupid like that.