I'd say you're good to leave your towel on the chair in this case. The OP was asking about putting a towel on a chair, then heading to breakfast. In which case, they were not ready to utilize the chair. If you are in the vicinity, ie in the pool/hot tub, then, IMHO, you are utilizing your chair.