There is no technical difference between the front door of your home and your phone outside the neccesity to be in a specific location. They are all points of contact in which you can ignore, hang up or slam the door shut, or engage the person trying to contact you. In a sense, outside of being inconvenient at times, it is little different than post mail.
If you told politicians they couldn't knock on your door or send you mail, would they have standing to complain? The bottom line is that if you have a means to be contacted, barring politicians from contacting or attempts, you would violate their first amendment rights. You do not need to sit and listen.