Agreed. Even if you're not talking to another person, it's a sign of respect to the other person that you're ready to give them your full attention.
My brother does something similarly rude (and I don't think he even knows it.) When we go out to eat, he'll take his phone out and put it on the table. Now sometimes it stays in his pocket, but not always. The next time he does this, I need to ask him why it's out on the table.