Me? I think I have more right not to be in physical pain for hours then they do to recline their seat.
It's not like it reclines enough to make a real difference to them anyway. And they hardly ever do it to "sleep".
In my experience, if I don't recline the seat about half way, I get serious back pain over the course of the flight. This is because the part of the seat with the headrest that angles forward is usually at the top of my back, which forces me to sit angled forward with my back at an awkward angle. So a very tall person's knees may hurt, but also consider some people may get back pain without reclining the seat.