Manual doesn't bother me in the same way. If I'm shifting gears, that's fine - the car does exactly what I expect it to do.
The thing that annoys me is shift pauses in automatic. The only thing I told it to do was go faster, and it takes a break do do something else. On some cars, this can be like a 1.5 second pause.