Known Problem (Score 1)

Basically the issue is that your router can prioritize outgoing packets, but your ISP controls the priority of incoming packets. The only sensible solution is to bandwidth limit your other uses (most bittorrent clients can do this) so that there's always spare capacity for VOIP.