I really think people are on the wrong track if they are suggesting that the inbound (to your modem) is the problem here and can't be adequately shaped. In my experience when you have cable or dsl with a disproportionately low egress (outbound) in relation to the ingress(inbound), you will have the increased latency you describe.
I think the first step would be to upgrade to the highest level of service your ISP provides short of business class. If you have comcast, we are in the same boat and you should get the 8/768 plan as you are essentially doubling your upload speed.
If you don't have the money, then I would setup a simple test to see how well your connection performs under stress. Since you are testing for VOIP latency, I would use a UDP ping on a machine you give higher priority to, and run a simple speed test on a machine that would be running BitTorrent...it won't simulate the number of connections that machine will put out, but for the purposes of the test you want to see how well your router is shaping traffic when you reduce the upload cap by 60%, 50%, 40%, etc...and you want to get an approximation for the amount of bandwidth being received and sent out. Try adjusting your inbound bandwidth in the same manner.
Now you mentioned that you are prioritizing based on MAC address, and not based on protocol or service. This is not actually QoS, but rather CoS and is only layer2. CoS is really ineffecient at lower bandwidth rates...it's not really meant for that little traffic and if you throw a lot of connections at it, it will definitely screw up queueing.
Prioritize the VoIP traffic by port or application if you can, and try the different algorithms available to your router...I would definitely suggest as has been stated above to try different firmware images to see if you get better results...Tomato may have been the best suggestion given it's apparent inclusion of Layer 7 (Application) matching using the L7-filter projects signatures. If you can make BitTorrent the lowest possible priority that would be good too;)
Something that someone else might not have mentioned, is the number of connections that are allowed to be setup for either a given computer or application. Connection/Session limiting on your router would definitely help out QoS to prevent BitTorrent from opening more and more connections, and basically increasing overhead. That change can be made in iptables on any linux-based router running a modified image or full blown linux(has to be done from the command line).