I had similar problems. The best and easiest way to do this would be to rate-limit all TCP transmission/reception to some percentage less than your overall Internet bandwidth, say 80%. That way, all applications (FTP, BitTorrent, HTTP, etc.) won't take up all of your bandwidth, and thus kill your VoIP. The problem that you're having is that you're saturating your uplink or downlink, and that's what's delaying the VoIP packets. You only need to rate-limit TCP, since Vonage (and most VoIP out there) is UDP-based. You want to let the UDP through without touching it. By limiting TCP, you're effectively reserving UDP bandwidth for your VoIP.
You can do this with a fairly inexpensive Cisco router (buy an 800-series off ebay). You can probably do it with DD-WRT as well.
For the true gearheads out there, I worked out something similar, except that my system actually senses when a call is in process and makes appropriate TCP rate-limiting for the duration of the call. Once the call is complete, the full bandwidth becomes available for other things. It's a clever use of Snort, pfSense, and a Cisco router. I call it AATQoS or Application Aware Triggered Quality of Service.
Whatever you end up doing, just understand that your call quality is suffering due to traffic congestion, and if you can alleviate that congestion enough to let the VoIP UDP packets flow, then your call quality will sound great.