Most delays are due to users connecting to their ADSL modem via Ethernet and not traffic managing properly.
On a congested link this can cause large delays as Ethernet normally has a 1000 packet buffer in the Linux kernel and the ADSL modem has a similar buffer. You only need a couple of heavy connections which want to go faster than the ADSL will support and those buffers start to fill up real fast. You can easily end up with latencies measured in seconds if you have a lot of connections running (say bittorrent).
There are several solutions to this but the best in my experience is to change the queuing discipline to SFQ and rate limit using HTB. This has been in the kernel for years and works extremely well. You need to limit the traffic upstream and downstream to slightly less (5% less) than the ADSL link speed. This ensures that the modem never queues traffic. Uplink you can use all sorts of fancy queuing but downlink all you can really do is policing of traffic unless you install the IMQ patch to the kernel.
I've a script which I got from somewhere a while ago, don't remember where though. I've put it at http://ams1.x31.com/~andy/ppp0-ratelimit.sh if anyone wants to look at it. It expects to work on ppp0 but can be adapted as required.
I've played a lot more recently with Linux kernel disciplines and it has produced surprising performance on congested links. One link is running mail, remote access and Internet access over a 1mbit symmetric link for about 60 users. in the morning it hits 95% link capacity at the start of work and stays there until everyone goes home but ssh sessions are fully interactive without noticeable lag all this time. Yes web browsing is a little slow but it is the same for everyone and one user can't flood the link and upset everyone else.
Linux QOS is the future, pity about the documentation