The trouble with relying on QoS is that this won't help a lot of users (particularly not-the-fastest DSL users) when someone, say, joins a torrent: the incoming requests will end up swamping the DSL modem's uplink. That is, the congestion is not between the client and the AP, it's between your next-hop and your modem. Your wireless AP's QoS controls are helpless to regulate this traffic. Slowing down the traffic between the AP and the client will maybe discourage your neighbor from attempting to use the line on the torrent, but it won't have a significant effect on decreasing the traffic to the DSL line, and if you start dropping more packets per QoS policy, it will just result in more retransmissions.
This all gets a lot easier when everyone has significantly faster lines, but ultimately this is not a problem that current technology does a great job of solving. Specifically, this gets easier (but is still a far cry from solved) when the bandwidth of the wireless fabric is about the same as the bandwidth of the ISP uplink.
It is also worth pointing out that even if your neighbors don't share your internet connection, if their wireless AP shares your channel they share your wireless bandwidth. But that is the wireless fabric bandwidth, which tends to be more abundant.