You're right, I'm in Australia. Google does have a presence here and I get a ping of about 64ms to 126.96.36.199.
8 google-public-dns-a.google.com (188.8.131.52) 82.579 ms 64.420 ms 65.664 ms
I've tried the 184.108.40.206 resolver a couple of times, and in all cases iTunes will give slow downloads, simply due to not optimal resolution of the CDN host. Switch it to another DNS resolver, and everything is fine again.
Querying the DNS of our ISP (Internode):
#nslookup a1.phobos.apple.com 220.127.116.11
a1.phobos.apple.com canonical name = a1.phobos.apple.com.edgesuite.net.
a1.phobos.apple.com.edgesuite.net canonical name = a1.da1.akamai.net.
8 203-206-129-16.deploy.akamaitechnologies.net (18.104.22.168) 81.438 ms 67.101 ms 67.139 ms
This kind of issue isn't exactly Unknown.
In addition, in Australia we have quotas for most of our internet plans. If you were on an ISP such as iiNet, then you could end up using up your quota since iiNet provides "unmetered" downloads from iTunes, on the condition that it comes from their mirror. iiNet mirrors Apple servers, and uses their DNS to redirect to those own mirrors. If you were to use 22.214.171.124 for someone on iiNet, you'd end up with them being charged extra since it probably wouldn't resolve to their mirror.