If you talk to people over the internet, latency is an issue. Like, you say something in Skype. The person at the other end hears it and replies. By the time you hear the reply, a regime change has taken place and there's a new president in power. Currently internet video chatting over long distances is an unpleasant experience due to the lag.
Callers usually notice roundtrip voice delays of 250ms or more. ITU-T G.114 recommends a maximum of a 150 ms one-way latency. Since this includes the entire voice path, part of which may be on the public Internet, your own network should have transit latencies of considerably less than 150 ms
I'm already getting cross country ping times of 65ms (round trip), so to be compliant with ITU-T G.114, my codec has 235 ms to do its work. I regularly talk on the phone with colleagues on the other side of the country using a VOIP system hosted here, and haven't noticed any latency problem. Even video calls using our Polycom have good latency (but not great, there's still a noticable lag, even when connected to local users)
I find voice latency on cell phones (even local calls) to be far more noticable and annoying than with cross country VOIP calls. I suspect the lag you're seeing is due to client side buffering and codec compression more than network latency.