I have to wonder what program you are using to output the audio. On my desktop, there are settings that allow the audio to be routed immediately to output without processing in PulseAudio or anything.
But, more to the point, your latency relies on the program itself much more. I've been doing quite a bit of low-latency audio programming using both ALSA and PulseAudio, and they both get extremely small if you know what you're doing. With ALSA, I've gotten to around 1ms, while I can easily get PulseAudio to approximately 5ms. Using a PC that has two cores, I've gotten the numbers cut down to around 0.5ms and 2ms, respectively.
If you're having a real touchy time getting response that low, you likely have to bump up priority of the process. Using ALSA, simply setting the application to realtime scheduling will do it. With PulseAudio, you're going to need to set the audio server itself to realtime as well as the application (I haven't done too much testing with that, but that seems to be the consensus online). As a parent suggested, you probably want to look at using JACK. It does most of the dirty work for you.
I learned most of this working on a low-latency audio application. Just yesterday I got the thing to route a guitar at low latency using ALSA, and I'm looking to finish up the Pulse portion this next week or so.