Latency is hairy because there are a bunch of ways to deal.
For example I us Aardvark Q10's and Sonar in a fairly decked out AMD system. I have zero* latency...
OK before everyone goes crazy, I have zero latency (maybe that's 2ms really) because I do not use sonar or the cpu for monitoring.
All monitoring of what I am currently recording (like doing a vocal track) goes into the Q10 and right back out of it into my headphone amp where it is mixed with the music coming from Sonar.
Some people do not like this because it means (for most intents and purposes) that what you are tracking at that time will come through the headphones dry.
Well, I have trouble singing in tune so dry is good for me because nothing is hidden.
Now, if what I say sounds like a pile of crap, then I would defer to the other posters who shared information that seems right in line with what I experienced before making my change over.
Jim