This seems to be caused by USB, indeed.
Try routing to Analog and see if the problem recurs, if it does, it's caused by USB buffers running out.
The reason for this is many. On OS X, the primary cause using Live seems to be hyperthreading. Disable hyperthreading. You can also try disabling power saving options. I've got a core i7 processor, and this solved the same problem. You get in, it works for 10 minutes, and then distortion starts.
There is another strange thing with live. I've noticed that whenever CPU usage goes above 50% in Live, the Virus sync/crackling/FUBAR errors start. This MIGHT be due to a bad USB chip but I don't have high hopes for that. I think that has more to do with the kernel. When you're running Virus, try to run a program that accesses disk frequently. That will also cause Virus crackling/sync errors.
Somehow USB transfer doesn't have high priority. I don't know if this is the same in OS X 10.6.x. But I'm pretty sure there is a strange behavior in OS X 10.7.x.
I also completely disabled the dynamic pager! That works very well, and it doesn't break memory if you have enough RAM.