I'm using an i7 too with a Gigabyte motherboard. One thing I found is that you need to disable most of the energy saving, processor throttling and whatever else options your motherboard has as they cause drop outs when they switch. The EasyTune utility causes glitches too, it's fine to leave this controlling the fan speed in stealth mode but the GUI for some reason causes a glitch every time it polls the motherboard.
Also, sometimes when you change the ASIO speed you need to restart your DAW, I use Cubase. I'm not sure exactly why, but it seems to flush the system and works fine. I can happily run at highspeed and get 3ms latency.
Be careful with things like firewalls, virus checkers and whatever else is running in the background. With the buffers down that low it doesn't give you much room for other software to start demanding attention, so you can end up with various glitches creeping in. As a rule I find Fast or Rapid work fine for me.