You will get some better latency by using the virus asio driver in your daw, but who wants to do that.
Actually, you should make sure you are using ASIO at all on Windows systems and make sure your buffer is set to 128 to 512.
If you would like to be able to use just the librarian part of the VC add your voice to this FR :
Just give us a librarian and let us deal with midi sync, latency , audio compensation and for me all that usb crap could go right where it belongs, the trash bin.
Nobody prevents you from using VC just as a librarian, use the analog outs and switch of sample accuracy with the Live Mode button.
Now it^s great, if I need the VC to get some patch or do some advance editing, I need to launch a second vst host, copy the patch to the ram, close that app, and now I can use my TI again from my main audio app. It was more fun 20 years ago
What host are you using that can't use VST plugins? If it can, you can also load VC into your "main audio app".