I agree it could be drivers and I would suspect the Nvidia ones but, I have also come across similar situations with network cards on the PCI slots causing these issues as well.
It is really a process of elimination and the easiest, and cheapest, is to try removing the network card. It doesn't cost anything and, if the problem is still there, you will know it isn't that.