Try this
- Go into device manager.
- Android device > Google nexus 4 ADB interface.
- Driver > Uninstall. (Make sure to check "delete the driver software for this device").
- Uninstall then take out usb and put it back again. It should work now.
I've always found setting up device debugging to be very temperamental.
Failing that:
In the past what worked for me was downloading the USB drivers.
You can find a list of devices and support from here http://developer.android.com/tools/extras/oem-usb.html
And sepecifically the nexus USB drivers here http://developer.android.com/sdk/win-usb.html