5 microsecond is a super small duration, you need more time just to switch to the kernel to process your recv system call. 4 ms though sounds a bit long to me.
The fastest you can do is set no timeout at all, use select to check if the socket have a pending packet.