Just add the difference between seconds:
udiff = (second.tv_sec - first.tv_sec) * 1000000 + (second.tv_usec - first.tv_usec)
You just have to check that the difference between two packets is less than ~2000 seconds to stay in the size of a 32-bits int
.