Is there a way I can prevent packets from arriving from a certain source, or any source besides the correct one?
Yes. Just connect()
the socket to that correct source. Then UDP will filter out all datagrams from other addresses. See man 2 connect, the paragraph about SOCK_DGRAM sockets.