You could try setting the parameter overwrite to false
My.Computer.FileSystem.CopyDirectory(folder, destination, false)
This will raise an exception when, in the destination folder, a file to be copied exist, but the missing files in the destination are still copied.
See the documentation on CopyDirectory(source, dest, overwrite) to know how to handle the exception