The "best" way to do what I think (based on the code you provided, I suppose you want the couples (IP source address, IP destination address) rather than IP packets) you want is :
>>> set((p[IP].src, p[IP].dst) for p in PcapReader('file.pcap') if IP in p)
You don't waste memory with the whole PCAP file, the set
object is built packet by packet.
If it's not fast enough, you can instruct Scapy not to dissect packets after the IP
layer since you don't use the IP
payload:
>>> IP.payload_guess = []
And then run the previous command.