I managed to figure this out and I'll leave this here for others to find.
The payload starts with an iphdr struct. The iphdr struct has a protocol field, for example tcp, if it is tcp then the data after the iphdr struct is a tcphdr struct, if it is udp, then there's another struct hdr for that, etc for icmp etc.
To access port, assume q_data is a pointer to a nfq_data struct:
unsigned char *data;
nfq_get_payload(q_data, (unsigned char**)&data);
struct iphdr * ip_info = (struct iphdr *)data;
if(ip_info->protocol == IPPROTO_TCP) {
struct tcphdr * tcp_info = (struct tcphdr*)(data + sizeof(*ip_info));
unsigned short dest_port = ntohs(tcp_info->dest);
} else if(ip_info->protocol == IPPROTO_UDP) {
//etc etc
}