I was able to achieve it by looking up inet hashtables (code snippet below). I am the server in this example. Make sure you lookup after 3 way handshake is complete.
const struct iphdr *iph;
const struct tcphdr *th;
struct sock *sk = NULL;
struct tcp_sock *tp;
.
.
.
.
sk = __inet_lookup_established(dev_net(skb->dev), &tcp_hashinfo,
iph->saddr, th->source,
iph->daddr, ntohs(th->dest),
skb->skb_iif);
// Sanity checks here.
tp = tcp_sk(sk);