Answering to myself because at the time I found a fuller answer.
It is possible to make sure the protocol is IP by the 'protocol' field of the sk_buff struct. For example:
if (skb->protocol != htons(ETH_P_IP)) return ERROR_NOT_IP; //ERROR_NOT_IP defined by me
struct iphdr* ip_header = ip_hdr(skb); //Now that we know the protocol we can use ip_hdr()