So after comparing the headers of an IDP initiated request and an SP initiated request, I noticed a difference. The IDP initiated request was missing the wctx parameter. Once I included this in my relaystate, the WIF RP app worked fine.
wctx=rm%3D0%26id%3Dpassive%26ru%3D%252FMyDesiredStartPage.aspx