The reason it was doing that was because the domain name wasn't setup correctly. I was using a custom domain to host the site, and umbraco was obviously not matching the incoming request to a site in the CMS.
By going to the Site and right clicking -> Culture and Hostnames, I could set the hostname.
As per this answer: Umbraco configured with IIS 7 having hard time with the site URL?
Did you set your hostname in Umbraco?
Right click on the site root in your solution and click "Culture and hostnames"
Fill out your hostnames and save