Found the problem and fixed it. Bottom line was that I had to configure the default ApplicationPool to use the identity of my authorized user, following the docs here.
I think what gave me trouble is that I had this directory mapped as a Virtual Directory in IIS, and mapped it as that same user. Apparently this doesn't give the permissions to access that info the way I thought it would. Probably not a great idea to change the application pool identity, but for an intranet based site it will get me where I want to go :)