Well, this isn't a good answer to the quesion "How can I determine the cause..." but it's what worked for me. What I did was open up STSSOAP.dll, the assembly containing the Lists service implemention, in reflector. I copied the relevant code to implment my own GetListItems method in my own service. Luckily, the actual Lists.GetListItems method code was minimal and just called other SharePoint methods and even luckier, those methods & members are all declared public so I was able to do this. I then replaced Lists.asmx with my version on the server and attached the debugger to get some info. What I found was my authentication cookie was being used and HttpContext.Current.Session.User.Identity.IsAuthenticated was true. So it knew I was logged in. But I was still getting an authentication error which I could see farther down in SharePoint's code, was converted to a 403.
So I know asp.net considers me authenticated but SharePoint says I don't have permissions. But the service account the app pool is running under is a SP admin so why don't I have permissions? So next I inspected the SPContext.Current.Web.User; SharePoint's current user. It was null! In other words, as far as SharePoint is concerned, I'm still logged in as an anonymous user because I haven't explicitly logged in with a domain user, and an anonymous user doesn't have permissions on the list I'm accessing or most of the rest of the API. The answer by Sean McDonough to this quesion lays it out.
Basically I need to run the involved code with elevated privileges to get the code to run under the asp.net service account that I originally thought it was running under. If using the API, you can use the RunWithElevatedPrivileges delegate. But if you're calling the web services, you can't do that. The few options I could think of were:
- Call the services on the base site that uses Windows Authentication. This would require me to embed the credentials in the client-side Silverlight application which is a security threat so I nixed this one.
- Implement my own web service to get the data for me.
- The web service could access the list using the SharePoint API and RunWithElevatedPrivileges.
- The web service could call the Windows Authentication site's services passing the current credentials or other embedded credentials.
For implementation ease, all I've done so far is the second option under number 2 and it's working. However, it is clunky and I may change to option 1 which I expect will also work.