Alright, finally got it working, thanks for the tips, Dan! Here are the remaining details that would have saved me a few hours - hopefully this is helpful to someone else.
1) As I expected and mentioned earlier, dev_appserver.py needs to be run with --host=0.0.0.0 so that it can listen on the local network.
2) In order to properly parse the ID token to make the user authentication work, you need to have the PyCrypto library installed - it is not installed by default, so just having it in your app.yaml isn't enough. I found the binaries for Windows here.
3) Then, in my generated Tictactoe.java-equivalent class, I had to change the DEFAULT_ROOT_URL to be http://<my-local-machine>:8080/_ah/api/
(which is where my local machine is running), so that requests went to the local network. Alternatively (and perhaps less invasively), you can use builder.setRootUrl to the same address from wherever you initialize your builder. This way you don't muck with your generated classes.
4) I also had to make the change Dan mentioned described here.
After doing those four things, everything seems to be working now and I can test my app locally - hooray!