After doing a bit of research, it seems that PageSpeed works similar to the Google search, in that it indexes your page and looks for images to compress. Seeing as Angular needs to load scripts before pages are loaded, the two just won't work together.
There is a possible solution found here:
http://www.yearofmoo.com/2012/11/angularjs-and-seo.html
For me this won't really work, as I have dynamic search pages that I can't really pre-render with PhantomJS. But for others, this might be useful.
Please alert me here if this ever changes!