I think I see what you're saying. The issue with Angular, using views, etc., is that a lot of does not have much to do with access control, but instead hiding what the user can or can't see. That's simply because these types of MV* framework are on the client-side. Usually the solution to this seems to be handling things on the server (privately), and then sending them to the client only if the client has access to those resources (by using, for example as you mentioned, a token.)
If you haven't looked into using Node.js, it makes it pretty easy to deal with this issue (Node.js isn't the only option, obviously; any other server can do this, but it will involve configuring the server. With Apache, for example, you can use the .htaccess
file.)
Anyway, if you use the Express engine with Node.js, it sets up the skeleton for your app, using a public folder, which holds only the static content that users will be able to access (and when I say static content, I mean .html, .css, .js files, etc.) If there's something you don't want to show the user unless they have authorization to see it, you can serve that resource up from another location and simply render it and send it off to the client side. Using Jade with Node.js/Express is a great example of this.