The reason the files in css folder were retrieved on every request was because my website required login and has anonymous access denied.
In my web.config I had explicitly allowed all users to App_Themes folder and that is why it seemed to be cached. So I added the following to my web.config and no more additional request for those files.
<location path="css">
<system.web>
<authorization>
<allow users="*" />
</authorization>
</system.web>
</location>
Those css files weren't used on the home login page so never occurred it could be the access issue.