It appears I was wrong, what takes time isn't to compile the templates. The browser is just slow to load all the page and basically the document.ready event took some time to be fired.
To fix this I only include my main template <script type="text/ng-template" id="/myMainTemplate.html">...</script>
in the page, and I load the other templates using an asynch $http call, based on this solution : https://gist.github.com/vojtajina/3354046