Honest to god I looked everywhere and finally saw the same quesition on github.
The answer is that you need to specify the tasks they dont get called by useminPrepare
in other words I need to make my task like this
grunt.registerTask('build', [ 'useminPrepare', 'concat', 'uglify']);
instead of
grunt.registerTask('build', [ 'useminPrepare']);
The answer was incredibly frustrating because it is nowhere in the documentation.