So here is how you do it: You don't.
The reason why I wanted to use the controller() function instead for path() is because Symfony will protect the URL from controller() from unauthorised requests. What you should do is to use path() and prefix the URLs with "esi/" and then protect that URL in your security.yml.
//app/config/security.yml
security:
# // ---
access_control:
- { path: ^/esi/.*, roles: IS_AUTHENTICATED_ANONYMOUSLY, ip: 127.0.0.1 }
If you want to clear the cache you just use the url as you normally would.
Thank you @jongotlin on Twitter for helping me with this.