The problem was that another bundle was messing with each request checking if the user was logged in or not. If the user wasn't logged in, a redirect response was generated to the login page.
No idea why this was done, I think it comes from an era where the original authors had less experience with Symfony.
But so it proves again, always check the logs. Very thoroughly.