I still don't get why, but I manage it to work.
What I had to do was to play the music(silent sound) for a little way while the app is active and then pause it.
After doing that, everything works if I resume the playback while being in the background.