If you tell a thread to run with a non-set affinity, then it'll be allowed to run on any core. This means however, that when one core is busy, it'll move your thread onto a different core, this stopping and possible moving is called a Context Switch. In most cases you'll never notice it, however, in cases like gaming consoles, context switch's can be a surprisingly expensive process.
In these cases it might be better to move something like the audio loop and the video loop onto "private" core's where they are locked to that core, and as such won't switch, giving possible optimisations.