I worked it out.
I need to fade back in with fadeTo rather than fadeIn as when fading out with fadeTo, it only adjusts the opacity so still treats it as visible:
$(document).on('mouseenter', '#player', function(){
$(this).stop().fadeTo('fast',1);
});
$(document).on('mouseleave', '#player', function(){
$(this).stop().fadeTo(10000,0.2);
});