Adding and removing tracks on a MediaStream object do not signal a renegotiation and there are also issues with a MediaStream having two tracks of the same type in chrome.
You should probably just add the separate mediastream to the peer connection so that it can fire a re-negotiation and handle the streams. The Track add/remove functionality in chrome is very naive and not very granular and you should move away from it as much as you can.