I have figured out why this happens. For some reason if you use a style that has no window background, even though you hide/show different fragments all of them will be drawn (for some reason I have no idea why). So my style which I added a
<item name="windowBackground">@null</item>
in order to reduce how many pixels are drawn, was the thing that was blocking me. Who knew.