I think I found the answer but I hope someone can verify, seems to work in the simulator.
I removed the issue by setting both of these values to disabled. It wasn't enough to just set one to disabled and the other to hidden, they both have to be disabled
VerticalScrollMode="Disabled"
VerticalScrollBarVisibility="Disabled"