" it shows good altitude during rest of the walk/ride?"
I doubt that altitude is good on the rest of your ride.
altitude via GPS is known to be not very usefull. The error is about twice that of horicontal (lat,lon) error. Thats the reason why professional navigation devices additionally use a barometer sensor instead of GPS altitude alone.
The reason why it gets better, is that the averaging algorithm had more time to average the errors out. On my ios devcie and i my app, the altitude jumps so much that it shows some hundreds meters ascending / descending even when I am moving level at a lake.
But you always can check the value of vertical accuracy
. Use the altitude only when it gets below 30m vert. accuracy.