Link your duration to the pixels you are having to move.
The duration in your code is locked at 500
. If I get the amount of pixels that have to be moved in either direction and multiply it by some milliseconds you can get a set speed at which the page will scroll.
Replace this:
$('html,body').animate({scrollTop:$(this.hash).offset().top}, 500);
With this:
$('html,body').animate({scrollTop:$(this.hash).offset().top},
Math.abs(window.scrollY - $(this.hash).offset().top) * 10);
Edit the 10
above to increase or decrease the duration.