You can create a Custom-control that overrides from VariableSizeGridView and listen, as you do, to the scrollviewer. Since you have access to the latest element of the GridView, after starting loading more elements, you can set the scrollviewer to the that latest position position. Get a visual reference to that element and then call this code:
FrameworkElement focusedElement = FocusManager.GetFocusedElement() as FrameworkElement;
GeneralTransform focusedVisualTransform = parent.TransformToVisual(_scrollViewer);
ApplyHorizontalScrolling(focusedElement, focusedVisualTransform);
private void ApplyHorizontalScrolling(FrameworkElement focusedElement, GeneralTransform focusedVisualTransform)
{
Rect rectangle = focusedVisualTransform.TransformBounds(new Rect(new Point(focusedElement.Margin.Left, focusedElement.Margin.Top), focusedElement.RenderSize));
double horizontalOffset = _scrollViewer.HorizontalOffset + (rectangle.Left);
_scrollViewer.ChangeView(horizontalOffset, 0, _scrollViewer.ZoomFactor);
}
Using and tunning this code, will help you preventing the scrollbar to go to the latest position.