Turns out the issue was CSS related. I had "position:fixed" on all web content. After changing it to "absolute" the problem was solved.
In my point of view it's still an unexpected behavior, because "fixed" is supposed to position things relative to the browser window. What is expected is that, as the webview is scrolled left, the "browser window" moves past the device screen.