I am not sure that is possible. As per specs:
If a src attribute of a media element is set or changed, the user agent must invoke the media element's media element load algorithm. (Removing the src attribute does not do this, even if there are source elements present.)
So if you touch the video.src the browser should invoke implicitly video.load(). In your case (setInterval) Chrome does this.
I guess you already went the route of saving the currentTime of the video before changing the src and applying it after the src change (wait for the canplay event in this case and call video.play() to resume playing)? I guess you would have some stuttering for 4 seconds refresh in your case.
It seems that you are trying to emulate a live stream as an on demand feed and I do not know a way to do this with progressive download of mp4 (read with un-fragmented MP4).
Related article.
Thanks