Though you've correctly used the non-greedy \{-}
quantifier, because there's no consumption before, it still will start matching at the first x
, and then match as few as possible. Because that works, there's no backtracking.
Now, you need to add a greedy match before your expression, yet do not consume those characters. This can be achieved with \zs
to let the match only start afterwards:
:s/.*\zsx.\{-}x4/foo/