elem isn't getting updated with the newly compiled element because .replaceWith doesn't return the new element. http://api.jquery.com/replacewith/
The .replaceWith() method, like most jQuery methods, returns the jQuery object so that other methods can be chained onto it. However, it must be noted that the original jQuery object is returned. This object refers to the element that has been removed from the DOM, not the new element that has replaced it
You need to store the compiled element and replace with that.
var compiled = $compile(svg.node)(scope);
elem.replaceWith(compiled);
elem = compiled;