So I figured this out. I had two spring services one with IGNORE and one with RETHROW. I was getting freemarker template object from that one with RETHROW and it seems that template object keeps ExceptionHandler setting based on origin configuration.
So when I processed template from RETHROW one in IGNORE one, it still used RETHROW because that template came from that config.
When I loaded template from IGNORE one it kept ignoring even on RETHROW one. This is definitely unexpected behavior for me.