It depends how the Smooks team have implemented their Camel component. The Camel error handler only kicks in, if an exception was thrown, which Camel can catch; or there was an exception explicit set on the Exchange using setException. If Smooks do not do that (maybe they catch the exception, and dont propagate that back to Camel), then Camel cannot detect that exception and react upon it.
If you want to see what goes on at runtime, you can enable the tracer http://camel.apache.org/tracer
Also mind that when you use a bean to handle the exception with the error handler. Then read this FAQ how to get access to the caused exception: http://camel.apache.org/why-is-the-exception-null-when-i-use-onexception.html