What you could do is to use the Java EE event mechanism to get a notification when your transaction fails and create a subsequent output message.
See
- (EE7) http://docs.oracle.com/javaee/7/tutorial/doc/cdi-adv005.htm
- (EE6) http://docs.oracle.com/javaee/6/tutorial/doc/gkhic.html
You need to use a new transaction to write to the out queue to avoid rolling back the message writing as well.
You will still have the message in the input queue which caused the exception, since the rollback will prevent the successfull consumption. You need to handle that separately, for example by the JMS provider.