I'm not sure that JTA or JMS alone would solve your problem, since even distributed transactions is still between transactional resources, such as a JMS broker and a database, not between applications.
I would still go for a transactional transport in your case, such as JMS. That would give you "guaranteed delivery", which perhaps would simplify the error handling.
1 c1 -> jms -> server -> jms -> c2 2
4 c1 <- jms <- server <- jms <- c2 3
If you do this right, you can be sure that c1 (and the server) eventually WILL receive a "result" from c2, good or bad.
If C2 crashes during processing and fails to send back a result jms message, the transaction will roll back, locally at c2, and c2 has to try again.
The down side of this solution is that messages might get "stuck", for instance if it cannot be processed at all by c2, however, it will never be lost. If you route a synchronous request (soap, RMI, simple tcp..), you can face a situation where the reply is lost and C1 will never know if C2 processed the message or not. This can to some point be avoided by making C2 idempotent and make C1 able to retry the transaction if there is no reply after some time.
As I see it, there is no "golden solution", but any option would do it pretty good. Good luck