I think there are 2 possible solutions:
1) The simplest one is to implement tputrace(3c) function and libutrace.so and catch tpcall or tpservice there. The downside of this approach is that you will have to turn on tmtrace for you application which I wouldn't rely on for business functionality. It's fine for monitoring, debugging, etc.
2) Implement custom TSAM+ Tuxedo plug-in. You can enable just your own plug-in and skip mondebug, monshm elephants. Or you can skip TSAM+ completely and implement plug-in for undocumented engine/tsam/agent interface.