If the application in question is not an Eclipse RCP application to which you could install SWTBot I don't think it's doable. I don't think SWTBot can control another process even if it is Java process.
Only thing that comes to mind is some kind of -Xbootclasspath: JAR that would boot some kind of "remote control server" in that application and allow SWTBot to issue commands to it.