I would suspect that the issue is not related to the certificates. If your BusinessWorks application did not trust the external vendors certificate you would get an entirely different error message.
A more likely suspect would be that your BusinessWorks application and the external vendor server could not find a matching protocol that both sides accept (such as SSLv3 or TLSv1). Try connecting using your favorite web browser or use tools such as openssl to find out which protocols the server supports.
Update: Try to execute the following commands and see if you get a handshake failure for any of them
openssl s_client -connect server:port -ssl2
openssl s_client -connect server:port -ssl3
openssl s_client -connect server:port -tls1
As an example, BW does not support SSL 2.0, so if that is the only protocol the server supports you need to modify the default security provider by setting the following property to your deployed .tra file
java.property.TIBCO_SECURITY_VENDOR=j2se