Or does the certificate only contain the public information and is the "private" information still in the keystore?
Yes.
is it a wise thing to implement my own TrustManager on server side so that i can manage trusted client certificate easily?
No. You should let the truststore system do what it wants to do for authentication, and then use a handshake listener to get the peer certificate to authorise it. Don't confuse or conflate these two steps: they are distinct, and both parts are required.