It turns out that there is a command line utility in Java that do what I need. In the Java runtime directory (C:\Program Files (x86)\Java\jre7\bin
on my machine) there is a utility called keytool
(my newlines for readability):
>keytool -importkeystore
-srckeystore clientStore.jks -srcstoretype JKS
-destkeystore outputStore.p12 -deststoretype PKCS12
This produces an output store called outputStore.p12
that I can load in .Net with X509Certificate2Collection.Import()