Bouncy Castle, like many other providers ignores the key password for PKCS#12. PKCS#12 is a standard concerning personal credentials exchange, one result of this is that most implementations assume only one password is required for both the keystore and the keys.
It is possible to have keys encrypted in a PKCS#12 file with a different password to that used to seal the keystore, however if you do this it is unlikely that you will find another application that can read the file. Or put another way, trying this is not a crazy idea, it's just that the KeyStore API and PKCS#12 are similar but different, so you will find idiosyncrasies exist - in some ways PKCS#12 is more fully featured as it allows the attachment of attributes to the objects stored in the file in a much more general way than the KeyStore API does.
We (as in Bouncy Castle) have recently attempted to provide a more general way to deal with this in a separate API for PKCS#12 which is in the current beta for 1.49. It will allow you to use different encryption passwords if you really want, but I would recommend against it if you wish the files you produce to be understood by anything else. The PKCS#12 API does allow you to make better use of attributes, but whether that is something you really need to be able to do (often you don't) I will leave to you to decide.
You will find that other formats like JKS allow for different passwords for keys and keystores. As I mentioned earlier though, the difference here is that JKS was not defined as a "personal" storage mechanism, unlike PKCS#12.
Regards,
David