Try using ssh
as the transport. Generally, Ansible uses paramiko
which is not as friendly for interactive sessions:
ansible all -c ssh -m ping
If that doesn't work, I didn't see anything on running Ansible with an ssh key pass phrase on the documentation or in the code, so you might have to remove it with something like this:
openssl rsa -in private_key_with_pass_phrase -out private_key_without_pass_phrase