Check the project settings, what is the "Default SSH key" listed?
If the value is "var/lib/rundeck/.ssh/id_rsa" then that indicates you have installed rundeck using either a Debian or RPM package. In this case Rundeck executes as a different user, explaining why the key in your home directory doesn't get used.