You'll need to show the command-line used to run rsync. I'm guessing it's one of two problems:
- Your rsync command is not running over ssh (See the "-e" option in the manual)
- Your ssh connection is using an alternative ssh key
For an example of running rsync over ssh (using ANT) see:
Finally, if all else fails (and provided rsync is using ssh), you could check the sshd log file on the remote server to see how the connection is being authenticated using ANT.