You need to define a .ssh/config file
which will reference your private key.
Host upstream
HostName my_app_name.rhcloud.com
User 1234567890
IdentityFile ~/.ssh/.ssh
Then use the SCP syntax for ssh url:
git pull upstream:~/git/app_name.git/