Found a way to do what I wanted. The problem was as I suspected around the "EOF / if"
Here's what I found :
#!/bin/bash
sftp -oIdentityFile=path_to_private_key USER@$IP1:PATH <<-EOF1
put myfile
quit
EOF1
if [[ $? -ne 0 ]]; then
sftp -oIdentityFile=path_to_private_key USER@$IP2:PATH <<-EOF2
put myfile
quit
-EOF2
if [[ $? -ne 0 ]]; then
echo "ERROR"
fi
fi
Hope it will help anyone who had this problem