The issue was in my deploy/production.rb, I had to mention the ssh port when defining the server. So, this will look like this:
set :stage, :production
set :branch, 'production'
set :user, ENV['CAP_PRODUCTION_USER']
# Simple Role Syntax
# ==================
# Supports bulk-adding hosts to roles, the primary
# server in each group is considered to be the first
# unless any hosts have the primary property set.
set :production_app_ip, "#{ENV['CAP_PRODUCTION_USER']}@#{ENV['CAP_PRODUCTION_IP']}"
set :production_web_ip, "#{ENV['CAP_PRODUCTION_USER']}@#{ENV['CAP_PRODUCTION_IP']}"
set :production_db_ip, "#{ENV['PRODUCTION_USERNAME']}@#{ENV['PRODUCTION_HOST']}"
role :app, [fetch(:production_app_ip)]
role :web, [fetch(:production_web_ip)]
role :db, [fetch(:production_db_ip)]
# Extended Server Syntax
# ======================
# This can be used to drop a more detailed server
# definition into the server list. The second argument
# something that quacks like a hash can be used to set
# extended properties on the server.
server ENV['CAP_PRODUCTION_IP'], user: ENV['CAP_PRODUCTION_USER'], roles: %w{web app}, port: ENV['SSH_PORT'], primary: true