I found this solution,admit its a bit hacky but al least works, if someone find a cleaner solution feel free to comment.
Added this to my application_controller.rb
before_filter :set_host_from_request
private
def set_host_from_request
ActionMailer::Base.default_url_options = { host: request.host_with_port }
end