if anyone is interested, this is a more elegant solution. just in the controller. then all links are the same, and it will open message if it exists. filtering by Conversation.participant(a).participant(b) was the key
def new
@recipient = User.find(params[:recipient])
@ongoing_conversation = Conversation.participant(current_user).participant(@recipient)
respond_to do |format|
format.html { redirect_to root_path }
if @ongoing_conversation.present?
format.js { redirect_to mailbox.conversations.find(@ongoing_conversation) }
else
format.js
end
end
end