I solved this issue by creating an invisible UIView on the Navigation Bar itself and then use that as the anchor point for the segue. The issue with that is the "arrow" of the popover when the uiviewcontroller gets displayed is awkwardly placed. Still trying to figure that out.
This is where I got the answer from