Your annotation's frame is probably too small. Even if you set the callout to be big enough, its parent, the annotation, itself is small. Had the same problem and it's really messy to solve it.
If you don't want to mess with frames and selected states and also because you already are using a custom view which is added, why don't you do the following :
- add the custom callout view as a completely new/different annotation/overlay when an annotation is selected ( not as a subview like now )
- remove it when your annotation is deselected
You only need to position it correctly and won't even need a gesture recognizer.