I ended up using this solution, which not only gets rid of the need for the first tap before long-tap-drag, but also is very smooth feeling for the user when dragging and has a large draggable area. There's a bit of an explanation from Azavea who implemented it, on their blog here.
I'm nearly done (famous last words) on the next version of my app which uses a slightly modified version of the above, namely that it (a) drags as in the original, (b) allows a tappable action on the annotation and (c) specifically ignores a long-tap on the annotation so that when attempting to drag you don't get annoyed by the unwanted tap action. Once I am done, I plan to fork or branch (I'm a bit new to what you do in this situation) the above github project to add that functionality. If I get really lucky I'd also like to be able to fix or at least configure the quick swipe issue they mention in the 2nd last paragraph of that blog I linked to.