I tried your code and could replicate the issue. After I set the length it stopped the issue.
Try setting the length of the attachment to 1
case UIGestureRecognizerStateBegan:
self.attachment = [[UIAttachmentBehavior alloc]initWithItem:self.cardView attachedToAnchor:touchPoint];
self.attachment.length = 1;
[self.animator addBehavior:self.attachment];
break;