Your code is both increasing the rotation amount you add to the transform, and applying the rotation to the already-rotated view transform. So each time it starts from where it left off and moves more.
Don't apply a rotation to the existing transform. When you begin your animation, save the starting transform into an instance variable. Then apply the rotation value you calculate from the pan gesture to the ORIGINAL transform and save that as the view's transform.