If you don't center the FFT phase reference to the center of the aperture (by using fftshift), rather than the front/back edge, then the phase of any signal discontinuity (between the beginning and end of the FFT aperture) alternates between FFT result bins.
An unwrap will possibly take the sign flip and add 2pi.
The phase sign alternates because the FFT is trying to measure the phase of the Sinc function that represents a rectangular window on any non-periodic-in-aperture data. An fftshift "twists" the complex Sinc function so it's phase no longer alternates.
Try your phase comparison using an fftshift (either pre-fft data rotation or post-fft alternate sign flipping), and see if that produces more reliable phase offset results.