I solved this problem by dividing the target size by the screen scale (screenScale). ScreenScale is 1.0 on normal screen and 2.0 on retina screens:
CGFloat screenScale = [[NSScreen mainScreen] backingScaleFactor];
float targetScaledWidth = sourceImage.size.width*scale/screenScale;
float targetScaledHeight = sourceImage.size.height*scale/screenScale;