CABasicAnimation to make NSView flip

Question 1

You ask:

How do I make the left side taller/stretched on flipping in, instead of making the right side smaller/stretched?

You also say that flipping out works fine but flipping in is wrong.

The difference between the two is in the sign of the perspective:

Flipping out code:

CATransform3D mt = CATransform3DIdentity;
mt.m34 = 1.0/1000; // note the lack of a minus sign
lr.transform = mt;

Flipping in code:

CATransform3D mt = CATransform3DIdentity;
mt.m34 = 1.0/-1000; // note the minus sign
lr.transform = mt;

If you want the two to look the same then they should most likely have the same perspective.

In my experience you usually want the negative perspective value (as you have done in the flipping in example). This has to do with the fact that the value represents the position of the "eye" / "camera" / "observer" or whatever you call it.

If you imagine a 3D scene where the position of the eye is (e_x, e_y, e_z) then the perspective part of the transform is:

perspective transform

Assuming that you are looking at right at the world (i.e. not looking at it from the side) the position would be (0, 0, e_z) which is the reason why we usually only set m34 (3rd column, 4th row) when adding perspective to a transform.

You can also see that this is how it is used in the Core Animation Programming Guide:

Listing 5-8 Adding a perspective transform to a parent layer
CATransform3D perspective = CATransform3DIdentity;
perspective.m34 = -1.0/eyePosition;

If the rotation looks wrong to you should probably rotate in the other direction (for example changing a rotation from 0 to π into a rotation from 0 to -π or the other way around: changing a rotation from π to 0 into a rotation from -π to 0.

Question 2

What about adding some scale while you flip the card?

You could even exaggerate it and it would look as if someone lifted the card in order to flip it.

Some code to scale a view inspired by this answer:

CABasicAnimation *scaleAnimation = [CABasicAnimation animationWithKeyPath:@"transform.scale"];
scaleAnimation.fromValue = [NSNumber numberWithFloat:1.0];
scaleAnimation.toValue = [NSNumber numberWithFloat:1.3];

Question 3

I understand, this is late answer, but I wrote easy usage solution.

FlipTransition

import Cocoa

public final class FlipTransition: NSObject {
    
    private var srcView, dstView: NSView?
    
    private var duration: TimeInterval = 0.3
    
    public func flip(
        from srcView: NSView,
        to dstView: NSView,
        duration: TimeInterval = 0.3
    ) {
        
        self.duration = duration
        self.srcView = srcView
        self.dstView = dstView
        
        srcView.isHidden = false
        dstView.isHidden = true
        
        // Get super layer
        
        guard let superLayer = srcView.superview?.layer else {
            return
        }
        
        // Setup super layer 3d perspective
        
        var transform3D = CATransform3DIdentity
        transform3D.m34 = -1 / 1000
        
        let translation = CATransform3DMakeTranslation(
            superLayer.bounds.midX,
            superLayer.bounds.midY,
            .zero
        )
    
        superLayer.sublayerTransform = CATransform3DConcat(
            transform3D,
            translation
        )
        
        // Set layer anchor & position to center
        
        [srcView, dstView]
            .compactMap(\.layer)
            .forEach { layer in
                layer.anchorPoint = .init(x: 0.5, y: 0.5)
            }
        
        // Start src view animation
        
        animate(
            srcView,
            from: CATransform3DIdentity,
            to: CATransform3DMakeRotation(CGFloat.pi / -2, 0, 1, 0)
        ) { f in
            self.startSecondStep()
        }
    }
    
    private func startSecondStep() {
        
        guard let srcView = srcView, let dstView = dstView else {
            return
        }
        
        srcView.isHidden = true
        dstView.isHidden = false
        
        animate(
            dstView,
            from: CATransform3DMakeRotation(CGFloat.pi / 2, 0, 1, 0),
            to: CATransform3DIdentity
        ) { f in
            self.finish()
        }
    }
    
    private func finish() {
        
        guard let srcView = srcView, let dstView = dstView else {
            return
        }
        
        srcView.layer?.removeAllAnimations()
        dstView.layer?.removeAllAnimations()
        
        [srcView, dstView]
            .compactMap(\.layer)
            .forEach { layer in
                layer.anchorPoint = .zero
            }
        
        srcView.superview?.layer?.sublayerTransform = CATransform3DIdentity
    }
    
    // MARK: - Animation Utility
    
    private class AnimationDelegate: NSObject, CAAnimationDelegate {
        
        private let completion: (Bool) -> Void
        
        init(completion: @escaping (Bool) -> Void) {
            self.completion = completion
        }
        
        func animationDidStart(_ anim: CAAnimation) { }
        
        func animationDidStop(_ anim: CAAnimation, finished flag: Bool) {
            completion(flag)
        }
        
    }
    
    private func animate(
        _ view: NSView,
        from: CATransform3D,
        to: CATransform3D,
        completion: @escaping (Bool) -> Void
    ) {
        
        let dstRotation = CABasicAnimation(keyPath: "transform")
        dstRotation.fromValue = from
        dstRotation.toValue = to
        dstRotation.duration = duration / 2
        dstRotation.fillMode = .forwards
        dstRotation.isRemovedOnCompletion = false
        dstRotation.delegate = AnimationDelegate(completion: completion)
        
        view.layer?.add(dstRotation, forKey: "flip")
    }
    
}

Usage

FlipTransition().flip(from: srcView, to: dstView)

View Structure

holderView {
    srcView,
    dstView
}

dstView initially should be hidden (dstView.isHidden = true). holderView, srcView and dstView should have equal sizes.