UIView animations are for animating the properties of a view, not for animating layers. To the best of my knowledge, you can't animate a view's layer using a UIView animateWithDuration: call.
You'll need to create a CABasicAnimation and add it to your view's layer yourself.
It's been a while since I've done CAAnimations, but let's see what I can dig up:
As atxe says, you should move the layer's anchor point to 1,.5 before doing this animation, so it rotates around the right edge.
Here is some code from our "Kevin & Kell" cartoon app, that does an animation of a door swinging open on it's hinge:
//Open the door.
rotateAnimation = [CABasicAnimation animation];
rotateAnimation.keyPath = @"transform";
rotateAnimation.fromValue = [NSValue valueWithCATransform3D:CATransform3DIdentity];
//Pivot on the right edge (y axis rotation)
theTransform = CATransform3DIdentity;
theTransform.m34 = -1/100.0;
theTransform = CATransform3DRotate(theTransform, degreesToRadians(70), 0, 1, 0);
rotateAnimation.toValue = [NSValue valueWithCATransform3D:theTransform];
rotateAnimation.duration = 0.5;
// leaves presentation layer in final state; preventing snap-back to original state
rotateAnimation.removedOnCompletion = NO;
rotateAnimation.fillMode = kCAFillModeBoth;
rotateAnimation.repeatCount = 0;
rotateAnimation.timingFunction = [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionEaseInEaseOut];
rotateAnimation.delegate = self;
[treeDoorImageView.layer addAnimation:rotateAnimation forKey:@"transform"];
The transform logic is all but identical to yours (I use an m34 value of -1/100 for a more exaggerated 3D effect. Your -1/500 setting is more normal.)
Note that instead of trying to do the animation from a UIView animation block, I create a CABasicAnimation, and set it's fromValue and toValue properties to the starting and ending transform values.
You should be able to use this code as a starting-point for your animation.