Your parameters actually will be setting the transform value, so you need to use a combined transform of what you want.
You should combine your rotation with your transform3D like this:
CATransform3D tranform1 = CATransform3DMakeScale(0.01f, 0.01f, 0.01f);
transform1 = CATransform3DRotate(transform1, -m_PI / 2.0f, 0.0f, 0.0f, 1.0f);
CATransform3D tranform2 = CATransform3DMakeScale(1.1f, 1.1f, 1.1f);
transform2 = CATransform3DRotate(transform2, -m_PI / 2.0f, 0.0f, 0.0f, 1.0f);
CATransform3D tranform3 = CATransform3DMakeScale(0.9f, 0.9f, 0.9f);
transform3 = CATransform3DRotate(transform3, -m_PI / 2.0f, 0.0f, 0.0f, 1.0f);
bounceAnimation.values = @[
[NSValue valueWithCATransform3D:transform1],
[NSValue valueWithCATransform3D:transform2],
[NSValue valueWithCATransform3D:transform3]];