First, find the normals by taking the cross-product of the vectors and then normalizing.
To take the cross-product of two vectors, A and B, use this formula:
Cx = Ay*Bz - Az*By
Cy = -Ax*Bz + Az*Bx
Cz = Ax*By - Ay*Bx
(Notice that the order matters. In general, AxB ≠ BxA.)
So for your two planes, the cross-products are (1,1,1) and (-1,0,0).
To normalize a vector, divide it by its magnitude. So the normal vectors of your planes are (1/sqrt(3))(1,1,1) and (-1,0,0).
Now to rotate on vector into another (I will assume you have atan2()
, and that you have the right-hand rule down cold):
1. Rotate around Ox: to get A into the XZ plane, rotate by atan2(Ay, Az).
2. Rotate around Oy: to get to the correct phi (angle from Oz)). PhiB is atan2(sqrt(Bx2+By2), Bz), so rotate by atan2(sqrt(Bx2+By2), Bz) - atan2(Ax, Az)
3. Rotate around Oz: to get to the correct "longitude", rotate by atan2(By, Bx) - atan2(Ay, Ax).
So in your example, you would rotate A around Ox by π/4 to get (sqrt(2/3), 0, sqrt(1/3)), then around Oy by π/2 - atan(sqrt(2)) to get (1,0,0), then around Oz by π to get (-1,0,0).