If you want those answers separately, then it should be fairly straighforward (i.e. subtract the partent location for the child's to get the child's relative displacement). However I assume you want to reconstruct all of them together. If you start writing down the math you'll figure out a system of equations that gives you those things (the rotations will need to factor into displacement and so on). The thing is that the computation is perfectly equivalent (same number of computations required) to inverting the matrix and doing the multiplication, so you don't get anything by doing it anyway else (likely you'll make a mistake or you make it less cache friendly).
What is usually the design for these things is to use a stack and always keep the changes relative to the parent. Then when you want to do something, you go through the object hierarchy and multiply the matrices as you go along, keeping a stack so that when you move up the parent, you do a pop on the stack instead of inverse multiplication of matrices. This way you get very good performance and also the objects are more stable because you don't make as many computations with them (multiplying a matrix with it's inverse will rarely give a perfect identity matrix).