In real-address mode segment register values are, indeed, shifted left 4 bits and then added to the offset to form a 20 bit linear address (which in this mode is the same as the physical address).
Obviously, this means than different segment:offset pairs may be translated to the same physical address (just as different linear addresses in protected mode with paging enabled may be translated to the same physical address). Since there is no 1:1 mapping between logical and linear addresses the only thing you can get from a linear address is a set of segment:offset pairs that are translated to it.
You can find more detailed description in the Intel Manuals, volume 3B, section 20.1.1.